Typesetting isn't selecting the typefaces to be used; that's part of book layout and design, and up to the book designer, though the typesetter may be consulted.
Typesetting is the arrangement of individual glyphs in space on a page.
Generally speaking, most typesetters either have a four year degree in design plus an apprenticeship, or they have the equivalent in on-the job training.
I had two years of apprenticeship, several classes and had to pass an exam based on the quality of my work.
After which I worked under a senior colleague's guidance.
In other words, there's a lot to learn. I'd start with reading Ken Umbach's /Research Guy's free pdf on self publishing. He's a member here, and it's linked in his sig.
I'd suggest going to your local public library and reading the sections in The Chicago Manual of Style on layout, design and topography.
And I'd make myself familiar with some basic concepts;
this site discusses some of them.
Look at a lot of commercially published books. Look at how they're designed, what type is used where, and how, and when. Notice the sizes, the leading, the justification and weight.
Some of what I'm saying here is about design rather than typography, but I suspect you're going to be your own designer as well.
Look at the front matter and how chapters open, and how page numbers are handled.
Notice how punctuation is handled. Pay attention to curly quotes and apostrophes, and how they behave in terms of punctuation marks like em-dashes.
Look for colophons at the very start or the very end of books; often they'll tell you the typefaces used, the size, and the leading. You'll have better luck with older books than newer ones in finding a colophon.