A story needs the characters it needs to move it forward. Fantasy is a varied genre, and there are plenty of authors (probably most, actually) who don't write huge epics with ten povs and dozens of secondaries like GRRM.
Hard to say how many is enough, depending on the nature of the story. What is the focus of the story? Are they on a long journey together? Then the story may no involved any more detailed characters.
My own story has the plot driving protagonist and two very important secondaries who have big arcs of their own (also important to plot movement) as povs, a group of maybe 5-6 important secondaries who aren't pov characters but still get a lot of "screen" time and impinge on the povs in important ways. Then there are a larger circle of tertiary and quaternary characters.
Each of them has a purpose or role in the story, and I've actually eliminated or combined some support characters who didn't do enough in my opinion. I'm actually worried about the sequel, because there are two more people who *want* to be povs as well, and I worry if that's too much.
The important thing is having any characters who are more than just spear carriers feel like they're real people with personalities and so on, even if we only see them a few times. Showing how your pov character feels about that person is important too.