I am very intrigued by the fact that you have two agents. I have often thought that this is something that could end up happening to me since I've written three novels in three different genres. My current agent is reading one of my other novels, but I don't know if he'll want to rep it. Could you share how this came to happen? Are you happy with your current arrangement?
If your current agent represents books in all the genres you write, I'd say you should always give him first crack at representing everything you write, since that makes the most sense career-wise and agent-relationship-wise. It sounds like you're already doing that.
If your agent turns your next book down and you still want to pursue publication through channels unavailable to the un-agented, though, I guess you wouldn't have much choice but to query others eventually. But I wouldn't rush into it. If the agent hasn't sold the first book but turns down your second, I'd probably wait to see what'll happen with the first book. If at all possible, having one person handling all your books is the best way to go. And if it does turn into a no, make sure you find out why and if it's something you can change so he will take it on. If you're just writing in several different genres of commercial fiction, they're probably still in your agent's wheelhouse.
That said, I've met writers who had more than one agent at the same time (even in fiction; one author had an agent for his thrillers and a different agent for his cozy mysteries, and they were both series that sold). I don't know why he made that choice, but everyone involved in the arrangement was happy with it.
For me, I was signed for fiction before I was signed for nonfiction, and hadn't even decided for sure that I was going to pursue agency representation for the nonfiction when I signed for fiction. My fiction agent represents a very small amount of nonfiction and it is always narrative-style nonfiction--memoir and travel writing, to be exact. She does not represent the genre of nonfiction I write in, and furthermore, nonfiction is sold very differently from fiction. My fiction agent has no experience with book proposals for nonfiction and no significant relationships with nonfiction imprints.
When I discussed it with her initially, she considered asking someone else at the same agency to represent my nonfiction, but ultimately we all decided nobody there had the experience or connections to sell this genre, since it's pretty niche (though the prez of the agency did have experience with book proposals for other kinds of nonfiction, but she didn't want to take on the project).
The agent I found was reassuringly established in the nonfiction world, to the point that she even sometimes runs workshops on nonfiction proposals when she attends conferences. When she first requested my proposal, I made it clear in writing that I had a fiction agent and she said that wasn't a problem whatsoever. When she called me on the phone to offer representation and I double-checked that she was okay with my fiction being represented by someone else, she told me I'm not her only client who has another agent; she said "I've had this situation with other authors too, and I think other agents are pretty good about it. It's not that unusual in the industry." After being offered representation with the nonfiction agent, I informed my fiction agent for the final "blessing" (even though she knew I was going to look for someone else for nonfiction), and we're all on the same page.
I don't think it'd be wise to chase multiple agents in most cases--though in mine it was necessary if I wanted the nonfiction to be agented--and if you do need to I of course recommend full disclosure at all levels of the process with everyone involved.