I'm so sorry to hear about your frustration, brs18041. I know it well because I went through a similar situation last year. I ended up deciding to fire that agent, but then I discovered he'd left the agency weeks earlier.
It's one thing if your agent is holding back your manuscript for a particular reason, but it's another for them to not communicate with you about that. Even if it takes a few days or so for your agent to get back to you on things, it's part of their job to communicate with you.
Are you sure this a reputable agent and agency? I'd call again and leave a message stating specifically why you are calling. I think before you decide whether to walk away, you should try to find out if your manuscript has been pitched anywhere, and if so, exactly to whom. You'll need to know that if you want to approach other agents with it. (The best scenario if you want to part ways would be if your agent hasn't subbed it anywhere.)
And you might want to read your contract again to be sure of all exit terms, including language around the agent's rights to commission on your manuscript and any waiting period that might be required before you can contact other agents.
But hopefully, there's still a good reason why your agent hasn't gotten back to you, and things will work out with them.
Good luck!