I've never read an Avengers comic book. Didn't know anything about Hawkeye, Black Widow or Samuel L Jackon's character (name's already slipped my mind). Only knew Iron Man and the comic-book Thor and Loki from the movies. I do know some Hulk canon.
The movie made perfect sense to me. Just enough backstory to ground the characters and away we go! Whedon is excellent.
On the Serenity DVD, in Joss's commentary track, he talks about the extreme effort he needed to introduce everyone who'd never seen Firefly (he wisely assumed that was most people) to the entire crew at once. If you recall, after the flashback (history lesson) within a flashback (hands of blue laboratory) within a flashback (the agent) the first scene introduces the entire crew in one near-continuous scene - as well as almost all of Serenity itself.
Knowing how seriously he took that one scene, and how successful he was at that, I had no worries that he'd be able to introduce the Avengers to non-comic-savvy moviegoers as well.