I've been thinking about this recently, because I know that I'm going to likely be the same. pixydust mentioned brand, which is of course important, but why does brand have to be anchored to genre? My first novel was adventure, the second is sci fi, the third will likely be crime. But while the stories are technically in different genres, they still share similar themes and structures. Consider Game of Thrones and Breaking Bad. The audience for those shows overlaps hugely. One is crime, one is fantasy, but they both deal with trying to attain power.
It depends on what you write, of course. Different books from different genres can obviously be hugely different. A whodunnit and a historical romance, for example.
Anyway, the theory is that an authors brand could be something like "dark, thrilling stories that deal with moral ambiguity and paranoia" as opposed to something more rigid like "crime" or "science fiction". But yes, this would probably be an issue with agents because many specialise by genre.