I'm in your boat in that I feel as though I look for clever/witty prose more than funny story lines or plots when I assess humor writing.
BUT, at the end of the day, I think it's definitely better to have a good plot, too.
I'm writing a memoir and it almost feels like it overlaps two genres. My early fragmented childhood cute/funny/eccentric memories tend to be written about humorously, jumping from one cute/funny/eccentric anecdote to another with clever/witty insights and commentary. Then my more recent and complete memories are written about in more of a traditional narrative memoir style with action scenes and plot that pertain to the book's overall main storyline.
While my early childhood memories aren't terribly critical toward the main story arc of the book, they do sorta have mini-plots and relevance in terms of theme (or at least I hope). Furthermore, my first two chapters are about recent events, so I'm hoping they suffice in getting the reader invested in the "characters" and story so that I can kind of coast through childhood on mostly humor (after the first two chapters I travel back in time to childhood and proceed chronologically).
I try not to be too tangential and too navel-gazey. I try to at least organize my funny/cute/eccentric anecdotes in some way, be it by character, theme, or some small story/plot.
But I really love to just write humorously and entertain through witty prose. Like, I can get lost in descriptive or expository writing, just describing and analyzing a character and his/her quirks. I love pointing out quirks that are eccentric but relatable. BUT, at the end of the day, I think you will have a broader audience if you also have that character DO SOMETHING.
My writing has been compared to David Sedaris. I haven't read a ton of his work, at least not recently. But I'd say "Santaland Diaries" was one of his best pieces and if I recall correctly there's a lot of action and conflict and like, situational comedic plot points in there. Then, I've read some weaker Sedaris pieces that feel as though he is dialing it in and it feels like he's just navel-gazing and offering up somewhat funny insights on mundane things but it feels disjointed, like nothing's happening, it's incoherent, and just his random musings.
Again, not familiar with Sedaris that much but many say he does plot his stories (or at least his better ones), albeit in a very formulaic way. They also say he has central themes for his stories.
Anyway, sorry for rambling...it's just that I'm totally struggling with this issue to. But I'm definitely striving to include plot and not just rely on humor.