Bonkers? No. Need to be cautious? Yes.
Many will disagree, but few women can write 100% convincing male POVs and dialog. To be fair, even fewer men can write females well.
After 26 years of grading thousands of papers, I can tell the gender of the writer without seeing him or her as easily as discerning Ernest Hemingway's prose from Stephenie Meyer's. It's tougher with teens, but there are still clues.
When I need to write a woman's dialog--and especially POV--I try to write with lots of "so" phrases, "that" clauses, sprinkles of "just," and more: That is just tips of the iceberg because the task involves so much more that helps to make it just sound so much better.
I run my better half passages past three very critical lady alpha and beta readers who always make helpful suggestions.
So far I haven't had any complaints about my male POV,but my readers are mainly (I assume) female romance readers. I did used to worry about writing my male POV characters, but not so much anymore. It's to do with my mind set when I'm inside their heads. Sometimes I ask my husband if a phrase I've written seems 'right' to him and he keeps me straight. I once did a paper during my MA socio-cultural linguistics about male v. female speech and I try to remember my research when I'm writing. But really, people are just...people, with different personalities and reactions. I haven't finished writing my male POV book yet, but hopefully my only male beta reader will tell me if there's anything that jarrs.