It's more entertaining to be a gardner. Everything will surprise and be creative. Architects have strict guidelines that makes it a bit harder for a story to grow.
A mix of being a gardener and an architect is the most entertaining for me. I currently have a project that relies heavily on the worldbuilding portion due to the fact it's meant to explore how a specific culture and religion might develop, and traditional fantasy tropes and clichés simply won't do it for this one. In fact, there's not really any specific story yet since that wasn't my original goal (I was actually just conlanging), but I'm hoping that at some point I can pinpoint a time and place suitable for a story to take place within the culture.
Then there's another story I have in mind that merely requires research into shows like Life After People and what would happen if Humans were no longer around (in sufficient numbers) to maintain the buildings, vehicles, et al that typify the modern world. I can be as ambiguous as I want (and I intend to be).
To me, it all depends on the intent of the project, and flexibility is crucial. The first project I mentioned above I'd never just start writing due to what I specifically want to do with it (it does involve non-White mythological beings and potentially a non-White culture, so I really can't just fudge it), while the second one benefits from a less specific world (so keeping in mind what would happen should Humans largely disappear, I can write whatever I please).
But it's fun! 