If someone wants to fight you, they don't care what stance you're in: aggressive, neutral, passive - they're going to throw no matter what.
If they're trained, your stance may influence what attack they open with based on their perception of your stance's weakness. Of course, you could use this to bait him into choosing an attack line of your choice and are prepared for. For example, if your hands are high, you encourage him to attack low and visa versa. Hands wide, you encourage him to strike up the middle.... As long as you're aware of the attributes of your stance and guard, I think most any one can work for you.
Being able to effectively attack or defend from a neutral stance without telegraphing is certainly a good (and advanced) skill to have. This does require coordination of hands, feet and body, as well as mind, to execute all of it simultaneously and the average Joe probably won't pull it off effectively. Handling a quick, true, sucker punch out of the blue? Jason Bourne, John Wick, and maybe one of you can. To be honest, chances are (and I'm pretty quick) I'd probably suffer some damage, but if I'm still on my feet, I would be very, very angry.
I think, in the end, you have to do what you are comfortable, and proficient, with, and control the action (actively or passively) wherever possible.