Two factors to consider are how long you're in close proximity to another person, and whether or not they get droplets on you. There's a big difference in passing by a group with no masks, compared with hanging out with a group with no masks. You could also get a face shield and wear that in addition to your mask. Mask protects others from you, face shield protects you from others. Just wash your hands before you touch your face, and wash the face shield.