A couple of thoughts:
1. Same mass and speed into a smaller area increases the force. Hitting with whole sole of the foot doesn't concentrate force like hitting with heel only.
2. Mass at the point of impact depends on technique. Kick with just leg has less mass than one you get your body mass into the foot or fist.
3. I think what you're calling "impact" is the same as force at contact. So yes, mass and speed are the keys. The physics are F=MA (Force = Mass x Acceleration). Since you don't change the mass of your foot, speed is what you control. However, weapons of greater mass change the equation: kicks have more mass than punches, so they're inherently more forceful/powerful.
4. Target selection is the hidden but, IMO, most important variable. I've trained a fair amount with firearms, pistols in particular. The decisive variable in handgun stopping power is bullet placement. It doesn't take board-breaking force against soft targets like the throat. And I've taken some amazing round kicks in the butt (sparring partners with poor skills)--left me sore but undamaged.
The simple answer is that your foot's mass doesn't change, so yes, speed is what affects power. Would you rather try to break a board with a slow kick or a fast one?