Combination of things, really.
Bore diameter does have a good bit of importance, since a 0.40" hole in someone is going to have more of an effect than a 0.355" hole.
However, what the bullet does once it's inside the target, has just as much, and probably more, importance. Bullet design is a critical aspect here, since today's premium hollowpoints have pretty much levelled the playing field, when it comes to cartridges of .38 Special +P, and anything more potent.
With today's premium designs, any of the serious defensive calibers will do the job, since such bullets are designed to expand at lower velocities than their older counterparts used to require. If anything, the developments in bullet designs have changed the way we think about ammo. Old "dirty words," such as "147 grain 9 mm hollowpoint," are no longer dirty, and some of the best performing designs are of the 147 grain subsonic 9 mm type (Federal's HST, Winchester's Ranger, Remington's Golden Saber, etc).
But... What happens when you're not using one of the above "serious defensive" calibers? If you're confident in being able to place your shots very accurately and consistently, then you're already way beyond most folks, and don't need advice on this matter. However, for the rest of us ordinary folks, we need to evaluate the gun, and ask if we're willing to sacrifice the extra margin of error when it comes to the better defensive calibers, just to have better portability.
With today's small firearms (Airweight J-frame revolvers, Kahr micro series, Rohrbaugh, Kel-Tec, etc), available in such serious defensive calibers, I simply prefer to stick with them. I find it hard to beat my Glock 26, stoked with 13 rounds of Winchester Ranger ammo (127 grain +P+), and that the ammo still has enough velocity to match a .357 magnum coming out of a snub nosed revolver.
In cases where I have to keep the gun concealed with a tucked shirt, and no overgarmets, that's where I'll use the S&W 342 (airweight), loaded with 5 rounds of Speer's 130 grain Gold Dot +P loads designed for shorter barrels.
The only lesser caliber that I carried was a .380 ACP (Sig P230), where i chose the heaviest premium hollowpoint available, that fed reliably (The Remington Golden Saber 102 grain), since penetration from a .380 ACP is going to be marginal.