The other thing to consider, at least when it comes to the Olympics, is the time factor. The olympics are every 4 years. It is very difficult to remain the top fighter in a division for that period of time. If a fighter were to make the Olympic team at 25 the next time they could fight in the Olympics would be when they are 29. Time, injury, etc. might prevent a fighter from competing in consecutive Olympic games; meaning the next young fighter would get the spot.
Also, and I might show my ignorance here, a country, save the host country, cannot send a full team with all weight classes. So, they are going to pick the fighter and the weight class that gives them thebest shot at winning a medal. That alone might keep a fighter out of the games.
But, all this only applies the the Olympics, which is what was referenced in the subject line.