Youre probably misinterpriting the one issue with being a Brown Belt.
When Youre a beginner, Youre challenged by Learning. When Youre intermediate, Youre challenged by Your Peers. When You get to a high rank, being paired with anyone of a substancially lower rank, possesses no challenge other than not doing 90% of the stuff You could be.
This is one of the reasons I think Rank can blinker people a bit, because the above is not factual. Its just the conclusion some people come to.
Training with betters, lessers, and equals is all essential. Optimally, it should be balanced, with a bit of a lean towards equals.