Buka, I think a big part of that is the time per belt. In TKD, for example, it can range anywhere from 1 month to 4 months between belts, depending on your school. Some schools will basically just push belts onto you, others will actually require you meet testing requirements, so it may be longer, but in general 1-2 month average per belt can be expected. In BJJ, you're looking at years per belt even before black belt.
My TKD school, before black belt, goes White-Yellow-Purple-Orange-Green-Green (1 Stripe)-Blue-(Blue 1, Blue 2)-Red (Red 1, Red 2) - Black. We test every 2 months, although some people test every 4, and a lot of people slow down around blue or red. Let's just go with 12 belts and 2 mo/test for comparison, because I think that's common of TKD.
A new BJJ Blue belt has probably been doing this for a year. That means he is the equivalent of a blue belt under the TKD system mentioned above. However, while the blue belt has tested several times, the BJJ student only has earned one belt. Now let's look at purple belt, that's another 2 years in grade. By this time, the TKD practitioner (if they've kept up) is probably a 1st Dan - 2nd Gup. Assuming they've slowed down, they're probably High Blue or Red. After another year and a half, the BJJ student is brown, and the TKD guy is almost definitely a black belt, probably 2nd degree closing in on 3rd if they've kept pace. A BJJ black belt is probably the equivalent of a TKD 3rd degree in terms of experience.
So yes, in TKD, green vs. orange isn't much different. Blue 2 vs. red isn't much different. But, comparing green to white belt, or green 1/blue to red 1/red 2...yeah, there's a difference. Comparing a fresh black belt to someone approaching 3rd degree testing, there's a big difference. I think that's why you see such a difference in BJJ between belts compared with other arts. You get a new belt every few years instead of months.
I'll also agree with your assessment that grappling arts are less about physical ability and more about knowledge of the art and practice with the application.