I haven't fully read through the responses, so this may have already been said.
I don't see why one should adjust for size/age/weight. This is comign from someone who is 5'6 and between 130-140 pounds (much less than most 'fighters'). IMO, if a black belt is expected to be able to punch with X force, and I am unable to do that because I weigh less than most people, I shouldn't be awarded that rank. An old sensei of mine told me "I dont have any child black belts. I don't have any issue with that idea, but I expect a child black belt to be better at sparring than an adult brown belt. If he's not, then he's missing something, and hasn't yet earned his black belt." I don't expect to be treated differently because of my weight, most female martial artists that I know don't expect to be treated differently because of their gender, and children shouldn't be treated differently for the same reason. Doing otherwise is an unfair bias against that person (by not respecting them) and against others (by expecting more of them.
Regarding your statement about "If the requirement is to "hold your own" with X number of people of the rank you're testing for, should that be people selected at random, or matched to the size/age/fitness of the person testing?" If a requirement for me was a kumite, and someone could not participate because of their size (again, I am 5'6-5/7 and 130-140 pounds, very short and skinny for a martial artist), I would walk out on that test, and lose respect for the tester. If the system requires me to be able to hold my own with someone of a certain skill level, I God Damn better be able to hold my own with that individual, and create strategy on the fly to handle their advantages, or else I do not deserve to be the same rank as them.