One of the fundamental theoretical difficulties in ever doing such a study (aside from the obvious practical issues) is the problem of selection bias.
Since we aren't amoral dictators who can draft experimental subjects and randomly assign them into test groups and force them to train their assigned art for a specified number of hours per week before mandated combat tests, we are limited to observing the outcomes for people who have selected particular schools and arts and training regimens.
Studies like this aren't unheard of. Might pare it down to 100 participants or something like that, focus the scope on the first year, and conduct it in a single area, but the feedback would be interesting and informative. We might even get some interesting data on attrition, if we tracked the participants for longer than a year. This is, of course, just thinking through how we might do the study. Would be cool... I think we need to get an academic to make it happen.
The typical result is that potential martial arts students who are already athletic, aggressive, motivated, and mentally tough are more likely to sign up for competitive combat sports or other martial arts training which encompasses that sort of intensity in training and skill testing.
I agree with your overt statement here. But you seem to be implying that people who don't have those attributes avoid competitive sports, which I don't think is true at all. On one hand, we have many, many examples of people who train in combat sports who do not have (or at least don't start with) those attributes. You are exhibit A. But we have a lot of women who train. And we also have a lot of lawyers, doctors, scientists, network engineers, etc, of both genders. I think it's a myth that combat sports people are all lunkheads. So, while people who are athletic, aggressive, motivated, and mentally tough will gravitate toward combat sports, that is not to the exclusion of folks who do not.
I would actually flip your statement around to say that the level of athleticism is higher in combat sports because combat sports, like other sports, builds athleticism, motivates the participants, and promotes mental toughness. When you walk into a gym, you can tell the new students from the veterans, but you can't really tell the lifelong athletes from the students who walked into the gym a year or 5 years ago with a beer gut and a little courage.
Now, all of that said, there are some schools out there that have a macho, aggressive culture. I'd say that the culture is what keeps them that way, though, and not something intrinsic to competition or athleticism. A lot of women train in BJJ, for example, but they don't tend to stay in schools where the instructor is a misogynist.
Students who lack those attributes are more likely to be drawn to schools where they won't encounter that same kind of pressure. As a result - even if the training in the two groups was equally effective for developing combative skill (it's not, but let's just imagine for the moment), the second category of schools would on average produce worst fighters just because their students started out at a lower level.
Were I to actually conduct this study, the participants (whether it's 100, 400, or whatever) would be distributed among the groups at random. So, the folks don't get to pick their group. 25% would be assigned to group 1, where they would train in some combative sport. 25% would be assigned to group 2, etc.
That said, you touch on a point I think is really important, which is the marketing that goes on. Non-competitive styles tend to bake in this notion that they can teach you to fight without the discomfort (physical, emotional, or mental) that comes with learning how to fight. It's a sales ploy, and in large part it's this con that really angers me when we talk about "self defense."
(BTW - I believe this is the secret of some top fighting gyms - it's not that the coaching is necessarily always that much better than other gyms so much as they recruit the most talented athletes to begin with, focus on them, and then the success of those athletes attracts more top talent. I think the real test of an instructor is what they can do with someone who is naturally untalented.)
Getting back to my own experience, I think one other benefit I got from my time in the Bujinkan is that it gradually toughened me up to the point where I could jump into more challenging martial endeavors without being completely overwhelmed. Obviously, not everybody needs that kind of remedial development, but it was useful for me.
I can definitely relate to this. It didn't take me quite as long. I trained in WC for while in high school (a little over a year, I think), and was on the wrestling team. But as an adult, I was looking for something to do with my kids. I didn't know much about martial arts, and found myself in a "self defense oriented" school for a few years before I decided to move on.