Hearing all of the stories from more experienced people about Hayes, doesn't make me feel like he is in any way legitimate.
It depends on what you mean by "legitimate."
Did Hayes train for many years in the Bujinkan?
Yes.
Was the vast majority of that time spent training and teaching in the U.S. without direct supervision and feedback from his instructors?
Yes.
Did he receive a relatively high rank (10th dan) in the Bujinkan from Hatsumi?
Yes.
Was Hatsumi in the habit of handing out ranks like candy to Westerners who came to visit Japan, so that he eventually had to raise the top belt rank in his system to 15th dan?
Yes.
Is Hayes no longer a member of the Bujinkan?
Yes.
Is the system he teaches primarily based on his personal interpretation and expression of what he learned in the Bujinkan?
Yes.
Have other practitioners split off from the Bujinkan to form their own organizations and teach their own interpretation of the X-kan arts?
Yes.
Are some of them widely considered legitimate instructors of those arts?
Yes.
Do lots of experienced martial artists consider Hayes to be skilled and knowledgeable?
Yes.
Are lots of other experienced martial artists much less impressed with him as a martial artist?
Yes.
Do some experienced practitioners in the Bujinkan, who spent much more time training in Japan under the direct supervision of Hatsumi and the shihans, consider Hayes's understanding of the art to be limited or off-base?
Yes.
Are they correct in that assessment?
Neither you nor I are qualified to say.
Are there serious doubts about the accuracy of the history presented by Hatsumi regarding 6 of the 9 arts which make up the Bujinkan?
Yes.
Are those arts and their derivatives, such as Toshindo, effective for modern day use regardless of their historical lineage?
Experienced martial artists will argue about such matters endlessly. I have my opinions, but you have no reason to trust my opinion over that of your instructor.
Bottom line - it's good to gather information from a variety of sources, but you have to make decisions based on your own experience and judgment, not because someone on a forum told you something.
The only legitimate people I view in the field of Ninjutsu, are the Japanese
There are non-Japanese who have spent decades and training in Japan.