If there is any roughness with the grabbing, that clearly is a danger to the child. However, there is also a problem if he is gentle in his touch.
First off, any childrens instructor should be familiar with the ideas of "2 deep leadership." If you are in contact with children, especially physically, you want another responsible adult around. Not only does that prevent you from taking advantage of the situation, but it also covers you in the event that an accusation is made.
This guy seems to be setting up a situation where he alone is teaching (correct me if I am wrong here) children and he is touching them a lot. He doesn't have a back up to testify that it was not any else. (look up "grooming" when you look up "2 deep leadership.") Now I am not saying this guy is doing anything inappropriate. But I am saying that he is opening himself to a ton of unneeded risk, by his actions.
I realize that martial arts involves contact. However, if your contact is limited to the moves in the arts curriculum and you have another adult that can vouch for it... that is very different than gently touching all the kids in ways that are not in the curriculum, and are not drills... especially with no second adult to vouch for you.
If it were my kid, I would not want them in a situation where it is normal for the person in authority, to have his hands on you, just because. That is not something that I want my kids to learn is normal. Its not. And too often, that happens in other places, where it definitely should not happen.
If you are uncomfortable in any way, find someplace else.