I'm going to disagree with you on the nature of this encounter, Bill. This was a Monkey Dance.
I said it was a monkey dance as well.
Let's clarify. A monkey dance is often described as a dominance struggle among male primates, in this case humans. It is meant to establish a pecking order. It is often an adolescent rite of passage, although some men never outgrow the need to attempt to establish dominance over other men into their 40s and 50s and even beyond.
When two parties voluntarily engage in the monkey dance, there may or may not be violence. Often one backs down, ending the confrontation and establishing that he is submissive to the one who doesn't back down, who is dominant.
However, when one party has no desire to engage and the other party does, there is nearly always going to be violence. Why? Because dominance is not established until both parties engage in the escalation and one backs down. If one party refuses to play, dominance is not established. Therefore, the male who wants to establish dominance is left with one-sided escalation until the other party is completely run off, or engages.
That is why I advised the OP to leave as soon as the dance begins - if possible.
If it is not possible, the dance continues, as we see from the OP's narrative. Eventually the OP did in fact escalate, even without meaning to, as you noted.
Here is where I differ from some of the others giving advice.
In my experience, the monkey dance at this point is going to escalate until there is an actual fight. Most self-defense laws in the USA (I am not a lawyer, this is not legal advice) state words to this effect:
"An individual who has not or is not engaged in the commission of a crime at the time he or she uses force other than deadly force may use force other than deadly force against another individual anywhere he or she has the legal right to be with no duty to retreat if he or she honestly and reasonably believes that the use of that force is necessary to defend himself or herself or another individual from the imminent unlawful use of force by another individual."
That's Michigan, but other states, IMHO, are similar.
It means you don't have to wait until the first punch is thrown. "Honestly and reasonably believes" are key here. The person being confronted has to believe that they in in imminent danger of unlawful force by Joe Shirtless here. And most of us have heard that the 'reasonable man' test means as close to objective as we can get - we can't just make up our own definition of what is 'reasonable' and what isn't.
However, given that we all seem to agree that the 'monkey dance' can and often does end in violence, it is pretty clear to me that once the dance starts - the threats, the intimidating gestures and especially the shirt removal, the fight has already begun. The danger is extreme, and unlawful force on the part of Joe Shirtless is about to begin. HIT THAT PUNK NOW, AND HIT HIM AS HARD AS YOU CAN, is my opinion. Do not stop hitting him until the threat is eliminated. I interpret that to mean he is no longer capable of hitting me.
The moment someone says to me,
"You ain't so tough, I'll bet I could..." It's go time. He threatened me. I feel threatened. Any reasonable man would feel threatened. I'm not going to let him screw up his courage to the sticking point, bump chests with me, throw fake punches to try to make me flinch, rip of his shirt, etc, etc. We're not going there. The fight has begun and I am legally defending myself from imminent assault by yon pimpleface.
Why wait? All it can do is increase the odds of you being hemmed in by a crowd, having his buddies wander over to witness his superlative display of testosterone, let him get pumped up into an agitated state, etc.
The initial response, as I said, is to try to leave. I also said one should loudly and clearly state that they do not want to fight, and make it clear with a non-threatening posture. The crowd needs to be able to say to the cops that the OP was trying to avoid fighting but was pushed into it by the guy who is laying in a bloody pool of mush on the floor.
As martial artists, we train to move from a non-threatening posture into full-on attack mode, and that's what should happen here. The moment dirtbag lifts his arm to punch, or moves in too close to chest-bump, or says he is about to blah blah blah, that's it. We're in full engagement mode. He is not expecting that response, because it's against the rules of the monkey dance. Yep, it is. Break those rules.
Buy the guy a beer, flirt with him, agree to step outside and close the door behind him. (OK, that's more a deflection than an actual derailment, but it works!) A different approach -- though it seems to have been a little unlikely to be available -- is to be the "big dog."
I only see this work with sentient beings who don't actually want to fight and are looking for a reason to back down without appearing to have lost. Teenagers are not sentient beings, not yet. Young men who start the dance at a party are not sentient beings.
It's all beer, women, and ego at that point. The fight is going to happen. The trick is in realizing that you're in a fight before the first punch gets thrown.
It is also a trick that requires considerable skill to master. Law enforcement types often have to react this way, and they develop the skills. There are a thousand ways you can say
"Hey, dude, let's just chill, let me buy you a beer," and 999 of those will get the beer bottle broken over your head. Tone, attitude, verbal and non-verbal cues, it's a tricky response to master, and it also requires a receptive audience, which I believe from the OP's description that this yahoo was not.
But I do agree -- once it's on, it's on. Handle it quickly, efficiently, and thoroughly enough to end it. How much harm you do depends on lots of factors, from personal beliefs to legal standards.
So I think the only place we may disagree is on where the 'on' switch is in this continuum of violence. No problem, I am willing to admit I could be wrong about the OP's stated situation - I was not there. And yes, personal beliefs enter into it. However, speaking for myself, I have done my due diligence on the laws of self-defense where I live. I feel I know where I stand with regard to when I may and may not engage in self-defense. And as to the degree of violence I am willing to inflict, that all depends on what Joe Brokennose feels like doing. He stands up after I knock him down, he's getting knocked down again. And I have no problems busting faces, bones, cartilage, or whatever else presents itself. His option to not get serious f'd up ended when he chose me as his target. He'd do the same to me if he could.