You are projecting every fight into a mutual confrontation. Sometimes it is and could be considered avoidable. Sometimes it isn't.
Now regarding that I will engage in a fight or at least in force if the other persons bad behaviour is too extreme regardless as to whether they are an immediate threat or if I conceivably could have de escalated.
That is the problem bear. While you and I know that it is just as likely that once turned and walking away, one of those little darlings will jump on our backs. There is still no justification to knock them clean out. You have to let it happen in that context. The good thing is that the wolf pack is usually a poodle pack. So merely turning around and being impassive does the trick.