In Chinese culture and religion, self-defense can be seen both as a low-tech endeavor (such as farming, fieldwork or bodyguard work) and also, especially in terms of cultivation culture and religion, as a means to bettering one's self and even to achieve enlightenment. Across Taoism and Buddism we find references to a culture of self-development based around a sort of fusion between yoga, qigong, philosophy, and martial arts.
I think the point should be made that as a sovereign spiritual entity (someone with a soul) it is important to be able to defend your self in the physical plane while still understanding that the other person is a sovereign spiritual entity in the same sense you are. Then, the removal of ego can be accomplished via the drive towards non-violent martial arts and the seeds of enlightenment will have been planted, leading towards the practitioner's karma being changed by a lifetime of practice.
Another aspect to the removal of violence from defense is the foundation of a society; a society aims to keep it's population safe. So when engaging in self-defense it is important to avoid cruelty and "excessive violence" whenever possible. The very definition of excessive becomes whatever society allows to seep through the cracks. If someone comes to attack you, this is already a violation of society. You are only allowed to defend yourself because you are able to point to society and say "You failed me;" in any other case your actions, in that they overstep society's right to protect you for you, are to be seen as excessive.
Tao Te Ching Chapter 36 seems to discuss a type of restraint from violence as in fact the pinnacle of kung fu; "It is better to maim than kill, it is better to break than to maim, it is better to strike than to break, it is better to push than to strike, it is better to throw out (push) than to strike, and the highest level is when the opponent does not feel the action of the push against him." Heavily paraphrased, of course.
So considering we get a free pass from the ancients and society allows it, does that mean it is truly acceptable for one on the spiritual path to learn self-defense? Is it possible to respond to true violence in a non-violent way?