I have my own views, which I'll share shortly, but I wanted to put this question out for discussion, as I'm sure views will differ, which is fine of course.
So, what do you feel it is? Is it the techniques in your art? Is it the avoidance aspect?
I think it depends very much on context. Self-defense could be defense of your reputation, your morals, your belief system, etc. But in the physical context, I'll give it a shot.
To me, self-defense is defense of my own life and physical well-being. Period. The 'how' of that is somewhat irrelevant to me, as I'll do whatever I can, whatever I feel I have to do, to try to survive a life-or-health endangering situation.
That means a lot of things.
It means I try to eat better food and get regular exercise, because disease and old age are risks to my life and well-being.
It means I try to avoid risky situations which I do not feel a strong need to engage in. I not only avoid paragliding, but I also stay out of bars and dangerous parts of town. That doesn't mean that I live in my house, quivering in the dark, but it does mean that I think about the risks versus the desire to take that action anyway. There are times when I will decide to take risk for pleasure, duty, or whatever other reason, but I try to keep in mind that I am engaging in risky behavior, and sometimes reminds me to be on my guard more than usual.
It means I have prepared for events that might impact my life that are beyond my control, but not beyond predicting, such as weather or other disasters, natural and man-made. It means I am armed in my home and proficient with my weapons, and that I've taken the training sufficient to know when I may and may not legally defend myself with deadly force. It means I've examined my conscience and come to the conclusion that I can and will defend my life by taking someone else's if I must.
It means I try to pay attention to my surroundings and otherwise engage in what is loosely termed 'situational awareness'.
It means I train twice a week in Isshin-Ryu karate, read books on the subject, practice in my spare time, and engage in mind experiments involving self-defense techniques and situations where this or that defense from a martial skill might be employed.
It means I try to have an honest assessment of myself in terms of my ability to engage in martial-arts-based self-defense against an actual aggressor, including my stamina, flexibility, strength, speed, level of training, and any physical disabilities. That leads to not thinking I can 'handle' things that I probably cannot.
It means I try to keep my options open and not consider things like saving face, engaging in pay-back, taking insults personally, vengeance, or extra-curricular law-enforcement actions. It means running away if that's what I feel the best survival option is at the moment, or handing over my wallet if that is what I feel gives me the best chance of survival. It means I keep in mind that self-defense is not pride-defense. I do not care about pride, I care about hide - mine.
It means committing to the actions I decide to take. If I am going to hit someone, hit them and keep on hitting them until the threat is eliminated. If I am going to draw a weapon, aim and fire it and not wave it around and hope the situation will magically get better.
It means trusting myself with myself, and not worrying about what others might think of any actions I might take.