Meh. A challenge in something I'm not particularly good at doesn't appeal. I have no need to prove myself in most things. A challenge to a chess match by someone who actually plays would bore me - I haven't played in about 30 years. Turning down a challenge that doesn't interest me leaves me with no regrets.I had someone challenged me arm wrestling in the public. You can always turn down the challenge. But you may not feel good about yourself for the rest of your life.
It's different if the challenge is something that I might learn from, in an area I am particularly interested in, etc. But even then, if it's someone who should be able to dominate me (say, sparring with a highly skilled boxer), I'm uninterested if they're actually challenging me. It's different if it's a friendly game where I'd expect them to hold back and have fun with it and still beat me.