Reading the original post, it could be your "evil looking" face scared him to the point that he doesn't consider being safe around you anymore, but it could be simply that the face offended him and he has decided your lack of ability to take a joke simply isn't worth your company.
I'm not going to describe myself as it isn't relevant, but I will say that in life I've had people give me what they considered intimidating or angry looks (from law enforcment, prosecutors, U.S. Attorneys, federal agents, legislative bodies, etc.), and I haven't been frightened since I was a young adolescent. I'm much older (read decades) than that now. If I were joking around, and someone gave me a look that indicated they were angry, I may ignore it, or I may apologize to keep the social function moving if they were a real friend to make them feel better (perhaps I did misspeak or they had a bad day). I may pick on the person some more to see if they were joking about the look, and call them out on it, or I might decide the person was an individual with a poor disposition and simply not worth my time and ignore them for the rest of my days. But I surely wouldn't be afraid of them.
Since the guy did approach you later and make a statement, or reference it, later on, I didn't get the impression he was afraid of you. But he did seem offended. There is a big difference.