Self-esteem is about loving yourself. Being "special" is like beauty in that it's in the eye of the beholder. Only an egocentric person thinks they're "special" or better than others. We're all "unique" though. :)
However if someone (chooses) you to be their friend, lover, or spouse (they) are in fact declaring that (you) are "special" when compared to others they've met. It's about how (they) relate to you or feel when around you.
Naturally it's possible to be considered "special" in the eyes of one person and be viewed as "ordinary" by another person.
As I said it's all in the eyes of the beholder.
Having "self-esteem" is what keeps you levelheaded rather than falling into the trap of being emotionally invested with what others think about you. Know yourself, Love yourself, Trust yourself.