I can only answer for myself, but most of my relationships have ended because the person I was dating had low self-esteem.
The problem with dating a person who lacks confidence and self-worth is that they are always second guessing your motives for being with them. They are frightened you are going to leave at any minute. Consequently they don't open themselves up to you and/or they don't allow themselves to depend on you in any way.
I can't be with someone who is so emotionally guarded, its a barrier to an honest relationship. If in spite of my best efforts I can't win their unconditional trust, I'll leave. Proving them right and perpetuating the problem.