I think it can be difficult to swap clothes if you feel like you want to get equal value. I have found it is best to pass on clothes (and things in general) without especially expecting anything in return. Two of my favorite dresses were given to me from girlfriends because they felt the dress suited me... That's how I try to pass on my clothes.
On a similar, but slightly different subject: pass on the baby clothes! No swapping needed-- new parents need all the help they can get, and except for the occasional weird stain, baby clothes are never worn out.