It varies from person to person. For some it may be their physical appearance, fashion/style, for others their confidence, personality, or sense of humor, and with others there is a "meeting of the minds" where you're naturally on the same page when it comes to your opinions about things you consider important.
It's really not about "them" a much as it is about how (you) feel when you are around them. Note: Everything is fluid. It's not uncommon to initially see someone as being "attractive" but after getting to (know them) you feel otherwise.