I believe how well you do on an exam should not be the only criteria for judging people; however, it is a significant aspect of the holistic review of a person.
Talent is not something that can always be judged by exams. You might fail your next physics test because that is not what your talent is, but you could just as well win an art competition because that is what you are passionate about.
Therefore, when people apply to colleges they don't just look at your SAT score and your GPA, but they consider your other achievemnets aswell. Surely, SAT scores (exam scores in general) are an indicator for how much effort you put in, but it is also true that maybe that is just not how your talent can be judged. Exams can judge those skilled academically to a certain degree, but what about someone who is a talented dancer, singer or artist? For those people, exams donot bring their talents to light.