Because we look at faces. We also look at the overall physicality of the person we're talking to, but that is generally only a quick glance. However, emotion is not just shown on the face but in the whole demeanour of a person. That is why body-language experts are so able to made evaluations as to how people are feeling, e.g. the defensive stance of arms folded across the chest.
What is particular relevant about faces, though, is that over time our habitual expressions leaves lines, creases that can be interpreted as a fairly accurate guide as to our more habitual mood. The happy person and the so-called 'smile creases.' around the eyes, the perpetual frowner and the lines between the eyebrows.
Certainly there are those who are really observant and, perhaps, have made a study of faces and can tell whether a person has had some significant trauma or upset in their lives. I recall a neighbour remarking that I - at the time I was nineteen and had broken up with a girlfriend - remarking that I had "pain lines in my face." This was laughted at and scoffed at by others who were present at the time. I, of course, said nothing about the incident. But I knew the woman was 'spot on.' For I was feeling pain at that time.
So, yes, emotional pain is shown in the face, be it a current or recent event, or something which might have happened years ago.