It isn't that they're "unable" to. It's more a matter of etiquette and generally being aware of normal social behavior. People may make eye contact when they first get in the elevator, or if there's reason to make contact during the ride they may exchange quick looks.
Other than that, people generally don't want to appear to either by trying to engage the other person in conversation, confrontational, or (for lack of a better term) "weird".
The fact that the elevator is closed in, and a crime could be committed behind the closed doors, makes it even more likely that people (especially men, concerned about appearing to be a potential threat) want to make it clear they have not interest in bothering the other person. Another reason is that people who don't want to catch the attention of a "weirdo" know that making eye contact may do that.
In many ways, and potential (relatively rare) threats aside, there is just a "normal social behavior" that kicks in more dramatically, as people find themselves in smaller spaces with strangers.