It can definitely lead one astray. Perhaps a person who looks angry simply has poor vision and is straining to see. Perhaps a person who will not look others in the eye during conversation is suffering an extreme grief that no one knows about. Perhaps a person who is fidgety has a disc in their lower back that is distracting them with pain. We need to give each other the benefit of the doubt rather than making assumptions.
That said, taking the time to get to know someone can lead us to understanding their tells. That takes time and effort, though, and it must be attempted with an understanding that we still do not know all the details.