In my youth I have knowingly slept with married women or those who were involved with another man. However I wouldn't consider myself being "the other man" since I had no "emotional investment" with these women.
In my opinion the "other woman/man" is a title used to define a person's role in a "relationship" (ie: boyfriend, fiancé, or husband.). The "other person" is usually expected to be faithful to the married person. (Waiting in the wings....) They don't usually get to spend time together on holidays. This leaves the single person going out alone to their company holiday party and countless other events. They are pretty much regulated to living life on "stand-by". If you are emotionally invested in the arrangement I would imagine at some point you would get tired of playing second fiddle. There is also a possibility that you are not the only "other person" as was the case with Tiger Woods. (There is not much to be evious about.)
If it's a situation where two people just get together to have sex every once in while with "no strings attached" they probably don't think of themselves as being (the other anything). The same could be true of a man who is regular client for an escort. He's having sex on the side without a relationship. Naturally the escort/prostitute does not think of herself as being "the other woman".