I wouldn't say it's about distance as much as it is about the circumstances. If someone is trailing behind you with their arms full of boxes you might hold the door open for them even if they're several feet away. The same could be true if they were running towards the door or elevator. As you noted it could make a difference if the person is elderly, in a wheel chair, or on crutches...etc
If someone is walking leisurely several feet behind, your holding a door open for them may cause them to feel (rushed) in order not to inconvenience you because of your "kind gesture". For me personally if it's a normal person walking further than 4 yards behind me I typically will not hold the door open.