I say call it like it is. The problem society has today is that too many people are offended by too many things, and because personalities all differ, it's very difficult to discern what may NOT be offensive to one and BE offensive to another.
Obvious derogatory terms notwithstanding such as you mentioned. Geezer, for example, is an obvious one. Old man not so obvious. If you are on in the years you are old. Even if old could potentially mean used up or out of useful commission. Old could also mean built better, more solid, or more ornate. An old house has character, for example. Nowadays it's just two-by-fours, cheap walls, and a roof.
Some blacks like to be called black. Others like to be called African-American. I've run into blacks who hated to be called African-American and blacks who hated to be called black.
When we buy too much into this sort of thing it just makes life more difficult, makes people look more fragile, and helps to create circumstances of avoidance. If I don't know what to call you I just won't talk to you...
Perhaps it's why people don't even bother to say hello on the streets anymore. SOMEONE might be offended you've said hello.
This is not the kind of world I want to live in. I don't think it's the kind of world you grew up in. And it's this kind of thinking that is quite literally ripping apart the fabric of our communities all over the country.