No, it's not wrong in some cases. Having spent considerable time with people dying of chronic degenerative diseases, I've seen a lot of the conflicting emotions that their loved ones go through. On the one hand, they want to keep this person with them. On the other, that person is in constant pain and wasting away before their eyes. When you know the suffering can only end with death, it may be the most loving thought in the world to wish they would die.
That said, if you wish them dead just because you can't stand them, there may be a degree of wrongness in that.