This is tricky, Unless your grandfather has dementia or some other kind of mental illness I think you shouldn't let it drive you crazy. Unless he's incapable of making his own decisions I think he should live his final years exactly the way he wants. He cant take the money with him, so without sounding cold I think you should let him enjoy his little escapade. Sounds like its not really hurting anyone on a deep level. And if this is the last relationship he has I guess he should get some good memories from it. Well at least his moneys worth.
Or you could just get a doctor to find him mentally incompetent and draw up a power of attorney with your lawyer. Then you can control the cash flow. Just make sure you take your grandpa out for some good times.