It depends on how it is asked, the inflection and syntax involved, and the conversation leading up to that question are important.
Sometimes people ask that question with tone of voice that says, "I'm going to invite you to my church, cuz you need to go, and you know it." . . . .that could be rude.
I'd say that it is not an offensive question at all if it's random and out of the blue; but if I've been telling you for weeks about how I love to hang out at topless bars and go to Richard Dawkins book club meetings. . . . .the you should already know what kind of reaction that you'd get from such a person.