All houses of worship were built by men, not God.
Not going to church, temple, mosque, etc. does not make you any less of a faithful man or woman in the eyes of God. You can pray in the shower, at your desk, or even at the gym. As long as you believe in God and live a good life, it doesn't matter where you pray, only that you DO.