There are two reasons, I think ~ one 'righteous' and one 'unrighteous'. The righteous reason is when the 'church' takes on the doctrines of men and starts imposing all kinds of works to be 'saved' or 'keep salvation'. Some of them twist the Word of God's true meaning and impose 'indoctrination' of the teachings of men instead. The unrighteous reason is when people fall away from the faith because they love to sin and thus claim, "I'm not religious" (meaning they have chosen to no longer follow Christ).
Jesus confirmed the 'righteous' reason for leaving a 'so-called church' in Matthew 23:13 "Woe to you, teachers of the law and Pharisees, you hypocrites! You shut the kingdom of heaven in men's faces. You yourselves do not enter, nor will you let those enter who are trying to."
Truly the Head of the 'Church' is Jesus (Col 1:18). He is the High Priest (Heb 4:14). All believers that follow Him, within or without the organized churches, are part of His Church. He said in Mat 18:20, "For where two or three have gathered together in My name, I am there in their midst."
May each one of us simply stay in the faith, loving God with all our hearts and loving our neighbors as ourselves (Mat 22:37-40).