I think there are a lot of reasonable explanations. One example comes to mind and that is my mother found a tiny fawn in her back yard once. It was adorable and you really wanted to keep it, but since it is against the law here, the game warden was called. The reason they said is because they're hooves are like razors and since they kick with them as a defense, they could easily kill someone. Of course this is after they're grown and stronger.
On the other hand, I don't believe it's fair to most animals. There are so many that have specific needs and even nutrients that can't be found in human table food or livestock feed. (Hard to find Tiger Chow at Pets R Us these days ;)
Too, some people would grow bored with the pet or become too busy to offer the time and attention it needs. Could you also imagine someone forgetting to close the door on their tiger's cage only to hear screams coming from the neighbors on the next block?
All too often we read of pythons, etc running away from home, something they normally only do when hungry or neglected in some other way.