I don't know that it's wrong if you have no other choice, but here's a story to tell why I don't give money to panhandlers:
Used to help run a restaurant that had a "streetperson" who would come by at closing time most weeknights for a handout. We'd usually give him some food and if someone felt like giving him money they would...I never did, but I had no problem with the food.
One day someone saw him come out of a building and get on a very nice mountain bike. He looked different than he normally did...more cleaned up and dressed better, but this person recognized him from the restaurant. Fortunately our streetperson did not recognize this person. So, this person struck up a conversation with him under the guise of asking him about the apartment building he came out of, and found out he lived there. The bike? That was his as well.
A few nights after hearing about this he came by the restaurant again, and I asked him about it. He made up a story that he didn't say those things, and just generally made a really poor attempt at lying. I told him that he would need to pay for his meals from that point forward, and we didn't see him again at the restaurant. I would still see him in the usual places around town, though, where he liked to beg.
Another time in New Orleans a guy came up to me dressed nicer than I was begging for change! Sorry, but I would prefer to buy someone a meal than give them money.