Let me ask this, do you occasionally give food to these people. If so, like stray cats, they will not leave. They figure if they pester you enough, you will cave.
If you have never feed, given money, or even the time of day, then I would contact your local law enforcement. If the panhandlers show up at irregular intervals then this may not be practical.
One last possibility includes a sign stating "This property protected by Smith and Wesson."