I would have them come over and interact with the kids a little and observe if its natural for them or if they arent quite sure how to interact with them.
I would ask what experience they have had with children and what AGES of kids (may only worked with older kids and not know what to do with your baby or only babies and not know how to tell your 8 year old NO when they are trying to get away with something).
If its not a teenager I would run a background check- I live in wisconsin and there is a website you can search people's names and see what comes up. I would ask for references and call them...be nosey! They are your kids!
If it is a teenager I would want to meet their parents as well...the apple doesnt fall far from the tree. Again, ask for referenecs. Ask if they have taken the babysitting class. I would have a hard time leaving my kids with a teenager that I didnt know though. I would be worried of them snaeking boyfriends over or smoking/drinking or just trying to get away with things without their parents around.
I would ask if they smoke (they would have to step outside and not be in sight of your kids then)
Search for them on facebook or myspace before they come. Look at their pictures and see what kind of things they are into-partying or down to earth. You can get a good idea before they come to talk to you who you are dealing with.
My biggest thing would be how they interact with the kids and if they seem trusting.