Olive oil is generally safe for humans, even infants. There are only two things you want to look out for. First is that you want to make sure to get Organic Extra Virgin Olive Oil, because it is free of harsh chemicals used in non-organic non-virgin brands. The second thing to be mindful of, is that your baby might have an allergy to olives or olive oil, so only try out a very very small amount first on his or her arm, then wait a few hours to see if a rash, red mark or welt develops. Then take it in small steps from there.
I'd like to suggest though, that while Olive oil is a good choice, I would get some Hemp Eaze. It's totally non-toxic, uses only fresh organic herbs and is great for dry skin. Get the "Baby and Me" spray, it works the best. =)