I can only speak about Kansai because I know it much better than the rest of Japan. I'd spend some time in Kyoto for sure, but personally I prefer Nara. Todaiji Temple and Nara Park are really nice relaxing places to stroll around. I think Todaiji is the largest wooden structure in the world.
Most guide books tell you to pretty much skip Osaka, but the food is wonderful and at night it really comes alive. It also has perhaps the best aquarium in Japan (but go on a weekday if you can!)
Back to Kyoto, Arashiyama is beautiful and Kiyomizu Temple and Kinkakuji make great photo opportunities.
In southern Kansai, Koya-san (Mount Koya) is a famous Buddhist retreat, with a beautiful cable-car journey required to get there. Also in the south is Shirahama, a hot spring resort with one of the most famous beaches in Japan (except for those in Okinawa).
One more thing, have you bought a Japan Rail Pass? It should save you a lot of money if you want to properly explore. You can use it on the bullet train and costs about the same as a round trip from Tokyo to Kyoto.