We have a 3 1/2 year old and 6 month old.  Here is a list of our top 5 things to do in the city.

1. Bay Area Discovery Museum-The best part of this museum is the outdoor play area.  There are no slides or swings but instead kids explore and interact with SF area icons like the Golden Gate Bridge under construction, sea vessels, tide pools, native animal homes, mazes made of large twigs, trees perfect for climbing, and wonderful caves.  Artists have also donated their work.  On the weekends docents roam the play area and interact with the kids.  When we were there one Sat. one docent was playing the guitar and singing with the kids.  The indoor activites are also worth the time.  My daughter loves the art studio and the Tot spot.  The Tot spot doesn't allow children over the age of 3 which can be difficult if you have older siblings, but it makes the experience solely about the younger children and they are less likely to be trampled.  The outdoor tot spot allows the children to watch as fish make their way down a man-made stream.  There are aprons to avoid their clothes from becoming too wet.  There is also a discovery theater with a wide range of performances for children.  The only down side is there is an additional admission fee.  I wish these performances were free with museum admission.  The Discovery Museum is located in Sausalito.  Exit Alexander Avenue, the first exit, when coming over the Golden Gate Bridge from SF and follow the signs and winding roads to the Discovery Museum.  The cost of admission is $8.50 for adults and $7.50 for children over one year of age.  We became members and felt like it was definitely worth it.  Even though it takes us 30 min. from our house in the city, we go quite frequently.  Check out their website if you're interested for more information.
2. Crissy Field-We love Crissy Field!  On a sunny day in SF, I can't think of a prettier place to go.  Views of the Golden Gate Bridge, Alcatraz and the city abound.  Our daughter loves to play in the sand and walk down to The Warming Hut for hot cocoa and a cookie.  On the weekend we like to go as a family and bring our dog.  Park in the lot closest to the Marina and dogs can play off leash.  We then walk down the flat path to The Warming Hut and back.  This is a great place to appreciate SF.
3. Golden Gate Park and the Carousel-What toddler doesn't love a carousel?  We went to the Carousel in Golden Gate Park this weekend and our daughter loved it.  It's a really nice carousel and we went on a great walk in the park afterward.  This experience will be even better when the playground next to the carousel is finished (it is under complete reconstruction) and the Academy of Sciences is completed.  Golden Gate Park will probably be one of our most favorite places really soon.
4. California Academy of the Sciences Steinhart Aquarium-Our daughter loved the aquarium especially the African Penguin feeding.  The feeding happens daily at 11 am and 3:30 pm with a biologist giving information.  The aquarium is open every day of the year 10-5.  On the third Thursday of the month the admission is only $5 for everyone and they are open late.  The aquarium is located at 875 Howard St.  The new site will be open in Golden Gate Park in 2008.  Check out their website for more info. 
   
5. Ride a cable car-What toddler doesn't love transportation?  Whether it's riding the muni train, a ferry or cable car, they are having fun.  Our toddler loves just driving down Lombard St, the crookedest street in the world.  The cable cars, although touristy, are fun.  Kids love to look around, wave at the other passing cable cars and feel the wind.  

For more info on activities in the city or any large city in the US, check out gocitykids.com.  They are a wonderful resource for daily activities and neighborhood activities for kids.