I say give them another go. The first couple books in the series are more kid-friendly, but around the fourth book they shift into a more adult, darker, serious tone. That's not to say they're dark and depressing: they have lots of humor and fun parts too.
Do you like fantasy? Have you seen the movies? Many people will tell you that the books have so much more to them than the movies, and they're right. I know fantasy's not for everyone, but HP is very accessible. It has many of the themes and archetypes of classic myths and the hero's journey.
Whether you like them may also be a generational thing. That's not to say that adults can't enjoy them--so many adults do, and HP is not written specifically for children. But HP was a big deal for the young generation, with the readers literally growing up alongside their heroes Harry, Ron, and Hermione.
Obviously, I'm a huge fan. But even if I weren't, I'd say to give them a try just because they were a great cultural phenomenon--like Beatlemania. You wouldn't want to say you've never listened to a Beatles song!