I've used both. I started with Joomla, then after my web sites got defaced a couple of times, I tried Wordpress. I haven't had a security incident since.
It's true that Joomla is a CMS (content management system). It is what makes joomla's underpinnings more complex, and thus more susceptible to code bugs, and consequently security vulnerabilities.
Wordpress originated as a blogging platform, but it has evolved to something much more--blurring the difference between it and other web site framework like Joomla. There are so many plugins for Wordpress that you pretty much can do most things you need it to do.
Today I use Wordpress for my personal blog and my personal business web site. It is easy to set up and maintain. Updates to it and its plugins are as easy as clicking a link.