Not necessarily. If you have respiratory issues, like asthma, it might aggravate it. Some people are allergic (usually sneeze). However, in my opinion, it is really only an issue when breathing in the dust. Like when you are taking handfuls out of the bag, etc. It usually settles and isn't bad after that. If you think it is an issue, try finding fresher bags/brands.
I think people have more issues with the dust from the beding than they do the hay.