Every event does not draw a big crowd and these are work days, so people cannot just leave their jobs and the attendance of the night events could possibly be attributed to the location -- not close to population centers.
Finally, you can see a lot more clearly on television. I imagine as the Olympics get closer to the finals, the attendance will pick up.
If you are not a swimmer, watching a swimming meet can get boring, until you get down to the finals. No hard evidence, just my speculations.
That's got a fair bit of attention down here, too. One of the theories is that the seats were reserved for offical types (employees of the sponsor companies, etc.) who just aren't using them. Another is that they're unsold tickets that were assigned to other countries, which just weren't returned so that they could be used. Or, it could be a combination of both, and a few other reasons.