Yes, you need to wear a large brimed hat while gardening.
The hat serves two purposes; first of all, it protects your head from direct sun rays. Secondly, the hat serves as a 'body cooler' when your body temperature elevates.
The hat, along with water consumption will keep you comfy and cool while gardening. Also, keep you safe from a possible heat stroke.