Yes, because they were put up as protection and probably for a damn good reason. As long as it stays up you are in your fort. Nothing can touch you and if need be, you can go on the attack.
Letting your guard down makes you vulnerable. I let my guard down recently and got burned! I learned that if someone isn't willing to give what they get, there is no reson to give it to them!
Keep your guard up, you are #1 in your life.