We have a 4 year old and to start, we never say punish. It has a much too negative connotation. We instead always say 'consequence' to him.
What we do is give him time outs if his behavior is really poor. We also explain why we're giving him the time out and do our best to not raise our voices to him.
We don't have set times for our time-outs. We instead wait until he calms down. It has sometimes taken upwards of 10 minutes before he calmed down enough.
We also have him apologize before he is allowed to get up. And, if he gets up before we give our OK, then he has to sit more. We also make sure to turn off music, TV, keep toys out of his reach, and do not provide any attention to him during that time. If he asks for anything, he also needs to wait until after the time-out is completed before he can have it.
If he is instead just being too rough with a toy, then we just take it away. As long as he calms down after that is taken away, then we do not provide a time-out.
If his behavior is otherwise poor and he is really looking forward to something, then he doesn't get to do that. For example, he loves to take rides with me to the store in my van. If his behavior is poor, I leave him home with my significant other and go by myself. We also explain why he has to stay home and tell him it's a consequence for bad behavior.
The other thing we do (unless it's extremely poor behavior) is give him the opportunity to correct his behavior. We tell him that he needs to be happy, stop doing [insert bad behavior], etc. or [insert consequence] will happen. If he doesn't stop, then the consequence happens.
The main thing is to follow-through with what you say you will do each and every time. If you follow-through every-time, then your little ones learns you mean it (though it takes patience before your little ones learn this).