As a longtime manager, I've seen many employees shoot their careers in the foot by being overly casual on Fridays, especially if they're not in a traditionally 'casual' type of workplace, such as IT often offers. It is best in business settings to do a preppy version of casual - perhaps combine nice (clean, not baggy) jeans with a shirt that could be worn under a jacket or blazer. That way, you can grab your jacket if an unplanned meeting hits or an unexpected client shows up.
Avoid T-shirts with a lot of art, or ones that look too much like underwear or the beach. Knit shirts with collars are casual but professional enough to get by. Dockers or khakis work well.
One unspoken piece of corporate culture is that your highest boss can show up looking like a slob on Friday, but can hold it against you if you follow his or her lead. When a boss dresses way too casual on Friday, many staffers below that level think that's the code everyone can follow. Nope; doesn't work that way. Dress like the managers right under the most senior person, and you might have a good guide for what really works on Casual Fridays. Those managers know the unwritten rules and they're advancing in their careers. That's where you want to be.