Amazon S3 is a service provided by amazon where anyone can backup files, and optionally serve them over the internet.

Amazon S3 can be a great tool for users to backup files, but it is also an invaluable resource as a file server especially with its affordable prices.

We would honestly recommend that you use Amazon S3 (and possibly CloudFront) wherever possible.

Amazon S3 used in combination with CloudFront allows you to serve your content from servers local to all of your users at an extremely low price.  Many sites don't spend more than a few pennies a month.

If you are interested in how to integrate Amazon S3 with Wordpress, check out this blog post on Integrating Amazon S3 with Wordpress.