I would still go, personally. Most people don't even show symptoms if they have it and those who do usually have mild symptoms that are gone pretty quickly. It seems no more dangerous (maybe even less so) than your average flu.
Obviously pregnant women are the exception, but I don't think any Olympic athletes are pregnant while competing. Seems that people with pre-existing conditions, the elderly, and very young are at a heightened risk of complications (same as with the flu) but again I don't think any of those criteria apply to Olympic athletes who are generally in very good health.
The Olympics are a special and sometimes a once in a lifetime opportunity for people - I wouldn't let this deter me at all.