If you are living in a moderate temperature, it's fine. Like, say the summers don't get any hotter than 80 degrees at any given time. THEN, it's fine.
If it gets "summer hot" and you are wearing a scarf....you look insane. Don't do it. You look like you're trying too hard to be "chic" and you'll come across as fashion illiterate.
Even if it gets fairly cool where you are, it just looks unseasonable and silly, and people will be like "why is that girl wearing a scarf?" So I'd avoid it.
BUT little summery scarves that tie around (gauze or cotton-SUPER lightweight kind) can be fun-with a tanktop. For a shawl in the eves like the girl below said.
You just REALLy have to pull it off correctly or you will end up looking silly and out of season.
Good luck and happy fashion!