Some do. I would say most do.
Just not the divine kind.
Not the absolute kind, like in karma or someone going to Heaven or Hell based on what they've done. But yes many atheists want and understand justice on some level. In fact this may be a human universal, a sense of fairness, justice, and reciprocity (the Golden Rule has existed in one form or another in nearly every culture).