Yes, "of" almost always is followed by whom. Who is used as the subject of a sentence. Whom is used as the object of a sentence.
Who is that masked man? (“Who” / subject)
The men, four of whom are ill, were indicted for fraud. (“whom” / object)