I am not sure the Bible gives a clear definition so it may be difficult to say exactly who they were, other than knowing what is written about them:
Genesis 6:1-4 (NIV)
1 When human beings began to increase in number on the earth and daughters were born to them, 2 the sons of God saw that the daughters of humans were beautiful, and they married any of them they chose. 3 Then the LORD said, “My Spirit will not contend with humans forever, for they are mortal; their days will be a hundred and twenty years.”
4 The Nephilim were on the earth in those days—and also afterward—when the sons of God went to the daughters of humans and had children by them. They were the heroes of old, men of renown.
It has been suggested they were the fallen angels.
I always wondered if they have anything to do with the ancient skeletons that are found and considered Neanderthals that are not quite human. It is just a thought. No proof to back it up.