You must get backlinks from the same niche . And links must not be in large numbers .
If you get backlinks from another niche or unrelated site , then ask that site owner to remove your links ,,, If you do this , you will rank higher in SERPs than that you do at present for sure .
I am gonna write a hub about it soon . Stay tuned ..