I believe it's their (potential) political capital that scares some people if they are granted full citizenship with voting rights. Hispanics are now the largest minority in the U.S. making up 17% of the population while blacks make up 13%.
Another concern for some folks is the shift immigrants have made in their approach to American society. It used to be an immigrant came here and (adapted to the customs and language) spoken here.
Thus creating a "melting pot". Today the U.S. is more likely to bend over backwards providing countless services in the immigrant's native language. Very little osmosis takes place.
I personally know of a couple from Mexico who have been here for almost 30 years and speak very little English. When I lived in California the drivers license exam was available in 5 different languages. Why bother to learn English?
Most TVs come with the SAP option to hear audio in Spanish and there are various newspapers, radio stations, and markets specifically catering to this segment of the population. In Garden Grove, CA there is a neighborhood called "Little Saigon" and just about every business sign is written in Vietnamese. If you blinked you'd think you were in another country!
The overall feeling is instead of today's immigrants (adapting) to the USA the country is adapting to them. I suspect this causes a lot of resentment in areas that have a higher influx of Latino immigrants.
The reality is no one can rise to higher paying careers in the U.S. without speaking English. Good intentions may be causing long-term harm. The real fear has to do with their power to vote.