Just comparing point A to point B is irrelevant without considering the time it took to go from one to another.
For example, if you move 30 feet across a room in a minute, you are fine. Cover the same distance in a microsecond and you are a splat on the wall.
Similarly moving 1 degree over a millennium is far different then over decades. Over a longer period of time biotic communities have the ability to gradually adapt to changing conditions. Too quickly and they can not.
And while generally an increase in a single variable such as temperature or CO2 levels may be good for plants, reality isn't that simple. Plants are dependent on a complex interaction of variables. You could get more CO2 but less water, or higher temperatures but also greater damage from insect pests. It just isn't that simple, and tinkering with global biomes and centuries of agricultural patterns is risky at best.
As for moving, well it depends. People of affluence, sure, they can move wherever they want, but just take a look at how disruptive the relatively small stream of refugees from Syria are, or immigrants into America from Mexico. Now just imagine that instead of 4 million or so it is 40 million or 400 million. Good luck with that.
It is also really important to keep in mind that alterations to climate will not be evenly disributed. It isn't like every place on the globe will get exactly a degree or so warmer at all times. Some place may get cooler, some hotter, some may end up with wider seasonal spreads, some might have more variability year to year. And again, none of this is inherently bad. It is only problematic if it happens in a timeframe which does not allow people and biotic communities to adapt.
Of course, all of this is without even considering some of the really nasty potential consequences of climate change.