Clearing land for agricultural use isn't a problem in the developed world. American farmers now grow exponentially more food on less acreage than they were using in the nineteenth century. The land no longer devoted to farming is now in forest (of which there is a somewhat more now than then), road right of way (of which there is much more now than then), and urban development (of which there is vastly more now than then). The developing world will presumably follow the same path: once it reaches an advanced stage, land starts reverting to trees and being given over to construction.
Most farmers (I think, not being one myself but knowing some) would say that modern farming methods are far less harmful to land, water and air than the ones used a hundred years ago. Given how much improvement we've seen in those things, I'd say they're probably right. That position appears to be true in both relative (i.e. per unit of produce) and absolute (i.e. overall) terms.
Based on those considerations, I'd say the basic premise is incorrect. More agricultural output does not equate to more environmental damage, and therefore there is no reason to pay more for food to avoid a problem that doesn't exist.