Of course this is mearley a metaphorical statement, although it is more often used in the sense of using your heart to 'feel' not think.
You either go with your head or your heart; you either go with your 'logical thinking' or your 'gut feeling'.
The heart is associated with love & feelings, this goes back a long way & is associated with the energy centre located around this point; the heart chakra.
The brain is where we analyse situations & do the 'thinking' this is the 'processor' so to speak of human beings.
The heart is not literally where we feel emotion but as it is associted as one of the main internal organs which also happens to be located at this energy centre, it is where we appear to 'feel' emotion.