Starting from the point of 'figment' being an invention of the mind, you've landed in what some call 'Cloud Cuckoo Land' if your invention is too fantastic for those of us of a conservative persuasion. Anybody who comes up with an idea that most find upsets their 'equilibrium' lives in Cloud Cuckoo Land. You can take the 'figment of your imagination' sort of reasoning to mean anything between 'get your head seen to' and 'you're dreaming'.
You could take it as a compliment if it doesn't outright rile you to distraction because nobody can understand what it is you're trying to put across. And you're in good company.
'I think we're being followed', said in the middle of the Gobi Desert would be answered by 'It's a figment of your imagination'. Anyone might think you're under the influence, delirious.
'I think we're being followed', said on top of a Manhattan skyscraper would definitely get the same answer. Those of us with imagination would look up into the air. There could be a chopper up there, with somebody watching and taking pictures from a distance. That's when it gets into the realms of 'persecution complex'. When both of you look up, that means there's trouble!