Not to sound skeptical of jaded, but do you have a clear diagnosis from a qualified physician regarding her mental illness? Whether you do or not, it's a difficult situation that I've experienced myself, though my mother-in-law lived with us for about fifteen years!! She claimed all sorts of illnesses, even went so far as to call all the family together to announce that she only had six months left to live......and that was sixteen years ago. She's still alive today at age 73. She was very disrespectful of both me and my husband, she was rude and caused constant turmoil between the extended family members and she undermined us, me especially, with our children.
I tried to remain composed and secure for the sake of my husband and kids. My children absolutely loved her and, even though he saw her faults, she was my husband's mother and he did feel responsible for her. My husband finally made the decision to put her in an apartment complex for the elderly.
However, what I learned later was the extent that she had hurt my children and how she was the cause of the often rocky relationsip we had with our own children. I didn't see it then because she had convinced the children to lie to me about it or to keep certain things hidden from me. Now that my children are older, they have admitted to me the things she used to say or do to them. Had I known then, I would have put my foot down and she would not have had all those years to influence them.
So, my advice? First, be sure there is a real illness there. Don't just take her word for it....or your husband's. He may have believed it without question since he was a child and may not see the real truth.
If there is an actual illness you need to deal with, look for other options. There are people trained to deal with this sort of thing. Does she need to be placed somewhere other than your home?
I know it would be a really tough thing to do but in the end, you have to protect your children, your husband and the unity of your own family.
I wish you luck!