Your question is dealt with reasonably by both Besarien and Tamara. You will find that many single women know that the man is married, and for the sake of balance, many single men know that the woman is married. This makes both single and married adulturous. Again, a point not mentioned is that the sex should be consensual.
Jimmy Carter once said that the Christ set impossible standards, as if you look at a married person or indeed a single person with lust in your heart, you are already committing adultery.
You have not asked this, but let us take it a bit further. In the higher spiritual life, lust and indeed sex would one day have to be transcended. Many of our Christian monasteries were built in the woods, hills and forests for this reason, as they knew the inner disturbances of mixing with the opposite sex.
Modesty in clothes such as wearing Habits, muslim and indian style dresses, were originally designed to minimise attraction and temptation, or lust, leading to lower carnal desires.
Our Christianity has many: the Cistercians, The Carmelites; the Trappist Monks; The Desert fathers; The Jesuits; the Francisians and many more contemplatives. Still, I am not asking you to run too fast, for you may break your legs. Slow and steady wins the race, and again, you should also feel that this 'calling' is yours.
Meanwhile, Paul advocates a life of self-control, of marriage, if one is unable to abstain. Abstaining before the 'call' is injurious, and promiscuity after the call is also injurious. Let us all act according to our nature and the dictates of the inner Will.
Let me end by saying that Desire is very necessary, and it is that which will push us from the animal life, to the life Divine. Again, I am also saying that a time will come when that life will give way to the descent of the Higher forces, in order to attract the purity of a greater Light. Much Love.