The reason for both existing is historical.
The 1611 King James Version of the Bible has "Holy Ghost" because that was how people refered to "spirits" then.
Luke: 1:35 "And the angel answered and said unto her, The Holy Ghost shall come upon thee, and the power of the Highest shall overshadow thee" (KJV)
Later, "ghost" (as Miles said, German-family word, from the Anglo-Saxon) came to refer only to spirits of the dead, and "spirit" (a new, Latinate, word) to spiritual beings of all kinds, but the term "Holy Ghost" stuck around because it was in the only Bible English speakers read before the 20th century, and which is still beloved today.
Personally, I like them both, in different contexts.