There are a lot of reasons for that, and they generally have little to do with many of the conspiracy theories you'll hear about from a question like this.This is a complex market place and there is not a simple answer.
One is Oil is priced in US dollars globally. So the value of the dollar directly effects the price of Oil. Recently the dollar has gained some strength relative to Japan for example. But the overall trend for the dollar has been lower for some time, hence making Oil and other commodities more expensive over the last decade plus.
Another issue is many of these enormous discoveries in the US in places like North Dakota still require the oil to be transported to a refinery.Since we haven't built a new refinery in a lifetime and won't because the regulatory environment makes it too costly, Oil must be transported from places like ND to existing refineries. With no pipeline having been completed, the oil is transported via rail cars and tractor trailer. The makes the process substantially more expensive. Although the process to transport oil via rail car has improved a lot over the last few years.
Another aspect is OPEC. Oil is a global commodity. Not everything produced in the US is sold in the US. That has to do with Geography. If you look at map, imagine transporting Oil from Alaska to NY versus transporting it to Russia. As a result of geography, new product is sold and delivered wherever it is more cost effective to deliver it. This global market means it is influenced by global demand, which is still growing in parts of the world faster than it is in the US. But being part of a global market makes us subject to the actions of the OPEC cartel, which is not US friendly and effects global demand. As demand declines, production will decline. It is in their best interest not to increase productivity as long as minimum demand is met. Most of the major Oil companies are in the business of buying Oil from OPEC controlled cartels and refining the product for use. They are mostly in the service industry. So when Oil has a spot price of $100 per barrel, that is not what the Oil company sells oil for...but rather what they are often paying to buy it...which translates into the price you will pay through the supply chain. And the last time I checked we imported about 60% of consumption. Yet the price of natural gas has come down dramatically as this is a much smaller global market and not subject to as much geopolitical risk.