How to Hack the Microsoft Kinect - Overview
In early November 2010, Microsoft introduced the Kinect. In essence it is a hands-free motion detection device for gaming systems. As such it can detect motion with a variety of sensors. The idea, of course, is to directly challenge the handheld device of the Wii.
The sensors include one camera, one infrared transmitter, one infrared receiver and four microphones. The visible light camera is a full-color light capture device that sits between the infrared emitter and detector. The infrared detectors are designed mainly to capture and, with the built in processor, determine depth. The color camera to capture the light ranges in normal human vision. Because Kinect also includes an infrared emitter the device is able to work in near darknes.
The arrangement of sensors is horizontal in a case. The case is meant to be mounted below the display of the gaming system. The base also includes a motor that causes the device to pivot up and down, tracking the motions of the players.
Kinect Field of View
Kinect's field of view is fifty-seven (57°) degrees horizontally and forty-three (43°) degrees vertically. The bar can tilt plus or minus twenty-seven (27°) degrees. The depth sensor can detect a range of four (4ft) feet to eleven and a half (11.5ft) feet or 1.2 meters to 3.5 meters. The infrared range camera resolution is at 320x240 16-bit depth at 30 frames per second. The color camera is 640x480 32-bit color also at 30 frames per second.
Audio is via four (4) microphones at 16-bit at 16 kHz.
The Kinect requires 12 volts DC to operate; it consumes 12Watts of power during operation. This is far greater than the 5 volts and 2.5Watts designed into the USB port in most home computers. This means the Kinect has an additional transformer type wall-plug power supply. Fortunately the data stream is well within the ranges used by USB.
Connection to USB
Because of the power requirements and the differences between that and the interface to the xBox (or any other computer) the Kinect uses a proprietary connector and an adapter to USB. The pinout of this connector (and USB) are in the photos below.
All of the above specifications are from Microsoft and should be viewed as a loose guide to the device's capabilities.
Problems Hacking the Kinect
Initially Microsoft objected to attempts to hack the device, but after Adafruit Industries offered an award of $5,000 for the first third-party open-source driver for the Kinect, Microsoft changed it's position and declared the technology open.
The winner of the prize, one Hector Martín, wrote an open-source Linux driver to interpret and capture the motion data from the Kinect.
This does not mean Micrsosoft is encouraging disassembling the machine and hacking into the built-in processor. It does mean Microsoft has no problem with users intercepting the outgoing data stream.
It should be noted that at this point in time only the most dedicated hacker is likely to be able to make much use of Hector's hack, but as time goes on simpler interfaces will grow out of Mr. Martin's research.
Why is the Kinect Important?
This isn't just about a new interface for gaming devices. The means and methods this device uses to detect player motion isn't that new1. Frankly a lot of it has been around for decades. What is new is the combination of the technologies, the already processed data stream coming from the device and most importantly just how inexpensive it is.
The initial price point of the Kinect, without the associated xBox player, is currently $150.00. This is the suggested retail price2! As the device sees wider use and economies of scale come into play I can clearly see the price point drop to well under one hundred dollars a unit. Imagine then, a sensor that can detect motion, depth, and dimension so cheap that it's the least expensive component of a larger construct.
This is exceptionally inexpensive for the uses I propose below. This in turn puts a useful and cleverly conceived sensing device into the hands of creative people who cannot afford the time and expense involved in cobbling together their own sensing devices. Devices that have historically costs thousands of dollars per unit. This does not include the time required to test, debug, and improve these designs.
Because the core problems with creating one's own sensor have been solved by Microsoft the experimenter is now only faced with how to get meaningful data out of the device. This will save a considerable amount of time, expense, and experimentation.
Hector Martin's Hack
What Kinect Sees and How it Sees it
The two videos below, one titled What Kinect Sees and the other titled Kinect with Nightshot go a long way toward explaining how the Kinect sensor can detect motion and distance. The second video in particular demonstrates that the infrared emitter does not just put out a single beam of light, but rather emits a variable pattern of various sized infrared dots. The size of the dot determines the distance from the emitter (and therefore the detector). Closer dots will be larger; smaller dots farther away. What I find most interested is the number of dots projected. There are literally thousands.
What Kinect Sees
The graphic below shows the connectors for the Kinect.
The graphic below shows the USB pinout.
USB Pin out Table
Kinect Technical Information Links
- OpenKinect | Google Groups
Open Source user group on Google. The group is dedicated to dumping and deciphering the codes generated by the Kinect. This is a great place to go to determine what the Kinect data stream contains and how to interpret it.
- Microsoft Kinect Teardown - iFixit
iFixit - Microsoft Kinect Teardown: The Kinect is a peripheral containing suite of sensors for Microsoft's Xbox 360 that provides a "controller-free gaming and entertainment experience." A how-to covers the deconstruction of the Kinect step-by-step.
Limited Game Availability vs Hacking for Something Better
So far Microsoft has released very few games optimized for the Kinect. Additionally, since the xBox itself retails for about $200 the Kinect, at $150, is seen by many gaming experts as overpriced and underwhelming.
Games that are designed specifically or to take advantage of the Kinect will be packaged with a purple color scheme.
Gaming is for Sissies
Within two weeks of product release the first hacks were being published online. Popular Science covered some of the best.
3D Room Mapping and Gesture Control
One of the first hacks of the Kinect involved mounting it atop the iRobot Romba.
Philipp Robbel, a Master of Science at Massachusetts Institute of Technology, has hacked into the Kinect infrared sensor and, by mounting the device atop a Roomba, has cobbled together a room mapping robot. As such the robot can "see" it's surroundings and create a three dimensional image map of it. Beyond that the robot can also sense arm movement from it's human master and move off in the direction indicated. Robbel has named the device KinectBot. Since the bot also has a built in WiFi connection it can transmit the data it gathers to a remote computer
"Minority Report" Object Manipulation
Another hacker, one floemuc of Germany, was able to hack into the Kinect and use it as an input and screen manipulation device in lieu of keyboard and mouse. His efforts have created the first user accessible "gesture" system for interacting with a windowed computer.
In this case the system using the input is run on Ubuntu Linux. In the example video below you can see the code-author's image, as seen by the Kinect, and the subsequent object manipulation on his computer.
Window Surface and Control
Another hack of the Kinect involves the projected image of a desktop and the Kinect as a mouse of sorts. Because the Kinect can detect both motion and depth this setup allows the user to make a mouse of his/her hands. Since other object can be sensed as well, PattenStudio projects that the technology could be used as an interactive entertainment device in public areas.
Potential interest by casinos, news departments, and commercial consumer stores us encouraged. I could also see artists taking advantage of this particular configuration to allow the viewer to interact with their works without any potential to damage those works.
Even though the device is quite good (not great though) at detection of depth and motion the four microphones embedded in it are also pretty good at recognizing voice commands and converting those commands into a useful interface.
The video below demonstrates xBox control of the desktop menu system and video streaming. This includes pause, rewind, fast forward etc.
I'm extremely encouraged both by the interest of the hacking community and Microsoft's willingness to support (or at least "look the other way") that communities efforts into delving into the device and exploring its potential.
This device not only represents a giant leap forward in game controller hardware, it has the potential for much greater and beneficial applications.
Not only is the device able to capture depth, motion and sound the combined effect is one that can be put to many beneficial and humanitarian purposes. I'll list a few below:
- As a means to detect human presence in collapsed buildings after earth-quakes, fires, and other natural disasters
- An improved vision/audio system for robotics
- A motion capture control for computer interface
- Time and motion studies
- As a means to capture the habits of fauna in the wild; because the Kinect can also detect depth this information would be much more useful to scientists who have, to date, only had flat video or still images to work from
- As a remote means of measuring dimensions; the Kinect, once calibrated can be quite accurate in measuring the dimensions of objects
- Interactive store displays in real-time
- Computer interface for those of us with debilitating injuries; for people with limited mobility or a reduced number of limbs the Kinect could become the prime computer interface for both control and dictation
Wild-Life Studies; Undersea: About a year ago scientists rigged a pair of visible light laser pointers to an underwater video camera. The object of the exercise was to capture the feeding habits of squid and simultaneously determine their size and distance from the camera. The problem was that the squid could see the lasers and avoided them. The Kinect does not project visible light, but better than that it's range of detection is well within the range the scientists needed for these studies. With an underwater housing enclosing a Kinect-like device a better means might have been available for these studies. Also, because the Kinect uses an infrared pattern for detection and distance the device would work quite well in a water-tight housing; unlike ultrasound.
Wild-Life Studies; Jungle/Savanna: The size of a particular species of big-cat is often an indication of age and health. It can also give the observer a fair indication of the big-cat's weight. With current methods a still camera is used to snap a picture when motion is detected. With a device like the Kinect the motion capture is still present, but as a video stream instead of a still image. Additionally, the infrared pattern projected and detected can give a better indication of size and distance from the Kinect.
Handicap Interface: To date, those of us humans with debilitating injuries, specifically paraplegics, have had to rely on sensors that are mouthed and designed to detect puffs of air from the operator. A Kinect like device would, instead, be able to detect head and eye motion, replacing a cumbersome interface with a much more useful and natural, to the operator, interface. Perhaps best of all, the operator would not be required to wear or attempt to manipulate anything more than his/her head.
Rescue Operations: Because the Kinect interface can detect depth and dimension mountings such a device on a robotic rescue "waldo" would give the remote operator much more meaningful and accurate information while exploring collapsed structures. Additionally, because the Kinect detects infrared, finding a trapped human would be more a sure thing, simply by sensing body heat.
I don't often do this. Usually I have no reason to. But I want to extend a heart-felt thanks to Microsoft for realizing the true importance of their device and extending an invitation to the public to use the Kinect "off-label."
This indicates a considerable amount of foresight and trust on the part of Microsoft. Publicly owned corporations do not often exhibit such generosity. They are, after all, designed to make a profit for their investors. This often means protecting, by any means legal, the intellectual property these devices represent.
Microsoft has realized that the built-in proprietary work within the device does not have to be compromised for the techie public to make use of it. All hackers need do is gain access to the stream of data coming out of the device. As such it's not a hack in the true sense of the term, but rather a simple use of that data for something other than gaming.
Kudos to Microsoft for its foresight.
December 1, 2010
Microsoft reports that they have sold two point five (2.5) million Kinect sensors. After twenty-five (25) days of availability. That is roughly equivalent to one hundred (100,000) thousand Kinect sensors per day. This figure does not distinguish between sensor only or bundled systems.
For the hacker this may not be such a good thing. As of now Amazon is showing the sensor only as back-ordered with no guarantee of delivery before Christmas. Target and other retailers are reporting similar supply problems. Walmart has some, but only as part of a bundle...at $250.00.
December 2, 2010
Availability of the Kinect is severely limited just now. This appears to be the "hot" Christmas gift this year. A quick scan of eBay shows a lot of questionable offers for this device including claims of "free." You are not going to get a Kinect free!
A quick scan of Amazon will get you this:
"This item is in high demand and supplies from Microsoft are limited. Its availability will fluctuate, and we cannot guarantee that we will receive additional quantities in a timely manner. For Kinect sensors shipped and sold by Amazon.com, we are limiting orders, when available, to one Kinect sensor per customer. We will not charge your credit card until we ship the product."
In short, be careful with your dollars. Chances are if you DO find a Kinect it will be for more than the MSRP for the foreseeable future.
February 21, 2011
Microsoft has announced that it will release a Kinect Software Development Kit (SDK) this spring. When that happens there will be another article on that subject titled "Hack Your Kinect with Mircosoft SDK."
1 Motion detection by infrared, ultrasound, and visible light has been around for decades.
2 It is already possible to buy the Kinect for less that $150 with careful research.
The author was not compensated in any way by Microsoft or any of the other vendors mentioned in this article.
The author does stand to make a small profit from the article and from any advertising linked to the article.
The author has no control over the advertising or the content of those ads.