For 25 years, the field of robotics has been bedeviled by a fundamental problem: If a robot is to move through the world, it needs to be able to create a map of its environment and understand its place within it. Roboticists have developed tools to accomplish this task, known as simultaneous localization and mapping, or SLAM. But the sensors required to build that map have traditionally been either expensive and bulky or cheap and inaccurate. Laser arrays cost a few thousand dollars and weigh several pounds, and the images they capture are only two-dimensional. Stereo cameras are less expensive, lighter, and can construct 3-D maps, but they require a massive amount of computing power. Until a reasonably priced, easier method could be designed, autonomous robots were trapped in the lab.

On November 4, a solution was discovered—in a videogame. That’s the day Microsoft released the Kinect for Xbox 360, a $150 add-on that allows players to direct the action in a game simply by moving their bodies. Most of the world focused on the controller-free interface, but roboticists saw something else entirely: an affordable, lightweight camera that could capture 3-D images in real time.

Within weeks of the device’s release, YouTube was filled with videos of Kinect-enabled robots. A group from UC Berkeley strapped a Kinect to a quadrotor—a small helicopter with four propellers—enabling it to fly autonomously around a room. A couple of students at the University of Bundeswehr Munich attached a Kinect to a robotic car and sent it through an obstacle course. And a team from the University of Warwick in the UK built a robot that had the potential to navigate around post-earthquake rubble and search for trapped victims. “When something is that cheap, it opens up all sorts of possibilities,” says Ken Conley of Willow Garage, which sells a $500 open source robotics kit that incorporates the Kinect. (The previous non-Kinect version cost $280,000.) “Now it’s in the hands of just about anybody.”

Robot freaks weren’t the only people to explore the Kinect’s possibilities. Researchers, visual artists, and pornographers have all begun cobbling together home-brewed Kinect projects and posting the results online. Artist Robert Hodgin built a makeshift motion-capture animation program that allows users to manipulate video of themselves on the fly, turning their bodies into bulbous cartoon characters or reflective mercury-like blobs. Two students at Germany’s University of Konstanz bolted a Kinect to a helmet, creating a bare-bones navigational system for the blind. And a company called ThriXXX built a rudimentary sex game that allows players to rub women’s body parts with a creepy disembodied hand.

None of these projects were sanctioned by Microsoft (especially that last one). Indeed, for the past few months, if you wanted to use the Kinect on anything other than an Xbox, you had to install homemade drivers cobbled together by a dedicated group of hackers. Yet the company’s official response to all this activity has gone from hostility to acceptance to vigorous support. In June, Microsoft expects to release a software development kit that makes it easier for any academic or hobbyist to build Windows applications using the Kinect’s camera and microphones. The company is also granting access to the high-powered algorithms that help the machine recognize individual bodies and track motion, unleashing the kind of power that was previously available to only a small group of PhDs. (Microsoft is also working on a commercial version of its software development kit, which will allow entire new businesses to be built using the Kinect’s technology.)

Major manufacturers have long recognized the value of letting customers modify their products, a fact obvious to anyone who has ever swapped out factory-issue tires for performance treads. Many successful technology companies have encouraged independent developers to build on top of their platforms—consider, for instance, Windows, Facebook, and the iPhone App Store. And over the years, modders have introduced several innovations that have grown into entire product categories—like mountain bikes, heart-lung machines, and rodeo kayaks.

But today, unsanctioned tinkerers have more power than ever. Sophisticated computers, sensors, and accelerometers are all common ingredients in personal electronics, available for $100 or less at Best Buy. As a result, the kind of equipment that was recently available only to research universities or major corporations is now accessible to anyone with a cell phone and a soldering iron. That has dramatically altered the kind of projects modders can take on. “If you’re talking about changing the spoiler on the back of a Ford, that serves a very specific purpose,” says Eric von Hippel, a professor of technological innovation at MIT’s Sloan School of Management. “But a depth camera or an accelerometer or a GPS chip enables not just one application but a wide range of new activities.”

When DIYers combine those cheap, powerful tools with the collaborative potential of the Internet, they can come up with the kinds of innovations that once sprang only from big-budget R&D labs. In 2009, a PhD student named Daniel Reetz turned two Canon PowerShot A590s into an improvised high-speed book scanner. He detailed the project on a website, DIYbookscanner.org, where readers have since posted hundreds of tweaks, suggestions, upgrades, and entirely new designs. The open source MPGuino project, which uses an Arduino microcontroller to track gas consumption as you drive, has inspired a small community of fans who help refine and customize the gizmo. In December 2007, a Carnegie Mellon PhD candidate named Johnny Chung Lee reverse-engineered the Nintendo Wiimote to create a 3-D display that responded to the position of his head. Videos of his hack garnered millions of views on YouTube. Eventually, Lee landed a job at Microsoft’s Applied Sciences Group, where he worked on the Kinect, among other projects.

• 1

  Microphone array

  Four mics pinpoint where voices or sounds are coming from while filtering out background noise.

• 2

  IR emitter

  Projects a pattern of infrared light into a room. As the light hits a surface, the pattern becomes distorted, and the distortion is read by the depth camera.

• 3

  Depth camera

  Analyzes IR patterns to build a 3-D map of the room and all objects and people within it.

• 4

  Tilt motor

  Automatically adjusts based on the object in front of it. If you’re tall, it tilts the box up. If you’re short, it knows to angle down.

• 5

  USB cable

  Transmits data to the Xbox via an unencrypted feed, which makes it relatively easy to use the Kinect with other devices.

• 6

  Color camera

  Like a webcam, this captures a video image. The Kinect uses that information to get details about objects and people in the room.

Illustration: Kate Francis/Brown Bird Design

For most companies, the response to such hacks falls somewhere between benign neglect and lawsuit. When users began jailbreaking the iPhone—modifying the operating system so it could run unapproved apps—Apple quickly declared the practice illegal. (Last July, federal regulators determined that jailbreaking was allowed under the Digital Millennium Copyright Act.) When a hacker named George Hotz published code allowing anyone to run applications or operating systems on the PlayStation 3, Sony responded by pursuing subpoenas against Hotz, investigating his PayPal account, and collecting the IP address of anyone who visited his website, before settling in April.

A few companies, though, have welcomed the hackers. When iRobot learned that academics and hobbyists were rewiring its robotic vacuum cleaner, the Roomba, the company released a special vacuumless version—the iRobot Create—designed explicitly to be modded. “It really builds awareness of our company,” says Kristen Stubbs, who until recently served as iRobot’s outreach program manager. “When people do cool things with our robot and our platform, it’s great exposure.”

But until now, no company has made it so easy to hack into a product as popular as the Kinect, the fastest-selling consumer-tech product of all time. The Kinect racked up 10 million sales in just four months. That means 10 million people now have fully functioning depth cameras (which measure the distance between the Kinect and objects in front of it) sitting in their living rooms. And Microsoft is giving every one of them the tools and its blessing to build new applications with those cams. “We’re trying to usher in a new era of computers, a world of tomorrow,” says Xbox general manager of incubation Alex Kipman, who adds that the Kinect’s gesture-based interface is an early example of how we will soon interact with all of our computers and appliances. “It’s going to take a lot of people—from within Microsoft as well as outside it—to enable that to happen.”

Kyle Machulis’ Berkeley apartment looks like a museum of obscure technology. His desk is cluttered with rare machines—a handheld computer for teens called the Cybiko; a 40-button controller created for the Xbox game Steel Battalion; a couple of haptic gadgets that jiggle and buck when touched, creating the illusion of texture—as well as an array of sex toys, oscilloscopes, and personal data-tracking devices. Machulis has already figured out how to control most of these gadgets, posting source code for drivers (which allow programmers to connect the devices to other operating systems or applications) on hacker websites like GitHub so that anyone can write software for them. “Basically I just pick up new hardware, whatever seems interesting at the time,” Machulis says.

In the fall of last year, what seemed interesting to Machulis was the forthcoming Kinect. A couple of years earlier, he had spent weeks building a rudimentary body-tracking program; if he held his hand in just the right spot, his computer could recognize it. The Kinect, with its built-in color and depth cameras, promised to make that process much easier and more sophisticated. On the day the Kinect was released, Machulis rushed to GameStop, eager to be the first to post open drivers for the new machine.

Machulis was not alone. One hour after he returned to his apartment, he saw a posting on the website of Adafruit Industries, the open source hardware company run by hacker impresario Limor Fried. The company was offering a $1,000 bounty to whoever posted the first open source Kinect drivers to GitHub. Would-be modders should imagine being able to use the Kinect with “Mac, Linux, Windows, embedded systems, robotics, etc.,” the announcement read. “Let’s reverse engineer this together,” it encouraged, and “make cool stuff!”

Not everyone was so enthused about the project. By 4 pm that same day, in response to a question about the so-called Open Kinect contest, a Microsoft spokesperson told a reporter from CNET that “Microsoft does not condone the modification of its products,” threatening to “work closely with law enforcement and product safety groups to keep Kinect tamper-resistant.” Adafruit reacted by bumping the bounty to $2,000. Four days later, Microsoft released another statement, pointing out that the company would not support the practice of using the Kinect with any device other than the Xbox and “strongly encourag[ing] customers to use Kinect for Xbox 360 with their Xbox 360 to get the best experience possible.” “This is silly,” the Adafruit team responded in an online comment, “so now we’ve made it $3,000.”

On November 9, Machulis had almost finished his driver. It took a bit longer than he expected, largely because capturing the data streaming out of the Kinect’s USB cable required a costly piece of equipment called a USB analyzer, which retails for around $1,000. Machulis convinced Adafruit to purchase the device and publish the results online, but that process took the better part of a week. Still, mere hours after Adafruit had posted the USB analyzer data, Machulis and a group of online collaborators had figured out how to control the Kinect motors and had initialized the camera. The Berkeley modder went to bed around 1 am, confident that nobody else would be working while he slept.

But Machulis didn’t take into account the fact that on that very day, the Kinect had been released in Europe—where it was still morning. By the time Machulis awoke, Hector Martin, a hacker from near Bilbao, Spain, had claimed the prize. Martin, who didn’t even own an Xbox, had submitted a video showing all of the color and depth-camera source code displayed on his Acer Aspire. The key to Martin’s speedy accomplishment was Microsoft’s decision not to encrypt the USB feed—thereby making the data Adafruit had extracted readily usable. “It was easier than I expected,” Martin says. “Microsoft was nice enough to say, ‘We aren’t going to tell you how it works, but we aren’t going to prevent you from figuring it out.'”

That benevolent attitude didn’t seem to mesh with Microsoft’s public statements. In the days after Martin’s victory, nobody knew how the company would respond to this public flouting of its stated wishes. “Everyone just assumed that we would get sued,” says Phil Torrone, Adafruit’s creative director.

But that didn’t stop the hackers. Now that the drivers were public, every day seemed to bring an exciting new innovation. A German company called Evoluce built a gesture-based control system for Windows. A group of interactive designers conjured a way to use the Kinect to turn any surface into a multitouch interface, so a user could control the action on a screen by dragging their finger across a desk, wall, or book. And two artists based in New York designed a digital puppet, a giant bird that moved in coordination with a user’s arm, wrist and hand. The projects found an eager audience online, where they were posted on popular tech sites like Engadget, Boing Boing, and TechCrunch. A new site, Kinecthacks .com, was created just to keep track of the massive output. (As of mid-May, 58 pages of videos were posted on the site.) Meanwhile, Machulis and other programmers continued to refine the open drivers, figuring out how to adjust the color camera’s white balance, how to control frames per second, and how to make it compatible with Mac computers.

If the hackers were worried about Redmond’s response, however, those fears were allayed on November 19, when Microsoft’s Alex Kipman appeared on Talk of the Nation, NPR’s call-in show, to discuss the Kinect. About 18 minutes into the program, the show’s host, Ira Flatow, read a question from a listener: “Are you going to talk about the guy who hacked the Kinect?” Kipman responded that technically, the Kinect had not been hacked. Nobody had accessed the Xbox’s proprietary algorithms or figured out a way to cheat the system. They had merely tapped into the USB connection, which Microsoft had left open by design.

“So no one is going to get into trouble?” Flatow asked.

“Absolutely not,” Kipman said.

Three months later, Microsoft would go even further. On the Microsoft Research site, the company announced that it would make it easier than ever to modify the Kinect by releasing its own software development kit. In a matter of weeks, Microsoft’s reputation within the hacker community had completely flipped. Instead of acting like a lumbering, power-mad hegemon, it had lent its support to what was shaping up to be one of the biggest and most successful open source development projects the world had ever seen.

And there was one more surprise to come. The same day as that Microsoft Research posting, Johnny Chung Lee, the Wii hacker who had worked at Microsoft on the Kinect, made an announcement of his own. In a blog post, he confessed that he was the one who had come up with the idea for the Open Kinect contest—while he was working for Microsoft. After months of fruitlessly trying to persuade the company to develop drivers for Windows, he wrote, he approached the Adafruit team and asked them to host the competition. He put up the prize money himself. Lee had left Microsoft for Google not long before the software development kit was announced, and now that Microsoft had committed to opening up the Kinect, he felt comfortable going public. “Best $3,000 I ever spent,” Lee wrote.

Lee’s announcement was a final triumph for the hacker community, which set about congratulating itself for showing Microsoft the error of its ways. “Any time Microsoft releases something from now on, they’ll think about this,” Adafruit’s Torrone predicts. “I think they got religion.”

Sitting in his corner office on the Microsoft campus, Craig Mundie doesn’t seem like he’s undergone any recent conversion. Mundie, the company’s chief research and strategy officer, has been at Microsoft for 19 years. He is wearing his ID on a lanyard around his neck, despite the fact that he is among the most powerful people at the company. Mundie can be quite eloquent when discussing the potential that the Kinect represents, a so-called “natural user interface” that lets people navigate the digital world through gestures rather than mouseclicks and keyboards. “We want humans to interact with computers much like they interact with other people,” Mundie says. “That’s our goal.”

But when the topic turns to the Kinect hackers, Mundie’s demeanor turns taciturn. Did he have any reaction to the Open Kinect contest? “No.” What are his opinions about Adafruit? “Haven’t looked at that one.” Any thoughts about Johnny Chung Lee? “I don’t know the guy.” And those hackers who bent the Kinect to their will, seemingly making it perform feats for which it was never designed? “They were just going down the path we planned.”

Indeed, Microsoft’s reaction to the entire Open Kinect kerfuffle seems to be: Nothing to see here. According to Mundie and Kipman, the company always expected tinkerers to experiment with the Kinect—that’s why they left the USB feed unencrypted. Microsoft always planned to release a software development kit, executives say, but simply couldn’t devote the necessary resources to it until after the Kinect had launched. And those threatening public statements that so exercised the Kinect modders? Simple miscommunication—a knee-jerk response from an untrained PR representative to a question about “hackers.”

Whoever deserves the credit, the relationship between Microsoft and the Kinect hackers has blossomed into an unlikely love affair. Microsoft is now openly courting modders. When the company announced the software development kit in February, it followed that news with a demonstration of the navigational system for the blind developed by the University of Konstanz team. “We can help create together,” says Anoop Gupta, a distinguished scientist at Microsoft Research who helped implement the software development kit. “I think we can do amazing things.” Meanwhile, the hackers are ready to help. “I think it’s sad that most companies can’t see the value of their products outside their initial idea,” says Hector Martin, the Spanish hacker who first cracked the Kinect. “There are millions of people who might have better ideas they would never think of.”

If Microsoft has any doubts whether it made the right call by inviting hackers into its fold, Sony’s recent experience should serve as an object lesson: Less than two weeks after Sony announced its settlement with George Hotz, hackers launched a full-on assault against Sony, breaking into its PlayStation Network, its Qriocity media-streaming service, its Online Entertainment division, and several other properties, potentially accessing the login and credit-card information of at least 70 million users. Sony’s PlayStation Network was down for more than three weeks while the company scrambled to repair the breach—a delay that annoyed customers and did untold damage to Sony’s reputation. Although the identity of the culprits remains a mystery, Sony has suggested that the break-in might be a retaliatory act from the hacker community for the company’s dogged pursuit of Hotz. In other words, by embracing hackers, Microsoft benefitted from their enthusiasm. By punishing them, Sony became a target of their considerable wrath.

More companies are beginning to adopt the Microsoft approach. Motorola recently amended its policy of locking down its Android phones, announcing that future devices will be easier to modify. Sony Ericsson has a web page devoted to helping hackers unlock its phones. In May, Google released its Android Open Accessory Development Kit, which gives the operating system to tinkerers who want to build their own Android-powered devices. (Even Sony’s PlayStation division has belatedly tried to join the effort. On March 2, even as it pursued its case against Hotz, the company announced that it would release a software development kit for its PlayStation Move controller in an attempt to “inspire applications that we could never have imagined.”)

Still, Microsoft isn’t waiting for a bunch of hackers to unlock the Kinect’s potential. The company is investing millions of dollars in the traditional in-house model. Teams of researchers have built a rough prototype of a kind of Kinect-powered holodeck—images, projected into the real world in 3-D, that respond to touch. Others are combining the Kinect’s camera with advanced optics to create an LCD screen that can beam different images to different viewers. And Microsoft is already preparing its first major Kinect software upgrade, which will capture facial expressions as well as body movements. All of this work is expensive and difficult and the result of many hours of dedicated labor—the kind that probably won’t be replicated by a loose-knit group of enthusiasts.

Or at least not today. That’s because high-end optics, next-generation LCD screens, and proto-holodecks cost a lot of money, and figuring out how to build them takes a lot of time and close collaboration. But depth cameras were once prohibitively expensive, too; the Kinect changed that. And as more and more technology becomes commoditized—and as the web continues to make it easier for far-flung individuals to work together as a team—hacker communities will grow more and more capable. As everyone gains access to the same resources, the best ideas will win. And millions of hobbyists will usually come up with more interesting ideas faster than a few thousand professionally employed engineers.

That’s how Johnny Chung Lee sees things, in any case. Lee is sitting on the patio outside Charlie’s, the famously lavish cafeteria of his new employer, Google. It’s a sunny April afternoon, and Lee is dressed in a monkey-head T-shirt from hacker outfitter ThinkGeek. Although Lee has become something of a hero since publicizing his involvement with the Open Kinect contest, he is reticent. He isn’t interested in rehashing his conversations within Microsoft or taking credit for the company’s decision to release a software development kit. It doesn’t really matter, he says. “What matters is having such an interesting new device embraced by so many smart people and having the fruits of that labor embraced by the company that created it.” Lee smiles. “I think it paints a bright future.”

Music: Pump Audio/Getty

Wired New York editor Jason Tanz (jason_tanz@wired.com) interviewed Square CEO Jack Dorsey in issue 19.06.

• 19.07
• features