The Tale of Touchless

Oct 7, 2008 at 6:12 PM
Edited Oct 8, 2008 at 8:45 AM

Greetings, wise fans of Touchless! I’m Mike Wasserman, creator of this webcam multi-touch SDK. I get a huge kick out of solving coding problems, especially when some relatively simple code can unleash a lot of power from ordinary webcams. Touchless enables multi-touch by using a webcam to track colored objects, which means touch without touching. I’d like to tell my story of working on Touchless and fill you in on the road leading up to this public release.

I’m a work force rookie who joined Microsoft after graduating from Columbia University just over a year ago. By day I code features and tackle bugs for Microsoft Office. But outside of my vital duties to Office, I spend some spare time working on my side project. Touchless is the continuation of one of my undergrad projects at Columbia. The original demo let the user transform simple shapes and draw with two markers. Even after joining the working world Touchless stayed in my mental periphery. I just couldn’t put a stop to my continuing passions for human-computer interaction, augmented reality, and computer vision. Also, despite the growing success and adoption of multi-touch technology (think MS Surface and iPhone), there is still a huge potential to reach a wider base of home users.

Working at Microsoft is great because we get peeks into Microsoft Research projects, and lots of my peers have similar creative outlets and nerdy passions. I rewrote my project using XNA for Microsoft Office Labs’ Productivity Science Fair and demonstrated it for a lot of people, including Andy Wilson (the brilliant mind behind Microsoft Surface). Some laughed at the goofy little marker pellets on my fingers but they still saw value in the project. Office Labs Community Projects soon thereafter expressed interest in helping me release the project as an SDK. I gladly accepted their support and direction (huge props to Gary Caldwell and intern extraordinaire Isabel Mattos) and intensified my own efforts. All that work has already paid off. At the next fair Touchless was voted by attendees as “most interesting project.” You really should have seen how much this little kid enjoyed playing with M&Ms to control the drawing and pong demos. I even caught the attention of Chris Pratley (General Manager of Office Labs) and even Stephen Elop (President of Microsoft Business Division)!

That brings us to today, releasing Touchless to the world outside Microsoft, as a *FREE* *OPEN-SOURCE* SDK with a low barrier to entry for developers and users. Touchless makes developing multi-touch capable software easy, and the results can be enjoyed by anyone with a webcam (and some M&Ms :) )! Touchless has come a long way, but its journey has just begun. I can imagine a variety of great directions to explore using Touchless. Perhaps support for the forthcoming Windows multi-touch API, video chat white-boarding, implementing mouse/game/hotkey/media control, and designing decked out Minority Report style file/media browsers.

Oct 8, 2008 at 1:26 AM
A little confused by this demo.

This is more of a object tracking than "multi touch" is it not?

I have been monitoring these kinds of things and the likes of ARToolkit (uses glyph tracking, VERY good success rate as well. High FPS. A windows mobile phone, can track a glyph @ 120fps).

I did some experimenting with your app. It's pretty cool. I have been investigating ways for Wiimote like functionality on PC's. I have been developing in C#/WPF with the Wiimote and its a very difficult beast to handle even the simplest motion tracking.

Doing 1:1 type motion tracking on the Wiimote is very difficult because all you have is 3 axis accelerometer. It's like coding a physics engine for an automated flight system basically it feels like.

I experimented with your library and used a flashlight as a marker. It tracked it decently but it still judders alot and not very smooth. High motion I think it's losing track of it as well.

Also noticed this is taking VERY high cpu. Not much room for gfx etc unlike ARToolkit which really dumbs the image down to black/white/low color count etc. But that is because it's detecting glyphs which are black and white.

Any thoughts on how to use this as a motion tracker for gaming? :) I think this market has huge potential because many people have webcams. Getting reliable x,y,z from a webcam of a tracked object via webcam has huge potential.

Heck even a Silverlight 2 tennis game could be made which would rival the wii!
Oct 8, 2008 at 2:41 AM

Thank you for your honest and rather well informed post, Synced.

You're right that the fundamental functionality of Touchless is color-based object tracking. I believe that Touchless can serve as an excellent low-cost *platform* that caters to a large user base, namely anyone with a webcam. This platform is well suited for building applications that explore alternative forms of computer interaction, including multi-touch object manipulation, and gesture based interaction.

I have worked with ARToolkit in my past life as a college student; It's undoubtedly good at glyph tracking. There are, however a variety of tradeoffs between glyph tracking and the style of colored object tracking that Touchless employs. Development work done with the Wiimote in this space is also exciting, but that has another set of tradeoffs. A few differentiating benefits exhibited by Touchless are the existing large user base of people with webcams, the integration with full-fidelity webcam imagery, the flexibility of choosing everyday objects for tracking at runtime, and the very low barrier to entry for developers and users of Touchless. Regarding my last point, I'll hopefully soon post our *15 minute* walkthrough video of how to build a rudimentary drawing application with Touchless, and perhaps a short demo video that shows more of how I like to setup my working environment for Touchless Demo.

I am glad you think my Touchless Demo is cool. I find it interesting that you tried an active marker (your flashlight), but you'll probably find that passive markers (my favorite are M&Ms) will work better. This is mostly because the appearance of the light emitted from the flashlight will vary greatly as you change the relative placement and orientation of the flashlight and camera. The flashlight will also often cause webcams to adjust the lighting in their image, and this causes an unstable tracking environment.

It somewhat difficult to characterize the processor usage of Touchless across a wide range of hardware and environments, but I frequently find rather low usage in most reasonable scenarios. Optimizations and algorithmic improvements can undoubtedly be made within the internal workings of the Touchless SDK, while still offering a stable platform for client development efforts. I hope that our quickly growing community will join me in this and other efforts toward making Touchless even cooler.

I think there is a great potential and increasing demand for games that use alternative input, and additional interest in games that offer augmented reality experiences. Touchless is incredibly well-suited as a platform to deliver alternative sources of user interaction. And I hate to sound like a broken record, but what we create can be enjoyed by anyone with a webcam! There's no need to buy more expensive hardware. I certainly have ideas for games that I would like to build using Touchless, and I've heard my friends' ideas, but I would rather see the community explore their own creative ideas than have everyone hang on my suggestions... but I would like to see an awesome multi-player missile firing and defense game :-).

Thanks again for the feedback, and let this serve as a warning: If you engage me, I'll respond at length.

Oct 10, 2008 at 11:18 PM

Hehe thanks for your feedback!

I'm new to this space of technology but I come from a mobile background (Windows Mobile, BlackBerry etc).

I see things like WPF/Silverlight and see great potential for rich applications. I see touch/multi touch/object recognition/motion controllers as a potential for interactive controls.

One problem as an application developers perspective is things like the Wiimote are very difficult to develop against. It's like developing a flight simulator trying to read those accelerometers and understand the orientation/where the device is.

Things like object tracking however has a much more 1:1 type potential I think without the craziness of writting physics/flight simulator code.

Although M&M tracking is cool and all those possibilities have other uses, I think where this library could really shine is if its capable of productizing its motion tracking.

Let me try and put it into perspective. Things like TrackIR do what their built for very well because they set you up for success. They use IR light, developers receive rotation angles, they don't have to handle all the nuances of the calculations.

The Wiimote example again shows where a library sets up a developer for failure. It's very complicated. Most games are bad because its difficult for a developer to "get it right".

If Touchless library could somehow setup users for success, developers can target this library, and use it for some very powerful situations.

In my quick trial of your sample, the motion was very jerky, the box was very sparatic. This means developers targetting the library are going to have a very difficult result set to try and settle down. Again like the Wiimote only as good as their own skill.

For a library like Touchless to really shine is if it can provide the developer with much richer end result. There are algorithms out there for smoothening motion results etc.

That with the combo of perhaps some really well known objects the library is excellent at tracking, if developers are getting very good x,y,z etc coordinates, I don't see why things like TrackIR capable drivers can't be written via Touchless if markers are setup properly etc.

Perhaps my vision is a bit too vast but when the general joe can have head tracking without 300 dollar cost, wiimote like 1:1 motion control for games/etc via a webcam, it can really unlock some powerful uses.

Just look at a video like ArmA with TrackIR. Very kickass cool stuff, but definately NOT in the realm of general joe because it costs $$ and not many games support it.

Enter a library making it capable via webcam and perhaps you have more developers jumping on board, and users.

Perhaps I'm too ambitious though :)

Oct 19, 2008 at 9:19 PM
Edited Oct 19, 2008 at 9:21 PM
Hi Synced,

You raise some good points, particularly how important it is that we shape our SDK to setup developers for success.
You're right that it takes good form and function to make the most of touchless.
I hope we'll continually improve the algorithms and interfaces in order to achieve this goal.

Improved marker smoothing is definitely something I'll work on.
Providing more information about the marker (such as it's axis of least inertia and roundness) is also an important work item.

I really appreciate the time you've taken to provide this good feedback.
I encourage you to continue, and perhaps review our API and/or code.
Suggestions in our Issue Tracker and submissions of code changes are great ways to improve Touchless.
Please also feel free to get in contact with me directly, especially if you'd like to become a contributor.

- Mike