Integration with Multipoint SDK

Feb 27, 2009 at 11:12 PM
I'd love to be able to use Touchless as a fake mouse input device, together with Microsoft Multipoint SDK (that supports multiple mice [can each have a fancy cursor like an animal head etc.]) for educational applications (combined with camera and projector and kids moving their colored banner to move their mouse each one arround or even wearing a colored hat and walking arround in the room)
Mar 5, 2009 at 4:16 AM
Hi birbilis,

Thanks for the comment, I've heard of Multipoint before, but haven't used it myself.
It would definitely be interesting to see multiple cursors under independent control using Touchless.

I can tell you from experience that kids enjoy non-traditional computer interaction, like Touchless.
One little dude was fascinated by drawing on the computer with M&Ms at my last fair demo.
You may be interested in the SMART Table project, more or less Surface aimed at kids.

Do you have any coding experience? I'd be happy to offer help with using Touchless.

- Mike
Mar 5, 2009 at 11:10 AM
Thanks for the pointer to SMART Table, very interesting one. Multipoint (multiple cursors), Touchless and MultiTouch (MS Surface) are very interesting technologies.

I've also looked a bit at Intel's OpenCV (Open Computer Vision) since I'm into robotics too (doing a PhD). I have plenty of programming experience, I'm a Microsoft MVP btw (J# / R&D).

Speaking of interesting technologies, have you looked into the new Microsoft Tag? Wouldn't it be interesting to also support it apart from plain colored markers? They seem to have developed a coloured-pattern-drawing method that is like a barcode-descendent, but it is a 2D square of small colored triangles. They can detect and read its data, even from blured and out of focus (or shot at an angle) photos. MS is suggesting its use with mobile phone cameras from what I've read, having customers shoot at such tags they find on products etc. and automatically get more info or extra options/services via the net about that product/service on their phone. Imagine Touchless also allowing people to print such random markers (maybe MS Tag provides software for that already) and register them via the webcam to use for visual manipulation.

BTW, seeing a video of Microsoft Surface usage, there were some handle-cubes that had a dotted pattern under them (a fat dot at the middle and some dots arround like a dice, but in a way that one could find 2D rotation of the cube since at least one dot on the cube's bottom side edges arround the central dot were missing)

Back to the point, one crude way to integrate MultiPoint with Touchless might be to implement a virtual mouse device driver that would use IPC to talk to a .NET process which would use Touchless. That  is unless the MultiPoint SDK provides some entry point for one to register virtual mice. Making the virtual mouse device driver would allow of course Touchless to be more generic and integrate with other stuff too on Windows
Mar 6, 2009 at 1:05 PM

Just found another project strongly reminding me of this one...

Mar 25, 2009 at 4:43 AM
Edited Mar 25, 2009 at 4:44 AM
I thought readers of this thread might be interested in a MultiPoint SDK update.
Here's a Channel9 article with a video interview and some additional info:
Who wants to get crackin on some MultiPoint integration?!?

- Mike
Mar 25, 2009 at 3:38 PM
Edited Mar 25, 2009 at 11:42 PM
Thanks all for those links. Some notes:

1) the Touchless library (for building MultiTouch applications [using FTIR method on some surface]) seems to have a virtual mouse driver for Windows (according to the YouTube video above). For starters a separate thing from MultiPoint would be to allow one or more colored tracker via Touchless to work with that virtual mouse driver and its multitouch functionality (even in the case the NUI group [who make the MultiTouch lib] don't have something generic for any windows version, MS have such multitouch functionality for Win7 from what I read [maybe to act on files, windows - say zoom them in/out - etc. maybe])

BTW, should also see the "Sixth Sense" presentation at TED from MIT people:
regarding intergration of technology like Touchless and MultiTouch

- the ImagineCup competition (for Student Developer Teams) has a MultiPoint app topic and the Unlimited Potential group that makes MultiPoint allow one to get an extra chance to win something for the best MultiPoint app (not necesserily the winner app of the ImagineCup MultiPoint theme from what I understand)
- the new MultiPoint SDK seems to support registering other input devices apart from mice too. Maybe it can more easily be integrated now with Touchless lib