Lets make this usable as a mouse!

Oct 11, 2008 at 10:59 PM
Edited Oct 11, 2008 at 11:00 PM
go it working on my desktop, and i am freaking amazed.
i put red electrical tape on the end of my finger, and boom, i was playing snake. that is , after i realized i didnt have .NET installed.(very non-descript error message.)

hows this, 2 colors of tape, on 2 fingers.
make the red one used to move the mouse, and when the 2nd color become visible its a click. maybe add another color for right click.

i have little experience with code, but i think this community can make it happen. I plan on doing some learning and actually contributing. C++ sound like a good place to start?
i plan on using this with my projector, maybe using LEDs as markers, as tape in the dark will, my guess, not work.

Who's in?
Oct 12, 2008 at 6:20 PM
Trum4n, I appreciate your enthusiasm!
Tape on your finger should work great. And using a second marker for a click is definitely one of the more popular ideas.It's pretty easy to hide your thumb behind your other fingers and use a "thumbs up" as a click.

If you're unfamiliar with the supported languages (C++, C#, and VB), I'd recommend C#; it's very easy to learn.
This might be a good place to start your research: http://msdn.microsoft.com/en-us/library/ms646970(VS.85).aspx

Using Touchless with a projector is a great idea, but you're right that it will require active markers (something that emits light like and LED).

You might want to add a feature request in the Issues section to setup a transformation between webcam input and marker on-screen coordinates.
That way you could easily setup direct manipulation with the camera at an angle that wouldn't get so much background noise from the projector's light.
I imagine lots of people will want this feature.
Dec 10, 2008 at 7:03 AM
Tried it, and it works well.

One problem is the 'shivering' marker, it wont stop even if you dont move the pointingobject. this makes clicks hard to do, plus the problem that 'drag and drop' operations get interrupted from time to time, resulting in a doubleclick.

I'm a noob, but I'll try to work on the marker-recognition algorithms, cause thats where i see the problem: changing markersize. I got a nice idea for a workaround :)
Dec 12, 2008 at 3:31 AM

If you have a mouse control demo/app using Touchless, I would love to try it for myself and share it with our community. I would like to help address some of the problems you're having; I could potentially help you code. I've had a TODO on my plate to add something like exponential decay for stabilizing the markers within Toucghless, but I don't know when I'll get to that. (I'm definitely seeking capable contributors!)

I've thought of app-side approach you could use to smooth the marker motion. Perhaps try using the marker location as a force to act upon the cursor location you're setting. Think distance-based attractive forces or emulating a spring; these methods introduce some delay between actual and tracked locations, but they would reduce the 'shivering' that you're seeing.

As for clicking, perhaps you could have a similar heuristic to make invocation and releasing-invocation more reliable. Imagine having a value aggregated over a small amount of time (say 30-300 ms) that would accumulate your current metric for invocation. Once this value reaches a threshold, it would cause the actual change in the invocation state. It would require additional logic to disambiguate clicks from mousedown-drag-mouseup, but it would certainly be more reliable.

- Mike
Dec 12, 2008 at 8:35 AM
Sorry Mike, at this point of time I cant share the code, yet.
Maybe in a while :)

I've send you an message with some more datils about the problem.

Thanks for your time
Dec 12, 2008 at 6:17 PM
Wow Pollux, nice work!  Really hope you are able to share it out soon.
Dec 14, 2008 at 5:03 AM
Edited Dec 14, 2008 at 5:06 AM

Here is my implementation of MouseDemo. Cursor is a bit shaky... but I think that can be solved if in detection algorithm we keep same size of Area (or use absolute center of detected area). I'll try that in the morning (must get some sleep now :).

Great library by the way! (if you don't wish to compile) (complete solution, removed source bindings) (contains just my additions... you'll need to download Touchless source if you don't have it - http://www.codeplex.com/touchless/SourceControl/ListDownloadableCommits.aspx)

Dec 18, 2008 at 9:21 AM
  Hello guys, why you don't  make  this code   to be an driver for CMPD (Camera pointing device)  who work like HID (Human interface device) on PC. That's gone be like mouse i think !??

Dec 24, 2008 at 1:07 AM
I have almost same idea as Machineria to be able use it as mouse. I can have cursor control by a marker. However it is shaky.Trying to figure out a way to stabilize it.
Dec 24, 2008 at 1:49 PM
Edited Dec 24, 2008 at 2:58 PM

Hi all and merry Christmas to all of you!

I recently created a code snippet using a marker in the cam image center area (80% of width and height) as mouse, mapped to the whole screen. I use the dead edges to perform different kinds of clicks when a marker appears here for a short time after being hidden "long enough".

This seems to work quite well but yet has the problem that you don't "see" where you hover this marker in order to click.

I will try to enhance this by overlaying the whole screen with a transparent camera image and will report back my progress...

maybe I will attach the code to the appropriate issue ticket (see #6424 under "issue tracker").

[edit: fixed typos, explained "image center area" and click edges, added reference to issue ticket #6424]

Dec 25, 2008 at 12:10 PM
Hi all,

I am new at coding as well using Touchless sdk. 
I tried the code posted by lepipele. But it doesn't seem to work for me.
I tried to do the same with my code but getting the same error when it comes to simulate the mouse click.
Can anyone help please?

The error is:
Managed Debugging Assistant 'PInvokeStackImbalance' has detected a problem in 'C:\Data\Downloads\Touchless\Source\Samples\TouchlessDemo\bin\Debug\TouchlessDemo.vshost.exe'.
Additional Information: A call to PInvoke function 'TouchlessDemo!TouchlessDemo.MouseDemo::mouse_event' has unbalanced the stack. This is likely because the managed PInvoke signature does not match the unmanaged target signature. Check that the calling convention and parameters of the PInvoke signature match the target unmanaged signature.

for line:
mouse_event(MOUSEEVENTF_RIGHTDOWN | MOUSEEVENTF_RIGHTUP, Cursor.Position.X, Cursor.Position.Y, 0, 0);

It would be nice if some one can upload the complete working project. Thanks in advance.
Dec 26, 2008 at 4:11 AM
Edited Dec 26, 2008 at 4:12 AM

What version of Visual Studio are you using?

Replace lines from #40 to #71 in MouseDemo with this, and tell me if it works (I've changed mouse_event signature, now parameters are uint's instead of long's):


        // send mouse clicks
        [DllImport("user32.dll", CharSet = CharSet.Auto, CallingConvention = CallingConvention.StdCall)]
        public static extern void mouse_event(uint dwFlags, uint dx, uint dy, uint cButtons, UIntPtr dwExtraInfo);

        private const int MOUSEEVENTF_LEFTDOWN = 0x02;
        private const int MOUSEEVENTF_LEFTUP = 0x04;
        private const int MOUSEEVENTF_RIGHTDOWN = 0x08;
        private const int MOUSEEVENTF_RIGHTUP = 0x10;

        private int _leftclickCooldown = 200;
        private DateTime _previousLeftClick = DateTime.MinValue;
        void MouseDemo_Click(object sender, MarkerEventArgs e)
            if (_previousLeftClick < DateTime.Now.AddMilliseconds(-_leftclickCooldown))
                _previousLeftClick = DateTime.Now;
                    (uint)Cursor.Position.X, (uint)Cursor.Position.Y, 0, UIntPtr.Zero);

        private int _rightclickCooldown = 1000;
        private DateTime _previousRightClick = DateTime.MinValue;
        void MouseDemo_RightClick(object sender, MarkerEventArgs e)
            if (_previousRightClick < DateTime.Now.AddMilliseconds(-_rightclickCooldown))
                _previousRightClick = DateTime.Now;
                    (uint)Cursor.Position.X, (uint)Cursor.Position.Y, 0, UIntPtr.Zero);

Dec 28, 2008 at 4:38 PM
I can approve the uint parameter type works, in my implementation, I use
private enum MouseEventDWFlags

public static extern void mouse_event(
UInt32 dwFlags, // motion and click options 
UInt32 dx, // horizontal position or change 
UInt32 dy, // vertical position or change 
UInt32 dwData, // wheel movement 
IntPtr dwExtraInfo // application-defined information 

internal static void MouseLeftClick()
0, 0, 0, new System.IntPtr());
0, 0, 0, new System.IntPtr());

My MouseLeftClick() method clicks at the current mouse pointer position, not the passed one, attention when copying...

You can surely defer the other mouse buttons or double clicks if needed...

Cursor.Position = ... will help you "move" the cursor using native .net code...

Jan 5, 2009 at 5:01 AM

Thanks for you Help Guys. I am using VS 2005.
The modification by lepipele did worked for me. Thank you. 
Feb 4, 2009 at 3:12 PM
Hi guys! am new on this Touchless SDK and I would like to participate on your project about implementing Touchless as mouse.
I know it's a bit hard to simulate CLICKING on this one as we know WE ARE NOT touching anything .. so a Gesture instead is a great idea and more flexible.

- Jayzon

Feb 9, 2009 at 9:02 AM
Download the latest source to use the new Mouse Demo in TouchlessDemo.
(No building required, just run bin\release\TouchlessDemo.exe)
The first draft of Mouse Demo is added with Change Set #28723.

Be careful, it really does control your mouse with markers!
Read the demo notes for instructions.

Thanks to all of you, particularly lepipele and eFloh for help and inspiration.
Improve the demo, suggest changes, report experiences, and post video links!

- Mike