Magic Wand - Wiimote Gestures
Overview
Senior Design
For my BioE senior design project, we had to do some motion tracking. Initially we were using a pretty capable unit with 6 degrees of freedom, but something with it wasn't working, and we didn't narrow it down to a hardware problem until late into the semester. So we needed an accelerometer, but didn't have time to get one shipped. Where we could turn to?
The Wiimote, of course! Hackers all over the intarwebs have been finding innovative uses for this widely availalbe and cheap piece of technology, not the least of which is Johnny Lee. I had previously purchased an extremely cheap bluetooth module off EBay with the intention of doing something with Wiimotes. Once I started working on Wiimote code for Senior Design, I quickly found that it was pretty simple to play with, so I decided to try out a project idea I had: modifying the Dollar Recognizer to recognize Wiimote gestures.
Wiimote Viewer
and repeater graphs!
The WiimoteViewer program. Seriously, those graphs are mesmerizing.
My first task was to learn to use Brian Peek's Wiimote Library for VB.NET. This library is great; it handles every hard part and just gives you the information you want.
First I just wanted to visualize the accelerometer data from the Wiimote. While the test program included with the WiimoteLib is great for viewing button and IR status, it's hard to read the accelerometer values. I wanted to see them change over time, so I implemented a small user control that I've been working on and finding more and more uses for: a Repeater Graph.
The repeater graph works sort of like an EKG display. As points are collected over time, they're added to the graph from left to right. Once it reaches the end, it just wraps around and starts adding new data the beginning again, clearing old data and keeping a leading space at all times. I built it off of a very useful .NET control called NPlot. NPlot has a lot of versatility; it can be used for lots of types of graphs, and the RepeaterGraph is essentially a specialized NPlot. The Repeater Graph control is available in the downloads section for anyone interested in using it in their own applications.
The WiimoteViewer is a very simple program: it has a button that connects to the Wiimote, and a repeater graph for each axis of acceleration. Simple as it is, it's sort of mesmerizing to watch the data stream in... The WiimoteViewer and source are also available in the Downloads section.
Dollar Recognizer to Dollar3D
dimension++
The Dollar Recognizer is a small software library designed by a group at the University of Washington to recognize mouse gestures; check it out here. Go ahead! It's small and efficient enough that they actually have it running on their page so you can get a feel for it.
To adapt the recognizer to Wii gestures, I needed to bump it up from 2 dimensions to 3. Most of that was straightforward enough; the code is well-written, and I could change a lot of it to 3 dimensions without even trying to understand the algorithms.
The Dollar Recognizer resamples and rotates points before comparing them to recorded gestures. This has can be good or bad depending on how you want the program to behave. If you want the program to recognize a gesture whether it's drawn extra large, tiny, or at any orientation, then Dollar is perfect. But if you want it to differentiate between a down movement and an up movement, it requires some (simple) modification. All that was needed was a removal of the rotation code, and Dollar3D was ready to go. You can download the modified source or library binaries at the downloads section.
MagicWand
this is where it all comes together
The MagicWand program
The MagicWand program is pretty simple. It simply connects to a Wiimote, waits for button presses, records points and sends them to the Dollar3D Recognizer. Pressing and holding the A button records a new gesture, and pressing and holding the B button attemps to recognize a gesture. Gestures are saved as .xml files (same as in the Dollar Recognizer, but with an extra coordinate) and reloaded automatically when the program starts. There's no double-integration happening here to move from acceleration to position; it's the raw accelerations that are recorded and recognized.
To my surprise, it actually worked pretty well! I can usually get a score of .6ish or better on recognition. Of course, I tried the tried the basic circle, triangle, etc, but Wingardium Leviosa was much cooler. Now I just need to have that motion actually do something.
Again, the program & source code can be found with the other downloads.
Future
There's a number of things I have planned for future iterations, if I ever get to them:
- multiple recordings to match a given gesture
- recognize scaling (how big of a circle was made?)
- recognize speed (spinning the lasso fast or slow?)
- continuous recognition (no need to press B to denote gesture)
