LeapMotion Robot controller

Sometimes impulse takes over and I end up buying something that seems really cool but in the cold light of day doesn't quite live up to its promise

In this case, about two years, I bought a LeapMotion controller.

The premise seemed great, a way of using hand gestures to intuitively control your computer. It seemed especially promising given the controller could track hand and finger positions for both hands. Unfortunately, my old laptop didn't have a good enough spec to do anything more than the most basic samples and even then the experience seemed too prone to errors.

The new laptop certainly has enough power but even now the tracking sometimes seems to get the jitters. So it seemed like this piece of kit was destined to lie in a cupboard like so many other pieces of technology around the world.

But cometh the hour, cometh the robot-arm and it occurred to me LeapMotion would make for an interesting control interface.

Starting simply, my intention was to control the open and shut movement of the grabber with my index finger and thumb. By moving them apart and together I hoped to get the grabber to follow this motion.

To begin I downloaded the LeapMotion 2.3.1 SDK. Using Eclipse, I created a new Java application and added LeapJava.jar (Right-click on the project, select Properties -> Java Build Path -> Libraries Tab and Add External Jars, so that it appeared in the project:

To access the tracking data I created a class that implemented the com.leapmotion.leap.Listener interface and added this to a Controller

  LeapMotionListener listener = null;
  Controller controller = new Controller();
  try
  {
   listener = new LeapMotionListener( writer, new TcpClient() );
   controller.addListener(listener);
   System.out.println("Press Enter to quit...");
   System.in.read();
  }
  catch( Exception e)
  {
   e.printStackTrace();
  }
  finally
  {
   if (listener != null)
   {
    controller.removeListener(listener);
   }
  }

In the Listener class I implemented onConnect and onFrame. The interesting implementation was onFrame. This is called in a separate thread every time data is sampled. The event details can be obtained with a Frame instance

@Override
 public void onFrame(Controller controller) {
    Frame frame = controller.frame();

The API then quite nicely allows you to obtain the information about the index finger and thumb in the shape of LeapMotion Vectors:

FingerList indexFingerList = frame.fingers().fingerType(Finger.Type.TYPE_INDEX);
Finger indexFinger = indexFingerList.get(0); //since there is only one per hand. I'm only holding  one hand in the field of view
FingerList thumbList = frame.fingers().fingerType(Finger.Type.TYPE_THUMB);
Finger thumb = thumbList.get(0);
Vector thumbTip = thumb.tipPosition();
Vector indexTip = indexFinger.tipPosition();

At this point I started investigating the behaviour of the sensor. Calculating the distance between finger and thumb is simple:

final float distance = thumbTip.distanceTo(indexTip);

Unfortunately this measure is too noisy to be able to reliably control the arm. So I decided on a velocity measure:

v = delta (distance) / delta (time)

If the velocity exceeded a threshold then the signal to drive the motor in the appropriate direction would be sent.

I setup the robot arm and Arduino as per my earlier blog; replacing the Android app with this LeapMotion client.

This is certainly a work in progress. For one the velocity is still a bit noisy. Also the motor being a simple DC motor is setup to only work at one speed. Finally, if the controller lost tracking of the hand the motor was left in the last known state. I'll add post in future to address some of these problems and make the source available on GitHub.