Creating LeapMotion Gestures

Now that I have my LeapMotion controller and I have successfully managed to model a simplified pinball machine, I am keen to link the two together; and be able to play my pinball machine using the LeapMotion controller.

The first step along the road to achieving this is to define a way of interacting with the pinball machine - by defining gestures for using the Plunger and the Flippers.

The Plunger's Gesture

Source: gametrailers.com

The plunger is used to propel the pinball in to the playfield with a varying degree of force - depending on how far back the spring-loaded rod is pulled, before it is released. 

With this in mind, I set out to create a gesture which would track when the user grabbed the plunger and how far, along the Z axis, they pulled back the plunger before releasing it.

Source: leapmotion.com

Due to the limitations of the LeapMotion controller, and for simplicity's sake, I modelled the action of grabbing the plunger as a horizontal two-fingered pinch. Once the Leap Motion controller identifies two fingers as being brought within a certain distance of each other (along the X axis) they are considered as being pinched together - the plunger has been grabbed. The code then keeps track of how far back (away from the LeapMotion controller/towards the user) the fingers are drawn (along the Z axis) before the pinch is released (the two fingers move a certain distance apart from each other on the X axis). The distance in which the pinch gesture has travelled along the Z axis can then be translated in to an amount of force to be applied to the pinball in order to launch it in to the playfield. 

Based on the documentation for setting up a Java project and the Sample Java class - which provides a nice introduction to the API for getting hold of the Frames and the information contained within each of the Frames (Hands, Fingers, Gestures etc.) (Sample.java can be found as part of the SDK's files) - I created a new project of my own.

In order for your application to get hold of the event information dispatched by the Controller, the API contains a Listener Class; a subclass of which can override a number of callback functions in order to get hold of the dispatched event. The most useful of these callback functions is the onFrame method which is called every time the Controller has a new Frame of information - which could be anything between every 2 ms to 33 ms (i.e. 30 to 500 fps). My PinchGestureListener class, therefore, extends the Listener class and overrides the onFrame method:

There are a few notable things to mention about what the above code does: 

1. First of all it makes sure that we only have one hand in view; that, that hand only has two fingers visible and it works on the assumption that it is the right hand; I chose to impose these constraints to make it easier to make the gesture robust. Production-ready code would require more complexity and a lot more testing. 
2. Any state outside of the constraints will effectively reset/end the gesture in order to prevent the code from ending up in an erroneous state e.g if a pinch gesture is recognised and then hand disapears/is taken out of view of the controller.
3. Thresholds are defined for how close the fingers need to be together to constitute a pinch and how far they need to be away from each other for the pinch to subsequently be released (measured along the X axis). Since the Controller loses track of one of the fingers when they touch, I found it necessary to not require the fingers to touch for the gesture to be complete and to handle this boundary case.

The result looks like this:

The Flippers' Gesture

Source: wikimedia.org

Flippers are levers used to redirect the pinball up the playfield. Depending on the timing on which the button is pressed, to active the flipper, they can be used to control the direction and velocity of the pinball. 

Keeping with the idea of simplicity, I next went on to create a gesture for the right-hand flipper - activated when only the right hand is in view of the Controller. The reason that I chose to impose this constraint is that from experimenting with the Diagnostic Visualizer I found that when both of my hands were in view, the Controller tended to get confused more often; losing track of fingers or have them jutting out at what would be agonisingly painful angles. Using this tool was, however, a great help and saved me many hours of frustration. I had originally set out with the image of a user creating a flipper gesture by rotating a palm-opened hand that was facing towards them around the wrist in a flicking motion. After some experimentation with the Diagnostic Visualizer, I found that extending just one finger, pointing it horizontally flat; facing across the body to the left-hand side and wiggling it back and forwards was the best mechanic for robustly identifying a gesture for activating a flipper. 

Similarly to my PinchGestureListener class, the RightHandFlipGestureListener class is a subclass of Listener and overrides the onFrame method:

Whilst simpler than the PinchGestureListener class, there are still a few notable things to mention about what the above code does: 

1. For simplicity, we the code ignores Frames that have more than one hand in them or where the hand doesn't have any visible fingers.
2. It doesn't strictly enforce only that only one finger can be in view, but attempts to isolate the finger that is furthest over to the left (smallest value on the X axis) as the one to follow - in a hope that the code is resilient to those few occasions/Frames when random fingers are detected by the Controller.
3. Thresholds are defined for how far above and below the Z axis the finger needs to be, to be considered as either completely up (flipper activated) or completely down (flipper deactivated).
4. Instead of making the flipper completely binary (i.e. either on or off), I wanted to introduce the idea of the flipper being in a "falling" state (changing from being activated, but having not come to rest yet), so that if it is activated during this state that it applies less force to the pinball. This models a situation where the flipper is press repeatedly in quick succession. 

The result looks like this: 

Prev (Handling User Input and Simple Joints in JBox2D)

Handling User Input and Simple Joints in JBox2D

Introduction to handling user input

Sticking with the original theme of a pinball machine, in this post I will demonstrate how we can model a simplified pinball machine plunger that responds to a user's input and how we can use Joints to create flippers for our pinball machine.

In this model, the plunger applies a certain amount of force to a ball based on how long a button is pressed (up to a maximum i.e. modelling the plunger being pulled back as far as it can go). 

With a view to eventually, in the future, transforming a gesture interpreted by the Leap Motion Controller as pulling back the plunger in to input for our pinball machine. 

The code below creates a simple test which responds to the user pressing the 's' key in order to launch the pinball - the longer the key is held down the harder the pinball is fired. The complete code can be found on Github. Let's work through the code and look at the key, new elements.

On lines 72 to 104, we provide our own implementations for the keyPressed, keyReleased and step methods:

• In the keyPressed method, the simulation acts based on the key which has been pressed (down) by the user. In this case we only handle when 's' is pressed down, but by adding new Cases to the Switch statement we can add more functionality. The logic in the body of the method is based on the knowledge that when a user holds down a key, the keyPressed method will continue to be called. As such, the amount of force that we should apply to the pinball is increased; up to a maximum. 
• When the user lifts their finger, and the pressed key's contact no longer connects with the underlying circuit, the keyReleased method is called. We use this as the trigger for applying the user-determined force to the pinball.
• In the step method, we are simply adding a message, with instructions for the users, to the testbed GUI.  

Joints in JBox2d

A problem with the simulation is that the modelled plunger is not attached to anything, so after the pinball has been launched, further interaction from the user (force applied) will cause the plunger to jump up the screen.

Looking at the Box2D documentation, we can see that Joints can be used to solve this problem; by affixing the plunger to the ground. 

• Joints - "Joints are used to constrain bodies to the world or to each other... Joints can be combined in many different ways to create interesting motions. Some joints provide limits so you can control the range of motion. Some joint provide motors which can be used to drive the joint at a prescribed speed until a prescribed force/torque is exceeded."
• Distance Joint - "One of the simplest joint is a distance joint which says that the distance between two points on two bodies must be constant."

Another piece of the puzzle is how we can use Joints to "create interesting motions" and "limit the range of motions". Looking back at the documentation on Joints reveals to us a Joint which will be very useful in creating the flipper for our pinball machine: the Revolute Joint:

• Revolute Joint - "A revolute joint forces two bodies to share a common anchor point, often called a hinge point... A joint limit forces the joint angle to remain between a lower and upper bound."

The joints in the simulation can be seen in blue

I chose to use rectangles as simple representations for the flippers. I positioned them towards the bottom of the canvas, affixed one end of each of the flippers to the canvas using a Revolute Joint to form a hinge and then configured each joint to constrain the range of motion of the flippers - to three quarters of a radian (roughly 42 degrees) which seems like a realistic enough range of motion. 

In the picture you can see the three Joints which I have created depicted with light blue lines; the simple Distance Joint and the two Revolute Joints acting as hinges. It's easiest to see the actual point which forms the Revolute Joint on the right-hand flipper due to the sharp kink in the blue line - the point where the line bends in the pinkish rectangle is the Joint which the flipper rotates about. 

Note: these blue lines are purely aesthetic, for debugging purposes and do not, directly, interfere with the physics in the simulation.

Along with adding in the flippers and the Joints, I also added a small amount of code to allow the user to active the flippers. The code for which can be seen below (The complete code can be found on Github):
With all of these new features in place, the simplified pinball machine simulation looks like this

Modelling simple physics behaviour with JBox2D

Now that we are confident that we can add tests to JBox2D's Testbed, it is time to start exploring how we go about modelling a simple world in JBox2D, how we add actors to that world and how we specify how these actors interact with each other.

The MJWTest2, sample test from the previous article was a useful example and a great start, but sooner or later we are going to want to constrain the actors in to a bounded area; let's look at how we go about creating an environment for our actors.

JBox2D doesn't have any detailed, specific technical documentation, but instead directs you towards Box2D's technical documentation - since it is just a port of this library. From the Core Concepts we can see the following useful definitions:

• World - "A physics world is a collection of bodies, fixtures, and constraints that interact together." 
• Shape - "A 2D geometrical object, such as a circle or polygon."
• Rigid body - "A chunk of matter that is so strong that the distance between any two bits of matter on the chunk is constant... we use body interchangeably with rigid body."
• Fixture - "A fixture binds a shape to a body and adds material properties such as density, friction, and restitution."

These 4 concepts are quite powerful, will be heavily used in and integral to my JBox2D projects, and with these concepts alone there are numerous possibilities.

You can see in the following code (which is an adaptation of the MJWTest2 test) how I've utilised these concepts to create a simple test in which a ball falls from the sky and lands on the ground (the green line) (The complete code can be found on GitHub):

As you can see in the code, we instantiate a World; giving it gravity and setting the centre of gravity.

Note: The world is modelled similar to a graph, with coordinates along the X and the Y axes; with the point where they cross being 0,0. I find it helpful to try and visualise these invisible axes when creating Bodies, initialising their position in the world etc. 

We then create a Body to represent the ground and register it with the World. We decide which Shape we want the ground to be - in this case an EdgeShape, we decide upon the dimensions and location of the ground, and then we affix that Shape to the Body using a Fixture.

• EdgeShape - "Edge shapes are line segments. These are provided to assist in making a free-form static environment for your game."

The next step is creating the ball and positioning it above the ground waiting to drop. The ball is created in a similar fashion to the ground - instantiating a Body, defining it's location and registering it with the World; choosing a Shape to represent the Body and using a Fixture to bind the two. The one notable difference is that we set the Body's type as dynamic.

From the Box2D technical documentation we see that a Body have have 3 possible types, and that they are defined as follows:

• Static - "A static body has does not move under simulation and behaves as if it has infinite mass. A static body has zero velocity... Static bodies do not collide with other static or kinematic bodies."
• Kinematic - "A kinematic body moves under simulation according to its velocity. Kinematic bodies do not respond to forces...  A kinematic body behaves as if it has infinite mass... Kinematic bodies do not collide with other static or kinematic bodies."
• Dynamic - "A dynamic body is fully simulated. They can be moved manually by the user, but normally they move according to forces. A dynamic body can collide with all body types. A dynamic body always has finite, non-zero mass."

Since we have a non-static Body which we want to collide with the static Body of the ground, the ball is created as a dynamic Body. 

A more complex test

A ball dropping on to the floor is a good start, but it would be nice if the ball had some other material properties (say it was able to bounce), if there were more dynamic Bodies to interact with and if we could introduce Force to model more interesting behaviour. The thought of lots of bouncing Bodies on the screen conjured up images of a moshpit - at least a cross-section of one, so the next progression of my simple test is to model a very simple crowd of jumping Bodies - Moshers.

I also decided that it might be fun to give each of the Bodies slightly different characteristics - how much they weigh, how much friction they have (predominantly this will be with the ground) and how much power is in their, initial*, jump.

The following test is how I modelled a very naive and simplistic version of this scenario (The complete code can be found on GitHub):

You'll notice a slight modification to the "Environment body" section of the code, where I add in walls - to stop the Bodies disappearing off of the screen.

What's more interesting is how you go about dynamically creating, adding material properties and positioning multiple Bodies.

The positioning is all based around factoring in the radius we assigned to the CircleShape (0.9) and the coordinates of our environment. If you place a Body overlapping another Body then the dynamic one (in this case our CircleShapes) will try and push away and detach itself; this is why the first Body in the simulation is located at (-19.1, 0) - the bottom left hand corner of the environment with its left hand side just touching the left hand wall. It's also the reason why I increment the position by 1.805 each time, so that each Mosher has a bit of breathing space*.

I then use select a random index from each of the lists of attributes and assign it to a Mosher to give it some characteristics and set it off moshing.

*A limitation to this simulation is that I currently only know how set an initial Force to act upon each of the Bodies - when it is initialised. And to get around the fact that the Mosher would soon run out of steam, I set the restitution close to the maximum (1), so that they would lose very little energy each bounce. I wanted to add crowd surfers and interaction between the Moshers, but this quickly ends up with a bunch of inactive Bodies, so I will have to leave these features as future enhancements for when I know more about JBox2D.

• Restitution - "Restitution is used to make objects bounce. The restitution value is usually set to be between 0 and 1. Consider dropping a ball on a table. A value of zero means the ball won't bounce. This is called an inelastic collision. A value of one means the ball's velocity will be exactly reflected. This is called a perfectly elastic collision."

The next thing that I'm interested in looking at is adding the ability for the user to press a button in order to interact with the simulation - this is a necessary step on the road to creating a Pinball Machine. This is part of what I will focus on in the next post - the other thing I will look at is the complex subject of Joints.

Adding tests to a JBox2D Testbed

In order to speed up the learning process and really get the most out of the JBox2D physics engine, I'm going to utilise the Testbed framework. There is some online documentation for the Testbed, however, I found that it is more of a companion guide for the jbox2d-testbed module's codebase; almost psuedocode, pointing you towards paramount parts of the code. Whilst this is useful once you are up and running, what I really needed to get me started was a complete and simple guide on how to go from a brand new project to launching the example test.

What I'm going to outline here are the steps which need to be performed in order to launch a test of some JBox2D modelled, physics behaviour. The full code for which can be found on GitHub.

As highlighted in the Adding Tests/Settings section of the documentation, we need an entry point to perform the following, high-level, tasks:

• Instantiate an instance of the Testbed
• Add our test(s)
• Launch the Testbed's GUI

The first thing that wasn't immediately clear from the online documentation, but that was cleared up by the codebase is that our entry point (we'll call it TestbedMain) is not a JUnit test, but a Java class with a Main method. The online documentation/example seems to be a bit out of date - below is the complete code for my TestbedMain class:  

The MJWTest2 class (seen referenced in the code above) is the test which we want to run, and the code for which is almost copied verbatim from the A Simple Test section of the documentation, with the exclusion of the following line which does not compile:
It's also likely at this point that your TestbedMain class does not compile because of the missing libraries/dependencies. I used Maven to manage these dependencies, and I think that this is the easiest way - the following dependency needs to be added to your pom.xml file:

You could, however, just add the dependencies' JAR files to the project by hand if you don't want to use Maven - here's a view of the dependencies needed:

You should then be able to execute the Main method of the TestbedMain class, and you will see the sample test started in the Testbed's GUI:

Getting started with JBox2D

The first step on the road to realising my Pinball machine project is learning how to use JBox2D. As a back-end Java programmer, by profession, I chose to start learning this over JavaFX for a few reasons:

1. The skill set required for JBox2D is more closely aligned to mine than that required for JavaFX - so the learning curve should be less steep. 
2. The JBox2D team have done a fantastic job of creating a testing framework for easily prototyping and visualising ideas. 
3. Learning how to make the physics engine model complex behaviour should be a lot of fun. 

With the wealth of example tests, documentation and a great toolset, I anticipated being up and running in no time.

Stumbling at the first hurdle 

I had expected there to be growing pains as I started to learn JBox2D, since I've never used a physics engine before and it is an involved subject, but you have a crisis of confidence when you stumble whilst trying to get up and running.

How to build the code

The QuickStart guide lays out 3 easy steps for getting started - checkout the code, import it as a Maven project in to an IDE and finally run the TestbedMain class to start the Testbed suite.

Unfortunately, when I followed the instructions, I encountered compilation errors which prevented the Testbed from running. Whilst I was able to find a work around, I've subsequently identified where I went wrong - and what isn't completely in the QuickStart guide.

Where step 2 says "Import to your IDE as a Maven project (using the pom.xml descriptor in the root folder)", I interpreted this as the parent pom.xml file in the jbox2d-read-only, root folder. but I've found that if instead you only import the module jbox2d-read-only\jbox2d-testbed using the pom.xml file from found within this directory that the Testbed suite launches - without having to mess around with deleting files or changing their package headers.

Note: You may, however, have to run mvn clean install on the jbox2d-read-only\jbox2d-library module in order to make the necessary JBox2D library dependency available to the Testbed module. 

Once everything is configured properly, the Testbed Swing GUI should launch and you can start experimenting with all of the pre-written, example tests.


Then it's on to writing our own tests and plugging them in to the Testbed!





Prev (Leaping in to action)               Next (Adding tests to a JBox2D Testbed)

Leaping in to action

The Leap Motion Developer Portal was recently opened to the wider Development Community, and so it was time to see what those privileged few had been working on, with their early Development Kits, behind closed doors.

Browsing around on the Developer's forums, I came across a fascinating article by José Pereda which not only whet my appetite for the challenges which lay ahead, but emphasised that a lot of progress has been made with Java's UI platform, JavaFX, and that the Java port of Box2D looks awesome. I am completely unfamiliar with JavaFX and JBox2D, but I foresee myself using these libraries heavily in the future, so whilst I wait for my Leap Motion controller to be delivered, this seemed like a great time to get stuck in with learning these technologies.

In José Pereda's article, he reference Toni Epple's use of JavaFX and JBox2D to create a pinball machine as a very useful resource. Looking over what Toni has done, I was reminded of childhood memories of Space Cadet Pinball and it struck me as a good idea for a fun little project to learn all about UI design with JavaFX and to start learning how to program using the JBox2D physics engine. So that's what I'll be doing in the short term – slowly trying to build up to something that looks like Space Cadet Pinball, using JBox2D and JavaFX, but isn't as full featured.

When the Leap Motion controller finally arrives, I can work on replacing the keyboard controls for the flippers and spring launcher with the appropriate hand actions. 

Next (Getting started with JBox2D)