Head Tracking with FaceTrackNoIR

I've been aware of head tracking solutions since I started playing Arma 2; the game (and it's sequel, Arma 3) has the ability to "freelook", meaning you can continue to walk or point your weapon in a certain direction, while you scan your surrounding environment. This is extremely useful from a tactical standpoint, but it's also particularly helpful when piloting vehicles (especially helicopters). Probably the most well-known commercial solution is TrackIR; this kit comes with everything you need to start using head tracking in games that support it, but it's a little on the expensive side (around £175). Despite some serious consideration, I never actually bought the kit myself; I didn't think it was worth it considering the relatively small number of games I play that would benefit from it, especially when consumer versions of both the Oculus Rift and HTC Vive are not far off. So I was extremely excited when I happened across a post discussing head tracking on the Elite: Dangerous subreddit that mentioned a free alternative: FaceTrackNoIR. As it's name suggests, this little bit of software can use a webcam attached to your machine to track your actual face; when you hit "start", it identifies the basic shape of your face, then translates your movements into the desired protocol for the game to understand.

Tracking my FACE!

The application has quite a few configuration options and it took me a fair amount of time tinkering with them to achieve the desired results, but to say I was impressed is an understatement: being able to simply move my head and look down at the spot where I was trying to land my heli was not only super cool, but useful too! Once I started using it, I didn't want to stop, however it does have a couple of drawbacks:

• It's sensitive to poor lighting - I actually found if it was particularly bright outside, the software would often lose or simply not be able to recognise my face.
• The CPU usage is fairly high - it seemed to fluctuate between roughly 5-10% when in use; that's a fair amount of precious CPU cycles that could be put to use by a game engine (Arma 3 being particularly CPU hungry)!

After having a play around with the different settings in the application and reading the FaceTrackNoIR wiki, I discovered that there are a few different tracking sources available, one of which works in a similar fashion to TrackIR. This helps to improve accuracy and/or reduce the CPU load, but unfortunately is still susceptible to poor lighting and also requires an IR clip providing three points to track. However, on the same Reddit post I mentioned earlier, somebody commented that they had bought a "relatively cheap" IR clip to use with this particular mode of operation: the DelanClip. Given the low cost, I thought I'd give it a go myself. The kit I bought is the most basic model and comes with a USB powered IR clip, two cable ties and a couple of sections of coloured film that can be used to create a light filter for your webcam:

Attached DelanClip

Webcam with filters attached.

After I'd finished setting up, I downloaded the latest version of the PointTracker plugin (overwriting the version that ships with FaceTrackNoIR), tweaked my config and fired it up. I found that I only really had to tweak the curves that define the translation between your actual head and your in-game character to my liking (here's my config file if you're interested) and make sure there were no direct light sources in the webcam's field of view.

3x IR point tracking

I've definitely had more luck using the PointTrack plugin than the FaceAPI; there has been much fewer incidents of the software "losing" me. If there's anybody reading this wondering just how beneficial free-look in games can be, let me just show off a couple of Elite: Dangerous screen caps I took while using FaceTrackNoIR:

Checking my flight path is clear before leaving the landing pad.

Tracking an enemy ship that would have otherwise flown out of my line of sight.

Of course, these actions could be achieve using regular input (mouse, keyboard or even a hat-switch on a joystick), but I honestly feel that head tracking provides a more natural and (sigh) immersive experience. The only thing that will surpass this (in my opinion) will be a full VR headset, which I'll definitely be wanting to get my hands on once consumer devices are available!