Easy Face and hand tracking browser detection with TensorFlow.js AI and MediaPipe

In March the TensorFlow team has released two new packages: facemesh and handpose for tracking key landmarks on faces and hands respectively. This release has been a collaborative effort between the MediaPipe and TensorFlow.js teams within Google Research.

The facemesh package finds facial boundaries and landmarks within an image, and handpose does the same for hands. These packages are small, fast, and run entirely within the browser so data never leaves the user’s device, preserving user privacy. You can try them out right now using these links:

• Facemesh
• Handpose

Facemesh package

Handpose package

These packages are also available as part of MediaPipe, a library for building multimodal perception pipelines:

• MediaPipe face tracking
• MediaPipe hand pose tracking

Deep dive: Facemesh

Performance characteristics

Facemesh is a lightweight package containing only ~3MB of weights, making it ideally suited for real-time inference on a variety of mobile devices. When testing, note that TensorFlow.js also provides several different backends to choose from, including WebGL and WebAssembly (WASM) with XNNPACK for devices with lower-end GPU's. The table below shows how the package performs across a few different devices and TensorFlow.js backends:

Installation

1. Through NPM:

   import * as facemesh from '@tensorflow-models/facemesh;

2. Through script tags:

   <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-core"></script>
   <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-converter"></script>
   <script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/facemesh"></script>

Usage

// Load the MediaPipe facemesh model assets.
const model = await facemesh.load();
 
// Pass in a video stream to the model to obtain 
// an array of detected faces from the MediaPipe graph.
const video = document.querySelector("video");
const faces = await model.estimateFaces(video);
 
// Each face object contains a `scaledMesh` property,
// which is an array of 468 landmarks.
faces.forEach(face => console.log(face.scaledMesh));

{
    faceInViewConfidence: 1,
    boundingBox: {
        topLeft: [232.28, 145.26], // [x, y]
        bottomRight: [449.75, 308.36],
    },
    mesh: [
        [92.07, 119.49, -17.54], // [x, y, z]
        [91.97, 102.52, -30.54],
        ...
    ],
    scaledMesh: [
        [322.32, 297.58, -17.54],
        [322.18, 263.95, -30.54]
    ],
    annotations: {
        silhouette: [
            [326.19, 124.72, -3.82],
            [351.06, 126.30, -3.00],
            ...
        ],
        ...
    }
}

Deep dive: Handpose

Performance characteristics

Installation

1. Through NPM:

   import * as handtrack from '@tensorflow-models/handpose;

2. Through script tags:

   <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-core"></script>
   <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-converter"></script>
   <script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/handpose"></script>

Usage

// Load the MediaPipe handpose model assets.
const model = await handpose.load();
 
// Pass in a video stream to the model to obtain 
// a prediction from the MediaPipe graph.
const video = document.querySelector("video");
const hands = await model.estimateHands(video);
 
// Each hand object contains a `landmarks` property,
// which is an array of 21 3-D landmarks.
hands.forEach(hand => console.log(hand.landmarks));

{
    handInViewConfidence: 1,
    boundingBox: {
        topLeft: [162.91, -17.42], // [x, y]
        bottomRight: [548.56, 368.23],
    },
    landmarks: [
        [472.52, 298.59, 0.00], // [x, y, z]
        [412.80, 315.64, -6.18],
        ...
    ],
    annotations: {
        indexFinger: [
            [412.80, 315.64, -6.18],
            [350.02, 298.38, -7.14],
            ...
        ],
        ...
    }
}

Looking ahead

Next steps

• Try out the models
• Learn more about facemesh from this Google Research paper: https://arxiv.org/abs/1907.06724
• Check out this Google AI blogpost announcing the release of facemesh as part of the Android AR SDK: https://ai.googleblog.com/2019/03/real-time-ar-self-expression-with.html
• Learn more about handpose as part of MediaPipe on this Google AI blogpost: https://ai.googleblog.com/2019/08/on-device-real-time-hand-tracking-with.html

Acknowledgements

The TensorFlow team worked together with the MediaPipe team, who generously shared their original implementations of these packages. MediaPipe developed and trained the underlying models, and designed the post-processing graph that brings everything together.

Luigi Nori

He has been working on the Internet since 1994 (practically a mummy), specializing in Web technologies makes his customers happy by juggling large scale and high availability applications, php and js frameworks, web design, data exchange, security, e-commerce, database and server administration, ethical hacking. He happily lives with @salvietta150x40, in his (little) free time he tries to tame a little wild dwarf with a passion for stars.