Problem
Traditional human-machine interaction methods, such as physical buttons, touchscreens, and voice recognition, have limitations in various environments.
Challenges:
- Physical contact dependency – Buttons and touchscreens require direct physical contact, which may not be ideal in hygiene-sensitive environments.
- Limited accessibility – Voice recognition may not work well in noisy environments or for users with speech impairments.
- Poor performance in low light – Vision-based gesture systems using RGB cameras struggle in low-light conditions.
Need for an alternative solution:
A contactless, intuitive, and robust interaction system is required, one that can work in various lighting conditions, ensure accessibility, and operate on low-powered embedded systems.
Solution
This project presents a low-resolution thermal imaging-based gesture recognition system that can:
- Accurately classify hand gestures (rock, scissors, paper) using a thermal camera and machine learning.
- Function effectively in complete darkness using heat signatures rather than visible light.
- Run on a low-powered microcontroller while maintaining real-time responsiveness.
- Offer intuitive control for various applications, such as smart home automation, assistive technologies, and interactive interfaces.
By leveraging thermal imaging and AI-driven classification, this system provides a hygienic, accessible, and efficient alternative to traditional human-machine interaction methods.
Materials used in this project
Hardware components
Particle Photon 2
Pimoroni MLX90640 thermal camera breakout
Adafruit 2.8″ TFT touch shield
SeeedStudio Grove – dual button
SeeedStudio Grove – 4-pin male jumper to Grove 4-pin conversion cable
Jumper wires
ArduEZ ONE stackable breadboard
MicroSD card
Smartphone tripod stand
Software apps and online services
Particle Console
Edge Impulse Studio
Visual Studio Code
Hand tools and fabrication machines
Soldering kit
Story
This project aims to develop a human machine interface (HMI) system that utilizes heat signatures to detect and classify hand gestures through machine learning. This system employs a 32×24 pixels low-resolution thermal camera and a low-powered microcontroller to capture thermal images of hand gestures, specifically rock, scissor, and paper. These images are then processed and classified using advanced machine learning algorithms to accurately identify the gestures.
The main goal of this project is to significantly improve user interaction by developing a seamless and contactless method for engaging with devices. This innovative approach will function effectively in various lighting conditions, including complete darkness and low-light environments, ensuring that users can effortlessly navigate and control their devices regardless of their surroundings. The system offers significant benefits, including efficiency due to the low-resolution thermal imaging, which reduces computational requirements while maintaining accuracy. It also provides an intuitive user experience, as natural hand gestures make the system easy to use and learn.
Hardware selection
This project needs a reliable, low-powered, and Wi-Fi enabled device that is also capable of running image classification models. We will be performing TensorFlow Lite model inference using Edge Impulse, and for this purpose, we will utilize the Particle Photon 2, which has ample memory for buffering thermal image data and storing ML models. We are using a thermal camera (MLX90640) with a 32×24 resolution, which is a far infrared array of 768 thermal sensors. Additionally, we are incorporating the Adafruit 2.8″ TFT touch shield for displaying thermal images and basic information during data collection and inferencing.
We used a stackable breadboard and jumper wires to assemble the hardware.
This breadboard has double-sided tie points. The Adafruit TFT display shield is mounted on the front with all connections using the jumper wires, and the Photon 2 and the thermal camera are mounted on the back.
Due to the inherently low sampling frequencies of thermal sensors, excessive movement can result in motion blur in the generated images based on the thermal readings. To mitigate this issue and ensure stable operation, we have opted to utilize a sturdy tripod stand.
We have added a Grove dual button to initiate certain actions during data collection.
Hardware wiring
We are providing clear pin maps for a better understanding of the connections.
The Adafruit TFT shield and microSD card are connected via the SPI interface, and its pins are based on the Arduino Uno R3 pinout.
TFT shield & microSD card connections
| Photon 2 Pin | Adafruit TFT Shield Pin | Description |
|---|---|---|
| D15 | D11 | SPI (MOSI) |
| D16 | D12 | SPI (MISO) |
| D17 | D13 | SPI (SCK) |
| D8 | D9 | SPI (DC) |
| D9 | D10 | SPI (CS) TFT display |
| D19 | D4 | SPI (CS) microSD card |
| GND | GND | Ground |
| VUSB | 5V | Power |
The thermal sensor is connected via the I2C interface.
Thermal sensor (I2C interface)
| Photon 2 Pin | Pimoroni MLX90640 Breakout Pin | Description |
|---|---|---|
| D0 | SDA | Wire (SDA) |
| D1 | SCL | Wire (SCL) |
| GND | GND | Ground |
| 3V3 | 3-6V | Power |
The Grove dual button utilizes two digital pins on the Photon 2.
Grove dual button connections
| Photon 2 Pin | Grove Dual Button Pin | Description |
|---|---|---|
| D7 | SIG1 | White button |
| D6 | SIG2 | Red button |
| GND | GND | Ground |
| 3V3 | VCC | Power |
The schematics are provided below.
Data collection
We will use Edge Impulse Studio to train and build a TensorFlow Lite model. We need to register an account and create a new project at Edge Impulse Studio.
We will collect thermal images for three hand gesture classes: rock, scissors, paper, and several background images called unknown. The thermal sensors capture real-time temperature readings, which are then converted into thermal images using color mappings. These images are saved onto a microSD card built into the TFT display, ensuring the data can be conveniently accessed later for training. Prior to uploading them to Edge Impulse Studio, the images are processed to ensure they conform to a uniform size, specifically being padded to become 32×32 pixel square images suitable for training purposes.
For demonstration purposes, the sample images below have been enlarged.
To upload image data, visit the Edge Impulse Data Acquisition page, click on the upload icon, and follow the provided instructions.
We have uploaded 379 images to Edge Impulse Studio to create a balanced dataset.
The complete application code for data collection is available in the GitHub repository at the following link:
GitHub Repository – Data Collection
Data collection demo
Building and training model
We will use Edge Impulse Studio to train and build a TensorFlow Lite model.
To create a machine learning pipeline, follow these steps:
- Go to the Impulse Design section, then select the Create Impulse page. We have opted for a 32×32 pixel image size in the “Image Data” form fields.
- Click on “Add a processing block” and choose “Image”. This step will pre-process and normalize the image data while also giving us the option to choose the color depth.
- Click on “Add a learning block” and choose “Classification”.
- Finally, click on the “Save Impulse” button to complete the process.
On the Image page, choose RGB as color depth and click on the Save parameters button. The page will be redirected to the Generate Features page.
Now we can initiate feature generation by clicking on the Generate features button. Once the feature generation is completed, the data visualization will be visible in the Feature Explorer panel.
On the Classifier page, we can define our model. We have developed a custom Convolutional Neural Network (CNN).
The architecture of this model is illustrated below.
Now, click the “Start training” button. The training process will take a few minutes to complete. Once the training is completed we can see the accuracy and metrics as shown below.
For such a small dataset 98.4% accuracy is pretty good so we will use this model.
Model testing
On the Model Testing page, click Classify All to initiate model testing with the trained model. The testing accuracy is 98.67%.
Model deployment
The Edge Impulse Studio officially supports Particle Photon 2, so on the Deployment page, select Particle Library.
For the Select Optimizations option, choose TensorFlow Lite and Quantized (Int8).
Click Build, and in a few seconds, the library bundle will be downloaded.
Inferencing demo
Use Case: Controlling an LED with Gestures
An ML model, no matter how sophisticated, lacks practical value without real-life examples to demonstrate its applicability. We developed a demo application that allows users to control an LED remotely using hand gestures. As demonstrated in the code snippets below, we have configured the inferencing application to publish results to the “gestureEvent” topic.
uint8_t max_prob_index;
float max_prob_value;
for (uint8_t index = 0; index < EI_CLASSIFIER_LABEL_COUNT; index++) {
ei_printf(" %s: ", ei_classifier_inferencing_categories[index]);
ei_printf("%.5f\r\n", result.classification[index].value);
if (result.classification[index].value > max_prob_value) {
max_prob_value = result.classification[index].value;
max_prob_index = index;
}
}
if ((prev_max_prob_index != max_prob_index) &&
(millis() - prev_pub_ms) > 1000) {
Particle.publish("gestureEvent",
ei_classifier_inferencing_categories[max_prob_index],
PRIVATE);
prev_pub_ms = millis();
}
We have set up another Photon 2 with minimal code, which subscribes to a topic and turns the onboard RGB LED on and off based on the received messages: “paper” and “scissor,” respectively.
#include "Particle.h"
SYSTEM_MODE(AUTOMATIC);
SYSTEM_THREAD(ENABLED);
SerialLogHandler logHandler(LOG_LEVEL_INFO);
void gestureEventHandler(const char *name, const char *data) {
String prediction(data);
Serial.printf("Event: %s Class: %s\n", name, prediction.c_str());
if (prediction == "Paper") {
RGB.color(255, 0, 0);
}
if (prediction == "Scissor") {
RGB.color(0, 0, 0);
}
}
void setup() {
Particle.subscribe("gestureEvent", gestureEventHandler, MY_DEVICES);
RGB.control(true);
RGB.color(0, 0, 0);
}
void loop() {}
Published events logs can be viewed at the Cloud Services > Events menu in the Particle Console.
Conclusion
This project marks an advancement in the realm of Human-Machine Interaction (HMI), seamlessly integrating the intuitive nature of low-resolution thermal imaging with the sophisticated capabilities of machine learning. The result is an effective gesture recognition system that is robust and user-friendly, operating on a low-cost and low-powered device. The versatility of this technology allows it to be adapted for various applications beyond the initial scope, such as accessibility for individuals with physical disabilities and integration into gaming systems for gesture-based controls.