Fujitsu Laboratories Ltd.,Fujitsu Research and Development Center Co., Ltd.

News Facts:

• New "Actlyzer" tech uses AI to accurately recognize when users follow hand washing guidelines.
• Automated checks eliminate need for intrusive, time-consuming visual confirmation at worksites.
• Fujitsu envisions future use cases in food industry work sites, hospitals, schools, hotels, and venues for large events.

Kawasaki, Japan, May 26, 2020

Fujitsu Laboratories Ltd. and Fujitsu Research and Development Center Co., Ltd. (FRDC) today announced the development of "Actlyzer" hand wash movement recognition technology, which leverages AI and machine learning techniques to identify complex hand washing movements from video data captured by camera.

Amidst the ongoing global COVID-19 pandemic, the importance of hand washing as a measure to protect people's health from bacteria, influenza, and other infectious diseases is gaining renewed attention worldwide. Under new regulations⁽¹⁾ planned to come into effect in June 2020 in Japan, food business operators will also be required to implement stronger measures to ensure hygiene in accordance with international HACCP⁽²⁾ food safety standards, creating an urgent need for a non-invasive approach to quickly and accurately confirm that handwashing is carried out in a proper manner.

Anticipating this, Fujitsu has expanded the recognition function of its existing "Actlyzer behavioral analysis technology", which can recognize a variety of subtle and complex human movements without relying on large amounts of training data. Specifically, Fujitsu has refined recognition capabilities for hand movements to create technology to automatically recognize complicated hand movements performed during hand washing.

This technology makes it possible to easily determine whether someone is following each of the 6 steps for hand washing recommended by the Japanese Ministry of Health, Labour and Welfare , reducing the number of man-hours required for intrusive visual checks by inspectors for on-site sanitation management.

Moving forward, Fujitsu envisions this technology being used in a variety of other contexts, including in medical facilities, schools, hotels, and venues for large events, and plans to conduct field trials and additional research and development into it as a potential solution for its AI portfolio in the future.

Developmental Background

The Japanese Ministry of Health, Labour and Welfare recommends that people follow 6 steps to ensure their hands are properly washed and prevent food poisoning as well as the spread of infectious diseases (Figure 1).

Figure 1: 6 Steps for Correct Hand Washing

At present, food service industry workers are required to perform these six steps when washing their hands and confirm that during each step the different parts of their hands have been rubbed more than a certain number of times. To this end, food service providers fill out a check sheet and make a self-report, and workers on-site undergo visual checks by supervisors. Because verification is still conducted manually, however, costs associated with human error, securing resources for inspectors, etc. remain a persistent challenge. As many manufacturers, including in the food industry, continue the march towards process automation for inspections by using machine learning technologies, hand washing checks represent a prime candidate for process optimization through automation.

Challenges

Gesture recognition using deep learning is a common technique for identifying hand and finger movements. This conventional technique can detect multiple feature points, such as joints and fingertips, from an image of the hand, and determine the hand gesture based on the positional relationship of the feature points (Figure 2 Left). However, one issue with the existing technology was that when people wash their hands correctly, both hands overlap and are lathered with soap, which obscures the detection points on the fingers and prevents accurate gesture recognition (Figure 2 Right).

Figure 2: Conventional hand tracking tech (Left: Gesture recognition results; Right: Hand wash behavior)

Newly Developed Technology

To resolve this challenge, Fujitsu Laboratories Ltd. and FRDC have developed a new AI technology that automatically and accurately recognizes hand movements under the conditions described above, expanding the recognition function of the original Actlyzer behavioral analysis technology.

With the new technology, the complex hand movements of handwashing are captured as a combination of hand shape and repetitive rubbing motions, detected by two deep learning engines: Hand Shape Recognition and Motion Recognition (Figure 3). The two-hand shape recognition engine uses a learned model of a basic shape of two hands, which is a typical form of hand movement in which hands are placed on top of each other, to determine the hand shape for each frame of the image. Focusing on the overall shape solves the problem of when fingertips and joint feature points cannot be correctly detected due to hand overlap or foam. In addition, Fujitsu's unique AI technology "High Durability Learning (High Durability Learning)", which can track data changes, is applied to ensure that the basic shape of both hands is recognized with high accuracy even when the camera position or lighting changes during operation. The motion recognition engine uses a learned model that detects periodically changing motion from successive frames and counts the number of iterations as the number of rubs from the iteration pattern and its period.

In addition, the results of these two recognition engines are fed back to each other to improve recognition accuracy. The motion recognition engine sets a threshold value for the magnitude of the motion to be judged in accordance with the steps recognized by the two-hand shape recognition engine to prevent detection of erroneous periods, such as hand tremors not related to foam motion or rubbing. The two-hand shape recognition engine improves detection accuracy by filtering the judgment result using the repetition pattern period detected by the motion recognition engine.

Figure 3: Recognition of complex two-hand finger movements as a combination of the overall shape and movement patterns of both hands

Effects

A hand-wash video data set with approximately 2000 variations; including people, camera positions, and soap types, was independently filmed and collected for learning and evaluation. It was confirmed that the accuracy of 6 steps of correct hand washing is average of 95% or more, and that the accuracy of the number of hands rubbing movements is 90% or more. When operating the system on site, omission of action could be prevented because the person washing their hands always knows how long to scrub their hands and which of the six stages they've completed because the system determines it's displayed on the screen until it's completed. In addition to this, the system automatically records data with the starting time stamp and length of action of completing each six steps, and the number of times of rubbing. (Figure 4).

Figure 4: Example of recognition display

This technology automates on-site hand washing checks for workplaces that require strict hygiene control, eliminating the number of man-hours needed for visual confirmation and manual recording. Additionally, because the system doesn't recognize incorrect or incomplete hand washing, Fujitsu expects the solution will help educate users and ensure standardization for the right way of hand washing.

Related Links

Demo movie: "Judging the 6 steps for correct hand washing and number of hands rubbing"

Fujitsu Develops New "Actlyzer" AI Technology for Video-Based Behavioral Analysis

Fujitsu Develops Technology for Maintaining Stable, High-Accuracy AI Operations