OATAO - Open Archive Toulouse Archive Ouverte Open Access Week

A Unified Deep Framework for Joint 3D Pose Estimation and Action Recognition from a Single RGB Camera

Pham, Huy-Hieu and Salmane, Houssam and Khoudour, Louahdi and Crouzil, Alain and Velastin, Sergio A. and Zegers, Pablo A Unified Deep Framework for Joint 3D Pose Estimation and Action Recognition from a Single RGB Camera. (2020) Sensors, 20 (7). 1-15. ISSN 1424-8220

(Document in English)

PDF (Author's version) - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader

Official URL: https://www.mdpi.com/1424-8220/20/7/1825


We present a deep learning-based multitask framework for joint 3D human pose estimation and action recognition from RGB sensors using simple cameras. The approach proceeds along two stages. In the first, a real-time 2D pose detector is run to determine the precise pixel location of important keypoints of the human body. A two-stream deep neural network is then designed and trained to map detected 2D keypoints into 3D poses. In the second stage, the Efficient Neural Architecture Search (ENAS) algorithm is deployed to find an optimal network architecture that is used for modeling the spatio-temporal evolution of the estimated 3D poses via an image-based intermediate representation and performing action recognition. Experiments on Human3.6M, MSR Action3D and SBU Kinect Interaction datasets verify the effectiveness of the proposed method on the targeted tasks. Moreover, we show that the method requires a low computational budget for training and inference. In particular, the experimental results show that by using a monocular RGB sensor, we can develop a 3D pose estimation and human action recognition approach that reaches the performance of RGB-depth sensors. This opens up many opportunities for leveraging RGB cameras (which are much cheaper than depth cameras and extensively deployed in private and public places) to build intelligent recognition systems.

Item Type:Article
Additional Information:Thanks to MDPI editor. The original PDF of the article can be found at Sensors website : https://www.mdpi.com/1424-8220/20/7/1825
HAL Id:hal-02970860
Audience (journal):International peer-reviewed journal
Uncontrolled Keywords:
Institution:French research institutions > Centre National de la Recherche Scientifique - CNRS (FRANCE)
Université de Toulouse > Institut National Polytechnique de Toulouse - Toulouse INP (FRANCE)
Université de Toulouse > Université Toulouse III - Paul Sabatier - UT3 (FRANCE)
Université de Toulouse > Université Toulouse - Jean Jaurès - UT2J (FRANCE)
Université de Toulouse > Université Toulouse 1 Capitole - UT1 (FRANCE)
Other partners > Aparnix (CHILE)
Other partners > Centre d'études et d'expertise sur les risques, l'environnement, la mobilité et l'aménagement - CEREMA (FRANCE)
Other partners > Queen Mary University of London - QMUL (UNITED KINGDOM)
Other partners > Universidad Carlos III de Madrid - UC3M (SPAIN)
Laboratory name:
Deposited On:30 Sep 2020 09:02

Repository Staff Only: item control page