Document Type

Conference Paper

Publication Date

2022

DOI

10.1117/12.2641898

Publication Title

Proceedings of SPIE, Applications of Machine Learning

Volume

12227

Pages

122270P (1-6)

Conference Name

SPIE Optical Engineering + Applications, August 21-26, 2022, San Diego, California

Abstract

Automatic classification of child facial expressions is challenging due to the scarcity of image samples with annotations. Transfer learning of deep convolutional neural networks (CNNs), pretrained on adult facial expressions, can be effectively finetuned for child facial expression classification using limited facial images of children. Recent work inspired by facial age estimation and age-invariant face recognition proposes a fusion of facial landmark features with deep representation learning to augment facial expression classification performance. We hypothesize that deep transfer learning of child facial expressions may also benefit from fusing facial landmark features. Our proposed model architecture integrates two input branches: a CNN branch for image feature extraction and a fully connected branch for processing landmark-based features. The model-derived features of these two branches are concatenated into a latent feature vector for downstream expression classification. The architecture is trained on an adult facial expression classification task. Then, the trained model is finetuned to perform child facial expression classification. The combined feature fusion and transfer learning approach is compared against multiple models: training on adult expressions only (adult baseline), child expression only (child baseline), and transfer learning from adult to child data. We also evaluate the classification performance of feature fusion without transfer learning on model performance. Training on child data, we find that feature fusion improves the 10-fold cross validation mean accuracy from 80.32% to 83.72% with similar variance. Proposed fine-tuning with landmark feature fusion of child expressions yields the best mean accuracy of 85.14%, a more than 30% improvement over the adult baseline and nearly 5% improvement over the child baseline.

Rights

Copyright © 2022 Society of Photo‑Optical Instrumentation Engineers (SPIE). One print or electronic copy may be made for personal use only. Systematic reproduction and distribution, duplication of any material in this publication for a fee or for commercial purposes, and modification of the contents of the publication are prohibited.

Included in accordance with publisher policy.

Original Publication Citation

Witherow, M. A., Samad, M. D., Diawara, N., & Iftekharuddin, K. M. (2022). Facial landmark feature fusion in transfer learning of child facial expressions. Proceedings of SPIE, 12227, 1-6, Article 122270P. https://doi.org/10.1117/12.2641898

ORCID

0000-0002-6578-4657 (Witherow), 0000-0002-8403-6793 (Diawara), 0000-0001-8316-4163 (Iftekharuddin)

Download

Share

COinS