A Teaching Assistant for Teacher-Student Learning: Knowledge Transfer from Skeleton to Inertial Sensing for Activity Recognition in Industrial Domains

Abstract

Human activity recognition (HAR) in industrial domains is important for workflow optimization, throughput estimation, and bottleneck detection. Skeleton-based models achieve high HAR accuracy by exploiting rich spatial and temporal cues, but they are difficult to deploy in industrial sites due to occlusions, camera placement, and privacy concerns. IMU sensors, especially smartwatches, are practical for deployment but lack spatial awareness, resulting in weaker performance. This work aims to enable robust HAR using only a wrist-worn IMU by distilling knowledge from richer modalities. Knowledge distillation allows transferring information from a skeleton teacher to a single-IMU student, but the large modality gap has limited the success of prior teacher–student approaches. To address this issue, we propose a teacher-assistant-student (TAS) learning framework, in which a multi-IMU assistant model bridges the skeleton-based teacher and the single-IMU student. To support TAS, we develop the following techniques: (i) Dense temporal Contrastive Learning, aligning structural representations of skeleton and IMU segments; (ii) Spatial Relationship Learning, guiding models to capture spatial priors from skeleton data; and (iii) Temporal Attention Transfer, distilling attention patterns for key atomic actions. We further boost the robustness to behavioral variation with motion-guided IMU data diversification using physics-based simulation. Experiments on industrial HAR sensor data show that our framework consistently improves single-IMU recognition across diverse operational scenarios, highlighting its potential for practical deployment.