The continuous miniaturization of integrated circuits and sensor technologies has greatly accelerated the advancement of wearable devices.
Modern wearables integrate a range of heterogeneous sensors that capture data about their wearers and their surroundings. Processing these data enables the extraction of valuable contextual information, such as the wearer’s activity, location, and more. In real-world scenarios, where data are highly variable and complex, deep learning models have demonstrated remarkable potential for effectively processing such data. Nevertheless, deploying these models on wearable platforms remains a considerable challenge due to their high computational and memory requirements. Because wearable devices are intended for daily use, they impose strict constraints on form factor and power consumption, which is why they are typically built around low-power microcontrollers.
This thesis addresses these constraints with the goal of proposing deep learning approaches to recognize the intrinsic and extrinsic contexts associated with eyeglass wearers. To this end, we first propose a multimodal, multitask deep learning approach that tackles two central challenges.
The first concerns the hardware restrictions that prohibit the execution of multiple independent inference models on a resource-constrained microcontroller. The second relates to improving the model’s generalization capacity by exploiting interdependencies across multiple context-recognition tasks. To evaluate this approach, we construct a real-world dataset and train the proposed model of heterogeneous sensor data, including audio, acceleration, and ambient light, with optional integration of visual data to assess its impact on performance. Empirical results show that our approach outperforms a range of single-task baselines employing diverse sensor fusion strategies.
In parallel, we propose a second approach based on an on-device adaptation of our multimodal, multitask model.
This approach adapts the model by continuously retraining it on data from unknown wearers. We validate this approach through an experimental protocol designed to reflect real-world deployment scenarios.
Furthermore, we introduce a data compression method that enables training samples to be stored within the internal memory of our microcontrollers, thereby further optimizing our model’s performance.
Finally, we present a hardware integration of our proposals, confirming their effectiveness in an environment subject to strict memory and energy constraints.