Department of Computer Science - University of Milan
I am a Ph.D. student at the Department of Computer Science of the University of Milan, and a member of EveryWare Lab.
My Ph.D. research currently focuses on Context-awareness, Graph Neural Networks, Self-supervised Learning, eXplainable AI, and Neuro-Symbolic AI (i.e., infusion of domain knowledge into a deep learning classifier) for Human Activity Recognition (HAR). My main research goal is to mitigate the data scarcity and the lack of transparency issues of deep learning models in the HAR domain.
In the general machine learning domain, solutions based on the integration of deep learning models with knowledge-based approaches are emerging. Indeed, such hybrid systems have the advantage of improving the recognition rate and the model's interpretability. At the same time, they require a significantly reduced amount of labeled data to reliably train the model. However, these techniques have been poorly explored in the sensor-based Human Activity Recognition (HAR) domain. The common-sense knowledge about activity execution can potentially improve purely data-driven approaches. While a few knowledge infusion approaches have been proposed for HAR, they rely on rigid logic formalisms that do not take into account uncertainty. In this paper, we propose P-NIMBUS, a novel knowledge infusion approach for sensor-based HAR that relies on probabilistic reasoning. A probabilistic ontology is in charge of computing symbolic features that are combined with the features automatically extracted by a CNN model from raw sensor data and high-level context data. In particular, the symbolic features encode probabilistic common-sense knowledge about the activities consistent with the user's surrounding context. These features are infused within the model before the classification layer. We experimentally evaluated P-NIMBUS on a HAR dataset of mobile devices sensor data that includes 14 different activities performed by 25 users. Our results show that P-NIMBUS outperforms state-of-the-art neuro-symbolic approaches, with the advantage of requiring a limited amount of training data to reach satisfying recognition rates (i.e., more than 80% of F1-score with only 20% of labeled data).
The sensor-based recognition of Activities of Daily Living (ADLs) in smart-home environments is an active research area, with relevant applications in healthcare and ambient assisted living. The application of Explainable Artificial Intelligence (XAI) to ADLs recognition has the potential of making this process trusted, transparent and understandable. The few works that investigated this problem considered only interpretable machine learning models. In this work, we propose DeXAR, a novel methodology to transform sensor data into semantic images to take advantage of XAI methods based on Convolutional Neural Networks (CNN). We apply different XAI approaches for deep learning and, from the resulting heat maps, we generate explanations in natural language. In order to identify the most effective XAI method, we performed extensive experiments on two different datasets, with both a common-knowledge and a user-based evaluation. The results of a user study show that the white-box XAI method based on prototypes is the most effective.
The sensor-based recognition of Activities of Daily Living (ADLs) in smart-home environments enables several important applications, including the continuous monitoring of fragile subjects in their homes for healthcare systems. The majority of the approaches in the literature assume that only one resident is living in the home. Multi-inhabitant ADLs recognition is significantly more challenging, and only a limited effort has been devoted to address this setting by the research community. One of the major open problems is called data association, which is correctly associating each environmental sensor event (e.g., the opening of a fridge door) with the inhabitant that actually triggered it. Moreover, existing multi-inhabitant approaches rely on supervised learning, assuming a high availability of labeled data. However, collecting a comprehensive training set of ADLs (especially in multiple-residents settings) is prohibitive. In this work, we propose MICAR: a novel multi-inhabitant ADLs recognition approach that combines semi-supervised learning and knowledge-based reasoning. Data association is performed by semantic reasoning, combining high-level context information (e.g., residents’ postures and semantic locations) with triggered sensor events. The personalized stream of sensor events is processed by an incremental classifier, that is initialized with a limited amount of labeled ADLs. A novel cache-based active learning strategy is adopted to continuously improve the classifier. Our results on a dataset where up to 4 subjects perform ADLs at the same time show that MICAR reliably recognizes individual and joint activities while triggering a significantly low number of active learning queries.
The recognition of human activities in sensorized smart-home environments enables a wide variety of healthcare applications, including the detection of early symptoms of cognitive decline. The most effective Human Activity Recognition (HAR) methods are based on supervised Deep Learning classifiers. Those models are usually considered as black boxes, and the rationale behind their decisions is difficult to understand for human beings. The recent advances in eXplainable Artificial Intelligence (XAI) offer promising tools to make HAR models more transparent. The state-of-the-art explainable HAR methods provide explanations for the output of classifiers that periodically predict the performed activity on short time windows (usually in the range 15-60 seconds). However, non-technical users may be more interested in investigating explanations associated with complete activity instances (e.g., an instance of the cooking activity may last 30 minutes). Unfortunately, temporally extending the time window harms the recognition rate of HAR classifiers. In this paper, we propose DeXAR++: a novel method that generates explanations for human activity instances based on deep learning classifiers. The sensor data time windows used for classification are encoded as images. DeXAR++ aggregates the explanations generated by a computer-vision XAI approach on each time window to obtain a single explanation for approximated activity instances. Moreover, DeXAR++ includes a novel visualization approach particularly suitable for non-expert users. We evaluate DeXAR++ with both automatic and user-based evaluation methodologies on a public dataset of activities performed in smart-home environments, showing that our results outperform the ones obtained by state-of-the-art methods.
Neuro-symbolic AI methods aim at integrating the capabilities of data-driven deep learning solutions with the ones of more traditional symbolic approaches. These techniques have been poorly explored in the sensor-based Human Activity Recognition (HAR) research field, even if they could lead to multiple benefits such as improving model interpretability and reducing the amount of labeled data that is necessary to reliably train the model. In this paper, we propose DUSTIN, a novel knowledge infusion approach for sensor-based HAR. DUSTIN concatenates the features automatically extracted by a CNN model from raw sensor data and high-level context data with the ones inferred by a knowledge-based reasoner. In particular, the symbolic features encode common-sense knowledge about the activities which are consistent with the context of the user, and they are infused within the model before the classification layer. We experimentally evaluated DUSTIN on a HAR dataset of mobile devices sensor data that includes 14 different activities performed by 26 users. Our results show that DUSTIN outperforms state-of-the-art neuro-symbolic approaches, with the advantage of requiring a limited amount of training data and training epochs to reach satisfying recognition rates.
While the sensor-based recognition of Activities of Daily Living (ADLs) is a well-established research area, few high-quality labeled datasets are available to compare the results of different approaches. This is especially true for multi-inhabitant settings, where multiple residents live in the same home performing both individual and collaborative ADLs. The reference multi-inhabitant datasets consider only environmental sensors data and two residents in the same home. In this paper, we present MARBLE: a novel multi-inhabitant ADLs dataset that combines both smart-watch and environmental sensors data. MARBLE includes sixteen hours of ADLs considering scripted but realistic scenarios where up to four subjects live in the same home environment. Twelve volunteers participated in data collection. We describe MARBLE also providing details on the design of data collection and tools. We also present initial benchmarks of ADLs recognition on MARBLE, obtained by applying state-of-the-art deep learning methods. Our goal is to share the result of a complex and time consuming data acquisition and annotation task, hoping that the challenge of improving the current baselines on MARBLE will contribute to the progress of the research in multi-inhabitant ADLs recognition.
Recognizing the activities of daily living (ADLs) in multi-inhabitant settings is a challenging task. One of the major challenges is the so-called data association problem: how to assign to each user the environmental sensor events that he/she actually triggered? In this paper, we tackle this problem with a contextaware approach. Each user in the home wears a smartwatch, which is used to gather several high-level context information, like the location in the home (thanks to a micro-localization infrastructure) and the posture (e.g., sitting or standing). Context data is used to associate sensor events to the users which more likely triggered them. We show the impact of context reasoning in our framework on a dataset where up to 4 subjects perform ADLs at the same time (collaboratively or individually). We also report our experience and the lessons learned in deploying a running prototype of our method.
The sensor-based detection of Activities of Daily Living (ADLs) in smart home environments can be exploited to provide healthcare applications, like remotely monitoring fragile subjects living in their habitations. However, ADLs recognition methods have been mainly investigated with a focus on singleinhabitant scenarios. The major problem in multi-inhabitant settings is data association: assigning to each resident the environmental sensors' events that he/she triggered. Furthermore, Deep Learning (DL) solutions have been recently explored for ADLs recognition, with promising results. Nevertheless, the main drawbacks of these methods are their need for large amounts of training data, and their lack of interpretability. This paper summarizes some contributions of my Ph.D. research, in which we are designing explainable multi-inhabitant approaches for ADLs recognition. We have already investigated a hybrid knowledge- and data-driven solution that exploits the high-level context of each resident to perform data association. Currently, we are studying semi-supervised techniques to mitigate the data scarcity issue, and explainable Artificial Intelligence (XAI) methods to make DL classifiers for ADLs more transparent.
Master Degree Thesis
- Luca Arrotta, Semi-Supervised Learning for Multi-Inhabitant Activity Recognition, University of Milan, 2019. PDF
- Travel Grant at ISACT 2021 summer school.
- TCCC Travel Grant at PerCom 2021 conference.
- 3 years research grant from the Italian national funding for Ph.D. courses at the Department of Computer Science, University of Milan, 2020.
Talks at International Conferences and Workshops
- IEEE DSAA, Virtual, October 2022
- ACM UbiComp, Cambridge (UK), September 2022
- IEEE SmartComp, Helsinki (Finland), June 2022
- EAI MobiQuitous, Virtual, November 2021
- IEEE MDM (PhD Forum), Virtual, June 2021
- I have reviewed papers for the IEEE MDM and IEEE PerCom conferences.
- Student Volunteer at PerCom 2022 conference.
- Ambient Intelligence and Domotics - Master course (Lab)
- Distributed and Pervasive Systems - Master course (Lab)
- Distributed and Pervasive Systems - Master course (Lab)
- Distributed and Pervasive Systems - Master course (Lab)
Department of Computer Science - EveryWare Lab (Room 7020), 7th floor
University of Milan
Via Celoria 18 - 20133 Milan, Italy