Luca Arrotta

Publications

Journals

• Arrotta, L., Civitarese, G., Bettini, C. (2023, November) Semantic Loss: a new Neuro-Symbolic approach for Context-Aware Human Activity Recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2023, to appear.
  
  

  Abstract:

  Deep Learning models are a standard solution for sensor-based Human Activity Recognition (HAR), but their deployment is often limited by labeled data scarcity and models’ opacity. Neuro-Symbolic AI (NeSy) provides an interesting research direction to mitigate these issues by infusing knowledge about context information into HAR deep learning classifiers. However, existing NeSy methods for context-aware HAR require computationally expensive symbolic reasoners during classification, making them less suitable for deployment on resource-constrained devices (e.g., mobile devices). Additionally, NeSy approaches for context-aware HAR have never been evaluated on in-the-wild datasets, and their generalization capabilities in real-world scenarios are questionable. In this work, we propose a novel approach based on a semantic loss function that infuses knowledge constraints in the HAR model during the training phase, avoiding symbolic reasoning during classification. Our results on scripted and in-the-wild datasets show the impact of different semantic loss functions in outperforming a purely data-driven model. We also compare our solution with existing NeSy methods and analyze each approach’s strengths and weaknesses. Our semantic loss remains the only NeSy solution that can be deployed as a single DNN without the need for symbolic reasoning modules, reaching recognition rates close (and better in some cases) to existing approaches.

  
  
• Arrotta, L., Civitarese, G., Bettini, C. (2023, April) Probabilistic Knowledge Infusion through Symbolic Features for Context-Aware Activity Recognition. Pervasive and Mobile Computing, Elsevier, 2023. (DOI: 10.1016/j.pmcj.2023.101780)
  
  

  Abstract:

  In the general machine learning domain, solutions based on the integration of deep learning models with knowledge-based approaches are emerging. Indeed, such hybrid systems have the advantage of improving the recognition rate and the model's interpretability. At the same time, they require a significantly reduced amount of labeled data to reliably train the model. However, these techniques have been poorly explored in the sensor-based Human Activity Recognition (HAR) domain. The common-sense knowledge about activity execution can potentially improve purely data-driven approaches. While a few knowledge infusion approaches have been proposed for HAR, they rely on rigid logic formalisms that do not take into account uncertainty. In this paper, we propose P-NIMBUS, a novel knowledge infusion approach for sensor-based HAR that relies on probabilistic reasoning. A probabilistic ontology is in charge of computing symbolic features that are combined with the features automatically extracted by a CNN model from raw sensor data and high-level context data. In particular, the symbolic features encode probabilistic common-sense knowledge about the activities consistent with the user's surrounding context. These features are infused within the model before the classification layer. We experimentally evaluated P-NIMBUS on a HAR dataset of mobile devices sensor data that includes 14 different activities performed by 25 users. Our results show that P-NIMBUS outperforms state-of-the-art neuro-symbolic approaches, with the advantage of requiring a limited amount of training data to reach satisfying recognition rates (i.e., more than 80% of F1-score with only 20% of labeled data).

  
  
• Arrotta, L., Civitarese, G., Bettini, C. (2022, April) MICAR: Multi-Inhabitant Context-Aware Activity Recognition in Home Environments. Distributed and Parallel Database, Springer, 2022. (DOI: 10.1007/s10619-022-07403-z)
  
  

  Abstract:

  The sensor-based recognition of Activities of Daily Living (ADLs) in smart-home environments enables several important applications, including the continuous monitoring of fragile subjects in their homes for healthcare systems. The majority of the approaches in the literature assume that only one resident is living in the home. Multi-inhabitant ADLs recognition is significantly more challenging, and only a limited effort has been devoted to address this setting by the research community. One of the major open problems is called data association, which is correctly associating each environmental sensor event (e.g., the opening of a fridge door) with the inhabitant that actually triggered it. Moreover, existing multi-inhabitant approaches rely on supervised learning, assuming a high availability of labeled data. However, collecting a comprehensive training set of ADLs (especially in multiple-residents settings) is prohibitive. In this work, we propose MICAR: a novel multi-inhabitant ADLs recognition approach that combines semi-supervised learning and knowledge-based reasoning. Data association is performed by semantic reasoning, combining high-level context information (e.g., residents’ postures and semantic locations) with triggered sensor events. The personalized stream of sensor events is processed by an incremental classifier, that is initialized with a limited amount of labeled ADLs. A novel cache-based active learning strategy is adopted to continuously improve the classifier. Our results on a dataset where up to 4 subjects perform ADLs at the same time show that MICAR reliably recognizes individual and joint activities while triggering a significantly low number of active learning queries.

  
  
• Arrotta, L., Civitarese, G., Bettini, C. (2022, March) DeXAR: Deep Explainable Sensor-Based Activity Recognition in Smart-Home Environments. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2022. (DOI: 10.1145/3517224)
  
  

  Abstract:

  The sensor-based recognition of Activities of Daily Living (ADLs) in smart-home environments is an active research area, with relevant applications in healthcare and ambient assisted living. The application of Explainable Artificial Intelligence (XAI) to ADLs recognition has the potential of making this process trusted, transparent and understandable. The few works that investigated this problem considered only interpretable machine learning models. In this work, we propose DeXAR, a novel methodology to transform sensor data into semantic images to take advantage of XAI methods based on Convolutional Neural Networks (CNN). We apply different XAI approaches for deep learning and, from the resulting heat maps, we generate explanations in natural language. In order to identify the most effective XAI method, we performed extensive experiments on two different datasets, with both a common-knowledge and a user-based evaluation. The results of a user study show that the white-box XAI method based on prototypes is the most effective.

  
  

International Conferences

• Arrotta, L., Civitarese, G., Bettini, C., (2023, October). SelfAct: Personalized Activity Recognition based on Self-Supervised and Active Learning. In Proceedings of the 20th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (Mobiquitous), EAI, 2023, to appear.
  
  

  Abstract:

  Supervised Deep Learning (DL) models are currently the leading approach for sensor-based Human Activity Recognition (HAR) on wearable and mobile devices. However, training them requires large amounts of labeled data whose collection is often time-consuming, expensive, and error-prone. At the same time, due to the intra- and inter-variability of activity execution, activity models should be personalized for each user. In this work, we propose SelfAct: a novel framework for HAR combining self-supervised and active learning to mitigate these problems. SelfAct leverages a large pool of unlabeled data collected from many users to pre-train through self-supervision a DL model, with the goal of learning a meaningful and efficient latent representation of sensor data. The resulting pre-trained model can be locally used by new users, which will fine-tune it thanks to a novel unsupervised active learning strategy. Our experiments on two publicly available HAR datasets demonstrate that SelfAct achieves results that are close to or even better than the ones of fully supervised approaches with a small number of active learning queries.

  
  
• Arrotta, L., Civitarese, G., Fiori, M., Bettini, C., (2022, October). Explaining Human Activities Instances Using Deep Learning Classifiers. In 2022 IEEE International Conference on Data Science and Advanced Analytics (DSAA), 2022. Online
  
  

  Abstract:

  The recognition of human activities in sensorized smart-home environments enables a wide variety of healthcare applications, including the detection of early symptoms of cognitive decline. The most effective Human Activity Recognition (HAR) methods are based on supervised Deep Learning classifiers. Those models are usually considered as black boxes, and the rationale behind their decisions is difficult to understand for human beings. The recent advances in eXplainable Artificial Intelligence (XAI) offer promising tools to make HAR models more transparent. The state-of-the-art explainable HAR methods provide explanations for the output of classifiers that periodically predict the performed activity on short time windows (usually in the range 15-60 seconds). However, non-technical users may be more interested in investigating explanations associated with complete activity instances (e.g., an instance of the cooking activity may last 30 minutes). Unfortunately, temporally extending the time window harms the recognition rate of HAR classifiers. In this paper, we propose DeXAR++: a novel method that generates explanations for human activity instances based on deep learning classifiers. The sensor data time windows used for classification are encoded as images. DeXAR++ aggregates the explanations generated by a computer-vision XAI approach on each time window to obtain a single explanation for approximated activity instances. Moreover, DeXAR++ includes a novel visualization approach particularly suitable for non-expert users. We evaluate DeXAR++ with both automatic and user-based evaluation methodologies on a public dataset of activities performed in smart-home environments, showing that our results outperform the ones obtained by state-of-the-art methods.

  
  
• Arrotta, L., Bettini, C., Civitarese, G., (2022, June). Knowledge Infusion for Context-Aware Sensor-Based Human Activity Recognition. In 2022 IEEE International Conference on Smart Computing (SmartComp), 2022. Online
  
  

  Abstract:

  Neuro-symbolic AI methods aim at integrating the capabilities of data-driven deep learning solutions with the ones of more traditional symbolic approaches. These techniques have been poorly explored in the sensor-based Human Activity Recognition (HAR) research field, even if they could lead to multiple benefits such as improving model interpretability and reducing the amount of labeled data that is necessary to reliably train the model. In this paper, we propose DUSTIN, a novel knowledge infusion approach for sensor-based HAR. DUSTIN concatenates the features automatically extracted by a CNN model from raw sensor data and high-level context data with the ones inferred by a knowledge-based reasoner. In particular, the symbolic features encode common-sense knowledge about the activities which are consistent with the context of the user, and they are infused within the model before the classification layer. We experimentally evaluated DUSTIN on a HAR dataset of mobile devices sensor data that includes 14 different activities performed by 26 users. Our results show that DUSTIN outperforms state-of-the-art neuro-symbolic approaches, with the advantage of requiring a limited amount of training data and training epochs to reach satisfying recognition rates.

  
  
• Arrotta, L., Bettini, C., Civitarese, G. (2022, February). The MARBLE Dataset: Multi-Inhabitant Activities of Daily Living Combining Wearable and Environmental Sensors Data. In: Hara T., Yamaguchi H. (eds) Mobile and Ubiquitous Systems: Computing, Networking and Services. MobiQuitous 2021. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 419. Springer, Cham. Online - Dataset
  
  

  Abstract:

  While the sensor-based recognition of Activities of Daily Living (ADLs) is a well-established research area, few high-quality labeled datasets are available to compare the results of different approaches. This is especially true for multi-inhabitant settings, where multiple residents live in the same home performing both individual and collaborative ADLs. The reference multi-inhabitant datasets consider only environmental sensors data and two residents in the same home. In this paper, we present MARBLE: a novel multi-inhabitant ADLs dataset that combines both smart-watch and environmental sensors data. MARBLE includes sixteen hours of ADLs considering scripted but realistic scenarios where up to four subjects live in the same home environment. Twelve volunteers participated in data collection. We describe MARBLE also providing details on the design of data collection and tools. We also present initial benchmarks of ADLs recognition on MARBLE, obtained by applying state-of-the-art deep learning methods. Our goal is to share the result of a complex and time consuming data acquisition and annotation task, hoping that the challenge of improving the current baselines on MARBLE will contribute to the progress of the research in multi-inhabitant ADLs recognition.

  
  
• Arrotta, L., Bettini, C., Civitarese, G., Presotto, R. (2020, July). Context-Aware Data Association for Multi-Inhabitant Sensor-Based Activity Recognition. In 2020 21st IEEE International Conference on Mobile Data Management (MDM) (pp. 125-130). IEEE. Online
  
  

  Abstract:

  Recognizing the activities of daily living (ADLs) in multi-inhabitant settings is a challenging task. One of the major challenges is the so-called data association problem: how to assign to each user the environmental sensor events that he/she actually triggered? In this paper, we tackle this problem with a contextaware approach. Each user in the home wears a smartwatch, which is used to gather several high-level context information, like the location in the home (thanks to a micro-localization infrastructure) and the posture (e.g., sitting or standing). Context data is used to associate sensor events to the users which more likely triggered them. We show the impact of context reasoning in our framework on a dataset where up to 4 subjects perform ADLs at the same time (collaboratively or individually). We also report our experience and the lessons learned in deploying a running prototype of our method.

  
  

International Workshops

• Arrotta, L., Civitarese, G., Presotto, R., Bettini, C. (2023, July). DOMINO: A Dataset for Context-Aware Human Activity Recognition using Mobile Devices. In 2023 24th IEEE International Conference on Mobile Data Management (MDM) Workshops. IEEE. Online - Dataset
  
  

  Abstract:

  Human Activity Recognition (HAR) with mobile and wearable devices has been deeply studied in the last decades. Research groups working on this topic evaluated their proposed methods mostly on public datasets. However, most of the existing datasets only include inertial sensor data, while it is well-known that additional context data (e.g., semantic location) has the potential to significantly improve the recognition rate. Only a few datasets for context-aware HAR are publicly available, and their annotations were mostly self-reported in-the-wild by the subjects involved in data acquisition. This method harms the quality of annotations, thus discouraging the application of supervised models. In this paper, we propose DOMINO, a new public dataset for context-aware HAR. DOMINO includes 25 users (wearing a smartphone and a smartwatch) performing 14 activities. During data acquisition, the mobile devices recorded both inertial and high-level context data while our team monitored the quality of the self-reported annotations. Our experiments on DOMINO show the positive impact of considering high-level context information for Human Activity Recognition.

  
  
• Arrotta, L. (2021, June). Multi-inhabitant and Explainable Activity Recognition in Smart Homes. In 2021 22nd IEEE International Conference on Mobile Data Management (MDM) (pp. 264-266). IEEE. Online
  
  

  Abstract:

  The sensor-based detection of Activities of Daily Living (ADLs) in smart home environments can be exploited to provide healthcare applications, like remotely monitoring fragile subjects living in their habitations. However, ADLs recognition methods have been mainly investigated with a focus on singleinhabitant scenarios. The major problem in multi-inhabitant settings is data association: assigning to each resident the environmental sensors' events that he/she triggered. Furthermore, Deep Learning (DL) solutions have been recently explored for ADLs recognition, with promising results. Nevertheless, the main drawbacks of these methods are their need for large amounts of training data, and their lack of interpretability. This paper summarizes some contributions of my Ph.D. research, in which we are designing explainable multi-inhabitant approaches for ADLs recognition. We have already investigated a hybrid knowledge- and data-driven solution that exploits the high-level context of each resident to perform data association. Currently, we are studying semi-supervised techniques to mitigate the data scarcity issue, and explainable Artificial Intelligence (XAI) methods to make DL classifiers for ADLs more transparent.

  
  

Technical Reports

• Arrotta, L., Civitarese, G., Bettini, C. (2023, June) Neuro-Symbolic Approaches for Context-Aware Human Activity Recognition. arXiv preprint arXiv:2306.05058. Online
  
  

  Abstract:

  Deep Learning models are a standard solution for sensor-based Human Activity Recognition (HAR), but their deployment is often limited by labeled data scarcity and models' opacity. Neuro-Symbolic AI (NeSy) provides an interesting research direction to mitigate these issues by infusing knowledge about context information into HAR deep learning classifiers. However, existing NeSy methods for context-aware HAR require computationally expensive symbolic reasoners during classification, making them less suitable for deployment on resource-constrained devices (e.g., mobile devices). Additionally, NeSy approaches for context-aware HAR have never been evaluated on in-the-wild datasets, and their generalization capabilities in real-world scenarios are questionable. In this work, we propose a novel approach based on a semantic loss function that infuses knowledge constraints in the HAR model during the training phase, avoiding symbolic reasoning during classification. Our results on scripted and in-the-wild datasets show the impact of different semantic loss functions in outperforming a purely data-driven model. We also compare our solution with existing NeSy methods and analyze each approach's strengths and weaknesses. Our semantic loss remains the only NeSy solution that can be deployed as a single DNN without the need for symbolic reasoning modules, reaching recognition rates close (and better in some cases) to existing approaches.

  
  
• Arrotta, L., Civitarese, G., Valente, S., Bettini, C. (2023, April) SelfAct: Personalized Activity Recognition based on Self-Supervised and Active Learning. arXiv preprint arXiv:2304.09530. Online
  
  

  Abstract:

  Supervised Deep Learning (DL) models are currently the leading approach for sensor-based Human Activity Recognition (HAR) on wearable and mobile devices. However, training them requires large amounts of labeled data whose collection is often time-consuming, expensive, and error-prone. At the same time, due to the intra- and inter-variability of activity execution, activity models should be personalized for each user. In this work, we propose SelfAct: a novel framework for HAR combining self-supervised and active learning to mitigate these problems. SelfAct leverages a large pool of unlabeled data collected from many users to pre-train through self-supervision a DL model, with the goal of learning a meaningful and efficient latent representation of sensor data. The resulting pre-trained model can be locally used by new users, which will fine-tune it thanks to a novel unsupervised active learning strategy. Our experiments on two publicly available HAR datasets demonstrate that SelfAct achieves results that are close to or even better than the ones of fully supervised approaches with a small number of active learning queries.

  
  

Master Degree Thesis

• Luca Arrotta, Semi-Supervised Learning for Multi-Inhabitant Activity Recognition, University of Milan, 2019. PDF