Luca Arrotta

Publications

Journals

• Arrotta, L., Civitarese, G., Bettini, C. (2023, March) Probabilistic Knowledge Infusion through Symbolic Features for Context-Aware Activity Recognition. Pervasive and Mobile Computing, Elsevier, 2023. (DOI: 10.1016/j.pmcj.2023.101780)
  
  

  Abstract:

  In the general machine learning domain, solutions based on the integration of deep learning models with knowledge-based approaches are emerging. Indeed, such hybrid systems have the advantage of improving the recognition rate and the model's interpretability. At the same time, they require a significantly reduced amount of labeled data to reliably train the model. However, these techniques have been poorly explored in the sensor-based Human Activity Recognition (HAR) domain. The common-sense knowledge about activity execution can potentially improve purely data-driven approaches. While a few knowledge infusion approaches have been proposed for HAR, they rely on rigid logic formalisms that do not take into account uncertainty. In this paper, we propose P-NIMBUS, a novel knowledge infusion approach for sensor-based HAR that relies on probabilistic reasoning. A probabilistic ontology is in charge of computing symbolic features that are combined with the features automatically extracted by a CNN model from raw sensor data and high-level context data. In particular, the symbolic features encode probabilistic common-sense knowledge about the activities consistent with the user's surrounding context. These features are infused within the model before the classification layer. We experimentally evaluated P-NIMBUS on a HAR dataset of mobile devices sensor data that includes 14 different activities performed by 25 users. Our results show that P-NIMBUS outperforms state-of-the-art neuro-symbolic approaches, with the advantage of requiring a limited amount of training data to reach satisfying recognition rates (i.e., more than 80% of F1-score with only 20% of labeled data).

  
  
• Arrotta, L., Civitarese, G., Bettini, C. (2022, March) DeXAR: Deep Explainable Sensor-Based Activity Recognition in Smart-Home Environments. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2022. (DOI: 10.1145/3517224)
  
  

  Abstract:

  The sensor-based recognition of Activities of Daily Living (ADLs) in smart-home environments is an active research area, with relevant applications in healthcare and ambient assisted living. The application of Explainable Artificial Intelligence (XAI) to ADLs recognition has the potential of making this process trusted, transparent and understandable. The few works that investigated this problem considered only interpretable machine learning models. In this work, we propose DeXAR, a novel methodology to transform sensor data into semantic images to take advantage of XAI methods based on Convolutional Neural Networks (CNN). We apply different XAI approaches for deep learning and, from the resulting heat maps, we generate explanations in natural language. In order to identify the most effective XAI method, we performed extensive experiments on two different datasets, with both a common-knowledge and a user-based evaluation. The results of a user study show that the white-box XAI method based on prototypes is the most effective.

  
  
• Arrotta, L., Civitarese, G., Bettini, C. (2022, March) MICAR: Multi-Inhabitant Context-Aware Activity Recognition in Home Environments. Distributed and Parallel Database, Springer, 2022. (DOI: 10.1007/s10619-022-07403-z)
  
  

  Abstract:

  The sensor-based recognition of Activities of Daily Living (ADLs) in smart-home environments enables several important applications, including the continuous monitoring of fragile subjects in their homes for healthcare systems. The majority of the approaches in the literature assume that only one resident is living in the home. Multi-inhabitant ADLs recognition is significantly more challenging, and only a limited effort has been devoted to address this setting by the research community. One of the major open problems is called data association, which is correctly associating each environmental sensor event (e.g., the opening of a fridge door) with the inhabitant that actually triggered it. Moreover, existing multi-inhabitant approaches rely on supervised learning, assuming a high availability of labeled data. However, collecting a comprehensive training set of ADLs (especially in multiple-residents settings) is prohibitive. In this work, we propose MICAR: a novel multi-inhabitant ADLs recognition approach that combines semi-supervised learning and knowledge-based reasoning. Data association is performed by semantic reasoning, combining high-level context information (e.g., residents’ postures and semantic locations) with triggered sensor events. The personalized stream of sensor events is processed by an incremental classifier, that is initialized with a limited amount of labeled ADLs. A novel cache-based active learning strategy is adopted to continuously improve the classifier. Our results on a dataset where up to 4 subjects perform ADLs at the same time show that MICAR reliably recognizes individual and joint activities while triggering a significantly low number of active learning queries.

  
  

International Conferences

• Arrotta, L., Civitarese, G., Fiori, M., Bettini, C., (2022, August). Explaining Human Activities Instances Using Deep Learning Classifiers. In 2022 IEEE International Conference on Data Science and Advanced Analytics (DSAA), 2022. Online
  
  

  Abstract:

  The recognition of human activities in sensorized smart-home environments enables a wide variety of healthcare applications, including the detection of early symptoms of cognitive decline. The most effective Human Activity Recognition (HAR) methods are based on supervised Deep Learning classifiers. Those models are usually considered as black boxes, and the rationale behind their decisions is difficult to understand for human beings. The recent advances in eXplainable Artificial Intelligence (XAI) offer promising tools to make HAR models more transparent. The state-of-the-art explainable HAR methods provide explanations for the output of classifiers that periodically predict the performed activity on short time windows (usually in the range 15-60 seconds). However, non-technical users may be more interested in investigating explanations associated with complete activity instances (e.g., an instance of the cooking activity may last 30 minutes). Unfortunately, temporally extending the time window harms the recognition rate of HAR classifiers. In this paper, we propose DeXAR++: a novel method that generates explanations for human activity instances based on deep learning classifiers. The sensor data time windows used for classification are encoded as images. DeXAR++ aggregates the explanations generated by a computer-vision XAI approach on each time window to obtain a single explanation for approximated activity instances. Moreover, DeXAR++ includes a novel visualization approach particularly suitable for non-expert users. We evaluate DeXAR++ with both automatic and user-based evaluation methodologies on a public dataset of activities performed in smart-home environments, showing that our results outperform the ones obtained by state-of-the-art methods.

  
  
• Arrotta, L., Bettini, C., Civitarese, G., (2022, April). Knowledge Infusion for Context-Aware Sensor-Based Human Activity Recognition. In 2022 IEEE International Conference on Smart Computing (SmartComp), 2022. Online
  
  

  Abstract:

  Neuro-symbolic AI methods aim at integrating the capabilities of data-driven deep learning solutions with the ones of more traditional symbolic approaches. These techniques have been poorly explored in the sensor-based Human Activity Recognition (HAR) research field, even if they could lead to multiple benefits such as improving model interpretability and reducing the amount of labeled data that is necessary to reliably train the model. In this paper, we propose DUSTIN, a novel knowledge infusion approach for sensor-based HAR. DUSTIN concatenates the features automatically extracted by a CNN model from raw sensor data and high-level context data with the ones inferred by a knowledge-based reasoner. In particular, the symbolic features encode common-sense knowledge about the activities which are consistent with the context of the user, and they are infused within the model before the classification layer. We experimentally evaluated DUSTIN on a HAR dataset of mobile devices sensor data that includes 14 different activities performed by 26 users. Our results show that DUSTIN outperforms state-of-the-art neuro-symbolic approaches, with the advantage of requiring a limited amount of training data and training epochs to reach satisfying recognition rates.

  
  
• Arrotta, L., Bettini, C., Civitarese, G. (2021, November). The MARBLE Dataset: Multi-Inhabitant Activities of Daily Living Combining Wearable and Environmental Sensors Data. In: Hara T., Yamaguchi H. (eds) Mobile and Ubiquitous Systems: Computing, Networking and Services. MobiQuitous 2021. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 419. Springer, Cham. Online - Dataset
  
  

  Abstract:

  While the sensor-based recognition of Activities of Daily Living (ADLs) is a well-established research area, few high-quality labeled datasets are available to compare the results of different approaches. This is especially true for multi-inhabitant settings, where multiple residents live in the same home performing both individual and collaborative ADLs. The reference multi-inhabitant datasets consider only environmental sensors data and two residents in the same home. In this paper, we present MARBLE: a novel multi-inhabitant ADLs dataset that combines both smart-watch and environmental sensors data. MARBLE includes sixteen hours of ADLs considering scripted but realistic scenarios where up to four subjects live in the same home environment. Twelve volunteers participated in data collection. We describe MARBLE also providing details on the design of data collection and tools. We also present initial benchmarks of ADLs recognition on MARBLE, obtained by applying state-of-the-art deep learning methods. Our goal is to share the result of a complex and time consuming data acquisition and annotation task, hoping that the challenge of improving the current baselines on MARBLE will contribute to the progress of the research in multi-inhabitant ADLs recognition.

  
  
• Arrotta, L., Bettini, C., Civitarese, G., Presotto, R. (2020, June). Context-Aware Data Association for Multi-Inhabitant Sensor-Based Activity Recognition. In 2020 21st IEEE International Conference on Mobile Data Management (MDM) (pp. 125-130). IEEE. Online
  
  

  Abstract:

  Recognizing the activities of daily living (ADLs) in multi-inhabitant settings is a challenging task. One of the major challenges is the so-called data association problem: how to assign to each user the environmental sensor events that he/she actually triggered? In this paper, we tackle this problem with a contextaware approach. Each user in the home wears a smartwatch, which is used to gather several high-level context information, like the location in the home (thanks to a micro-localization infrastructure) and the posture (e.g., sitting or standing). Context data is used to associate sensor events to the users which more likely triggered them. We show the impact of context reasoning in our framework on a dataset where up to 4 subjects perform ADLs at the same time (collaboratively or individually). We also report our experience and the lessons learned in deploying a running prototype of our method.

  
  

International Workshops

• Arrotta, L. (2021, June). Multi-inhabitant and Explainable Activity Recognition in Smart Homes. In 2021 22nd IEEE International Conference on Mobile Data Management (MDM) (pp. 264-266). IEEE. Online
  
  

  Abstract:

  The sensor-based detection of Activities of Daily Living (ADLs) in smart home environments can be exploited to provide healthcare applications, like remotely monitoring fragile subjects living in their habitations. However, ADLs recognition methods have been mainly investigated with a focus on singleinhabitant scenarios. The major problem in multi-inhabitant settings is data association: assigning to each resident the environmental sensors' events that he/she triggered. Furthermore, Deep Learning (DL) solutions have been recently explored for ADLs recognition, with promising results. Nevertheless, the main drawbacks of these methods are their need for large amounts of training data, and their lack of interpretability. This paper summarizes some contributions of my Ph.D. research, in which we are designing explainable multi-inhabitant approaches for ADLs recognition. We have already investigated a hybrid knowledge- and data-driven solution that exploits the high-level context of each resident to perform data association. Currently, we are studying semi-supervised techniques to mitigate the data scarcity issue, and explainable Artificial Intelligence (XAI) methods to make DL classifiers for ADLs more transparent.

  
  

Master Degree Thesis

• Luca Arrotta, Semi-Supervised Learning for Multi-Inhabitant Activity Recognition, University of Milan, 2019. PDF