PROJECT DESCRIPTION
Personalization in physical interaction
BENEFICIARY: ORTELIO L.T.D.
OBJECTIVES: In this project, the ESR will develop a cloud-based, GPU compatible SLAM library to enable autonomous robotic movement via the cloud. Both GPU and CPU processing will be supported. The structure of the library will be compatible with the NOOS robotic app store – RAPP – which means that containers (such as Kubernetes) will be used in conjunction with Python, ROS and KERAS. The library will support a wide number of actuators and sensors by focusing on hardware abstraction via modelling instead of explicit support on multiple specific sensors. From the end-user point of view, the library will enable robots with increased autonomy and context awareness, especially with respect to contact with humans. Robots will be able to reconfigure their tasks as and when the task context changes. Possible obstacles, machinery or humans can be detected and recognised as such by the use of cloud data processing. Real-time maps can be generated, and new paths and motion planning can be drawn.
EXPECTED RESULTS: An open-source library, to be used in the cloud, that can manage the Localization and Mapping of a robot in dynamic environments like hospitals, elderly care homes, or elderly people’s houses. The aim the library is to that allow an easy personalized configuration of the robot in a new environment when it is required to accomplish tasks that include goal-oriented and user-oriented navigation.
BENEFICIARY: UNIMORE
OBJECTIVES: The aim of the project is to research new algorithms for people recognition, that effectively scale to real-world data. This project will research and design novel algorithms for human body pose estimation during interactions by effectively mix convolutional networks for body part location with regression network for 3D pose estimation in conjunction with differentiable rendering techniques capable of projecting parametric body model on the scene and optimizing the pose estimation process through plain back-propagation. The objective will be to provide real time pose estimation algorithms and exploiting the extracted 2D and 3D body poses for classifying and eventually predicting actions. A possible research line will be the exploitation for the latter task of generative recurrent models trained for both action classification and intention prediction. This task has been effectively applied to pedestrian trajectories, but a very limited number of pioneering works deal with complex full-body human actions and are limited to surveillance scenarios.
EXPECTED RESULTS: State of the art algorithms for real time body pose estimation that can be used by the robot for planning and deciding the proper strategy for the HRI. Generative models for action classification and intention prediction in order to potentially generate multiple futures and provide the robots several hypotheses on which the planning and control can be based. In synergy with concurrent European projects (MOVECARE, WISER), we will test the recognition of typical home objects (glasses, remote controllers, keys, etc.) which exhibit different point of manipulation and interaction modalities.
BENEFICIARY: TUM
OBJECTIVES: ESR3 will exploit how to transfer complex tasks to the robot by using kinaesthetic demonstrations, without the need for expert knowledge or advanced programming skills of the user. Such tasks include multi-modal conditions such as positions, forces or grasp status. Both internal joint torque sensors and external force torque sensor will be leveraged, in order to learn and generate robot behaviours, which depend on the contact location with the human or the environment. This shall be achieved by an interactive teaching scheme between user and robot. Hereby, the robot shall request further demonstrations of the user in order to resolve unseen situations and to add alternative behaviours according to the environmental state.
EXPECTED RESULTS: A robot will learn how to adapt to environmental conditions. This will be learned from expert’s demonstration in order to be used in rehabilitation settings. Desired interaction force profile will be identified and learned while a human teacher demonstrates the interaction tasks with envi- ronment via kinaesthetic teaching. The desired interaction force for the tasks and the interaction force from human’s physical guidance will be distinguished.
BENEFICIARY: PAL Robotics
OBJECTIVES: User-centred design and development of novel imitation learning techniques for enhancing the TIAGo robot. The goal is to learn human actions while modelling the physical capabilities of the human and mapping them on the robot. This includes upgrading the robot’s hardware and software (100% ROS compatible) according to the experimental results.
BENEFICIARY: UNINA
OBJECTIVES: The project aims to investigate the role of dynamical properties of the body movements as well as non-verbal cues in making the robot motion and interaction behaviour legible as much as possible, in order to allow the user predicting and estimating the interaction state or the robot intention. Both a by-design approach as the possibility of learning such behaviour will be considered. To take into account user’s preferences, interactive machine learning will be explored as a meaningful learning method where leaned behaviour can take into account the individual user’s preferences. This introduces new challenges that require a better understanding of the functionality and needs of the end user, but also a number of modelling challenges on how interactively include the target person in the learning “cycle” considering both direct and indirect feedback and/or personal goals an integral part of the model.
EXPECTED RESULTS: The proposed behavioural and learning models will be tested in a domain with both interaction tasks and non-interacting ones. Results will be evaluated in terms of the ability of the users in recognizing the intents beyond the robot movements and actions in the space and in predicting the next actions.
Personalization in COGNITIVE interaction
BENEFICIARY: TUM
OBJECTIVES: A novel human intention recognition model that integrates information on low‐level motions and higher‐level activity in order to achieve context-based human activity interpretation. Recognized activity, activity model, and context understanding are exploited to estimate future human activities or intentions. Human intention is predicted at two levels including the low-level motion trajectory and the high-level semantic activity with accompanying prediction confidence. Both levels are exploited to on-line adapt the robot’s behaviour for the successful execution of physical HRI tasks.
EXPECTED RESULTS: Effective solutions for fast and accurate human activity recognition and prediction will be developed and integrated into a framework of learning from demonstrations. It will allow on-line robot’s behaviour seamless adaptation during the execution of collaborative tasks.
BENEFICIARY: UNIMORE
OBJECTIVES: To develop new approaches that bridge together perception, language and action in robotic scenarios, fostering a natural interaction between humans and robots. Objectives of the ESR include: (i) the development of language and vision-based navigation algorithms, in which a mobile agent is trained to perform actions or reach a target destination via natural language instructions; (ii) the investigation of solutions for interacting with robotic agents in natural language, by endowing the robot with the capability of describing its current state, and understanding inputs in natural language; (iii) the training of navigation and interaction algorithms on simulated environments and their deployment on real robots.
EXPECTED RESULTS: State-of-the-art algorithms for navigation and visual-semantic tasks which can bring the interaction between human and robots feasible in natural language, and which can effectively connect vision and language on robotic systems. Novel and state of the art approaches for real-time HRI in natural language, with a specific focus on semantically challenging domains. Deployment of such algorithms on real robots.
BENEFICIARY: UMAN
OBJECTIVES: This project aims at the design of a computational architecture for ToM. This will extend the ToM model developed by Vinanzi et al., with the specific goal to support personalized interaction. The design of ToM skills will be based on developmental principles, which show the incremental acquisition of ToM from pure mechanical agency representations to actional agency and meta-representations, as in Leslie’s ToM-System1-2 theory.
EXPECTED RESULTS: The project will lead to the design and test of a computational implementation of developmental ToM architecture in cognitive robots. This will follow the methods of developmental robotics, e.g. with the testing of three incremental ToM levels: mechanical, actional and meta-representational agency. These will take into consideration the individual differences and needs of each user, to support the personalization of the robot’s interaction with the user. Experiments will validate this cognitive architecture in experiments on joint task (e.g. joint manipulation of objects in a game-like scenarios, with the robot adapting its strategy to the personalized preferences of the user).
Requirements:
– Advanced programming skills
Other elements (not requirements):
– Experience of robot programming and/or human-robot interaction research
– Knowledge of machine learning methods
BENEFICIARY: SHU
OBJECTIVES: This project aims at user cognitive modelling for improving robot collaborative behaviour and make the human-robot interfaces more intuitive for the individual user. To this end, it will create a general model of how the user think and make decisions, then, use the data collected during the HRI experiments to refine the model and customise the interaction to the specific person and task. Variables to be monitored are those that affect human cognition, such as fatigue, emotion, stress, and distraction. The final model will allow the cognitive architecture to have the capacity to infer inner user intents, which are not always consistent with behaviour, and call upon expert systems for advice when needed.
EXPECTED RESULTS: This project goal is to create a modular cognitive architecture of the robot able to assess the inner cognitive status of the user and use this to reduce the cognitive load of the user and make more effective the collaboration. The robot will have the capacity to infer user intent from the interaction, store information from experiences similarly to human memory, and increasingly personalise the interaction.
Requirements:
– Advanced programming skills
Other elements (not requirements):
– Experience of robot programming and/or human-robot interaction research
– Knowledge of machine learning methods
BENEFICIARY: SHU
OBJECTIVES: Exploit closer collaboration among cognitive robotics and education psychology for personalized robotic teaching assistants. Carry-out experi- ments in the classroom to collect data to build and train the robot’s cognitive architecture while studying children’s personal reactions to the robot. To this end, the project will mix neuro-cognitive modelling, computer vision and HRI interface design to provide robots of a controllable autonomy that can be programmed and supervised by teachers and parents.
EXPECTED RESULTS: A novel class of robotic teaching assistants that could behave like peers, i.e. capable to mimic the behaviours of children when learning mathematics. These robots can autonomously lead educational activities in form of a game, during which they interact with speech and gestures to guide the learner through learning procedures and prompt the children to identify errors in the robot behaviours. Raw data from children’s experiment will constitute an open benchmark database for testing novel machine learning algorithm.
Requirements:
– Advanced programming skills
Other elements (not requirements):
– Experience of robot programming and/or human-robot interaction research
– Knowledge of machine learning methods
Personalization in SOCIAL interaction
BENEFICIARY: UMAN
OBJECTIVES: For successful HRI, a robot must know when, and how, to interact with one, or more people, and adapt its behaviour to the individual’s prefer- ences and states (e.g. emotion). This project will take inspiration from existing deep models for action selection in social HRI scenarios, complementing it with other CNN approaches to emotion and face recognition. This will be used for personalized interaction in a robot companion for the elderly, e.g. in collaboration with the H2020 MoveCare project.
EXPECTED RESULTS: Extend the initial training dataset collected by Romeo et al. for deep learning training experiments on action selection in social inter- action. This will include a variety of interactions between elderly and robots in a social care context. The ESR will then integrate such a model with other modules on face identification and emotion recognition, to achieve a personalized and adaptive social interaction architectures. Pilot experiments with target groups of elderly will validate the integrated architecture, for further revision and extension. Some experiments will integrate ESR8 work.
BENEFICIARY: UNINA
OBJECTIVES: Design and develop a set of solutions for a multimodal user interface to maintain engagement through personalized social interaction and to enrich such interaction with social cues that can influence persuasion. To pursue this goal, the project will hybridize domain knowledge with on-line learning of the social strategies that are more effective for each specific user. The expected benefit is the reduction of the time needed for familiarisation with the robot and the support of the long-lasting use of it, minimizing the risk of rejection by frustration. Selection and social colouring of recommendations and instructions will be introduced by balancing the candidate output with the estimated receptiveness and engagement of the user.
EXPECTED RESULTS: The developed system is expected to dynamically recognize the user state (e.g., engagement) and to adapt its social cues accordingly. A/B testing of the Social Robot/User interaction in a controlled environment, and then on a realistic scenario for recommendation purposes. The goal is to assess the system usability, and the user satisfaction for received recommendations.
BENEFICIARY: UNIBI
OBJECTIVES: The project will be grounded in empirical experimental studies that integrate the expertise in social robotics and social psychology represented by the applicants. Studies on cognitive vs. affective trust and their differential behavioural correlates will be realized using various robot platforms (e.g., PAL Robotics platforms or those available at UNIBI as Pepper, NAO, iCub or Meka). The degree of adaptation and personalization (e.g., through user preference profiles, similarity, etc.) will be manipulated, while user factors (e.g., gender, negative attitudes towards robots, prior robot experience, trust propensity, trust in automation, demographics etc.) will be assessed to contribute further to an extended understanding of human-robot trust, both in short-term and in long-term interactions. Moreover, cognitive and affective trust will be manipulated to identify and study differential behaviours that might result from high vs. low levels of cognitive vs. affective trust elicited by a robot.
EXPECTED RESULTS: Validate key aspects that determine judgments of cognitive and affective trust in robots, focusing on user- and robot-related character- istics. Shed light on core psychological determinants on the part of the user, while simultaneously identifying core robot features that predict differential levels of cognitive and affective trust in HRI. Explore the effects of the different types of trust on behaviours that are reflective of and specific for cognitive vs. affective trust, respectively. As a use case, a human-robot learning/gaming paradigm will be validated. The results will provide guidelines on how to increase the development and maintenance of long-term trust in robots through personalization and adaptation.
BENEFICIARY: SSSA
OBJECTIVES: The project aim is threefold: to identify the design criteria necessary for developing pleasant robots; to discuss the problem of deception and the possibility that (especially vulnerable) users may be lead to attribute features and qualities to the machine that, instead, machines do not possess (e.g. ability to feel emotions and build emotional bond with humans etc.); to discuss ethical and legal aspects of such interaction. For this matter, ESR14 will take into consideration the technological, philosophical and psychological research carried out at other partner institutions (also participating in said research, thanks to the exchange periods of the duration of up to 6 months duration, over the three year program), the European ethical and legal principles (dignity, equality, self-determination), including the initiatives on the development of ethical guidelines for trustworthy AI (and advanced robotics) currently undertaken at the European level. In particular, the project will consider the challenges of standardization in ethics, considering alternative ethical approaches, such as the utilitarian and deontological (neo-kantian).
EXPECTED RESULTS: Analysis of the legal, social and ethical problems connected to the human machine interaction, identification of criteria for the development of the ethically aligned design; elaboration of policy guidelines on the topic; assessment of how fundamental principles and rights may be interpreted and applied to determine licit, and socially desirable uses of said applications.
BENEFICIARY: UNIVIE
OBJECTIVES: The aim of the project is to define the user’s emotional perspective and her moral rights in care situations. First, the persuasive technologies and social robots and the discourses of critique and legitimation surrounding them. Second, the ethical questions regarding the role and tasks, moral agency and responsibility assigned to care and companion robots in the context of elderly care. Third, trust, deception and data privacy. Finally, the implications for policymaking considering that currently healthy persons pay into the social insurance system in the acceptance of being cared for by people.
EXPECTED RESULTS: The project delivers a case study for moral deliberation in research and engineering of “persuasive technologies” and social robots for the concrete use case of care and companion robots targeted on elderly people. It will insight for the elaboration of RRI guidelines with respect the moral implication of the use of assistive robotics.