Human beings often display empathy when talking to one another. They take into account the speaker’s facial expression and what they are saying to respond to them empathetically. If the speaker smiles, the human listening smiles back. If the speaker is expressing discontent or sadness, the listener listens patiently and tells the speaker that things would get better to encourage the listener or impart hope in the listener through words.
We propose a computational model that responds empathetically to the human considering their emotions into account. The computational model that we propose uses Llama2. We make Llama2 multimodal so that it can take into consideration both the human’s facial expressions as well as what they are saying. Llama2 is finetuned on existing emotional dialogue dataset that has empathetic responses to a wide range of queries.
The finetuned model responds to human in an empathetic manner. For example when the human says “I am tired of this. I can’t take it anymore”, the agent responds “Don’t worry, you’ll get through it”. We validate our study by performing experiments in Human-Robot Interaction scenario using the robot Pepper and conducting a survey regarding how participants thought of the interaction.