Background
Schizophrenia is a mental illness characterized by distortions in thinking, emotions, behavior, language, perception, sense of self (World Health Organization, 2019), and impairment in communication. These impairments can be observed through verbal and non-verbal communications. In verbal communication, impairments as a manifestation of thought disorder include unintelligible and disorganized speech, problems expressing thought through language, problems in differentiating verbalized and external speech, and difficulty interpreting long and complex sentences (Kuperberg, 2010). In non-verbal communication, impairments involve poor social perception, impaired gesture production, deficit tool use, and poor gestural knowledge (Walther et al., 2015).
Schizophrenia can be treated with pharmacotherapeutics and psychosocial supports, which are considered effective (Kahn et al., 2015; World Health Organization, 2019), such as social skills training and cognitive-behavioral therapy. One of the ways to support the treatment of patients with schizophrenia, particularly for communication problems, is using healthcare robots (Ozeki et al., 2020; Ujike et al., 2019). A study found that patients with schizophrenia conversed with robots well, whether robots were talkative or not. When the robot did not talk, the patients themselves initiated the conversation with these robots (Ozeki et al., 2020). Through conversing, answering, and asking questions to healthcare robots, patients with schizophrenia showed improvements toward establishing communication progress (Ujike et al., 2019).
Van Wynsberghe (2013) clarified that care robots are used for patient care in healthcare settings. Different types of healthcare robots were used for different purposes, for example, PALRO robot for communication use (Ozeki et al., 2020), PARO robot for behavioral therapy (Sabanovic et al., 2013), and Pepper for interactive dialogue and physical exercise guidance.
Pepper (SoftBank Robotics, n.d.) is a humanoid robot designed to interact with humans through conversation and touch screen with a tablet attached to its chest. Pepper can recognize faces and identify basic emotions while engaging with humans (SoftBank Robotics, n.d.). Attempts are being made to provide patients with schizophrenia the opportunity to enjoy talking to Pepper. However, it is relatively common to find verbal dysfunctions in patients with schizophrenia in clinical practice. Thus, they might have different interactive communication characteristics: 1) to make his/her speech clearer, more comprehensible, more informative; 2) to summarize a part of his/her own speech; 3) to choose the appropriate meaning of an ambiguous polysemic word on the basis of the context provided by the sentence or the conversation; and, 4) the difficulty describing or recognizing other people’s intentions (Bazin et al., 2005).
In the highly demanding technological world involving nursing and healthcare, nurses need to be aware that it is important to understand disruptive information with technologies to serve nurses and patients well in nursing practice (Aungsuroch & Gunawan, 2019). Nursing in the technologically-demanding world of healthcare requires technologically competent nursing practice (Locsin, 2005). When healthcare robots, such as Pepper, are used for patient care, especially for communication purposes, nurses should understand how robots can function as interactive technologies.
Previous case studies had examined Pepper interaction with an older person with a mental health condition (Tanioka, Betriana, et al., 2021). In that study, Pepper conversed only with one patient, and the interaction resulted on the occasion of joy (Tanioka, Betriana, et al., 2021). However, how Pepper would function during the interaction with more than one person at one time and how the interaction would be different between patients with a mental health condition and those who are healthy are not well studied and known.
Understanding the characteristic of communication between Pepper and multiple persons at the same time is important in the human-robot interaction. This information can inform nurses and roboticists to design and program healthcare robots based on the expected communication characteristics according to different users. Therefore, this study aimed to describe the characteristics of interactive communication between Pepper as a healthcare robot with patients with schizophrenia and healthy persons.
Case Presentation
Study Design
This is a descriptive case study involving two cases that employed the Intentional Observational Research Design (IOCRD) (Tanioka, Locsin, et al., 2021). The IOCRD is a research design for simultaneous data generation using advanced technological devices intended for clinical phenomena involving healthcare robots. IOCRD combined quantitative, qualitative, and intentional observation approaches for data generation (Tanioka, Locsin, et al., 2021). However, in this study, data were collected using intentional observation and interviews, which are essential parts of the IOCRD. The interaction of the two cases—two patients with schizophrenia with Pepper and the interaction of two healthy persons with Pepper—was observed and audio-video data were recorded simultaneously. After the conversations, participants were asked about their experiences, including feelings and impressions during their interactions with Pepper. All data, including researchers’ notes, were analyzed and interpreted.
Description of Participants
Four participants were selected for inclusion in the study: Patient A was a female patient in her 70s who was diagnosed with schizophrenia; Patient B was a female patient in her 40s who was also diagnosed with schizophrenia. In addition, healthy person C was a woman in her 60s; and healthy person D was a woman in her 50s. All the participants were consented and signed the Informed Consent Form.
Patients’ information
Patient A was diagnosed with paranoid schizophrenia, parkinsonism, and an affective disorder. Her current symptoms included active grandiose delusions, paranoia, and a mild hand tremor. She has been admitted to the hospital for seven years, and she was independent in performing activities of daily living (ADL), such as having meals, bathing, and using the toilet. She was prescribed olanzapine 20 mg, lithium carbonate 200 mg, and biperiden hydrochloride 2 mg per day.
Patient B was diagnosed with schizophrenia and epilepsy. She had symptoms of autism spectrum or disorganized schizophrenia. She could not keep her personal belongings tidy, and when nurses tried to keep things tidy, she got angry. She has been admitted to the hospital for 14 years. Her ADL, including having meals, bathing, and using the toilet, is independent. Her prescribed medications were sodium valproate 800 mg, risperidone 3 mg, and biperiden hydrochloride 3 mg per day.
Setting and Data collection
The experiment was set in the situation of a conversation between Pepper and two participants at the same time. In this situation, researchers would be able to confirm whether Pepper could interact appropriately with more than one person. Meanwhile, a different room was set for the operator and assistant operator so that participants did not see the operators and were not distracted by other activities while conversing with Pepper. Data were collected in July and December 2020. The length of each conversation time with Pepper was about 20 to 30 minutes.
In the case where patients were involved, the setting of the experiment was a ward in a psychiatric hospital in Western Japan. In the ward, a room was set for conversation interaction, while the other part of the room was set where the operator and assistant could hear and see patients’ interaction and control the Pepper.
In the case of interaction with healthy persons, the setting was a university laboratory. Inside the laboratory, two rooms were prepared for the conversation room and the operators’ room. Since the setting of the environment did not enable the operator and assistant to see and hear participants clearly, monitors and the video conferencing software developed by Zoom Video Communications, Inc. were used in both rooms to display the live interaction so that the operator and assistant could see and hear the conversation clearly (see Figure 1 and Figure 2).
The interaction between Pepper and the two participants was recorded by two digital cameras, and three observers intentionally observed and noted significant events occurring during the conversation. Pepper was set in the middle facing the two participants. On Pepper’s chest, a tablet was attached by default for Pepper’s operating functions, but it did not show the contents of the conversation. The situation of Pepper interacting with participants is presented in Figure 1.
In the operator’s room, computer tablets and keyboards were prepared. They monitored Pepper and the conversation with the participants. In order for participant conversational activities to operate accurately and appropriately, the operator could synchronize technologies involved in the methodology (Figure 2). The operator role was to insert the text to be uttered by Pepper by typing in the keyboard. The operator’s assistant role was inputting Pepper’s nodding movement, conversation backchannel, acknowledgment gestures, and turning Pepper’s face to the front during the conversation. These were also inputted by typing in the keyboard or clicking the functions on the tablet.
Ethical Considerations
Approvals from the Ethics Committee of the Tokushima University Hospital (#3046) and the Mifune Hospital Clinical Research Ethics Review Committee (#201180502) were obtained. Information about the study was provided to all participants before data collection started. Informed consent was obtained from the guardians (for patients with Schizophrenia) and the participants (for healthy participants). In reporting this study, images of participants’ faces were blurred to prevent identification.
Results
Patients with Schizophrenia
Significant observations of the conversational situations between Pepper and patients with schizophrenia were identified and grouped into four categories. These observations include the conversation, Pepper’s gaze, Pepper’s response time, and patients’ impression of Pepper.
Conversation
The conversation started with Pepper greeting the two patients by saying “Hello,” and asking about several topics, such as favorite food, trouble in hospitalization, and favorite entertainers. The two patients answered Pepper’s questions appropriately and asked Pepper some questions. When Pepper said, “Tell me more,” Patient A told Pepper about food. Next, Pepper encouraged the participants to sing the songs of their favorite entertainers, and both patients A and B enjoyed singing the songs they knew during the conversation. After the patients sang, Pepper praised them and said, “Good . . . good.” After singing, Patient A asked Pepper to sing, but Pepper said it could not sing.
Pepper’s gaze
Eye gaze is defined as the pointer from the viewers’ eyes to an object (He et al., 2015). In this study, Pepper’s gaze refers to the direction point from Pepper’s eyes to participants. During the conversation, two patients maintained their gaze at the Pepper until the end of the conversation. However, Pepper’s gaze did not always meet patients’ gaze. Pepper appeared to be pointing in a nearly intermediate direction between the two participants.
Response time
It was observed that there was an inappropriate timing response from Pepper during the conversation. Once, Pepper asked a question to Patient A. When Patient A was still answering Pepper’s question, Pepper said other things.
Patients’ impression toward Pepper
After the conversation, both patients were interviewed regarding their impressions during the conversation with Pepper, and both patients replied that they enjoyed talking with Pepper. During the experiment, there was a scene where Pepper froze in the middle of a conversation and got stuck. At that time, Patient A said, “Because it is a robot, it must have a good brain. I thought Pepper was thinking about something.” “There was no problem with the conversation,” Patient A said. Patient B was delighted that she could talk to Pepper, and she said, “Pepper turned to me. The distance to Pepper was also good. I want to talk to Pepper again if I have the opportunity.”
Below is the figure depicting the conversation situation between Pepper and two patients with schizophrenia (Figure 3).
Healthy Participants
The following are the results of the conversational situation between Pepper and two healthy participants.
Conversation
At the beginning of the conversation, Pepper greeted the two participants, “I am Pepper. Thank you for coming today.” The conversation continued about family, food, work, daily life, and entertainers. Next, Pepper gave questions to both participants. After Participant C answered, Pepper then asked Participant D. When asking a question to Participant D, the operator could not manipulate Pepper’s face to turn to her.
The conversation functioned as a two-way conversation. After Pepper asked, both participants also asked some questions to Pepper, such as, “Pepper, do you have a friend?” “Pepper, how old are you?” Additionally, Participant C asked Pepper to sing twice. However, Pepper could not sing.
When both participants talked, Pepper nodded and said, “Oh yes…,” as if Pepper intelligibly followed the conversation. This gesture was noticed by both participants as they said, “It’s good that Pepper nodded when I talked.” Pepper also moved its hand while talking.
The conversation lasted around 30 minutes.
Response time
On some occasions, it was observed that Pepper delayed in responding to participants’ questions, making them seem to be confused about whether to wait for Pepper to respond or to go ahead and say something else. The other time, Pepper talked while the participant was still talking.
Pepper’s gaze
Pepper turned to Participant C and spoke during the conversation. Pepper sometimes uttered Participant D’s name but did not turn to her. There were also scenes when Pepper did not turn to Participant C nor Participant D but faced the middle of these two participants when it spoke to one of them. This averted gaze was noticed by both participants.
At the start of the talking between Pepper and Participant C, Pepper faced Participant C. Afterward, our operation switch just turned Pepper’s face to the front. However, Pepper could not meet the gaze of Participant D because Pepper only turned its face to Participant C and the middle of both participants.
Participants’ impression toward Pepper
After the conversation, both participants were interviewed regarding their impressions during conversing with Pepper. Both participants expressed amazement at how Pepper could engage in the conversation. Participant C said, “I could have a conversation without any discomfort. It was easy to talk. It felt like a natural conversation, Pepper’s eyes moved, and its eye color changed during talking.” She also added, “Pepper is good. It always saw me when it talked to me.”
However, Participant D seemed less satisfied and said, “I talked with Pepper, but Pepper did not look at me. It called my name, but it did not turn to me.” (See Figure 4). Participant D also said that Pepper asked a question and then another question without giving enough time for her to answer each question.
Discussion
This case study addressed the characteristics of communication between Pepper and patients with schizophrenia and with healthy persons, with results expressed as similarities and differences in communication.
Similarities were found in the enjoyment of users, both patients with schizophrenia and healthy persons, in their interaction with Pepper. In both cases, it was observed that both participants and Pepper had two-way conversations. After Pepper asked, the participants replied and asked Pepper some questions, which were answered by Pepper. In both cases, participants asked Pepper to sing. However, Pepper could not sing. This situation highlighted that those users of robots, either patients or healthy persons, expected Pepper to entertain during communication. While the current application could not control Pepper’s function to sing songs, this finding encouraged improvement for a future design that will allow the application to switch converse and sing songs.
During the conversation, both patients could answer Pepper’s questions appropriately, and although not as much, patients could ask and maintain the conversation with Pepper. Similar to the previous study (Ujike et al., 2019), communication between Pepper and patients with schizophrenia is regarded as intentional communication, in which communication was intended and initiated by the patients.
Differences in interaction are found in the impression of participants toward Pepper. In this study, Pepper was operated remotely by an operator and an assistant. While healthy persons were amazed and wondered how Pepper could maintain the conversation, patients with schizophrenia thought Pepper initiated the conversation of his own will. Therefore, when there was a silent moment when Pepper did not speak, patients thought that Pepper was thinking. This finding conforms to a previous study, which found that patients with schizophrenia had a distorted main perception in which they tended to perceive a humanoid robot to have mental capacities to plan and act on his will (Raffard et al., 2018).
Furthermore, it was observed that Pepper could maintain eye contact with one person while talking in both cases and faced the middle of participants. When the next speaker spoke, the operator’s assistant moved Pepper’s face to the front. That is, Pepper’s face could only be turned to the front but not facing the second speaker. Pepper tended to face the first speaker because this application did not allow Pepper to gaze at the second speaker. Although it did not affect patients with schizophrenia, it made sense of uncomfortable feeling for healthy persons. In the case of patients with schizophrenia, Pepper appeared to be looking in a nearly intermediate direction between both participants. Therefore, the patient evaluated that she could talk to Pepper even when their eyes were not aligned.
For patients with schizophrenia, their symptoms were characterized by not meeting their gaze or avoiding gaze (de la Asuncion et al., 2015), and abnormalities in the eye gaze perception in which patients misinterpret averted gaze or as being directed toward them and watching them (Hooker & Park, 2005; Seymour et al., 2016). Whether this abnormality is caused by social cognitive deficits or perceptual impairment is unknown; however, this gaze misinterpretation is implied as to the manifestation of the later conscious stage of gaze processing (Seymour et al., 2016).
In the experiment of two healthy persons, Pepper was observed mostly to face one direction (Participant C) and not the other. This was noticed by Participant D. Participant D seemed less satisfied, especially when she said, “Pepper called my name, but it did not turn his face to look at me.” For healthy people, the value of the gaze contains the message being sent. If a person averts the gaze, it will send the signal of not being willing to interact with another person (Cañigueral & Hamilton, 2019). Therefore, when Pepper did not direct the gaze to Participant D, Participant D perceived that Pepper was not interested in interacting with her.
In a group conversation, sometimes only a particular pair of participants talk to each other, and the other particular side participants lose the opportunities to join in the conversation (Kobayashi, 2016). This condition highlighted the need for improvement in the design of a robot that is expected to be able to gaze at another person when necessary and join in group conversations. Tanioka, Yokotani, et al. (2021) reported that inefficient gaze activity is one of the development issues in the current healthcare robot.
To participate in collaborations with people, robots must not only see and talk with people but also use the conventions of conversation to connect with their human counterparts (Sirithunge et al., 2021). A group conversation is a form of conversation in which three or more participants talk to each other about a topic on an equal footing. In a group conversation, the robot system should understand the conversational situation such as who is speaking, to whom he is speaking, also to whom the other participants pay attention (Matsusaka et al., 2003). A sensor network system embedded within the environment was proposed for these responsive behaviors to build a human-like interaction (Chikaraishi et al., 2008). With the sensor network system, an android was enabled to display human-like behavior, such as turning its head to a person when that person walks (Chikaraishi et al., 2008).
Examples of an experiment on the interaction between an older person and a robot using a telenoid have been reported (Ogawa et al., 2011). As a result, telenoid induced a positive attitude in the older person. Older people moved independently and actively talked, which also affected the behavior of the telenoid operator. Telenoid can nod and make simple gestures and can be held by the older person. Therefore, communication in contact with the robot is possible. When the older person holds a telenoid, the older person can adjust the gaze to the telenoid.
In the case of Pepper, users cannot hold it, so the robot needs to be able to move and direct the gaze to its users. If the gaze and movement of the robot do not match the dialogue partner and dialogue content, it was considered that the person would feel uncomfortable. In this respect, it is necessary to improve the control aspect to perform more complicated movements that change the line of sight without discomfort by synchronizing with the dialogue content and another person.
Alternatively, Pepper is capable of shaking hands and has a more human-like form than telenoids so that Pepper can communicate more closely with humans. With this, it will be possible to encourage the patients to take positive actions (speaks) by using various gestures (such as beckoning and pointing) that Pepper can realize in dialogue. Even if operators are prompted to sing a song, Pepper can supplement it with actions, such as clapping, nodding, etc. It has an advantage over non-human robots like telenoid. In the future, it will be necessary to consider whether the content of the conversation suits the situation and whether the atmosphere and impression during the dialogue are positive or negative. However, since there is little knowledge about the interaction between patients with schizophrenia and robots, it is necessary to accumulate cases in the future.
Furthermore, Pepper was set to interact with two participants in each case in this study. With this situation, researchers can confirm the case of the effectiveness of Pepper interacting with more than one participant, including whether Pepper could meet the gaze. Thus, these two cases could confirm communication characteristics between Pepper and humans, both patients and healthy persons.
The summary of characteristics of communication in this study is presented in Table 1.
| Characteristic of interactive communication between Pepper robot and patients with schizophrenia, and with healthy persons | |
|---|---|
| Similarities | |
| Enjoyed talking with Pepper | |
| Expected Pepper to entertain, such as singing | |
| Two ways conversations (Pepper asked questions, and subjects in both cases asked Pepper too) | |
| Differences | |
| Patients | Healthy persons |
| Not noticing Pepper’s inaccurate response time | Appreciating Pepper’s gestures correctly, e.g., appreciating Pepper’s nods when talking, and noticing Pepper’s averted gaze |
| Perceived Pepper as a full automaton that thinks and speaks on his own will | Noticing Pepper’s inaccurate response time Amazed and wondered how Pepper could maintain the conversation |
Implications for Nursing Practice and Nursing Research
With the increasing use of healthcare robots for patient care in the hospital and other healthcare institutions, the responsibility to operate healthcare robots during engagements with patients might require nursing expertise. As such, this situation becomes important for nurses to have competency with robotics and understand of communication patterns occurring between healthcare robots and patients, particularly those with mental illness and those not suffering from mental health conditions, as was represented by the healthy persons in this study. Furthermore, nurses are encouraged to see healthcare robots as significant partners in facilitating patient care activities; therefore, nurses need to be educated in quality practice through the integration of these technologies (Betriana et al., 2020). In addition, nurses are encouraged to be cognizant of multidisplicinary research, in which teams, especially engineers and robotics experts, are well informed about nursing practice so that the future designs of healthcare robots can be more in synchrony with the demands of patient care in specific nursing situations.
Limitations
This study presents two cases of interactive communication between Pepper and patients with schizophrenia and Pepper with healthy persons. Generalizing the findings of the study may not be appropriate because of the limited number of interactive engagements derived from the four participants. However, in this human-robot interaction as a case study, findings showed basic information, which can be useful for improving Pepper’s ability for effective use in situations involving different users in the future.
Conclusion
This study described two cases describing the characteristics of communication between patients with schizophrenia and healthy persons in interactive communication with Pepper. Findings showed that the similarity of both cases is participants were found to enjoy conversing with Pepper. However, differences are found in terms of participants’ impressions of Pepper. Patients with schizophrenia perceived Pepper as a full automaton that plans and acts based on its own will. Nevertheless, healthy persons were amazed by Pepper’s ability to engage in the conversation but noticed that Pepper’s responses to questions were often time constraints, and Pepper displayed inaccurate gazes. Findings clarified that Pepper was effective for communication treatment that provides enjoyment for users, including patients with mental health conditions and healthy persons. For the better use of robots in the future, gaze, appropriate response time during the conversation, and entertaining functions are suggested to be improved.