As part of the EU Erasmus+ Careerbot project we have undertaken a short literature review of chatbots and education. The literature review was intended primarily as a means of the project partners from Spain, Austria, Greece and Ireland gaining an understanding of the development, technology, pedagogy and use of Chatbots. For this purpose, all partners were involved in identifying relevant literature sources and adding them to an online template which as well as bibliographic details included a summary of the literature source. Obviously search strings varied in different countries and languages but were based around chatbots, education, employment, careers, guidance.
It should be stressed our aim was not to undertake a systematic literature review, which would be beyond the resources and time frame of the project, but to contribute to a shared understanding of the potential of chatbots in education, the technologies behind chatbots and the design and development of chatbots.
If anyone can point me to more references which I should include I would be extremely grateful.
Definition of Chatbots
A Chatbot can be described as a digital system that can be interacted with entirely through natural language via text or voice interfaces. Chatbots are intended to automate conversations by simulating a human conversation partner and can be integrated into software, such as online platforms, digital assistants, or be interfaced through messaging services.
There has been a growing interest in using chatbots for career advice, guidance, and counseling in recent years. This is because chatbots can provide immediate and personalized information to users, 24 hours a day. They can also be accessed easily and conveniently, making them a useful tool for individuals who may not have access to other forms of career advice, guidance, and counseling.
In “A Systematic Literature Review on Chatbots in Education Are We There Yet?” Wollny, Schneider, Di Mitri, Weidlich, Rittberger, and Drachsler (2021) examined different frameworks for classifying chatbots. One classification framework is defined through “flow chatbots”, “artificially intelligent chatbots,” “chatbots with integrated speech recognition”, as well as “chatbots with integrated context-data” (Winkler and Soellner, 2018). They say that “by specifying text interfaces as “Button-Based” or “Keyword Recognition-Based” (Smutny and Schreiberova, 2020), text interfaces can be subdivided.” “Text interfaces have advantages for conveying information, and speech interfaces have advantages for affective support.” They also draw attention to the importance of the personality of chatbots deriving “four guidelines helpful in education: positive or neutral emotional expressions, a limited amount of animated or visual graphics, a well-considered gender of the chatbot, and human-like interactions.”
In their conclusions they consider the position of Chatbots on the Gartner hype cycle, suggesting technology around chatbots in education may currently be in the “Innovation Trigger” phase. “This phase is where many expectations are placed on the technology, but the practical in-depth experience is still largely lacking.”
The use of chatbots outside education
Outside of education, typical applications of chatbots are in customer service (Xu et al., 2017), counseling of hospital patients (Vaidyam et al., 2019), or information services in smart speakers (Ram et al., 2018). Outside of education, typical applications of chatbots are in customer service (Xu et al., 2017), counseling of hospital patients (Vaidyam et al., 2019), or information services in smart speakers (Ram et al., 2018).
Mehault (2021) outlines the advantages of the use of Chatbots from an employer point of view.
In an article asserting that the significance of human resource (HR) management lies in its being able to manage human resources for maximum efficiency and being able to plan the operations of the organization and lead it to success. Jitgosol, Y., Kasemvilas, S., Boonchai, P. (2019) present the design of an HR chatbot that could. They say, improve human resource management, particularly welfare concerns. In the case study, a HR chatbot application designed to answer questions and provide advice on employee welfare helping reduce costs within the organization.
Chatbots in education
Wollny, Schneider et al (2021) say “one central element of chatbots is the intent classification, also named the Natural Language Understanding (NLU) component, which is responsible for the sense-making of human input data. Looking at the current advances in chatbot software development, it seems that this technology’s goal is to pass the Turing Test (Saygin et al., 2000) one day, which could make chatbots effective educational tools. Therefore, we ask ourselves “Are we there yet? - Will we soon have an autonomous chatbot for every learner?” The December 2022 release of the ChatGPT chatbot application by Open AI has excited much comment and experimentation by educational researchers and developers in this regard. However there remain major concerns: “the creators of such models confess to the difficulty of addressing inappropriate responses that 'do not accurately reflect the contents of authoritative external sources'" (Birhane and Raji, 2022).
Chatbots incorporate generic language models extracted from large parts of the Internet and enable feedback by limiting themselves to text or voice interfaces. For this reason, they have also been proposed and researched for a variety of applications in education (Winkler and Soellner, 2018). Recent literature reviews on chatbots in education (Winkler and Soellner, 2018; Hobert, 2019a; Hobert and Meyer von Wolff, 2019; Jung et al., 2020; Pérez et al., 2020; Smutny and Schreiberova, 2020; Pérez-Marín, 2021) have reported on such applications as well as design guidelines, evaluation possibilities, and effects of chatbots in education.
Researchers have pointed to three different pedagogical roles for chatbots in education: a supporting learning role, an assisting role, and a mentoring role (Wollny, Schneider et al, 2021).
Pedro Antonio Tamayo , Ana Herrero , Javier Martín ,Carolina Navarro & José Manuel Tránchez (2020) report that “within the process of progressive digitization of materials and tools for teaching and distance learning of a subject of introduction to Microeconomics (quarterly, in year three of the Degree in Social Work), taught by the authors at the National University of Distance Education (UNED), a virtual assistant in the form of chatbot, or conversational robot, called EconBot, has been designed and made available to students from 2017.” Their paper “presents the reasons that led to its adoption, the process of its development, differentiating two phases, its characteristics and functions, the assessment of its usefulness and the role of teachers in the implementation of this type of technological innovation.”
Kumar (2021) discusses educational chatbots (ECs) designed for pedagogical purposes. “These chatbots are strategized to provide personalized learning through the concept of a virtual assistant that replicates humanized conversation. Nevertheless, in the education paradigm, ECs are still novel with challenges in facilitating, deploying, designing, and integrating it as an effective pedagogical tool across multiple fields, and one such area is project-based learning.” The chatbots were found to improve learning performance and teamwork and to facilitate collaboration among team members. “Nevertheless, affective-motivational learning outcomes such as perception of learning, need for cognition, motivation, and creative self-efficacy were not influenced by ECs.”.
Wollny, Schneider et al (2021) concluded their review of 74 relevant publications for chatbots’ application in education by pointing to three main research challenges: “1) Aligning chatbot evaluations with implementation objectives, 2) Exploring the potential of chatbots for mentoring students, and 3) Exploring and leveraging adaptation capabilities of chatbots.”
Smutny & Schreiberova (2020) evaluated 47 educational chatbots “using the Facebook Messenger platform based on the analytic hierarchy process against the quality attributes of teaching, humanity, affect, and accessibility.” They found that educational chatbots on the Facebook Messenger platform “vary from the basic level of sending personalized messages to recommending learning content. Results show that chatbots which are part of the instant messaging application are still in its early stages to become artificial intelligence teaching assistants.”
Tom (2021) reports on the development of a digital assistant by the UK Open University and called Taylor that is designed to have a dialogue with students who disclose a disability. “A conversation with Taylor has two purposes. The first is for the student to provide information, covering things such as the nature of their disabilities, any assistive technologies they use, and areas where they could require support or adjustments in study. Taylor is an alternative to the usual process of filling in forms to provide this information.
The second purpose is to help each student to better understand what Open Univeristy study entails and the support that they could benefit from. Students are given some introductory information on key topics, and they can also ask Taylor questions. This is designed to be part of the conversation throughout, with the idea that students can learn from the conversation and give better answers to the questions they are asked.”
Bolton College in the UK uses a chatbot called Ada to answer both staff and students’ questions about college life (Jisc, 2022). The chatbot is integrated with other platforms used by Bolton College, such as its information management system, which means that the chatbot can provide answers that are specific to any given user. A student, for instance, can get information about their timetable. Meanwhile, a teacher can get information about attendance figures in one of their classes.
Chatbots for Careers Guidance, Counseling and Advice
There is very limited literature on the use of chatbots for Careers Guidance, Counselling and Advice, probably reflecting the limited development of this use case to date.
Zahour et.al (2020) say “this branch of research is just emerging in the scientific community, therefore, in our article, we set up a chatbot in the field of educational and professional guidance which is based on the theory of John Holland and the RIASEC questionnaire in order to determine the dominant type of personality of undergraduate and graduate students that wants to enter the job market.:
Attwell, G. Hughes, D, Bekiaridis G, Percy, C. (2021) report on the development of a careers chatbot (CiCi) in the that works alongside careers and employability professionals in the UK and its potential for new forms of blended careers support.
Another study examined the use of chatbots for providing career advice, guidance, and counseling to individuals with disabilities. The study found that chatbots can be an effective tool for providing personalized information and support to individuals with disabilities, who may face barriers to accessing other forms of career advice, guidance, and counseling. The study also found that chatbots can help to reduce stigma and discrimination against individuals with disabilities, by providing a safe and confidential space for them to explore their career options.
Overall, the research suggests that chatbots can be a useful tool for providing career advice, guidance, and counseling, particularly to young people and individuals with disabilities. However, it is important to note that chatbots are not a replacement for human career advisers and should be used as part of a broader career advice, guidance, and counseling strategy.
References
Attwell, G. Hughes, D, Bekiaridis G, Percy, C. (2021) CiCi: The little powerhouse that supports your career, Careers Development Institute
Birhane, A., and Raji, D., (2022) ChatGPT, Galactica, and the Progress Trap. Wired, https://www.wired.com/story/large-language-models-critique/
Hobert, S. (2019a). “How Are You, Chatbot? Evaluating Chatbots in Educational Settings - Results of a Literature Review,” in 17. Fachtagung Bildungstechnologien, DELFI 2019 - 17th Conference on Education Technologies, DELFI 2019, Berlin, Germany, Sept 16–19, 2019. Editors N. Pinkwart, and J. Konert, 259–270. doi:10.18420/delfi2019_289
Hobert, S., and Meyer von Wolff, R. (2019). “Say Hello to Your New Automated Tutor - A Structured Literature Review on Pedagogical Conversational Agents,” in 14th International Conference on Wirtschaftsinformatik, Siegen, Germany, Feb 23–27, 2019. Editors V. Pipek, and T. Ludwig, (AIS).
Jitgosol, Y., Kasemvilas, S., Boonchai, P.(2019) Designing an HR chatbot to support human resource management, pp. 165, 170, December 2019 https://www.jobology.fr/blog/chatbot-automatisation-processus-recrutement/
Jung, H., Lee, J., and Park, C. (2020). Deriving Design Principles for Educational Chatbots from Empirical Studies on Human-Chatbot Interaction. J. Digit. Contents Society, 21, 487–493. doi:10.9728/dcs.2020.21.3.487
Kumar, J.A. Educational chatbots for project-based learning: investigating learning outcomes for a team-based design course. Int J Educ Technol High Educ 18, 65 (2021).
Mehault, M. (2021) Outils de recrutement, Automatisation des processus de recrutement: le chatbot, https://www.jobology.fr/blog/chatbot-automatisation-processus-recrutement/
Pérez-Marín, D. (2021). A Review of the Practical Applications of Pedagogic Conversational Agents to Be Used in School and University Classrooms. Digital 1 (1), 18–33. doi:10.3390/digital1010002
Pérez, J. Q., Daradoumis, T., and Puig, J. M. M. (2020). Rediscovering the Use of Chatbots in Education: A Systematic Literature Review. Comput. Appl. Eng. Educ. 28, 1549–1565. doi:10.1002/cae.22326
Ram, A., Prasad, R., Khatri, C., Venkatesh, A., Gabriel, R., Liu, Q, et al. (2018). Conversational Ai: The Science behind the Alexa Prize, in 1st Proceedings of Alexa Prize (Alexa Prize 2017). ArXiv [Preprint]. Available at: https://arxiv.org/abs/1801.03604.
Saygin, A. P., Cicekli, I., and Akman, V. (2000). Turing Test: 50 Years Later. Minds and Machines 10 (4), 463–518. doi:10.1023/A:1011288000451
Schroeder, R. (2022) Artificial Intelligence to Assist, Tutor, Teach and Assess in Higher Ed, in Inside Higher Education, 5 January 2022
Smutny P. & Schreiberova P. (2020) Chatbots for learning: A review of educational chatbots for the Facebook Messenger, Computers & Education (2020), doi: https://doi.org/10.1016/ j.compedu.2020.103862
Smutny, P., and Schreiberova, P. (2020). Chatbots for Learning: A Review of Educational Chatbots for the Facebook Messenger. Comput. Edu. 151, 103862. doi:10.1016/j.compedu.2020.103862
Tamayo, P.a., Herrero, A., Martín, J. , Navarro, C., & Tránchez, J.M. (2020) Design of a chatbot as a distance learning assistant, Universidad Nacional de Educación a Distancia (UNED) (Spain) https://search.informit.org/doi/pdf/10.3316/informit.219384622220499
Tom (2021) How digital assistants are promoting enhanced accessibility at the Open University, Jisc National Centre for AI, https://nationalcentreforai.jiscinvolve.org/wp/2021/09/27/how-digital-assistants-are-promoting-enhanced-accessibility-at-the-open-university/
Vaidyam, A. N., Wisniewski, H., Halamka, J. D., Kashavan, M. S., and Torous, J. B. (2019). Chatbots and Conversational Agents in Mental Health: A Review of the Psychiatric Landscape. Can. J. Psychiatry 64 (7), 456–464. doi:10.1177/0706743719828977
Winkler, R., and Soellner, M. (2018). Unleashing the Potential of Chatbots in Education: A State-Of-The-Art Analysis. in Academy of Management Annual Meeting Proceedings 2018 2018 (1), 15903. doi:10.5465/AMBPP.2018.15903abstract
Wollny S, Schneider J, Di Mitri D, Weidlich J, Rittberger M and Drachsler H (2021) Are We There Yet? - A Systematic Literature Review on Chatbots in Education. Front. Artif. Intell. 4:654924. doi: 10.3389/frai.2021.654924
Xu, A., Liu, Z., Guo, Y., Sinha, V., and Akkiraju, R. (2017). “A New Chatbot for Customer Service on Social media,” in Proceedings of the 2017 CHI conference on human factors in computing systems, Denver, Colorado, USA, May 6–11, 2017, ACM, 3506–3510. doi:10.1145/3025453.3025496
Zahour, O., Benlahmar, E.H., Eddaoui, A. et.al. (2020) A system for educational and vocational guidance in Morocco: Chatbot E-Orientation. Procedia Computer Science 175 (2020) 554-559
