人工智能聊天机器人对HPV感染患者教育的准确性

文献信息

中文摘要

目的： 聊天机器人是使用大型语言模型回答问题并模拟类人对话的人工智能（AI）工具。患者常使用聊天机器人进行医学问题的自我教育。通过聊天机器人进行患者自我教育搜索所获得的关于人乳头瘤病毒（HPV）感染和复发性呼吸道乳头状瘤病（RRP）信息的准确性尚不清楚。本研究旨在分析聊天机器人对关于喉部HPV感染和RRP的常见患者问题的回答的各个方面。
方法： 作者向三种不同的聊天机器人模型（Claude、ChatGPT和Gemini）提出了八个问题，以模拟真实世界的患者查询。两名接受过专科培训的喉科医生使用5点李克特量表对每个模型的八个回答进行盲法评分，评分维度包括准确性、全面性以及与评分医生本人会给出的回答的相似性。进行了描述性和统计分析。
结果： Gemini的总体平均分最高（4.7，SD 0.5），Claude最低（4，SD 0.7）。Gemini在每个评估方面的单项得分最高。在所有平台分析的各维度中，准确性得分最高（4.4，SD 0.6），但差异并不稳健。单因素方差分析显示三个平台之间存在总体显著差异（P < 0.0001）。使用事后Tukey检验进行平台间的配对比较，结果仅在Gemini分别与ChatGPT和Claude比较时具有显著性（P < 0.0001）。
结论： Gemini在回答准确性、全面性以及与医生评分者会提供给患者的回答的相似性方面，获得了最高的总体和单项评分。在关于喉部HPV感染/RRP的常见患者问题的回答方面，它是唯一一个与其他平台相比表现出稳健差异的聊天机器人平台。

英文摘要

OBJECTIVES: Chatbots are artificial intelligence (AI) tools that use large language models to answer questions and simulate human-like conversations. Chatbots are commonly used by patients for self-education regarding medical issues. The accuracy of information obtained through patient self-education searches using Chatbots for Human Papillomavirus (HPV) infection and recurrent respiratory papillomatosis (RRP) is unknown. This work is intended to analyze aspects of chatbot answers to common patient questions regarding laryngeal HPV infection and RRP.
METHODS: Eight questions were asked by the authors to mimic real-world patient queries to three different Chatbot models (Claude, ChatGPT, and Gemini). Two fellowship-trained laryngologists blindly rated eight answers for each model using a 5-point Likert scale regarding accuracy, comprehensiveness, and similarity to a response the rating physician would themselves offer. Descriptive and statistical analyses were performed.
RESULTS: Gemini had the highest overall mean score (4.7, SD 0.5), and Claude had the lowest (4, SD 0.7). Gemini had the highest individual scores for each aspect evaluated. Between the aspects analyzed across all platforms, accuracy was scored highest (4.4, SD 0.6), but the difference was not robust. The one-way ANOVA test showed an overall significant difference between the three platforms (P < 0.0001). A paired comparison between platforms was performed using a post hoc Tukey test, and the results were only significant when Gemini was compared individually with ChatGPT and Claude (P < 0.0001).
CONCLUSION: Gemini had the highest overall and individual rated scores for answer accuracy, comprehensiveness, and similarity to what physician raters would offer their patients. It was the only chatbot platform that demonstrated a robust difference when compared to the others regarding answers to common patient questions on laryngeal HPV infection/RRP.