耳鼻喉科Pubmed文献追踪每日 14:00 同步
← 返回全部文献
Article record

基于指南的生成式人工智能中文回答的多维评估：一项梅尼埃病研究

Multidimensional Evaluation of Guideline-Based Generative Artificial Intelligence Responses in Traditional Chinese: A Ménière's Disease Study.

AI/ML耳科IF 3.9Q1

文献信息

PMID42942234
原文在 PubMed 查看原文 ↗
发表日期2026
作者Mien-Jen Lin
查看其他 3 位作者

Yun-Chiao Wen；Li-Chun Hsieh；Chin-Kuo Chen

作者单位Department of Medical Education, Chang Gung Memorial Hospital, Taoyuan 333423, Taiwan.
查看其他 6 条单位信息

Department of Otolaryngology-Head and Neck Surgery, Linkou Chang Gung Memorial Hospital, Taoyuan 333423, Taiwan.；Department of Otolaryngology-Head and Neck Surgery, Mackay Memorial Hospital, Taipei 104217, Taiwan.；Department of Audiology and Speech Language Pathology, Mackay Medical University, New Taipei 252005, Taiwan.；Department of Medicine, Mackay Medical University, New Taipei 252005, Taiwan.；Department of Otolaryngology-Head and Neck Surgery, Keelung Chang Gung Memorial Hospital, Keelung 204201, Taiwan.；School of Medicine, College of Medicine, Chang Gung University, Taoyuan 333323, Taiwan.

期刊Life (Basel, Switzerland)
SCI 分区Q1
IF3.9
研究类型AI/ML · 临床
所属专科耳科

中文摘要

背景： 生成式人工智能在中文医学信息方面的可靠性仍不确定。本研究评估三种生成式人工智能系统在生成梅尼埃病中文信息时与临床实践指南的一致性及语言表现。
方法： 根据美国耳鼻咽喉头颈外科学会指南的关键行动声明改编了17个问题，向ChatGPT o4-mini-high、Gemini 2.5 Pro和Grok 3提问（共51份回答）。采用配对分析（Cochran's Q检验和Friedman检验）评估回答的指南一致性、沟通特征和可读性，并对九项沟通特征进行Holm-Bonferroni校正。
结果： 三个模型的正确率无显著差异（ChatGPT o4-mini-high：100%，Gemini 2.5 Pro：100%，Grok 3：94.1%；Q = 2.00，p = 0.368）。九项沟通特征中有六项存在显著差异，效应量为中等至较大（Kendall's W = 0.26-0.76），包括指南引用、引文质量、要点强调及指南之外的建议。难词比例也存在显著差异（p = 0.0033）。Gemini 2.5 Pro的难词比例低于ChatGPT o4-mini-high（校正p = 0.040）和Grok 3（校正p = 0.0002）。
结论： 高指南一致性并不一定意味着引文可靠、沟通有效或语言通俗易懂。这些发现反映的是单次提问条件下生成的回答，而非模型稳定一致的表现，凸显了临床使用中专家监督的必要性。

英文摘要

BACKGROUND: The reliability of generative artificial intelligence for Chinese medical information remains uncertain. This study evaluates the concordance and linguistic performance of three generative artificial intelligence systems in generating Chinese information on Ménière's disease against clinical practice guidelines.
METHODS: Seventeen questions, adapted from the Key Action Statements of the American Academy of Otolaryngology-Head and Neck Surgery guidelines, were posed to ChatGPT o4-mini-high, Gemini 2.5 Pro, and Grok 3 (51 total responses). Responses were assessed for guideline concordance, communication features, and readability with matched analyses (Cochran's Q and Friedman tests), with Holm-Bonferroni correction across nine communication characteristics.
RESULTS: Correctness rates did not differ significantly among the three models (ChatGPT o4-mini-high: 100%, Gemini 2.5 Pro: 100%, Grok 3: 94.1%; Q = 2.00, p = 0.368). Six of the nine communication characteristics differed significantly, with moderate to large effect sizes (Kendall's W = 0.26-0.76), including guideline quotation, citation quality, key point emphasis and recommendations beyond the guideline. The proportion of difficult words also differed significantly (p = 0.0033). Gemini 2.5 Pro had a lower proportion of difficult words than ChatGPT o4-mini-high (adjusted p = 0.040) and Grok 3 (adjusted p = 0.0002).
CONCLUSION: High guideline concordance does not necessarily indicate reliable citations, effective communication, or accessible language. These findings reflect responses generated under single-query conditions rather than consistent model performance, highlighting the need for expert oversight in clinical use.