MediEval: A Unified Medical Benchmark for Patient-Contextual and Knowledge-Grounded Reasoning in LLMs | AGIDaily
AGI Daily
资讯日报词条论文人物新创
返回论文列表

暂无章节

返回论文列表
目录

MediEval: A Unified Medical Benchmark for Patient-Contextual and Knowledge-Grounded Reasoning in LLMs

收录于 ACL 2026 Outstanding Paper;Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers);DOI 10.18653/v1/2026.acl-long.734被引 6 次
打开论文来源

MediEval: A Unified Medical Benchmark for Patient-Contextual and Knowledge-Grounded Reasoning in LLMs

MediEval:面向大语言模型的统一医学基准,用于患者语境与知识锚定推理

Zhan Qu and Michael Färber

TU Dresden and ScaDS.AI, Germany
{zhan.qu, michael.faerber}@tu-dresden.de

Abstract

摘要

Large Language Models (LLMs) are increasingly applied to medicine, yet their adoption is limited by concerns over reliability and safety. Existing evaluations either test factual medical knowledge in isolation or assess patient-level reasoning without verifying correctness, leaving a critical gap. We introduce MediEval, a benchmark that links MIMIC-IV electronic health records (EHRs) to a unified knowledge base built from UMLS and other biomedical vocabularies. MediEval generates diverse factual and counterfactual medical statements within real patient contexts, enabling systematic evaluation across a 4-quadrant framework that jointly considers knowledge grounding and contextual consistency. Using this framework, we identify critical failure modes, including hallucinated support and truth inversion, that current proprietary, open-source, and domain-specific LLMs frequently exhibit. To address these risks, we propose Counterfactual Risk-Aware Fine-tuning (CoRFu), a DPO-based method with an asymmetric penalty targeting unsafe confusions. CoRFu improves by +16.4 macro-F1 points over the base model and eliminates truth inversion errors, demonstrating both higher accuracy and substantially greater safety.

大语言模型(LLM)正越来越多地被用到医学中,但可靠性和安全性方面的担忧限制了它的落地。现有评估要么单独考查事实性医学知识,要么评估患者层面的推理却不核实对错,中间留下了一块关键空白。我们提出 MediEval,这一基准把 MIMIC-IV 电子病历(EHR)与由 UMLS 及其他生物医学词表构建的统一知识库连在一起。MediEval 在真实患者语境中生成多样的事实性与反事实医学陈述,从而可以在同时考虑知识锚定和语境一致性的四象限框架下做系统评估。借助这一框架,我们识别出当前专有、开源和领域专用大语言模型经常出现的关键失效模式,包括幻觉式依据和真相倒置。为应对这些风险,我们提出基于 DPO、带有非对称惩罚、专门针对不安全混淆的反事实风险感知微调(CoRFu)。相对基座模型,CoRFu 的宏平均 F1 提高了 16.4 个百分点,并消除了真相倒置错误,说明它既提高了准确率,也显著增强了安全性。

1 Introduction

1 引言

Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse domains, with medicine being among the most high-impact areas of application. In clinical contexts, LLMs have been explored for tasks such as summarizing electronic health records (EHRs), generating discharge instructions, providing clinical decision support, and answering medical questions (Rajpurkar et al., 2022; Singhal et al., 2025; Moor et al., 2023; Singhal et al., 2023; Pal et al., 2022). Their appeal lies in the ability to integrate unstructured text with medical knowledge, potentially reducing documentation burden and assisting clinicians in decision-making.

大语言模型(LLM)已在许多领域展现出很强的能力,医学是其中影响最大的应用方向之一。在临床场景里,人们尝试用大语言模型做电子病历(EHR)摘要、生成出院指导、提供临床决策支持和回答医学问题(Rajpurkar et al., 2022; Singhal et al., 2025; Moor et al., 2023; Singhal et al., 2023; Pal et al., 2022)。它的吸引力在于能够把非结构化文本和医学知识结合起来,有望减轻文书负担,并协助临床医生做决策。

Translating research prototypes into real-world deployment hinges critically on reliability and safety. Unlike generic NLP tasks, medical reasoning requires not only factual correctness but also contextual grounding in patient-specific data while adhering to verified medical knowledge. Errors in this setting are not mere degradations in performance but risks that can directly translate into patient harm (Thirunavukarasu et al., 2023; Yang et al., 2023; Haltaufderheide and Ranisch, 2024).

要把研究原型变成可实际部署的系统,关键取决于可靠性和安全性。与一般的自然语言处理任务不同,医学推理不仅要求事实正确,还要把结论锚定在患者的具体数据上,并遵守已经核实的医学知识。在这种设定里,错误不只是表现变差,而可能直接转化为对患者的伤害(Thirunavukarasu et al., 2023; Yang et al., 2023; Haltaufderheide and Ranisch, 2024)。

A critical challenge is that LLMs often fail to apply medical knowledge consistently within the heterogeneous and noisy context of patient records (Zhou et al., 2025b). For example, a model may state that metformin, a therapy for type 2 diabetes, is contraindicated in severe renal impairment, yet fail to apply this knowledge when the condition appears in a noisy and heterogeneous patient record. The cause of such errors may be that current models are trained to recall facts in isolation rather than to integrate them with diverse patient information. Such inconsistencies expose a critical gap between knowing medical facts and using them safely.

一个关键困难是,大语言模型往往无法在异构、嘈杂的患者记录中一致地运用医学知识(Zhou et al., 2025b)。例如,模型可以正确说出二甲双胍作为 2 型糖尿病疗法在严重肾功能损害时禁用,可一旦这一病情出现在嘈杂、异构的患者记录里,它又未必用得上这条知识。这类错误的原因,可能是现有模型被训练成孤立地回忆事实,而不是把事实与多样的患者信息结合起来。这样的不一致暴露了“知道”医学事实和“安全地使用”这些事实之间的关键差距。

Figure 1: For each admission, MediEval uses both unstructured discharge summaries and structured EHR tables (diagnoses, procedures, and medications). First, clinically relevant sec

Figure 1: For each admission, MediEval uses both unstructured discharge summaries and structured EHR tables (diagnoses, procedures, and medications). First, clinically relevant sections are extracted from the discharge summary to form the patient context. In parallel, structured medical codes are semantically normalized into UMLS Concept Unique Identifiers (CUIs), which are then linked in an ontology graph to identify clinically meaningful relations, including multi-hop connections when needed. Based on these relations, MediEval constructs candidate statements and evaluates them under a four-quadrant framework defined by factual correctness with respect to biomedical knowledge (True vs. False) and contextual grounding with respect to the extracted patient context (Supported vs. Not Supported): Q1 = True–Supported, Q2 = True–Unsupported, Q3 = False–Supported, and Q4 = False–Unsupported. The resulting benchmark is used to evaluate LLMs and to support Counterfactual Risk-Aware Fine-tuning (CoRFu). Blue text marks the entities or statements extracted from the real patient example.

图 1:对每一次入院,MediEval 同时使用非结构化的出院小结和结构化的电子病历表(诊断、操作与用药)。首先,从出院小结中抽出临床相关章节,形成患者上下文。与此同时,结构化医学编码被语义归一为 UMLS 概念唯一标识符(CUI),再在本体图中相连,以识别有临床意义的关系,必要时也包括多跳连接。基于这些关系,MediEval 构造候选陈述,并在一个由相对于生物医学知识的事实正确性(真 / 假)和相对于抽出的患者上下文的语境依据(有依据 / 无依据)共同定义的四象限框架下评估它们:Q1 = 真–有依据,Q2 = 真–无依据,Q3 = 假–有依据,Q4 = 假–无依据。得到的基准用于评估大语言模型,并支持反事实风险感知微调(CoRFu)。蓝色文本标出从真实患者示例中抽出的实体或陈述。

Existing evaluation paradigms only partially address this gap. Medical data are uniquely challenging because patient records are heterogeneous with free-text clinical notes and tabular entries coded in different systems for diagnoses, procedures, and medications. Medical knowledge is hierarchical and ontology-driven (e.g., UMLS, SNOMED CT, RxNorm), but its large scale, noise, and limited cross-vocabulary connectivity make consistent reasoning difficult. Benchmarks based on EHRs assess the ability to extract or reason over structured data, but often reduce the task to retrieval or serialization without verifying medical soundness (Lovón-Melgarejo et al., 2025). In contrast, knowledge-based evaluations test whether LLMs can handle logical transformations of medical facts (Zhou et al., 2025a; Sung et al., 2021), but do not connect reasoning to real patient contexts. The field thus lacks a unified framework that probes whether LLMs can (i) remain faithful to medical knowledge and (ii) apply it consistently to individual patient records.

现有评估范式只部分填补了这一空白。医学数据的独特困难在于:患者记录本身就是异构的,既有自由文本的临床笔记,也有按不同系统编码的诊断、操作和用药表格。医学知识是层次化、由本体驱动的(例如 UMLS、SNOMED CT、RxNorm),但其规模大、噪声多,跨词表连通又有限,要做一致推理并不容易。基于电子病历的基准会考查从结构化数据中抽取或推理的能力,却常常把任务简化成检索或序列化,而不去核实医学上是否站得住(Lovón-Melgarejo et al., 2025)。相反,基于知识的评估考查大语言模型能否处理医学事实的逻辑变换(Zhou et al., 2025a; Sung et al., 2021),却不把推理接到真实患者语境上。因此,这一领域仍缺少一个统一框架,去探测大语言模型能否(i)忠实于医学知识,并且(ii)把知识一致地用到每一份患者记录上。

In this paper, we address this gap with MediEval (Figure 1), a benchmark and evaluation framework that links real patient records (MIMIC-IV) (Johnson et al., 2023) with a unified biomedical knowledge base built from UMLS (Bodenreider, 2004), SNOMED CT (Donnelly and others, 2006), and RxNorm (Liu et al., 2005; Nelson et al., 2011). To ensure rigorous construction, MediEval constructs evaluation statements by applying graph-guided substitutions and recombinations within biomedical ontologies, with plausibility checks to ensure that true cases remain clinically valid while false ones are realistic and challenging. These statements are embedded in real patient contexts, enabling evaluation of quadrant-level reasoning, defined as True/False with respect to medical knowledge and Supported/Unsupported with respect to the patient record. To capture safety-critical errors, we introduce two new metrics: Hallucinated Support Rate (HSR) and Truth Inversion Rate (TIR). We then conduct a comprehensive evaluation of proprietary, open-source (general-purpose), and biomedical LLMs under this unified protocol, revealing consistent weaknesses across quadrants.

本文用 MediEval(图 1)来填补这一空白。它是一个基准和评估框架,把真实患者记录(MIMIC-IV)(Johnson et al., 2023)与由 UMLS(Bodenreider, 2004)、SNOMED CT(Donnelly and others, 2006)和 RxNorm(Liu et al., 2005; Nelson et al., 2011)构建的统一生物医学知识库连在一起。为了保证构造严谨,MediEval 在生物医学本体内做图引导的替换和重组来生成评估陈述,并用合理性检查确保真陈述在临床上仍然成立,假陈述则既现实又有难度。这些陈述被嵌入真实患者语境,从而可以评估象限层面的推理,即相对于医学知识的真 / 假,以及相对于患者记录的有依据 / 无依据。为捕捉安全关键错误,我们引入两项新指标:幻觉式依据率(HSR)和真相倒置率(TIR)。我们随后在这一统一协议下,对专有、开源(通用)和生物医学大语言模型做了全面评估,并揭示了各象限中反复出现的薄弱环节。

To mitigate the identified risks, we introduce Counterfactual Risk-Aware Fine-tuning (CoRFu), a DPO-based method that uses quadrant-structured preference pairs and an asymmetric penalty to push models to prefer “True+Supported” responses over safety-critical confusions (e.g., “False+Supported”). This risk-aware, counterfactual approach targets precisely the failure modes that conventional preference optimization overlooks. Our code is publicly available[^1]. Our key contributions are:

为缓解已经识别出的风险,我们提出 Counterfactual Risk-Aware Fine-tuning(CoRFu)。这是一种基于 DPO 的方法,使用按象限组织的偏好对,并以非对称惩罚推动模型更偏好“真 + 有依据”的回答,而不是安全关键的混淆(例如“假 + 有依据”)。这种带风险意识的反事实做法,正好针对常规偏好优化容易忽略的失效模式。我们的代码已公开提供[^1]。主要贡献如下:

1 https://github.com/ZhanQu945/MediEval

  • MediEval Benchmark. A unified benchmark linking EHRs with biomedical ontologies to test both factual grounding and patient consistency, with novel safety evaluation metrics.
  • MediEval 基准。 把电子病历与生物医学本体相连的统一基准,用于同时检验事实锚定和患者一致性,并配有新的安全性评估指标。
  • Comprehensive Evaluation. Cross-model evaluation of proprietary, open-source, and biomedical LLMs under identical settings, revealing systematic quadrant-level weaknesses and safety-critical error patterns.
  • 全面评估。 在相同设定下对专有、开源和生物医学大语言模型做跨模型评估,揭示系统的象限级薄弱环节和安全关键错误模式。
  • Counterfactual Risk-Aware Fine-Tuning (CoRFu). A risk-aware extension of DPO with asymmetric penalties and a quadrant-aware approach that reduces safety-critical errors in medical reasoning tasks.
  • 反事实风险感知微调(CoRFu)。 在 DPO 上加入非对称惩罚和象限感知策略的风险感知扩展,用于降低医学推理中的安全关键错误。

2 Related Work

2 相关工作

A growing body of work investigates whether LLMs can handle complex information in EHRs. One study focused on data serialization and retrieval, with performance sensitive to prompt design and feature selection (Lovón-Melgarejo et al., 2025). Another work has explored long-context modeling, showing some architectures can process entire patient timelines with >>>10k events and achieve greater robustness to irregular temporal patterns (Wornow et al., 2025). Additional approaches integrate structured and unstructured data, for instance, by using small auxiliary models as “knowledge triggers” to support tabular prediction (Yan et al., 2025) or by designing code-aware representations that capture semantic and temporal structure in diagnosis prediction (Tan et al., 2025). More recent work has explored longitudinal clinical reasoning through structure-aware retrieval and geometric representations of patient trajectories, enabling more clinically consistent next-event forecasting (Qu and Färber, 2026a). Related efforts model evolving patient care as a continuous temporal reasoning problem, using structured memory and agentic context management to support long-horizon inference over streaming EHRs (Qu and Färber, 2026b). These studies demonstrate progress in surfacing and encoding patient information, but whether model outputs are consistent with established clinical knowledge remains underexplored.

越来越多的工作在研究大语言模型能否处理电子病历中的复杂信息。一项研究关注数据序列化和检索,表现对提示设计和特征选择很敏感(Lovón-Melgarejo et al., 2025)。另有工作探索长上下文建模,表明某些架构可以处理包含超过 1 万条事件的完整患者时间线,并对不规则的时间模式更稳健(Wornow et al., 2025)。还有方法把结构化和非结构化数据结合起来,例如用小型辅助模型作为“知识触发器”来支持表格预测(Yan et al., 2025),或设计对编码敏感的表示,以捕捉诊断预测中的语义和时间结构(Tan et al., 2025)。更新的工作通过结构感知检索和患者轨迹的几何表示来探索纵向临床推理,从而更贴近临床地预测下一事件(Qu and Färber, 2026a)。相关工作则把不断演进的患者诊疗建模为连续的时间推理问题,用结构化记忆和智能体式的上下文管理,在流式电子病历上做长程推断(Qu and Färber, 2026b)。这些研究在呈现和编码患者信息方面取得了进展,但模型输出是否与既有临床知识一致,仍较少被深入考查。

Another line of research investigates the reliability of the medical knowledge encoded in LLMs. Dynamic probing methods show that models often achieve low joint accuracy when facts are rephrased (Zhou et al., 2025a), while multifaceted evaluations reveal brittle performance on tasks requiring comparison, verification, or rectification (Zhou et al., 2024). Broader studies also report steep declines when moving from factual recall to scenario-based reasoning as cognitive complexity increases (Zhou et al., 2025b). However, even when scenarios are considered, these evaluations do not disentangle whether errors arise from factual misunderstanding, failures of contextual grounding, or both.

另一条研究线索考查大语言模型中所编码医学知识的可靠性。动态探测方法表明,当事实被改写后,模型的联合准确率往往较低(Zhou et al., 2025a);多层面评估则显示,在需要比较、核实或纠正的任务上,表现较为脆弱(Zhou et al., 2024)。更广的研究也报告,从事实回忆转到基于情景的推理时,随着认知复杂度上升,表现会急剧下降(Zhou et al., 2025b)。不过,即便引入了情景,这些评估仍没有把错误拆开:它到底来自事实理解出错、语境锚定失败,还是两者都有。

Several approaches have sought to improve the factual reliability of LLMs by grounding them in structured biomedical knowledge. These include triplet generation with knowledge graph verification (Su et al., 2025), alignment with biomedical knowledge graph embeddings (Sakhovskiy and Tutubalina, 2025), and graph-based retrieval augmentation for medical question-answering (Wu et al., 2025). Scholars have also critiqued the evaluation paradigms: some argue that multiple-choice questions reward shallow pattern recognition rather than genuine medical knowledge (Griot et al., 2025), while others stress that medical benchmarks must prioritize construct validity to remain clinically meaningful (Alaa et al., 2025). Automatic evaluation frameworks, such as AutoMedEval (Zhang et al., 2025), further highlight the need for scalable yet reliable assessment pipelines.

已有若干方法试图通过把大语言模型锚定到结构化生物医学知识上来提高事实可靠性。这包括用知识图谱核实三元组生成(Su et al., 2025)、与生物医学知识图谱嵌入对齐(Sakhovskiy and Tutubalina, 2025),以及面向医学问答的基于图的检索增强(Wu et al., 2025)。也有学者对评估范式提出批评:有人认为选择题奖励的是浅层模式识别,而不是真正的医学知识(Griot et al., 2025);另一些人则强调,医学基准必须重视构念效度,才可能在临床上有意义(Alaa et al., 2025)。AutoMedEval(Zhang et al., 2025)这类自动评估框架,也进一步说明需要既可扩展又可靠的评估流程。

3 Methodology

3 方法

3.1 MediEval Framework: Data Construction

3.1 MediEval 框架:数据构建

MediEval is a benchmark constructed from the MIMIC-IV database (Medical Information Mart for Intensive Care IV) (Johnson et al., 2023) in combination with biomedical ontologies such as UMLS (Bodenreider, 2004). It is designed to evaluate whether large language models (LLMs) can perform clinical inference by verifying whether medical statements are factually correct and properly grounded in patient records. MediEval integrates structured EHR tables, unstructured clinical notes, and ontology-derived knowledge into a unified dataset of context–statement pairs annotated with correctness and support labels (see Figure 1).

MediEval 是一个由 MIMIC-IV 数据库(Medical Information Mart for Intensive Care IV)(Johnson et al., 2023)与 UMLS(Bodenreider, 2004)等生物医学本体共同构建的基准。它用来评估大语言模型(LLM)能否做临床推断,即核实医学陈述是否在事实上正确,并且是否恰当地锚定在患者记录中。MediEval 把结构化电子病历表、非结构化临床笔记和由本体得到的知识,整合成带有正确性与依据标签的上下文–陈述对统一数据集(见图 1)。

Electronic Health Records (EHRs) as Foundation.

以电子病历(EHR)为基础。

MIMIC-IV is a large-scale publicly available collection of de-identified electronic health records (EHR) comprising 546,028 hospital admissions for 223,452 unique individuals. Two modalities are utilized: (i) structured tabular data (lists of diagnoses, procedures, and medications recorded with ICD-CM, ICD-PCS, and NDC codes), and (ii) unstructured discharge summaries, which typically contain about 22 sections such as Brief Hospital Course, History of Present Illness, Major Surgical or Invasive Procedures, and Discharge Diagnoses.

MIMIC-IV 是一套大规模公开的去标识电子病历(EHR),涵盖 223,452 名独立患者的 546,028 次住院。我们使用两种模态:(i)结构化表格数据(用 ICD-CM、ICD-PCS 和 NDC 编码记录的诊断、操作和用药列表);(ii)非结构化出院小结,通常约有 22 个章节,例如Brief Hospital Course、History of Present Illness、Major Surgical or Invasive Procedures 和 Discharge Diagnoses。

For each admission aaa, we define the structured event set and the discharge summary section set as

对每一次入院 aaa,我们把结构化事件集合和出院小结章节集合定义为

Sa={ Ziadiag,  Zjaproc,  Zkamed },DSa={ sa(1), sa(2), …, sa(ma) }.(1)\displaystyle S_{a} \\ \displaystyle=\{\,\mathcal{Z}^{\text{diag}}_{i_{a}},\;\mathcal{Z}^{\text{proc}}_{j_{a}},\;\mathcal{Z}^{\text{med}}_{k_{a}}\,\}, \\ \displaystyle DS_{a} \\ \displaystyle=\{\,s^{(1)}_{a},\,s^{(2)}_{a},\,\ldots,\,s^{(m_{a})}_{a}\,\}. \tag{1}Sa​={Zia​diag​,Zja​proc​,Zka​med​},DSa​={sa(1)​,sa(2)​,…,sa(ma​)​}.(1)

Here Ziadiag\mathcal{Z}^{\text{diag}}_{i_{a}}Zia​diag​, Zjaproc\mathcal{Z}^{\text{proc}}_{j_{a}}Zja​proc​, and Zkamed\mathcal{Z}^{\text{med}}_{k_{a}}Zka​med​ denote the lists of diagnosis, procedure, and medication codes for admission aaa, containing iai_{a}ia​, jaj_{a}ja​, and kak_{a}ka​ elements respectively, while DSaDS_{a}DSa​ represents the set of mam_{a}ma​ segmented sections from the discharge summary. See Figure 1 for a real sample from MIMIC-IV.

这里 Ziadiag\mathcal{Z}^{\text{diag}}_{i_{a}}Zia​diag​、Zjaproc\mathcal{Z}^{\text{proc}}_{j_{a}}Zja​proc​ 和 Zkamed\mathcal{Z}^{\text{med}}_{k_{a}}Zka​med​ 分别表示入院 aaa 的诊断、操作和用药编码列表,元素个数为 iai_{a}ia​、jaj_{a}ja​ 和 kak_{a}ka​;而 DSaDS_{a}DSa​ 表示从出院小结中切出的 mam_{a}ma​ 个章节组成的集合。图 1 给出了一个来自 MIMIC-IV 的真实样本。

Step 1: Context Extraction.

步骤 1:上下文抽取。

Discharge summaries are divided into multiple sections, not all of which are equally informative. For MediEval, we define the evaluation context Ca⊂DSa\mathcal{C}_{a}\subset DS_{a}Ca​⊂DSa​ as the subset of discharge summary sections that best capture patient trajectory and clinical decision-making. In practice, we select sections covering diagnoses, procedures, treatments, medical history, and hospital course, as these jointly provide the most comprehensive evidence. Importantly, the reasoning that connects these entities is not explicitly structured but embedded in free-text narratives, requiring models to infer relations such as diagnosis–medication from natural language.

出院小结被分成多个章节,并非每一章都同样有信息量。对 MediEval 而言,我们将评估上下文 Ca⊂DSa\mathcal{C}_{a}\subset DS_{a}Ca​⊂DSa​ 定义为最能刻画患者轨迹和临床决策的那一部分出院小结章节。实践中,我们选取覆盖诊断、操作、治疗、病史和住院过程的章节,因为它们合在一起能提供最完整的证据。重要的是,把这些实体连起来的推理并没有被显式结构化,而是嵌在自由文本叙述里,因而模型需要从自然语言中推断出例如诊断–用药这样的关系。

Step 2: Semantic Normalization of Medical Codes.

步骤 2:医学编码的语义归一。

Electronic health records employ heterogeneous coding systems: diagnoses and procedures in MIMIC-IV are stored using ICD-9/10, while medications are represented with National Drug Codes (NDC). To reason across modalities, these codes must be projected into a unified semantic space. We achieve this by leveraging the Unified Medical Language System (UMLS) (Bodenreider, 2004), which integrates over 60 families of biomedical vocabularies and provides explicit mappings between them. Specifically, ICD codes are aligned with SNOMED CT concepts, NDC codes are aligned with RxNorm drug identifiers, and all are normalized into Concept Unique Identifiers (CUIs) that serve as universal nodes for cross-system integration. Formally,

电子病历使用异构的编码系统:MIMIC-IV 中的诊断和操作以 ICD-9/10 存储,用药则以国家药品代码(NDC)表示。要跨模态推理,必须把这些编码投影到统一的语义空间。我们借助统一医学语言系统(UMLS)(Bodenreider, 2004)来完成这一点。UMLS 整合了 60 多个家族的生物医学词表,并提供它们之间的显式映射。具体来说,ICD 编码与 SNOMED CT 概念对齐,NDC 编码与 RxNorm 药物标识符对齐,再全部归一为作为跨系统集成通用节点的概念唯一标识符(CUI)。形式化为

f:z↦CUI(z),  Ua={f(z)∣z∈Sa}.(3)f:z\mapsto\text{CUI}(z),\;U_{a}=\{f(z)\mid z\in S_{a}\}. \tag{3}f:z↦CUI(z),Ua​={f(z)∣z∈Sa​}.(3)

Here, zzz denotes a raw code that is being mapped to a CUI(z)\text{CUI}(z)CUI(z), and UaU_{a}Ua​ is the set of normalized CUIs for admission aaa. This normalization ensures that diagnoses, procedures, and medications from different coding systems can be compared, linked, and reasoned over in a unified space.

这里 zzz 表示正在被映射到 CUI(z)\text{CUI}(z)CUI(z) 的原始编码,UaU_{a}Ua​ 是入院 aaa 归一后的 CUI 集合。这一归一保证了来自不同编码系统的诊断、操作和用药可以在同一空间中比较、关联和推理。

Step 3: Ontology Graph Construction and Relation Extraction.

步骤 3:本体图构建与关系抽取。

From the normalized concepts across the cohort, we construct a semantic graph G=(V,E)G=(V,E)G=(V,E), where VVV are CUIs and EEE are relations drawn from the UMLS Metathesaurus, including treated_by, has_associated_procedure, and is_a etc. A dictionary mapping synonyms and abbreviations to CUIs supports robust entity matching in text. Many clinically valid associations are not encoded as direct links but instead mediated through intermediate nodes such as therapeutic classes, anatomical structures, or related conditions. To capture these, we allow multi-hop traversal in GGG, limited to paths of length at most three in order to preserve clinical plausibility.

我们从整个队列中归一后的概念出发,构造语义图 G=(V,E)G=(V,E)G=(V,E),其中 VVV 是 CUI,EEE 是取自 UMLS Metathesaurus 的关系,包括 treated_by、has_associated_procedure 和 is_a 等。一本把同义词和缩写映射到 CUI 的词典,用于在文本中做稳健的实体匹配。许多临床上成立的关联并不编码为直接边,而是经由治疗类别、解剖结构或相关疾病等中间节点传递。为捕捉这些关联,我们允许在 GGG 中做多跳遍历,并把路径长度限制在最多三跳,以保持临床合理性。

Formally, for concepts h,t∈Vh,t\in Vh,t∈V, let d(h,t)d(h,t)d(h,t) be their hop distance; we define their relations as

形式上看,对概念 h,t∈Vh,t\in Vh,t∈V,令 d(h,t)d(h,t)d(h,t) 为它们的跳数距离;我们把它们之间的关系定义为

R(h,t)={ r∣(h,r,t)∈G,  d(h,t)≤3 }(4)R(h,t)=\{\,r\mid(h,r,t)\in G,\;d(h,t)\leq 3\,\} \tag{4}R(h,t)={r∣(h,r,t)∈G,d(h,t)≤3}(4)

Direct relations. The semantic graph captures clinically meaningful associations as single-edge links. For instance, it encodes that the antihypertensive drug Lisinopril (CUI C0023861) is linked to the condition Hypertension (CUI C0020538) through a treated_by relation. Similarly, it represents that the procedure Coronary Angioplasty (CUI C0001979) is associated with the diagnosis Coronary Artery Disease (CUI C0010054) via a has_associated_procedure relation.

直接关系。 语义图把有临床意义的关联刻画为单条边。例如,它编码抗高血压药物 Lisinopril(CUI C0023861)通过 treated_by 关系与疾病 Hypertension(CUI C0020538)相连。类似地,它表示操作 Coronary Angioplasty(CUI C0001979)通过 has_associated_procedure 关系与诊断 Coronary Artery Disease(CUI C0010054)相关联。

Multi-hop inference. Not all clinically valid relations are encoded as direct edges between specific entities in UMLS. For example, many drug–disease links are only represented at the therapeutic class level rather than for individual drugs. As a result, the relation between Lisinopril and Hypertension may absent as a direct edge. Instead, the graph encodes that Lisinopril (CUI C0023861) belongs to the class Angiotensin-Converting Enzyme Inhibitors (CUI C0003028) via an is_a relation, and that this class as a whole is linked to Hypertensive disease (CUI C0020538) via a treats relation:

多跳推断。 并非所有临床上成立的关系都在 UMLS 中被编码为特定实体之间的直接边。例如,许多药物–疾病链接只表示在治疗类别层面,而不是针对单个药物。因此,Lisinopril 与 Hypertension 之间可能并不存在直接边。图中编码的是:Lisinopril(CUI C0023861)通过 is_a 关系属于 Angiotensin-Converting Enzyme Inhibitors 类别(CUI C0003028),而这一类别整体又通过 treats 关系与 Hypertensive disease(CUI C0020538)相连:

(C0023861)→is_a(C0003028)→treats(C0020538).\text{(C0023861)}\xrightarrow{\text{is\_a}}\text{(C0003028)}\xrightarrow{\text{treats}}\text{(C0020538)}.(C0023861)is_a​(C0003028)treats​(C0020538).

By traversing this two-hop path, the system can infer the treatment relation that is missing at the leaf level. Such multi-hop reasoning is essential for capturing clinically valid associations that are indirectly encoded in UMLS.

沿着这条两跳路径遍历,系统就能推断出在叶子层面缺失的治疗关系。这种多跳推理,对于捕捉那些在 UMLS 中只被间接编码、但临床上成立的关联是必要的。

In summary, UaU_{a}Ua​ provides the admission-specific set of normalized concepts, while GGG encodes both direct and inferred relations among these concepts. Together they form the foundation for structured fact extraction in MediEval.

总之,UaU_{a}Ua​ 提供本次入院特有的归一概念集合,而 GGG 编码这些概念之间的直接关系和推断关系。二者共同构成 MediEval 中结构化事实抽取的基础。

Step 4: Quadrant-based Statement Construction.

步骤 4:基于四象限的陈述构造。

For each fact (h,r,t)(h,r,t)(h,r,t), we generate a set of samples covering the four MediEval quadrants:

对每一个事实 (h,r,t)(h,r,t)(h,r,t),我们生成覆盖 MediEval 四个象限的一组样本:

Ya(h,r,t)={yQ1,yQ2,yQ3,yQ4}.(5)\mathcal{Y}_{a}(h,r,t)=\{y^{Q1},y^{Q2},y^{Q3},y^{Q4}\}. \tag{5}Ya​(h,r,t)={yQ1,yQ2,yQ3,yQ4}.(5)

Quadrants are defined by the cross of factual correctness (true vs. false) and contextual grounding (supported vs. unsupported), as summarized in Figure 1. This design ensures that evaluation covers both straightforward comprehension and more subtle failure modes such as plausible contradictions, relational mismatch, and erroneous information.

象限由事实正确性(真 / 假)与语境依据(有依据 / 无依据)交叉定义,如图 1 所概括。这样的设计保证评估既覆盖直截了当的理解,也覆盖似是而非的矛盾、关系错配和错误信息等更细的失效模式。

The quadrants are constructed as follows. Q1 (True–Supported, Supported Fact) represents the baseline case: a correct fact directly attested in the patient context Ca\mathcal{C}_{a}Ca​. Q2 (True–Unsupported, Plausible Contradiction) is generated by replacing an entity with a closely related alternative from the ontology graph GGG (e.g., another entity sharing the same parent concept) that is absent from Ca\mathcal{C}_{a}Ca​, yielding a statement that is medically correct in general but unsupported for this patient (see Figure 2 for an example). Q3 (False–Supported, Relational Mismatch) recombines entities from distinct true facts, creating a false relation even though all entities appear in Ca\mathcal{C}_{a}Ca​, thereby testing whether a model can resist misleading but superficially plausible associations. Finally, Q4 (False–Unsupported, Erroneous Information) introduces a semantically distant distractor entity from GGG, producing a statement that is both false and unsupported.

各象限的构造如下。Q1(真–有依据,有依据的事实) 是基线情形:一条在患者上下文 Ca\mathcal{C}_{a}Ca​ 中直接得到证实的正确事实。Q2(真–无依据,似是而非的矛盾) 则用本体图 GGG 中一个密切相关的替代实体(例如共享同一父概念、但未出现在 Ca\mathcal{C}_{a}Ca​ 中的实体)做替换,得到一条在医学上通常正确、但对这位患者并无依据的陈述(示例见图 2)。Q3(假–有依据,关系错配) 把来自不同真实事实的实体重新组合,即使这些实体都出现在 Ca\mathcal{C}_{a}Ca​ 中,关系却是假的,用以检验模型能否抵抗表面上说得通、实则误导的关联。最后,Q4(假–无依据,错误信息) 从 GGG 中引入语义距离较远的干扰实体,生成既错误又无依据的陈述。

This procedure requires ontology-guided substitutions, recombinations, and plausibility checks to guarantee that true statements remain clinically valid while false ones are both plausible and challenging. The resulting dataset integrates curated narrative context, ontology-based code normalization, and graph reasoning into a unified framework for rigorous evaluation of factual reliability and context adherence. See Figure 2 and Appendix A for real samples of each quadrant.

这一流程需要本体引导的替换、重组和合理性检查,以保证真陈述在临床上仍然成立,假陈述则既看似合理又有难度。最终数据集把整理过的叙述性上下文、基于本体的编码归一和图推理,整合进一个统一框架,用于严格评估事实可靠性和语境遵从。各象限的真实样本见图 2 和附录 A。

Figure 2: Example of statement verification against patient records (Quadrant 2: True-Unsupported). The statement is medically correct, yet in this case, GERD is treated with omepr

Figure 2: Example of statement verification against patient records (Quadrant 2: True-Unsupported). The statement is medically correct, yet in this case, GERD is treated with omeprazole, a different class of medication, so the statement is not supported by the context.

图 2:对照患者记录做陈述核实的例子(第二象限:真–无依据)。该陈述在医学上是正确的,但在这个例子里,GERD 实际使用的是另一类药物奥美拉唑治疗,因此该陈述并不被上下文所支持。

3.2 MediEval Framework: LLM Evaluation

3.2 MediEval 框架:大语言模型评估

Quadrant Formulation as a Four-Way NLI Task.

将四象限表述为四分类自然语言推理任务。

Given a patient context ccc and a candidate medical statement yyy, the evaluation task is to assign yyy to one of four quadrants q∈{Q1,Q2,Q3,Q4}q\in\{Q1,Q2,Q3,Q4\}q∈{Q1,Q2,Q3,Q4}. This formulation extends natural language inference (NLI) into a clinically grounded setting: rather than only judging whether a statement is correct, the model must also decide whether it is grounded in the patient record. By framing correctness and support jointly in a 2×2 structure, MediEval captures safety-critical confusions that cannot be identified when these dimensions are assessed separately. Although motivated by medicine, this setup provides a general template for domains where factual validity and contextual grounding are both essential. To ensure a fair comparison, all non-proprietary models are fine-tuned under an identical supervised setup with LoRA adapters and a classification head on the four quadrants, ensuring that differences in performance reflect model capabilities rather than variations in training.

给定患者上下文 ccc 和一条候选医学陈述 yyy,评估任务是把 yyy 分配到四个象限 q∈{Q1,Q2,Q3,Q4}q\in\{Q1,Q2,Q3,Q4\}q∈{Q1,Q2,Q3,Q4} 之一。这一表述把自然语言推理(NLI)扩展到有临床锚定的设定:模型不仅要判断陈述是否正确,还必须决定它是否锚定在患者记录中。通过在 2×22\times 22×2 结构里同时刻画正确性和依据,MediEval 能够捕捉那些在两个维度被分开评估时无法识别的安全关键混淆。这一设定虽然由医学激发,但也给那些既要求事实成立、又要求语境锚定的领域提供了一个通用模板。为公平比较,所有非专有模型都在同一套有监督设定下,用 LoRA 适配器和面向四个象限的分类头进行微调,从而保证表现差异反映的是模型能力,而不是训练方式的不同。

Baseline Metrics.

基线指标。

We evaluate models on a test set Dtest={(ci,yi,qi)}\mathcal{D}_{\text{test}}=\{(c_{i},y_{i},q_{i})\}Dtest​={(ci​,yi​,qi​)} with gold quadrant labels qiq_{i}qi​ and predictions q^i\hat{q}_{i}q^​i​. Overall accuracy is defined as

我们在测试集 Dtest={(ci,yi,qi)}\mathcal{D}_{\text{test}}=\{(c_{i},y_{i},q_{i})\}Dtest​={(ci​,yi​,qi​)} 上评估模型,其中 qiq_{i}qi​ 是黄金象限标签,q^i\hat{q}_{i}q^​i​ 是预测。总体准确率定义为

Accuracy=1∣Dtest∣∑iI(q^i=qi).(6)\text{Accuracy}=\frac{1}{|\mathcal{D}_{\text{test}}|}\sum_{i}\mathbb{I}(\hat{q}_{i}=q_{i}). \tag{6}Accuracy=∣Dtest​∣1​i∑​I(q^​i​=qi​).(6)

In addition, we report macro-averaged Precision, Recall, and F1 across quadrants, together with per-quadrant F1:

此外,我们报告跨象限的宏平均精确率、召回率和 F1,以及分象限 F1:

F1(Qj)=2 Prec(Qj) Rec(Qj)Prec(Qj)+Rec(Qj),j=1,…,4.(7)F1(Q_{j})=\frac{2\,\text{Prec}(Q_{j})\,\text{Rec}(Q_{j})}{\text{Prec}(Q_{j})+\text{Rec}(Q_{j})},\quad j=1,\ldots,4. \tag{7}F1(Qj​)=Prec(Qj​)+Rec(Qj​)2Prec(Qj​)Rec(Qj​)​,j=1,…,4.(7)

These metrics capture overall discrimination ability and expose potential biases, such as over-predicting “supported” statements.

这些指标既刻画总体判别能力,也能暴露潜在偏向,例如过度预测“有依据”的陈述。

Critical Error Rates.

关键错误率。

Beyond global performance, clinical safety requires avoiding specific misclassifications. A model may state a correct medical fact but wrongly claim it is supported by the patient record, or elevate an incorrect statement to seemingly valid evidence. To capture these entangled risks, we introduce two error rates:

在总体表现之外,临床安全还要求避免特定的误分类。模型可能说出一条正确的医学事实,却错误地声称它被患者记录所支持;也可能把一条不正确的陈述抬成看似有效的证据。为捕捉这些缠在一起的风险,我们引入两个错误率:

Hallucinated Support Rate (HSR). The fraction of true but unsupported statements (Q2) misclassified as true and supported (Q1):

幻觉式依据率(HSR)。 真实但无依据的陈述(Q2)被误分为真实且有依据(Q1)的比例:

HSR=∑iI(qi=Q2∧q^i=Q1)∑iI(qi=Q2).(8)\text{HSR}=\frac{\sum_{i}\mathbb{I}(q_{i}=Q2\land\hat{q}_{i}=Q1)}{\sum_{i}\mathbb{I}(q_{i}=Q2)}. \tag{8}HSR=∑i​I(qi​=Q2)∑i​I(qi​=Q2∧q^​i​=Q1)​.(8)

A high HSR indicates that the model hallucinates evidence, presenting correct medical knowledge as if it were grounded in the record.

HSR 高,说明模型在幻觉出依据,把正确的医学知识说得像已经锚定在记录中一样。

Truth Inversion Rate (TIR). The proportion of false but superficially supported statements (Q3) misclassified as true and supported (Q1):

真相倒置率(TIR)。 虚假但表面上有依据的陈述(Q3)被误分为真实且有依据(Q1)的比例:

TIR=∑iI(qi=Q3∧q^i=Q1)∑iI(qi=Q3).(9)\text{TIR}=\frac{\sum_{i}\mathbb{I}(q_{i}=Q3\land\hat{q}_{i}=Q1)}{\sum_{i}\mathbb{I}(q_{i}=Q3)}. \tag{9}TIR=∑i​I(qi​=Q3)∑i​I(qi​=Q3∧q^​i​=Q1)​.(9)

A high TIR indicates that the model elevates incorrect or unsafe claims to seemingly valid evidence, a critical failure mode in clinical contexts.

TIR 高,说明模型把不正确或不安全的主张抬成了看似有效的证据,这在临床语境中是一种关键失效模式。

Interpretation of Metrics.

指标的解释。

These metrics disentangle complementary aspects of model behavior: Overall accuracy and macro-F1 capture balanced classification performance. Per-quadrant F1 identifies systematic weaknesses (e.g., under-recognition of unsupported truths). HSR and TIR directly target the most safety-critical confusions between quadrants: hallucinating support and inverting truth.

这些指标把模型行为中互补的几个方面拆开:总体准确率 和 宏平均 F1 刻画均衡的分类表现。分象限 F1 识别系统性薄弱环节(例如对无依据真陈述识别不足)。HSR 和 TIR 则直接针对象限之间最关键的安全混淆:幻觉出依据,以及把真相反转。

3.3 Counterfactual Risk-Aware Fine-Tuning

3.3 反事实风险感知微调

Large language models are typically fine-tuned in two broad ways. The most direct approach is supervised fine-tuning (SFT), where the model is trained to predict gold-standard labels with a cross-entropy loss. While simple and effective, SFT treats each label independently. In contrast, preference-based optimization exploits comparisons between a preferred and a dispreferred output. A prominent example is Direct Preference Optimization (DPO), which updates the model to increase the relative likelihood of preferred responses while staying close to a reference model (Rafailov et al., 2023).

大语言模型的微调通常有两条宽路径。最直接的是有监督微调(SFT),用交叉熵损失训练模型去预测金标准标签。这种方法简单有效,但 SFT 把每个标签当作彼此独立。相比之下,基于偏好的优化 利用优选输出和劣选输出之间的比较。一个突出例子是直接偏好优化(DPO),它更新模型以提高优选回答的相对似然,同时保持接近参考模型(Rafailov et al., 2023)。

Building on these foundations and the supervised fine-tuning results established in our evaluation task, we propose CoRFu (Counterfactual Risk-aware Fine-tuning), a contrastive fine-tuning strategy tailored to the clinical safety challenges revealed by MediEval. Training proceeds on preference triplets (c,yw,yl)(c,y_{w},y_{l})(c,yw​,yl​), where ccc is the clinical context, ywy_{w}yw​ is a factually supported statement, and yly_{l}yl​ is a statement drawn from the counterfactual quadrants of MediEval. The key innovation is the CoRFu loss function, which augments the standard DPO objective with an asymmetric penalty that more severely punishes safety-critical errors.

在这些基础以及我们在评估任务中建立的有监督微调结果之上,我们提出 CoRFu(Counterfactual Risk-aware Fine-tuning),这是一种针对 MediEval 所揭示的临床安全挑战而设计的对比微调策略。训练在偏好三元组 (c,yw,yl)(c,y_{w},y_{l})(c,yw​,yl​) 上进行,其中 ccc 是临床上下文,ywy_{w}yw​ 是有事实依据的陈述,yly_{l}yl​ 则取自 MediEval 的反事实象限。关键创新是 CoRFu 损失函数,它在标准 DPO 目标上加入非对称惩罚,对安全关键错误施加更重的处罚。

Let SSS denote the DPO preference margin:

令 SSS 表示 DPO 的偏好间隔:

S(c;yw,yl)=βlog⁡[πθ(yw∣c) πref(yl∣c)πref(yw∣c) πθ(yl∣c)].(10)S(c;y_{w},y_{l})=\beta\log[\frac{\pi_{\theta}(y_{w}|c)\,\pi_{\text{ref}}(y_{l}|c)}{\pi_{\text{ref}}(y_{w}|c)\,\pi_{\theta}(y_{l}|c)}]. \tag{10}S(c;yw​,yl​)=βlog[πref​(yw​∣c)πθ​(yl​∣c)πθ​(yw​∣c)πref​(yl​∣c)​].(10)

and define the CoRFu loss as:

并把 CoRFu 损失定义为:

LCoRFu=−E[log⁡σ(S)]+λ⋅E[I(S<0)⋅S2].(11)\mathcal{L}_{\text{CoRFu}}=-\mathbb{E}\big[\log\sigma(S)\big]+\lambda\cdot\mathbb{E}\big[\mathbb{I}(S<0)\cdot S^{2}\big]. \tag{11}LCoRFu​=−E[logσ(S)]+λ⋅E[I(S<0)⋅S2].(11)

The first term is the standard DPO objective, which encourages the model πθ\pi_{\theta}πθ​ to prefer ywy_{w}yw​ over yly_{l}yl​ relative to a reference policy πref\pi_{\text{ref}}πref​. The second term introduces an asymmetric penalty that activates only when the model ranks yly_{l}yl​ above ywy_{w}yw​ (S<0S<0S<0). Scaling this penalty quadratically in SSS disproportionately punishes high-confidence mistakes, aligning optimization pressure with clinical safety by discouraging unsafe misinterpretations of evidence.

第一项是标准 DPO 目标,鼓励模型 πθ\pi_{\theta}πθ​ 相对于参考策略 πref\pi_{\text{ref}}πref​ 更偏好 ywy_{w}yw​ 而不是 yly_{l}yl​。第二项引入非对称惩罚,仅在模型把 yly_{l}yl​ 排在 ywy_{w}yw​ 之上时(S<0S<0S<0)被激活。把这一惩罚按 SSS 的二次方放大,会不成比例地处罚高置信错误,从而把优化压力与临床安全对齐,抑制对证据的不安全误读。

4 Experimental Setup

4 实验设置

4.1 Dataset

4.1 数据集

The final MediEval dataset is derived from MIMIC-IV v3.1 and consists of 2,015 unique hospital admissions, linked to 8,350 medical statements and yielding 37,144 samples (see Appendix B for distribution of relation types). The context length varies from 340 to 2,827 tokens. To prevent data leakage across splits, we retain only one admission per patient, since each patient may have multiple hospitalizations. The dataset is partitioned into training, validation, and test sets in an 80/10/10 ratio, balanced across the four quadrants. To ensure correctness, 200 test samples were randomly selected for human evaluation (details in Appendix C).

最终的 MediEval 数据集来自 MIMIC-IV v3.1,包含 2,015 次独立住院,关联 8,350 条医学陈述,共得到 37,144 个样本(关系类型分布见附录 B)。上下文长度从 340 到 2,827 个 token 不等。为防止划分之间发生数据泄漏,每位患者只保留一次入院,因为同一患者可能有多次住院。数据集按 80/10/10 划分为训练、验证和测试集,并在四个象限上保持均衡。为核实正确性,我们从测试集中随机抽取 200 个样本做人工评估(细节见附录 C)。

4.2 Large Language Models and Baselines

4.2 大语言模型与基线

To enable systematic comparison across model sizes and training regimes, we selected 15 LLMs spanning proprietary, open-source, and biomedical domain-specific families. These include GPT (OpenAI) (OpenAI et al., 2024), LLaMA (Meta) (Grattafiori et al., 2024), Mistral (Mistral AI) (Jiang et al., 2024), Qwen (Alibaba) (Yang et al., 2025), and the biomedical models Meditron (Chen et al., 2023), Med42 (Christophe et al., 2024), and ClinicalCamel (Toma et al., 2023). For implementing CoRFu, we used the Llama-3.1–8B-Instruct and Qwen3-8B models, with direct preference optimization (DPO) serving as a baseline.

为了在不同模型规模和训练制度之间做系统比较,我们选取了 15 个覆盖专有、开源和生物医学领域专用家族的大语言模型。其中包括 GPT(OpenAI)(OpenAI et al., 2024)、LLaMA(Meta)(Grattafiori et al., 2024)、Mistral(Mistral AI)(Jiang et al., 2024)、Qwen(Alibaba)(Yang et al., 2025),以及生物医学模型 Meditron(Chen et al., 2023)、Med42(Christophe et al., 2024)和 ClinicalCamel(Toma et al., 2023)。实现 CoRFu 时,我们使用 Llama-3.1–8B-Instruct 和 Qwen3-8B,并以直接偏好优化(DPO)作为基线。

4.3 Training Setups and Evaluation Metrics

4.3 训练设置与评估指标

LLM evaluation with supervised fine-tuning.

有监督微调下的大语言模型评估。

All non-proprietary models are fine-tuned under an identical supervised setup to ensure comparability. Each model is equipped with a sequence classification head and trained on the four MediEval quadrants using weighted cross-entropy loss. LoRA adapters are applied for parameter-efficient fine-tuning, with uniform hyperparameters across models (batch size, learning rate, and number of epochs). Proprietary models are evaluated in their base zero-shot generative form without fine-tuning, and their results are shown in Appendix E.

所有非专有模型都在同一套有监督设定下微调,以保证可比性。每个模型都配有序列分类头,并用加权交叉熵损失在 MediEval 的四个象限上训练。参数高效微调采用 LoRA 适配器,超参数在各模型间保持一致(批量大小、学习率和训练轮数)。专有模型以其未经微调的零样本生成形态评估,结果见附录 E。

Evaluation of CoRFu.

CoRFu 的评估。

We construct preference pairs where Q1 (supported truth) is always the preferred class. We explore three regimes:

我们构造偏好对,始终把 Q1(有依据的真陈述)作为优选类别。我们考查三种制度:

  • Pairwise: contrast Q1 with a specific error type (Q2, Q3, or Q4).
  • 成对: 把 Q1 与某一种特定错误类型(Q2、Q3 或 Q4)对照。
  • Mixed: contrast Q1 against a pooled set of Q2/3/4 in a single stage.
  • 混合: 在单一阶段里把 Q1 与汇合后的 Q2/3/4 对照。
  • Curriculum: train sequentially on Q1 vs. Q2, then Q1 vs. Q3, and finally Q1 vs. Q4, reusing the updated model at each stage.
  • 课程: 按顺序训练 Q1 对 Q2,再训练 Q1 对 Q3,最后训练 Q1 对 Q4,每一阶段复用更新后的模型。

The mixed setup exposes the model to diverse error types simultaneously, while the curriculum regime introduces them in stages of increasing difficulty, reducing interference from heterogeneous signals. By encoding MediEval’s safety perspective directly into both the loss and the training setups, CoRFu aligns model fine-tuning not only with preference correctness but also with clinical reasoning.

混合设定让模型同时接触多样的错误类型;课程制度则按难度递增分阶段引入,以减少来自异质信号的相互干扰。通过把 MediEval 的安全视角直接写进损失和训练设定,CoRFu 使微调不仅对齐偏好正确性,也对齐临床推理。

Evaluation Metrics.

评估指标。

Performance is measured by accuracy, macro-precision, macro-recall, macro-F1, and per-quadrant F1. Furthermore, critical error rates are quantified by the hallucinated support rate (HSR) and the truth inversion rate (TIR), for which lower values indicate better safety.

表现用准确率、宏平均精确率、宏平均召回率、宏平均 F1 以及分象限 F1 来衡量。此外,安全关键错误用幻觉式依据率(HSR)和真相倒置率(TIR)量化,二者越低表示越安全。

5 Results and Discussion

5 结果与讨论

Table 1: MediEval benchmarking results (best results underlined) and CoRFu results (best results in bold).

Overall Performance Per-Quadrant F1-Scores Critical Error Rates
Type Model Acc. Prec. Rec. F1 F1_Q1 F1_Q2 F1_Q3 F1_Q4 HSR TIR
Open-source Llama-3.1-8B-Instruct 67.8 66.4 66.8 61.5 83.6 64.0 38.2 60.0 28.2 21.1
(with SFT) Llama-3.2-1B-Instruct 64.1 57.6 64.3 50.3 80.9 0.0 68.1 52.2 27.3 15.8
Llama-3.2-3B-Instruct 64.1 61.8 63.8 61.6 77.1 53.8 44.8 70.9 40.9 31.6
Llama-3.3-70B-Instruct 73.9 73.3 73.2 70.7 86.7 70.0 65.9 60.0 21.2 21.1
Mixtral-8x7B-Instruct-v0.1 65.4 69.6 66.0 63.8 51.4 68.3 69.3 66.4 20.5 15.3
Mistral-7B-Instruct-v0.3 60.5 61.0 60.7 59.7 67.8 55.6 46.2 69.2 29.1 26.3
Vicuna-13B-v1.5 59.3 59.3 59.2 59.3 62.6 56.1 66.8 51.6 22.7 15.5
Qwen3-4B 61.7 61.3 61.7 61.4 71.7 49.0 63.3 61.6 31.8 26.3
Qwen3-8B 63.7 64.4 63.7 59.9 70.0 58.6 50.0 60.8 28.2 31.1
Qwen3-32B 62.9 64.6 63.8 62.1 72.9 62.6 51.4 61.3 28.2 21.1
Domain-specific Meditron-70B 64.4 62.3 65.2 68.0 66.2 64.3 66.7 74.9 24.8 26.3
(with SFT) Med42-70B 65.6 63.3 65.3 62.8 58.8 55.8 58.5 76.4 23.9 25.6
ClinicalCamel-70B 62.0 61.7 62.6 62.0 55.4 54.3 63.3 75.0 24.8 25.6
Llama-3.1-8B-Ins. + DPO (Q1 vs. Q2) 65.6 59.7 64.2 59.5 53.5 55.1 56.7 72.7 32.7 27.9
+ CoRFu (Q1 vs. Q2) 76.8 77.2 77.0 77.9 73.2 76.6 78.9 79.9 18.2 0.0
+ CoRFu (Q1 vs. Q3) 72.7 66.8 74.2 72.9 84.2 69.1 71.1 67.1 20.1 11.5
+ CoRFu (Q1 vs. Q4) 64.5 74.7 63.6 69.0 70.6 67.4 70.0 68.1 19.9 17.7
+ CoRFu (Q1 vs. mix) 69.1 70.2 69.9 69.6 67.7 68.6 72.1 71.7 23.1 12.9
+ CoRFu (Curriculum) 64.5 75.0 63.6 70.6 70.6 70.5 72.3 68.8 19.1 3.5
Qwen3-8B + DPO (Q1 vs. Q2) 51.0 51.6 50.9 51.1 51.2 49.1 56.7 47.4 22.7 4.4
+ CoRFu (Q1 vs. Q2) 70.7 71.5 71.1 71.0 65.1 66.7 78.0 74.3 21.8 0.0
+ CoRFu (Q1 vs. Q3) 61.1 52.0 59.9 53.0 52.9 55.3 49.3 54.4 22.3 8.3
+ CoRFu (Q1 vs. Q4) 60.3 55.2 59.6 50.7 54.3 50.1 51.3 46.7 18.9 7.6
+ CoRFu (Q1 vs. mix) 59.3 60.7 60.3 59.3 53.3 52.4 68.9 62.7 19.3 13.3
+ CoRFu (Curriculum) 48.2 46.0 48.1 47.7 40.8 45.3 41.1 49.2 17.3 14.5

表 1:MediEval 基准结果(最优结果以下划线标出)和 CoRFu 结果(最优结果以粗体标出)。

总体表现 分象限 F1 关键错误率
类型 模型 准确率 精确率 召回率 F1 F1_Q1 F1_Q2 F1_Q3 F1_Q4 HSR TIR
开源 Llama-3.1-8B-Instruct 67.8 66.4 66.8 61.5 83.6 64.0 38.2 60.0 28.2 21.1
(经 SFT) Llama-3.2-1B-Instruct 64.1 57.6 64.3 50.3 80.9 0.0 68.1 52.2 27.3 15.8
Llama-3.2-3B-Instruct 64.1 61.8 63.8 61.6 77.1 53.8 44.8 70.9 40.9 31.6
Llama-3.3-70B-Instruct 73.9 73.3 73.2 70.7 86.7 70.0 65.9 60.0 21.2 21.1
Mixtral-8x7B-Instruct-v0.1 65.4 69.6 66.0 63.8 51.4 68.3 69.3 66.4 20.5 15.3
Mistral-7B-Instruct-v0.3 60.5 61.0 60.7 59.7 67.8 55.6 46.2 69.2 29.1 26.3
Vicuna-13B-v1.5 59.3 59.3 59.2 59.3 62.6 56.1 66.8 51.6 22.7 15.5
Qwen3-4B 61.7 61.3 61.7 61.4 71.7 49.0 63.3 61.6 31.8 26.3
Qwen3-8B 63.7 64.4 63.7 59.9 70.0 58.6 50.0 60.8 28.2 31.1
Qwen3-32B 62.9 64.6 63.8 62.1 72.9 62.6 51.4 61.3 28.2 21.1
领域专用 Meditron-70B 64.4 62.3 65.2 68.0 66.2 64.3 66.7 74.9 24.8 26.3
(经 SFT) Med42-70B 65.6 63.3 65.3 62.8 58.8 55.8 58.5 76.4 23.9 25.6
ClinicalCamel-70B 62.0 61.7 62.6 62.0 55.4 54.3 63.3 75.0 24.8 25.6
Llama-3.1-8B-Ins. + DPO (Q1 vs. Q2) 65.6 59.7 64.2 59.5 53.5 55.1 56.7 72.7 32.7 27.9
+ CoRFu (Q1 vs. Q2) 76.8 77.2 77.0 77.9 73.2 76.6 78.9 79.9 18.2 0.0
+ CoRFu (Q1 vs. Q3) 72.7 66.8 74.2 72.9 84.2 69.1 71.1 67.1 20.1 11.5
+ CoRFu (Q1 vs. Q4) 64.5 74.7 63.6 69.0 70.6 67.4 70.0 68.1 19.9 17.7
+ CoRFu (Q1 vs. mix) 69.1 70.2 69.9 69.6 67.7 68.6 72.1 71.7 23.1 12.9
+ CoRFu (Curriculum) 64.5 75.0 63.6 70.6 70.6 70.5 72.3 68.8 19.1 3.5
Qwen3-8B + DPO (Q1 vs. Q2) 51.0 51.6 50.9 51.1 51.2 49.1 56.7 47.4 22.7 4.4
+ CoRFu (Q1 vs. Q2) 70.7 71.5 71.1 71.0 65.1 66.7 78.0 74.3 21.8 0.0
+ CoRFu (Q1 vs. Q3) 61.1 52.0 59.9 53.0 52.9 55.3 49.3 54.4 22.3 8.3
+ CoRFu (Q1 vs. Q4) 60.3 55.2 59.6 50.7 54.3 50.1 51.3 46.7 18.9 7.6
+ CoRFu (Q1 vs. mix) 59.3 60.7 60.3 59.3 53.3 52.4 68.9 62.7 19.3 13.3
+ CoRFu (Curriculum) 48.2 46.0 48.1 47.7 40.8 45.3 41.1 49.2 17.3 14.5

5.1 MediEval Benchmarking Results

5.1 MediEval 基准评估结果

Overall Performance.

总体表现。

Table 1 reports macro metrics (accuracy, precision, recall, F1) and safety scores (HSR, TIR) on MediEval. Overall performance is limited given the clinical reasoning requirements, with accuracies ranging from 59.3% to 73.9% and macro F1 from 50.3% to 70.7% across base models. The strongest model is Llama-3.3-70B-Instruct, which achieves the highest overall results in accuracy (73.9%), precision (73.3%), recall (73.2%), and F1 (70.7%). Among smaller models, Llama-3.1-8B-Instruct is competitive, reaching 61.5% macro F1. Although pretrained on clinical corpora, biomedical LLMs do not dominate. These results suggest that lexical familiarity and domain exposure alone are insufficient; fine-grained reasoning to disentangle factual correctness from contextual grounding requires explicit supervision.

表 1 报告了 MediEval 上的宏平均指标(准确率、精确率、召回率、F1)和安全性分数(HSR、TIR)。就临床推理的要求而言,总体表现有限:各基座模型的准确率在 59.3% 到 73.9% 之间,宏平均 F1 在 50.3% 到 70.7% 之间。最强的是 Llama-3.3-70B-Instruct,它在准确率(73.9%)、精确率(73.3%)、召回率(73.2%)和 F1(70.7%)上都取得最高总体结果。在较小的模型里,Llama-3.1-8B-Instruct 也有竞争力,宏平均 F1 达到 61.5%。尽管在临床语料上做了预训练,生物医学大语言模型并没有占据主导。这些结果表明,仅有词汇熟悉度和领域接触是不够的;要把事实正确性从语境锚定中拆开,精细推理需要显式监督。

Per-Quadrant Performance.

分象限表现。

Performance is uneven across the four quadrants, but consistent with their expected difficulty. Q1 (Supported Fact) is easiest (best 86.7 by Llama-3.3-70B), consistent with models’ strength on recognized truthful patterns in-context. Q2 (Plausible Contradiction) is harder, with the best model below 70%, showing that LLMs often treat true but irrelevant statements as supported. Q3 (Relational Mismatch) stresses relation-level reasoning; the best score reaches only 69.3, showing limited resistance to superficially coherent recombinations. Domain-specific models excel on Q4 (Erroneous Information) (74.9 to 76.4%), likely because clinical pretraining gives them stronger priors for rejecting clearly false medical statements, though their broader reasoning across quadrants remains limited. Notably, even within a single model (e.g., Llama-3.1-8B-Instruct), performance varies sharply across quadrants, showing that strength in one type of reasoning does not transfer to others. This unevenness highlights persistent weaknesses in how models combine factuality with patient grounding.

四个象限上的表现并不均匀,但与各自预期难度一致。Q1(有依据的事实)最容易(最好成绩是 Llama-3.3-70B 的 86.7),这与模型在上下文中识别已知真实模式的长处相符。Q2(似是而非的矛盾)更难,最好的模型也低于 70%,说明大语言模型常常把真实但无关的陈述当成有依据。Q3(关系错配)考验关系层面的推理,最好成绩只有 69.3,说明模型对表面上连贯的重组抵抗有限。领域专用模型在 Q4(错误信息)上表现突出(74.9% 到 76.4%),很可能是因为临床预训练给了它们更强的先验,去拒绝明显错误的医学陈述,尽管它们在各象限上更广的推理仍然有限。值得注意的是,即便在同一个模型内部(例如 Llama-3.1-8B-Instruct),各象限表现也相差很大,说明一种推理上的长处并不会自动转移到其他类型。这种不均衡凸显了模型在把事实性与患者锚定结合起来时持续存在的薄弱环节。

Safety-Critical Errors.

安全关键错误。

Safety-critical metrics expose risks not captured by aggregate F1. Mixtral-8x7B consistently achieves the lowest error rates, with HSR (Q2 ⁣→\!\to→Q1) of 20.5% and TIR (Q3 ⁣→\!\to→Q1) of 15.3%, whereas Llama-3.2-3B exhibits the highest risk on both metrics (HSR 40.9%, TIR 31.6%). This indicates that Mixtral is the safest model with respect to HSR/TIR, even though it does not attain the highest macro F1. In contrast, Llama-3.3-70B achieves the best macro F1 (70.7%) yet fails to minimize HSR/TIR (21.2/21.1), demonstrating that higher accuracy does not necessarily translate into safer clinical reasoning. Overall, the divergence between F1 and safety metrics shows that models can perform well on average while still making risky errors.

安全关键指标暴露了汇总 F1 捕捉不到的风险。Mixtral-8x7B 始终取得最低错误率,HSR(Q2 ⁣→\!\to→Q1)为 20.5%,TIR(Q3 ⁣→\!\to→Q1)为 15.3%;而 Llama-3.2-3B 在两项指标上都风险最高(HSR 40.9%,TIR 31.6%)。这说明就 HSR/TIR 而言,Mixtral 是最安全的模型,尽管它的宏平均 F1 并不是最高。相比之下,Llama-3.3-70B 取得了最好的宏平均 F1(70.7%),却未能把 HSR/TIR 压到最低(21.2/21.1),表明更高的准确率并不必然转化为更安全的临床推理。总体来看,F1 与安全性指标的分流说明,模型可以在平均意义上表现不错,却仍然犯下有风险的错误。

5.2 CoRFu Evaluation Results

5.2 CoRFu 评估结果

Overall Effects.

总体效果。

Across both backbones, CoRFu outperforms the base SFT and DPO models, but the strongest gains concentrate in a single pairing. On Llama-3.1-8B-Instruct, Q1 vs. Q2 achieves the best results at 76.8% accuracy and 77.9% macro F1 (+16.4 F1 over SFT), while other variants are lower (e.g., Q1 vs. Q3 at 72.9% F1, Q1 vs. mix at 69.6%, curriculum at 70.6%). For Qwen3-8B, the pattern is sharper: only Q1 vs. Q2 yields a substantial aggregate improvement (70.7% Acc, 71.0% F1; +11.1 F1 over base), whereas Q1 vs. Q3, Q1 vs. Q4, and curriculum underperform on macro metrics. This shows that contrasting supported with unsupported truths provides the most reliable signal for improving overall reasoning, while other pairings mainly target specific weaknesses.

在两个骨干模型上,CoRFu 都优于基座的 SFT 和 DPO 模型,但最强的增益集中在一种配对上。对 Llama-3.1-8B-Instruct,Q1 对 Q2 取得最好结果,准确率 76.8%,宏平均 F1 77.9%(比 SFT 高 16.4 个 F1 点),其他变体则较低(例如 Q1 对 Q3 的 F1 为 72.9%,Q1 对混合为 69.6%,课程为 70.6%)。对 Qwen3-8B,这一模式更明显:只有 Q1 对 Q2 带来可观的总体提升(准确率 70.7%,F1 71.0%;比基座高 11.1 个 F1 点),而 Q1 对 Q3、Q1 对 Q4 以及课程在宏平均指标上表现不佳。这说明,把有依据的真陈述与无依据的真陈述对照,是改进总体推理最可靠的信号,其他配对则主要针对特定薄弱环节。

Per-Quadrant Improvements.

分象限改进。

CoRFu lifts per-quadrant F1 scores, but the biggest gains come from targeted pairings rather than curriculum. With the Llama-3.1-8B-Instruct backbone, Q1 vs. Q2 achieves the best balance across quadrants (Q1/Q2/Q3/Q4 = 73.2/76.6/78.9/79.9), improving both factual grounding (Q2) and overall accuracy. Targeted objectives produce predictable effects: training on (Q1 vs. Q2) substantially boosts Q2 F1 (+12.6 over base), while (Q1 vs. Q3) increases Q3 F1. These results show that targeted pairings not only correct specific weaknesses but also provide complementary improvements across quadrants.

CoRFu 提高了分象限 F1,但最大增益来自有针对性的配对,而不是课程。以 Llama-3.1-8B-Instruct 为骨干时,Q1 对 Q2 在各象限上最为均衡(Q1/Q2/Q3/Q4 = 73.2/76.6/78.9/79.9),既改进了事实锚定(Q2),也提高了总体准确率。有针对性的目标会产生可预期的效果:在(Q1 对 Q2)上训练会显著提高 Q2 的 F1(比基座高 12.6),而(Q1 对 Q3)会提高 Q3 的 F1。这些结果表明,有针对性的配对不仅纠正特定薄弱环节,还能在各象限上带来互补改进。

Safety-Critical Reductions.

安全关键错误的下降。

Improvements in accuracy do not automatically translate into safety, but CoRFu reduces both HSR and TIR compared to SFT and DPO. On Llama-3.1-8B, the (Q1 vs. Q2) run achieves the lowest HSR at 18.2% and eliminates TIR entirely. On Qwen-8B, curriculum yields the lowest HSR at 17.3%, while (Q1 vs. Q2) again minimizes TIR to 0.0. These results show that different contrastive objectives selectively mitigate different risks, confirming that aggregate accuracy and safety-critical errors must be disentangled and addressed with tailored supervision.

准确率的提高并不会自动转化为安全性,但与 SFT 和 DPO 相比,CoRFu 同时降低了 HSR 和 TIR。在 Llama-3.1-8B 上,(Q1 对 Q2)这一轮取得了最低的 HSR(18.2%),并彻底消除了 TIR。在 Qwen-8B 上,课程取得最低的 HSR(17.3%),而(Q1 对 Q2)同样把 TIR 降到 0.0。这些结果表明,不同的对比目标会有选择地缓解不同风险,也证实必须把总体准确率和安全关键错误拆开,并用有针对性的监督去处理。

Ablation Study.

消融实验。

We conducted an ablation study for the value of λ\lambdaλ. When λ=0\lambda=0λ=0, the method reduces to vanilla DPO, with performance reported in Table 1. Values between 0.5 and 1.0 yield the best macro-F1 while significantly lowering HSR and TIR. For λ>1.0\lambda>1.0λ>1.0, over-penalization occurs, leading to a decrease in macro-F1 and an increase in error rates. See Appendix F & G for more analyses.

我们就 λ\lambdaλ 的取值做了消融。当 λ=0\lambda=0λ=0 时,方法退化为普通 DPO,表现见表 1。取值在 0.5 到 1.0 之间时,宏平均 F1 最好,同时 HSR 和 TIR 显著下降。当 λ>1.0\lambda>1.0λ>1.0 时会出现过度惩罚,导致宏平均 F1 下降、错误率上升。更多分析见附录 F 和 G。

Overall Insight.

总体认识。

Taken together, the results show that contrastive objectives shape model behavior in complementary ways. Q1–Q2 training provides the strongest overall gains, improving accuracy, macro F1, and grounding while driving HSR to its lowest levels. Q1–Q3 supervision instead strengthens supported reasoning and sharply lowers TIR, though with smaller aggregate benefits. Curriculum is less competitive on headline metrics but can reduce HSR further for some backbones. Overall, these results demonstrate that contrastive fine-tuning can be directed to improve specific weaknesses, and that MediEval makes such effects visible by disentangling accuracy from safety-critical errors.

合在一起看,结果说明对比目标会以互补的方式塑造模型行为。Q1–Q2 训练带来最强的总体增益,提高准确率、宏平均 F1 和锚定质量,并把 HSR 压到最低。Q1–Q3 监督则加强有依据的推理,并显著降低 TIR,尽管总体收益较小。课程在头条指标上不那么有竞争力,但对某些骨干可以把 HSR 压得更低。总体而言,这些结果说明对比微调可以被引导去改进特定薄弱环节,而 MediEval 通过把准确率与安全关键错误拆开,使这类效果能够被看见。

6 Conclusion

6 结论

We introduced MediEval, a benchmark that jointly evaluates factual verification and patient-contextual reasoning by linking biomedical ontologies with real-world patient records. Our results show that current LLMs struggle with quadrant-level reasoning and make safety-critical errors not reflected in aggregate accuracy. To mitigate these risks, we proposed Counterfactual Risk-Aware Fine-tuning (CoRFu), which reduces unsafe misclassifications through asymmetric preference optimization.

我们提出了 MediEval,这一基准通过把生物医学本体与真实世界的患者记录相连,同时评估事实核实和患者语境推理。结果表明,当前大语言模型在象限级推理上仍有困难,并会犯下汇总准确率反映不出来的安全关键错误。为缓解这些风险,我们提出 反事实风险感知微调(CoRFu),通过非对称的偏好优化减少不安全的误分类。

For future work, MediEval could be extended to other clinical tasks such as question answering, adapted to safety-critical domains beyond medicine, and integrated with methods such as retrieval-augmented generation or models designed for temporal patient reasoning. We hope this work provides a foundation for more rigorous and risk-aware evaluation of LLMs.

在未来工作中,MediEval 可以扩展到问答等其他临床任务,适配到医学以外的安全关键领域,并与检索增强生成,或面向患者时间推理的模型等方法结合。我们希望这项工作能为更严格、更有风险意识的大语言模型评估提供一个基础。

Limitations

局限性

Dataset size and coverage. MediEval is constructed from approximately 2k admissions in MIMIC-IV, yielding around 37k statements. This scale reflects a deliberate design choice to prioritize clinical plausibility, careful quality control, and strict prevention of patient-level information leakage, which are particularly important in safety-critical medical evaluation. The dataset construction pipeline is fully automated and released with the codebase, making MediEval readily extensible to larger cohorts, additional institutions, or alternative clinical subsets as future work.

数据集规模与覆盖。 MediEval 由 MIMIC-IV 中约 2 千次入院构建,得到大约 3.7 万条陈述。这一规模反映了一种有意的设计选择:优先保证临床合理性、细致的质量控制,以及严格避免患者层面的信息泄漏,这些在安全关键的医学评估中尤为重要。数据构建流程完全自动化,并随代码库发布,因而 MediEval 很容易在未来工作中扩展到更大队列、更多机构或其他临床子集。

Annotation and validation. A subset of the test set was independently reviewed by medically trained annotators, with GPT-5 used as an auxiliary reference to inspect disagreements. High agreement suggests that the ontology-grounded construction leads to largely unambiguous labels. While broader expert validation could further strengthen confidence, the reliance on structured biomedical knowledge and deterministic generation reduces subjectivity compared to fully manual annotation.

标注与核实。 测试集的一个子集由受过医学训练的标注者独立审阅,GPT-5 被用作辅助参考来检查分歧。较高的一致性表明,这种以本体锚定的构造所得到的标签大体是明确的。更广泛的专家核实还可以进一步增强信心,但依赖结构化生物医学知识和确定性生成,已经比完全手工标注减少了主观性。

Data and ontology noise. As with any benchmark grounded in real-world clinical data and large biomedical ontologies, MediEval inherits noise and inconsistencies from MIMIC-IV and resources such as UMLS, SNOMED CT, and RxNorm. We mitigate these effects through ontology-guided plausibility checks, constrained multi-hop traversal, and explicit grounding in patient context. Further advances in ontology curation and clinical data quality are likely to directly benefit benchmarks such as MediEval.

数据与本体噪声。 和任何锚定在真实临床数据与大型生物医学本体上的基准一样,MediEval 会继承来自 MIMIC-IV 以及 UMLS、SNOMED CT、RxNorm 等资源的噪声和不一致。我们通过本体引导的合理性检查、受约束的多跳遍历,以及显式锚定到患者上下文来缓解这些影响。本体整理和临床数据质量上的进一步进展,很可能直接惠及 MediEval 这类基准。

Acknowledgments

致谢

The authors acknowledge the financial support by the Federal Ministry of Research, Technology and Space of Germany and by Sächsische Staatsministerium für Wissenschaft, Kultur und Tourismus in the programme Center of Excellence for AI-research „Center for Scalable Data Analytics and Artificial Intelligence Dresden/Leipzig“, project identification number: ScaDS.AI.

作者感谢德国联邦研究、技术与空间部,以及萨克森州科学、文化与旅游部,在人工智能卓越研究中心计划“Center for Scalable Data Analytics and Artificial Intelligence Dresden/Leipzig”(项目编号:ScaDS.AI)中提供的资助。

The authors acknowledge the computing time made available to them on the high-performance computer at the NHR Center of TU Dresden. This center is jointly supported by the Federal Ministry of Research, Technology and Space of Germany and the state governments participating in the NHR (www.nhr-verein.de/unsere-partner).

作者感谢在 TU Dresden 的 NHR 中心高性能计算机上获得的计算时间。该中心由德国联邦研究、技术与空间部和参与 NHR 的各州政府共同支持(www.nhr-verein.de/unsere-partner)。

References

参考文献

Alaa et al. (2025) A. Alaa, T. Hartvigsen, N. Golchini, S. Dutta, F. Dean, I. D. Raji, and T. Zack Position: medical large language model benchmarks should prioritize construct validity . In Forty-second International Conference on Machine Learning Position Paper Track , Cited by: §2 .

Bodenreider (2004) O. Bodenreider The unified medical language system (umls): integrating biomedical terminology . Nucleic acids research 32 ( suppl_1 ), pp. D267–D270 . Cited by: §1 , §3.1 , §3.1 .

Chen et al. (2023) Z. Chen, A. H. Cano, A. Romanou, A. Bonnet, K. Matoba, F. Salvi, M. Pagliardini, S. Fan, A. Köpf, A. Mohtashami, et al. Meditron-70b: scaling medical pretraining for large language models . arXiv preprint arXiv:2311.16079 . Cited by: §4.2 .

Christophe et al. (2024) C. Christophe, P. K. Kanithi, P. Munjal, T. Raha, N. Hayat, R. Rajan, A. Al-Mahrooqi, A. Gupta, M. U. Salman, G. Gosal, et al. Med42–evaluating fine-tuning strategies for medical llms: full-parameter vs. parameter-efficient approaches . arXiv preprint arXiv:2404.14779 . Cited by: §4.2 .

Donnelly et al. (2006) K. Donnelly et al. SNOMED-ct: the advanced terminology and coding system for ehealth . Studies in health technology and informatics 121 , pp. 279 . Cited by: §1 .

Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models . arXiv preprint arXiv:2407.21783 . Cited by: §4.2 .

Griot et al. (2025) M. Griot, J. Vanderdonckt, D. Yuksel, and C. Hemptinne Pattern recognition or medical knowledge? the problem with multiple-choice questions in medicine . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 5321–5341 . Cited by: §2 .

Haltaufderheide and Ranisch (2024) J. Haltaufderheide and R. Ranisch The ethics of chatgpt in medicine and healthcare: a systematic review on large language models (llms) . NPJ digital medicine 7 ( 1 ), pp. 183 . Cited by: §1 .

Hurst et al. (2024) A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card . arXiv preprint arXiv:2410.21276 . Cited by: Appendix E .

Jiang et al. (2024) A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, L. Saulnier, M. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao, T. Gervet, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed Mixtral of experts . External Links: 2401.04088 , Link Cited by: §4.2 .

Johnson et al. (2023) A. E. Johnson, L. Bulgarelli, L. Shen, A. Gayles, A. Shammout, S. Horng, T. J. Pollard, S. Hao, B. Moody, B. Gow, et al. MIMIC-iv, a freely accessible electronic health record dataset . Scientific data 10 ( 1 ), pp. 1 . Cited by: §1 , §3.1 .

Liu et al. (2005) S. Liu, W. Ma, R. Moore, V. Ganesan, and S. Nelson RxNorm: prescription for electronic drug information exchange . IT professional 7 ( 5 ), pp. 17–23 . Cited by: §1 .

Lovón-Melgarejo et al. (2025) J. Lovón-Melgarejo, M. Mouysset, J. Oleiwan, J. G. Moreno, C. Damase-Michel, and L. Tamine Evaluating llm abilities to understand tabular electronic health records: a comprehensive study of patient data extraction and retrieval . In European Conference on Information Retrieval , pp. 153–168 . Cited by: §1 , §2 .

Moor et al. (2023) M. Moor, O. Banerjee, Z. S. H. Abad, H. M. Krumholz, J. Leskovec, E. J. Topol, and P. Rajpurkar Foundation models for generalist medical artificial intelligence . Nature 616 ( 7956 ), pp. 259–265 . Cited by: §1 .

Nelson et al. (2011) S. J. Nelson, K. Zeng, J. Kilbourne, T. Powell, and R. Moore Normalized names for clinical drugs: rxnorm at 6 years . Journal of the American Medical Informatics Association 18 ( 4 ), pp. 441–448 . Cited by: §1 .

OpenAI et al. (2024) OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V. Balcom, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L. Bogdonoff, et al. GPT-4 technical report . External Links: 2303.08774 , Link Cited by: §4.2 .

Pal et al. (2022) A. Pal, L. K. Umapathi, and M. Sankarasubbu Medmcqa: a large-scale multi-subject multi-choice dataset for medical domain question answering . In Conference on health, inference, and learning , pp. 248–260 . Cited by: §1 .

Qu and Färber (2026a) Z. Qu and M. Färber GRAIL: geometry-aware retrieval-augmented inference with llms over hyperbolic representations of patient trajectories . arXiv preprint arXiv:2602.12828 . Cited by: §2 .

Qu and Färber (2026b) Z. Qu and M. Färber TRACE: temporal reasoning via agentic context evolution for streaming electronic health records (ehrs) . arXiv preprint arXiv:2602.12833 . Cited by: §2 .

Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn Direct preference optimization: your language model is secretly a reward model . In Proceedings of the 37th International Conference on Neural Information Processing Systems , pp. 53728–53741 . Cited by: §3.3 .

Rajpurkar et al. (2022) P. Rajpurkar, E. Chen, O. Banerjee, and E. J. Topol AI in health and medicine . Nature medicine 28 ( 1 ), pp. 31–38 . Cited by: §1 .

Sakhovskiy and Tutubalina (2025) A. Sakhovskiy and E. Tutubalina BALI: enhancing biomedical language representations through knowledge graph and language model alignment . In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval , pp. 1152–1164 . Cited by: §2 .

Singh et al. (2025) A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthram, et al. Openai gpt-5 system card . arXiv preprint arXiv:2601.03267 . Cited by: Appendix E .

Singhal et al. (2023) K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, et al. Large language models encode clinical knowledge . Nature 620 ( 7972 ), pp. 172–180 . Cited by: §1 .

Singhal et al. (2025) K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, M. Amin, L. Hou, K. Clark, S. R. Pfohl, H. Cole-Lewis, et al. Toward expert-level medical question answering with large language models . Nature Medicine 31 ( 3 ), pp. 943–950 . Cited by: §1 .

Su et al. (2025) X. Su, Y. Wang, S. Gao, X. Liu, V. Giunchiglia, D. Clevert, and M. Zitnik KGARevion: an AI agent for knowledge-intensive biomedical QA . In The Thirteenth International Conference on Learning Representations , External Links: Link Cited by: §2 .

Sung et al. (2021) M. Sung, J. Lee, S. Yi, M. Jeon, S. Kim, and J. Kang Can language models be biomedical knowledge bases? . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , Online and Punta Cana, Dominican Republic , pp. 4723–4734 . External Links: Document Cited by: §1 .

Tan et al. (2025) Y. Tan, H. Lv, Y. Zhan, G. Ma, B. Xiong, and C. Yang BoxLM: unifying structures and semantics of medical concepts for diagnosis prediction in healthcare . In Forty-second International Conference on Machine Learning , Cited by: §2 .

Thirunavukarasu et al. (2023) A. J. Thirunavukarasu, D. S. J. Ting, K. Elangovan, L. Gutierrez, T. F. Tan, and D. S. W. Ting Large language models in medicine . Nature medicine 29 ( 8 ), pp. 1930–1940 . Cited by: §1 .

Toma et al. (2023) A. Toma, P. R. Lawler, J. Ba, R. G. Krishnan, B. B. Rubin, and B. Wang Clinical camel: an open expert-level medical language model with dialogue-based knowledge encoding . arXiv preprint arXiv:2305.12031 . Cited by: §4.2 .

Wornow et al. (2025) M. Wornow, S. Bedi, M. A. F. Hernandez, E. Steinberg, J. A. Fries, C. Re, S. Koyejo, and N. Shah Context clues: evaluating long context models for clinical prediction tasks on EHR data . In The Thirteenth International Conference on Learning Representations , Cited by: §2 .

Wu et al. (2025) J. Wu, J. Zhu, Y. Qi, J. Chen, M. Xu, F. Menolascina, Y. Jin, and V. Grau Medical graph RAG: evidence-based medical large language model via graph retrieval-augmented generation . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Vienna, Austria , pp. 28443–28467 . External Links: Document , ISBN 979-8-89176-251-0 Cited by: §2 .

Yan et al. (2025) J. Yan, J. Chen, C. Hu, B. Zheng, Y. Hu, J. Sun, and J. Wu Small models are LLM knowledge triggers for medical tabular prediction . In The Thirteenth International Conference on Learning Representations , Cited by: §2 .

Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report . External Links: 2505.09388 , Link Cited by: §4.2 .

Yang et al. (2023) R. Yang, T. F. Tan, W. Lu, A. J. Thirunavukarasu, D. S. W. Ting, and N. Liu Large language models in health care: development, applications, and challenges . Health Care Science 2 ( 4 ), pp. 255–263 . Cited by: §1 .

Zhang et al. (2025) X. Zhang, Z. Ouyang, L. Wang, G. De Melo, Z. Cao, X. Wang, Y. Zhang, Y. Wang, and L. He AutoMedEval: harnessing language models for automatic medical capability evaluation . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Vienna, Austria , pp. 6272–6285 . External Links: Document , ISBN 979-8-89176-251-0 Cited by: §2 .

Zhou et al. (2024) Y. Zhou, X. Liu, C. Ning, and J. Wu MultifacetEval: multifaceted evaluation to probe llms in mastering medical knowledge . In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence , pp. 6669–6677 . Cited by: §2 .

Zhou et al. (2025a) Y. Zhou, X. Liu, C. Ning, X. Zhang, and J. Wu Reliable and diverse evaluation of LLM medical knowledge mastery . In The Thirteenth International Conference on Learning Representations , Cited by: §1 , §2 .

Zhou et al. (2025b) Y. Zhou, X. Liu, C. Yan, C. Ning, X. Zhang, B. Li, X. Fu, S. Wang, G. Hu, Y. Wang, and J. Wu Evaluating LLMs across multi-cognitive levels: from medical knowledge mastery to scenario-based problem solving . In Forty-second International Conference on Machine Learning , Cited by: §1 , §2 .

Appendices

附录

A Real Samples for Each Quadrant

A 各象限的真实样本

This appendix provides representative examples of MediEval samples from each of the four quadrants (see Figure 2 for Q2). Each example consists of a patient context extracted from a real MIMIC-IV discharge summary, a constructed medical statement, and its corresponding quadrant label.

本附录给出 MediEval 四个象限各自的代表性样本(Q2 见图 2)。每个例子都包含从真实 MIMIC-IV 出院小结中抽出的患者上下文、一条构造出的医学陈述,以及相应的象限标签。

Figure 3: Example of statement verification against patient records (Quadrant 1: True-Supported). The statement is medically correct, and GERD is indeed treated with omeprazole, ma

Figure 3: Example of statement verification against patient records (Quadrant 1: True-Supported). The statement is medically correct, and GERD is indeed treated with omeprazole, making it true and supported.

图 3:对照患者记录做陈述核实的例子(第一象限:真–有依据)。该陈述在医学上是正确的,而且 GERD 确实用奥美拉唑治疗,因此它既为真,也有依据。

Figure 4: Example of statement verification against patient records (Quadrant 3: False-Supported). The crafted statement is medically incorrect, as Atenolol is indicated for hypert

Figure 4: Example of statement verification against patient records (Quadrant 3: False-Supported). The crafted statement is medically incorrect, as Atenolol is indicated for hypertension. Since both GERD and Atenolol appear in the patient context, the statement may seem supported, even though it is false.

图 4:对照患者记录做陈述核实的例子(第三象限:假–有依据)。这条构造出的陈述在医学上是不正确的,因为阿替洛尔的适应证是高血压。由于 GERD 和阿替洛尔都出现在患者上下文中,该陈述看起来可能有依据,尽管它是假的。

Figure 5: Example of statement verification against patient records (Quadrant 4; False-Unsupported). The statement is medically incorrect, since insulin is not indicated for GERD.

Figure 5: Example of statement verification against patient records (Quadrant 4; False-Unsupported). The statement is medically incorrect, since insulin is not indicated for GERD. Moreover, insulin does not appear in the patient context, making the statement false and unsupported.

图 5:对照患者记录做陈述核实的例子(第四象限:假–无依据)。该陈述在医学上是不正确的,因为胰岛素并不是 GERD 的适应证。此外,胰岛素也没有出现在患者上下文中,因此该陈述既为假,也无依据。

B Distribution of Relation Types

B 关系类型的分布

This appendix reports the distribution of biomedical relation types used in MediEval. Relations are derived from UMLS and associated vocabularies after semantic normalization and ontology-guided traversal. The distribution reflects the clinical diversity of the benchmark, covering treatment, diagnostic, pharmacologic, and associative relations.

本附录报告 MediEval 中所用生物医学关系类型的分布。这些关系在语义归一和本体引导的遍历之后,来自 UMLS 及相关词表。这一分布反映了基准在临床上的多样性,覆盖治疗、诊断、药理学和关联性关系。

Table 2: Distribution of relation types in the dataset

Relation Type Count
treats 17,540
may_be_treated_by 4,552
related_to 4,008
classified_as 3,774
contraindicated_class_of 1,250
subset_includes 1,568
has_pharmacologic_class 854
diagnoses 738
prevents 748
causes 808
associated_with 588
authorized_value 282
co-occurs_with 130
defining_characteristic_of 230
manifestation_of 56
sign_or_symptom_of 12
may_be_diagnosed_by 4
part_of 2
Total 37,144

表 2:数据集中关系类型的分布

关系类型 数量
treats 17,540
may_be_treated_by 4,552
related_to 4,008
classified_as 3,774
contraindicated_class_of 1,250
subset_includes 1,568
has_pharmacologic_class 854
diagnoses 738
prevents 748
causes 808
associated_with 588
authorized_value 282
co-occurs_with 130
defining_characteristic_of 230
manifestation_of 56
sign_or_symptom_of 12
may_be_diagnosed_by 4
part_of 2
合计 37,144

Table 3: MediEval benchmarking results of Proprietary GPT Models.

Overall Performance Per-Quadrant F1-Scores Critical Error Rates
Model Acc. Prec. Rec. F1 F1_Q1 F1_Q2 F1_Q3 F1_Q4 HSR TIR
GPT-5 45.7 36.4 46.7 30.4 68.3 53.3 0.0 0.0 19.3 0.0
GPT-4o 49.3 31.6 50.0 29.1 66.3 50.0 0.0 0.0 46.7 0.0

表 3:专有 GPT 模型在 MediEval 上的基准结果。

总体表现 分象限 F1 关键错误率
模型 准确率 精确率 召回率 F1 F1_Q1 F1_Q2 F1_Q3 F1_Q4 HSR TIR
GPT-5 45.7 36.4 46.7 30.4 68.3 53.3 0.0 0.0 19.3 0.0
GPT-4o 49.3 31.6 50.0 29.1 66.3 50.0 0.0 0.0 46.7 0.0

C Human Evaluation

C 人工评估

We conducted human validation on 200 balanced samples. Two medically trained annotators labeled each item independently. In cases of disagreement, GPT-5 was consulted as an auxiliary signal for sanity checking, but final labels were determined by the human annotators. Agreement was 97% (194/200), with Cohen’s kappa≈0.96kappa\approx 0.96kappa≈0.96, indicating strong agreement. All disagreements occurred between Q2 and Q3, which is one of the subtle distinctions that the benchmark is designed to probe. Because our extraction pipeline is fully ontology-driven and deterministic, the errors stem from noise or inconsistencies in the underlying ontologies or EHR data.

我们在 200 个均衡样本上做了人工核实。两名受过医学训练的标注者各自独立标注每一条。出现分歧时,会把 GPT-5 作为辅助信号做合理性检查,但最终标签由人工标注者确定。一致性为 97%(194/200),Cohen 的 κ≈0.96\kappa\approx 0.96κ≈0.96,表明一致性很强。所有分歧都发生在 Q2 与 Q3 之间,而这正是该基准有意要探测的细微差别之一。由于我们的抽取流程完全由本体驱动并且是确定性的,这些错误来自底层本体或电子病历数据中的噪声与不一致。

D Implementation Details

D 实现细节

All training was performed on an NVIDIA L40S GPU with 46 GB memory. All reported results are averaged over 3 random seeds (42, 43, 44). For the MediEval supervised baselines, we fine-tuned models for 3 epochs with per-device batch size 4, gradient accumulation 8, maximum sequence length 4096, learning rate 2×10−52\times 10^{-5}2×10−5, weight decay 0.01, and warmup ratio 0.03. When enabled, LoRA used rank r=8r=8r=8, α=16\alpha=16α=16, and dropout 0.050.050.05. For the CoRFu experiments, we fine-tuned with maximum sequence length 4096, per-device batch size 1, gradient accumulation 16 (effective batch size 16), and trained for 1 epoch. We used the AdamW optimizer with learning rate 1×10−41\times 10^{-4}1×10−4, β1=0.9\beta_{1}=0.9β1​=0.9, β2=0.999\beta_{2}=0.999β2​=0.999, weight decay 0.01, warmup ratio 0.05, and gradient clipping at 1.0. LoRA was configured with rank r=16r=16r=16, α=32\alpha=32α=32, and dropout 0.10.10.1. The CoRFu loss used β=0.1\beta=0.1β=0.1 and λ=0.5\lambda=0.5λ=0.5.

全部训练都在一块显存 46 GB 的 NVIDIA L40S GPU 上完成。所有报告结果都对 3 个随机种子(42、43、44)取平均。对 MediEval 的有监督基线,我们微调 3 个 epoch,单卡批量大小为 4,梯度累积为 8,最大序列长度为 4096,学习率为 2×10−52\times 10^{-5}2×10−5,权重衰减为 0.01,预热比例为 0.03。启用 LoRA 时,秩为 r=8r=8r=8,α=16\alpha=16α=16,dropout 为 0.050.050.05。对 CoRFu 实验,我们以最大序列长度 4096、单卡批量大小 1、梯度累积 16(有效批量大小为 16)微调,并训练 1 个 epoch。优化器为 AdamW,学习率 1×10−41\times 10^{-4}1×10−4,β1=0.9\beta_{1}=0.9β1​=0.9,β2=0.999\beta_{2}=0.999β2​=0.999,权重衰减 0.01,预热比例 0.05,梯度裁剪为 1.0。LoRA 配置为秩 r=16r=16r=16,α=32\alpha=32α=32,dropout 0.10.10.1。CoRFu 损失使用 β=0.1\beta=0.1β=0.1 和 λ=0.5\lambda=0.5λ=0.5。

E Evaluation of Proprietary GPT Models

E 对专有 GPT 模型的评估

GPT-4o Hurst et al. (2024) and GPT-5 Singh et al. (2025) are evaluated under a realistic clinical setting, following their typical usage: zero-shot inference without task-specific fine-tuning. This reflects practical deployment constraints, where adapting proprietary models is often infeasible. Our goal is not a direct head-to-head comparison, but rather to highlight two complementary observations: (1) the task remains challenging even for frontier models without adaptation, and (2) smaller models equipped with targeted supervision (SFT/CoRFu) can outperform general-purpose LLMs on safety-critical dimensions.

GPT-4o(Hurst et al., 2024)和 GPT-5(Singh et al., 2025)在贴近现实的临床设定下评估,沿用它们的典型用法:不做任务专用微调的零样本推断。这反映了实际部署中的约束,因为改造专有模型往往不可行。我们的目标不是做直接的一对一比较,而是突出两点互补观察:(1)即便对未经适配的前沿模型,这项任务仍然困难;(2)配上有针对性监督(SFT/CoRFu)的较小模型,可以在安全关键维度上超过通用大语言模型。

For proprietary models, the prompt consists of the following components: (1) context, (2) statement, (3) definitions of the four quadrants, (4) an instruction to select exactly one label, and (5) the list of candidate labels.

对专有模型,提示由以下部分组成:(1)上下文,(2)陈述,(3)四个象限的定义,(4)一条要求恰好选择一个标签的指令,以及(5)候选标签列表。

Overall, proprietary models struggle on this task. GPT models achieve only 29.1% (GPT-4o) and 30.4% (GPT-5) macro-F1. Since both models are evaluated in their base generative form without task-specific adaptation, this performance reflects both the difficulty of MediEval and the lack of alignment to the evaluation protocol.

总体来看,专有模型在这项任务上表现吃力。GPT 模型的宏平均 F1 分别只有 29.1%(GPT-4o)和 30.4%(GPT-5)。由于两个模型都以其未经任务适配的基座生成形态评估,这一表现既反映了 MediEval 的难度,也反映了它们尚未对齐到这一评估协议。

In terms of safety errors, HSR (Q2 ⁣→\!\rightarrow→Q1) ranges from 19.3% for GPT-5 to 46.7% for GPT-4o. Notably, proprietary models never predict Q3, resulting in zero TIR. This behavior indicates miscalibration rather than genuine safety awareness.

就安全错误而言,HSR(Q2 ⁣→\!\rightarrow→Q1)从 GPT-5 的 19.3% 到 GPT-4o 的 46.7%。值得注意的是,专有模型从不预测 Q3,因而 TIR 为零。这一行为说明的是校准失当,而不是真正的安全意识。

F Ablation Study

F 消融实验

Figure 6: Effect of the regularization coefficient $\lambda$ on macro-F1, HSR, and TIR for Qwen3-8B and Llama-3.1-8B-Instruct under Q1 vs. Q2 CoRFu training.

Figure 6: Effect of the regularization coefficient λ\lambdaλ on macro-F1, HSR, and TIR for Qwen3-8B and Llama-3.1-8B-Instruct under Q1 vs. Q2 CoRFu training.

图 6:正则系数 λ\lambdaλ 对 Qwen3-8B 和 Llama-3.1-8B-Instruct 在 Q1 对 Q2 的 CoRFu 训练下宏平均 F1、HSR 和 TIR 的影响。

Figure 6 illustrates the effect of the regularization coefficient λ\lambdaλ on performance and error rates, under Q1 vs. Q2 CoRFu training. For both models, moderate values of λ\lambdaλ (around 0.50.50.5–1.01.01.0) achieve the best trade-off: macro-F1 is maximized, while HSR and TIR are substantially reduced compared to λ=0\lambda=0λ=0, which corresponds to vanilla DPO. As λ\lambdaλ increases further, performance degrades and error rates rise, indicating over-penalization of negative preference margins. This ablation is necessary because λ\lambdaλ directly controls the strength of the corrective penalty introduced by CoRFu, balancing preference optimization against error suppression. In contrast, we fix the DPO temperature β\betaβ, since it only rescales the preference margin and its effect can be absorbed into λ\lambdaλ.

图 6 展示了正则系数 λ\lambdaλ 在 Q1 对 Q2 的 CoRFu 训练下对表现和错误率的影响。对两个模型,中等的 λ\lambdaλ 取值(约 0.50.50.5–1.01.01.0)都能取得最好的权衡:宏平均 F1 最高,同时 HSR 和 TIR 相对 λ=0\lambda=0λ=0(即普通 DPO)大幅下降。λ\lambdaλ 再增大时,表现会变差,错误率上升,说明对负偏好间隔出现了过度惩罚。这项消融是必要的,因为 λ\lambdaλ 直接控制 CoRFu 所引入校正惩罚的强度,需要在偏好优化和错误抑制之间取得平衡。相比之下,我们固定 DPO 温度 β\betaβ,因为它只是对偏好间隔做缩放,其作用可以被 λ\lambdaλ 吸收。

G Qualitative Evaluation

G 定性评估

We further analyze whether model behavior correlates with input characteristics such as context length or relation type, across all evaluated settings, including SFT and CoRFu. We observe no consistent correlation between any of the reported evaluation metrics and context length, nor systematic performance differences across relation types for any model. These results indicate that the observed performance trends are not driven by input complexity or relation semantics, but instead reflect differences in training objectives and supervision strategies.

我们进一步分析,在包括 SFT 和 CoRFu 在内的全部评估设定下,模型行为是否与上下文长度或关系类型等输入特征相关。我们没有观察到任何报告指标与上下文长度之间存在稳定相关,也没有在任何模型上看到各关系类型之间系统的表现差异。这些结果表明,观察到的表现趋势并非由输入复杂度或关系语义驱动,而反映的是训练目标和监督策略的不同。

Instead, errors predominantly arise from confusions between Q2 and Q1, indicating difficulties in distinguishing between partially incorrect and fully incorrect cases. We provide a qualitative analysis of these failure modes below.

相反,错误主要来自 Q2 与 Q1 之间的混淆,说明模型难以区分部分不正确和完全不正确的情形。我们在下面对这些失效模式做定性分析。

The statement is medically correct in general; however, the specific treatment (aluminum hydroxide) is not mentioned in the clinical context. While GERD is documented and treated with pantoprazole and other antacids, the generated statement introduces an unsupported medication, and is therefore labeled as Q2.

该陈述在医学上通常是正确的;不过,具体疗法(氢氧化铝)并未在临床上下文中出现。虽然 GERD 被记录,并用泮托拉唑及其他抗酸药治疗,但生成的陈述引入了一种并无依据的药物,因此被标为 Q2。

Almost all models fail on this example due to a combination of lexical bias and treatment-level over-generalization. The presence of strong surface cues such as GERD, reflux, and references to antacid use biases models toward conceptual matching, leading them to infer support even when the specific medication (aluminum hydroxide) is not mentioned.

几乎所有模型都在这个例子上失败,原因是词汇偏向和治疗层面的过度泛化叠加在一起。GERD、反流,以及对使用抗酸药的提及,这些强烈的表层线索会把模型推向概念匹配,从而在具体药物(氢氧化铝)并未被提及的情况下,仍推断出“有依据”。

In addition, models tend to collapse distinctions within therapeutic classes, treating different antacids as interchangeable. This error is further exacerbated by clinical complexity rather than context length: the note is dominated by severe acute pancreatitis and post-partum complications, while GERD appears as a secondary issue, making fine-grained verification of drug-level evidence particularly challenging. Together, these factors reveal a persistent difficulty in jointly verifying medical correctness and strict contextual grounding.

此外,模型往往抹平治疗类别内部的差别,把不同的抗酸药当成可以互换。这一错误被临床复杂性进一步放大,而不是被上下文长度放大:这份记录的主体是严重急性胰腺炎和产后并发症,GERD 只是次要问题,因而要在药物层面做细粒度的证据核实就特别困难。这些因素合在一起,揭示了同时核实医学正确性和严格语境锚定方面持续存在的困难。