Debating with More Persuasive LLMs Leads to More Truthful Answers

与更具说服力的大语言模型辩论可得到更真实的答案

Akbir Khan Affiliation:  University College London Correspondence to: akbir.khan.13@ucl.ac.uk    John Hughes Affiliation:  MATS Affiliation:  Speechmatics    Dan Valentine Affiliation:  MATS    Laura Ruis Affiliation:  University College London    Kshitij Sachan Affiliation:  Anthropic Affiliation:  Redwood Research    Ansh Radhakrishnan Affiliation:  Anthropic    Edward Grefenstette Affiliation:  University College London    Samuel R. Bowman Affiliation:  Anthropic    Tim Rocktäschel Affiliation:  University College London    Ethan Perez Affiliation:  Anthropic Affiliation:  FAR AI

Abstract

摘要

Common methods for aligning large language models (LLMs) with desired behaviour heavily rely on human-labelled data. However, as models grow increasingly sophisticated, they will surpass human expertise, and the role of human evaluation will evolve into non-experts overseeing experts. In anticipation of this, we ask: can weaker models assess the correctness of stronger models? We investigate this question in an analogous setting, where stronger models (experts) possess the necessary information to answer questions and weaker models (non-experts) lack this information but are otherwise as capable. The method we evaluate is debate, where two LLM experts each argue for a different answer, and a non-expert selects the answer. On the QuALITY comprehension task, we find that debate consistently helps both non-expert models and humans answer questions, achieving 76% and 88% accuracy respectively (naive baselines obtain 48% and 60%). Furthermore, optimising expert debaters for persuasiveness in an unsupervised manner improves non-expert ability to identify the truth in debates. Our results provide encouraging empirical evidence for the viability of aligning models with debate in the absence of ground truth.

对齐大语言模型（LLM）与期望行为的常见方法高度依赖人类标注数据。但随模型日益精巧，它们将超越人类专长，人类评估的角色将演变为非专家监督专家。为此我们预先发问：较弱模型能否评估较强模型的正确性？我们在一个类比设定中研究该问题：较强模型（专家）掌握回答问题所需信息，较弱模型（非专家）缺乏该信息但能力其余方面相当。所评估的方法是辩论：两名 LLM 专家各为不同答案辩护，由非专家选定答案。在 QuALITY 阅读理解任务上，辩论持续帮助非专家模型与人类作答，准确率分别达 76% 与 88%（朴素基线为 48% 与 60%）。此外，以无监督方式优化专家辩手的说服力，会提高非专家在辩论中识别真相的能力。结果为在无真值标签时用辩论对齐模型的可行性提供了令人鼓舞的经验证据。

1 Introduction

1 引言

Keywords:

Most existing approaches to align LLMs rely on the availability of labelled data (Ouyang et al. 2022; Menick et al. 2022). However, faced with models that can answer questions in increasingly broad context, obtaining such data requires domain expertise (OpenAI 2023; Gemini Team et al. 2023). As these systems continue to advance, they will surpass expert knowledge. Consequently, there will be no ground truth to rely on, rendering most alignment approaches unusable. We need mechanisms that provide scalable oversight (Amodei et al. 2016; Christiano et al. 2018; Irving et al. 2018; Bowman et al. 2022): alignment methods that scale with model capability.

关键词：

多数现有 LLM 对齐方法依赖标注数据（Ouyang et al. 2022；Menick et al. 2022）。但面对能在日益宽广语境中答题的模型，获取此类数据需要领域专长（OpenAI 2023；Gemini Team et al. 2023）。系统继续进步后将超越专家知识，届时将无真值可依，多数对齐方法将不可用。我们需要提供可扩展监督的机制（Amodei et al. 2016；Christiano et al. 2018；Irving et al. 2018；Bowman et al. 2022）：能随模型能力扩展的对齐方法。

Figure 1: Across all judges, using debate helps to answer questions better than baselines. Accuracy of non-expert judges using expert(s) to answer comprehension questions, and base- and toplines. Experts and expert judges have access to the underlying text. Non-expert judges do not have access to the underlying text. Naive judges have no access to experts. Protocols where two experts adversarially compete (debate and interactive debate) outperform non-adversarial baselines (consultancy and naive). Across all protocols, human judges outperform LLMs, suggesting limitations of current LLMs for automated scalable oversight. Experts use best-of-16 sampling with GPT-4-Turbo.

图 1：在所有裁判上，使用辩论都比基线更好地回答问题。 非专家裁判借助专家回答理解题的准确率，以及底线与顶线。专家与专家裁判可访问原文。非专家裁判不可访问原文。朴素裁判无专家可依。两名专家对抗竞争的协议（辩论与交互式辩论）优于非对抗基线（咨询与朴素）。所有协议中人类裁判均优于 LLM，提示当前 LLM 用于自动化可扩展监督仍有局限。专家使用 GPT-4-Turbo 的 best-of-16 采样。

Figure 2: An illustration of our set-up. We investigate three protocols for evaluating stronger models (experts) in the QuALITY dataset. Expert models, who can access the underlying text from a project Gutenberg science-fiction story, argue for a specific answer to a comprehension question. Weaker (non-expert) judges, who cannot access the underlying text, evaluate the arguments and choose an answer. In debate, two experts simultaneously present arguments for a number of rounds. In interactive debate, the judge may ask clarifying questions after each round. In consultancy, a single expert argues for one of the two answers while the judge asks questions.

图 2：设定示意。 我们在 QuALITY 数据集上研究三种评估较强模型（专家）的协议。专家模型可访问 Project Gutenberg 科幻故事原文，为理解题的某一具体答案辩护。较弱（非专家）裁判无法访问原文，评估论证并选定答案。在辩论中，两名专家同步进行若干轮论证。在交互式辩论中，每轮后裁判可提澄清问题。在咨询中，单名专家为两答案之一辩护，同时裁判提问。

A promising paradigm to overcome the need for ground-truth labels is using less capable models to align stronger models (Cotra 2021; Bowman et al. 2022; Burns et al. 2023). Fundamental to these approaches is the assumption that it is easier to identify or critique the correct answer than it is to generate it (Goodfellow et al. 2014; Bai et al. 2022; Saunders et al. 2022). However, evaluating the correctness of a critique can be difficult, in which case a critique of a critique can help. In similar spirit, Irving et al. 2018 propose debate as a method to align superhuman AI. The idea is that a human or a weaker model can judge the correctness of an answer more accurately by using the adversarial critiques that are generated by a debate.

一种有望摆脱对真值标签依赖的范式，是用较弱能力模型对齐较强模型（Cotra 2021；Bowman et al. 2022；Burns et al. 2023）。这些方法的根本假设是：识别或批评正确答案比生成它更容易（Goodfellow et al. 2014；Bai et al. 2022；Saunders et al. 2022）。但评估批评的正确性可能很难，此时对批评的再批评会有帮助。类似精神下，Irving et al. 2018 提出用辩论对齐超人类 AI：人类或较弱模型可借助辩论产生的对抗性批评，更准确地判断答案正确性。

Towards addressing the challenge of evaluating models without ground truth, we investigate the question: can debate[^1] aid weaker judges in evaluating stronger models? To simulate stronger and weaker models, we use information-asymmetric debates in a reading comprehension task (Michael et al. 2023; Radhakrishnan 2023). We give the debaters full access to the underlying text while judges have no access to the text. This setup renders the debaters experts (i.e. stronger) and the judges non-experts (i.e. weaker) in the task. Experts are provided with a quote tool, which allows them to present externally verified quotes from the text. The judges use the arguments from the debate to answer the reading comprehension question. We test this setup for both human and LLM judges and evaluate against a non-adversarial baseline called consultancy, in which a single expert model argues for one particular answer.

为应对无真值时评估模型的挑战，我们研究：辩论[^1]能否帮助较弱裁判评估较强模型？为模拟强弱模型，我们在阅读理解任务中使用信息不对称辩论（Michael et al. 2023；Radhakrishnan 2023）。辩手可完整访问原文，裁判则不可。该设定使辩手成为专家（更强）、裁判成为非专家（更弱）。专家配备引用工具，可展示经外部核验的原文引文。裁判用辩论中的论证回答理解题。我们对人类与 LLM 裁判均测试该设定，并与非对抗基线「咨询」（单名专家为某一答案辩护）对照。

To investigate how debate will scale with increased model capabilities, we introduce a metric called persuasiveness. Persuasiveness is measured by judge approval, meaning it does not require ground-truth labels. We optimise model outputs for persuasion with inference-time methods, generating a range of expert models. The resulting models are evaluated by unseen LLM judges whose preferences have not been used in optimisation. Finally, we evaluate debate and consultancy with human judges, conducting a large-scale study with over a thousand hours of annotator time spent providing judgements. Our findings are as follows:

为研究辩论如何随模型能力扩展，我们引入说服力指标。说服力由裁判认可度度量，因而无需真值标签。我们用推理时方法优化模型输出以提升说服力，得到一系列专家模型；再由未见过、偏好未被用于优化的 LLM 裁判评估。最后，我们用人类裁判评估辩论与咨询，开展超过一千小时标注时间的大规模研究。主要发现如下：

1. Weak judges can supervise strong debaters. The result holds both when using LLMs and when using humans to judge outputs. Specifically, for the most persuasive models we find that non-expert human judges achieve 88% and non-expert LLM judges achieve 76% accuracy with debate, where naive performance is 60% and 48% respectively. Debate also outperforms the single-model baseline consultancy, with which human and LLM judges achieve 78% and 54%, respectively.
2. Optimising debaters for persuasiveness improves a judge’s ability to identify truth in debates. Using inference-time methods such as best-of-$N$ and critique-and-refinement, we find that models optimised for judge approval (i.e. persuasiveness) become better at arguing for the correct answer relative to arguing for the incorrect answer. In particular, using persuasive debaters leads to higher judge accuracy. By contrast, judge accuracy decreases as consultants are more persuasive. We find that this effect generalises to unseen judges whose preferences have not been optimised for.
3. Human judges are well calibrated and achieve a lower error rate with debate. Based on confidence ratings of human judges, we find that judges are better calibrated with debate than with consultancy. Furthermore, debate achieves higher judge accuracy than consultancy across all rejection thresholds.

1. 弱裁判可以监督强辩手。 用 LLM 与人类裁判均成立。具体地，对最具说服力的模型，非专家人类裁判借助辩论达 88% 准确率，非专家 LLM 裁判达 76%；朴素表现分别为 60% 与 48%。辩论亦优于单模型基线咨询：人类与 LLM 裁判在咨询下分别为 78% 与 54%。
2. 为说服力优化辩手，会提高裁判在辩论中识别真相的能力。 使用 best-of-$N$、批评– refinement 等推理时方法，为裁判认可（即说服力）优化的模型，相对为错误答案辩护，更擅长为正确答案辩护；用有说服力的辩手时裁判准确率更高。相反，顾问越有说服力，裁判准确率越低。该效应可推广到偏好未被优化过的未见裁判。
3. 人类裁判校准良好，且辩论下错误率更低。 基于人类裁判的置信度评分，辩论下校准优于咨询；在所有拒答阈值上，辩论的裁判准确率均高于咨询。

Although greater access to information is only one way in which future models may be stronger than their supervisors, our results pave the way for further research on adversarial oversight methods. We provide empirical evidence in one domain indicating that as models get more capable, debate enables scalable oversight by both human and model judges.

尽管信息优势只是未来模型可能强于其监督者的方式之一，我们的结果为进一步研究对抗式监督方法铺路。我们在一个领域提供经验证据：随模型能力增强，辩论使人类与模型裁判都能实现可扩展监督。

2 Methods

2 方法

We are concerned with protocols that enable non-experts to elicit truth from experts. Here, we discuss the protocols we investigated and the task setting in which they are evaluated, as illustrated in Figure 2. Furthermore, we introduce unsupervised metrics for evaluating the strength of the debaters.

我们关心使非专家从专家处引出真相的协议。此处讨论所研究的协议及评估所用任务设定，如图 2 所示。此外引入无监督指标以评估辩手强度。

2.1 Protocols

2.1 协议

Debate — We first introduce debate, a protocol in which two expert models (debaters) argue for opposing answers to a question. Debate runs for a pre-determined number of rounds $N$, during which a transcript of the debaters’ arguments is kept. In each round, debaters see the arguments from previous rounds and simultaneously generate their arguments for the next round. After $N$ rounds, a judge reads the transcript and attempts to choose the correct answer. Each debater tries to convince the judge to pick their answer, and the judge is tasked with picking the correct answer. The adversarial nature of the protocol stems from the conflicting incentives between the debaters, as each debater strategically presents arguments to explain why their opponent’s claims are false. At the start of a round, debaters receive nearly-identical prompts explaining the game, their assigned answer, and the current transcript.

辩论 — 首先介绍辩论：两名专家模型（辩手）为问题的对立答案辩护。辩论进行预定的 $N$ 轮，期间保留辩手论证的记录。每轮中，辩手看到此前各轮论证，并同步生成下一轮论证。$N$ 轮后，裁判阅读记录并尝试选择正确答案。每位辩手试图说服裁判选己方答案，裁判任务是选出正确答案。协议的对抗性源于辩手之间冲突的激励：每位辩手策略性地提出论证，说明对手主张为何为假。每轮开始时，辩手收到几乎相同的提示，说明规则、各自被分配的答案以及当前记录。

Interactive debate — Similar to debate, interactive debate uses two experts arguing for opposing answers. In addition to the mechanisms above, after each round, the judge may make a statement. This enables the judge to highlight sub-problems, ask for clarifications, and direct questions to either debater. The statement is appended to the transcript and is visible to all players for the rest of the debate.

交互式辩论 — 与辩论类似，交互式辩论由两名专家为对立答案辩护。除上述机制外，每轮后裁判可发表陈述，从而突出子问题、请求澄清，并向任一方辩手提问。该陈述追加到记录中，供其余辩论过程中所有玩家可见。

Consultancy (Baseline) — For comparison with debate, we use the consultancy baseline established by (Michael et al. 2023). In consultancy, a single expert model (the consultant) is assigned a specific answer and aims to persuade the judge that their answer is correct. The judge aims to elicit the correct answer, asking the consultant probing questions. Consultancy runs for a pre-determined number of rounds $N$ (fixed to be the same as debate), in which the consultant and judge sequentially make statements. At the end of consultancy, the judge decides which answer to choose. At the start of each round, the consultant is provided with a prompt containing the rules of the game, their assigned answer and the current transcript. Consultancy builds up a transcript of a dialogue between the consultant and judge as rounds continue. In all our evaluations, we run consultancy for both the correct and incorrect answers; producing the same 50/50 prior as debate.

咨询（基线） — 为与辩论比较，我们使用 Michael et al. 2023 建立的咨询基线。咨询中，单名专家模型（顾问）被分配某一答案，旨在说服裁判其答案正确；裁判旨在引出正确答案，并向顾问提出探究性问题。咨询进行预定的 $N$ 轮（与辩论相同），顾问与裁判依次发言。结束时裁判决定选择哪个答案。每轮开始时，顾问收到含规则、被分配答案与当前记录的提示。咨询随轮次推进积累顾问与裁判的对话记录。全部评估中，我们分别对正确与错误答案运行咨询，以产生与辩论相同的 50/50 先验。

2.2 Experimental Details

2.2 实验细节

To operationalise the oversight problem, we conduct information-asymmetric experiments (Michael et al. 2023) in which non-experts do not have access to the necessary information to answer questions, but the experts do.

为操作化监督问题，我们进行信息不对称实验（Michael et al. 2023）：非专家没有回答问题所需信息，专家则有。

Task — We evaluate the ability of non-expert judges to answer questions from the reading comprehension dataset “Question Answering with Long Input Texts, Yes!” (Pang et al. 2022, QuALITY;). Judges are not allowed access to the original comprehension text, restricting their ability to answer questions and inducing their non-expert status. We motivate this setting and provide implementation recommendations in Appendix C. We use texts from the Project Gutenberg science-fiction story subset, which have a median length of approximately 7000 tokens. From these questions, we select the hard subset, where all untimed annotators chose the correct answer and rated the answer as unambiguous. For each question, we provide two answers: the correct answer and the best distractor. We use two data splits for LLM judge experiments: $T_{L}$ (400 train set questions) and $D_{L}$ (291 development set questions). For human experiments, where a story can only appear once, we use $T_{H}$ (153 drawn from both sets) and $D_{H}$ (47 drawn from both sets). For question filtering details, see Appendix D.1.

任务 — 我们评估非专家裁判回答阅读理解数据集「Question Answering with Long Input Texts, Yes!」（Pang et al. 2022，QuALITY）问题的能力。裁判不得访问原理解文本，从而限制答题能力并诱导非专家地位。设定动机与实现建议见附录 C。我们使用 Project Gutenberg 科幻故事子集文本，中位长度约 7000 token。从这些问题中选取困难子集：所有无计时标注者都选对且将答案评为无歧义。每题提供两答案：正确答案与最佳干扰项。LLM 裁判实验用两个数据划分：$T_{L}$（训练集约 400 题）与 $D_{L}$（开发集约 291 题）。人类实验中一故事只能出现一次，用 $T_{H}$（两集合共 153 题）与 $D_{H}$（共 47 题）。题目过滤细节见附录 D.1。

Quote verification tool — In each protocol, expert models have access to a quotation tool to present quotes from the text in tags. Similar to Radhakrishnan 2023, if the quote directly matches a portion of the text, the tool generates verified else, it generates unverified tags (see Appendix D.2). The judge is instructed to trust only verified quotes (see Appendix D.8). We assess the importance of quoting in Appendix A.6.

引用核验工具 — 各协议中，专家模型可访问引用工具，在 标签中展示原文引文。类似 Radhakrishnan 2023：若引文与原文某段直接匹配，工具生成已核验的 ，否则生成未核验的 （见附录 D.2）。裁判被指示仅信任已核验引文（见附录 D.8）。引用重要性评估见附录 A.6。

Number of rounds and word limit — We run protocols for three rounds. For ablations with the number of rounds, see Appendix A.8. To control for the quantity of information presented to the judge across protocols and mitigate the LLM judge verbosity bias, we restrict transcripts to 900 words in total, limiting consultants to 300 words per argument and debaters to 150 words (see Appendix D.6).

轮数与字数限制 — 协议进行三轮。轮数消融见附录 A.8。为控制各协议向裁判呈现的信息量，并缓解 LLM 裁判的冗长偏好，我们将记录总字数限制为 900，顾问每轮论证限 300 词、辩手限 150 词（见附录 D.6）。

Language models — We evaluate five large language models as debaters and consultants: GPT-4-Turbo, GPT-3.5-Turbo (OpenAI 2023), Claude 2.1, Claude 1.3 (Bai et al. 2022) and Mixtral 8x7B (Jiang et al. 2024). Debaters are presented with an egocentric view of the transcript, in which their arguments appear first. We present all models with a scratchpad (Wei et al. 2022; Nye et al. 2022), which allows the model to output tokens enclosed in tags that other players will not see (see Appendix G.1).

语言模型 — 我们评估五种大语言模型作为辩手与顾问：GPT-4-Turbo、GPT-3.5-Turbo（OpenAI 2023）、Claude 2.1、Claude 1.3（Bai et al. 2022）与 Mixtral 8x7B（Jiang et al. 2024）。辩手看到以自我为中心的记录视图，己方论证排在前面。所有模型配备草稿本（Wei et al. 2022；Nye et al. 2022），可在 标签中输出其它玩家看不到的 token（见附录 G.1）。

Figure 3: Stronger debaters result in better judge accuracy. We conduct a tournament between four different LLMs under best-of-n (bo$N$) and critique-and-refinement (c$N$) optimisations. (Left) Elo ratings calculated from matches with different debaters: Debaters arguing for the correct answer are more persuasive than debaters arguing for the incorrect answer, suggesting there is an advantage to being assigned the correct answer. (Middle) We quantify this advantage as (correct rating - incorrect rating) and plot against aggregate rating (an unsupervised metric). When debaters become more persuasive, they get relatively better at arguing the correct answer than the incorrect answer, and the advantage becomes larger. (Right) On a withheld dataset, we run debates with copies of the same models. We find that persuasive debaters have a large advantage, which in self-play matches leads to high judge accuracy. Claude2.1 is the zero reference for ratings.

图 3：更强辩手带来更高裁判准确率。 我们在四种不同 LLM 上，用 best-of-n（bo$N$）与批评–refinement（c$N$）优化做锦标赛。（左）由不同辩手对阵算出的 Elo：为正确答案辩护的辩手比错误答案更具说服力，表明被分配正确答案有优势。（中）将该优势量化为（正确评分 − 错误评分），相对总体评分（无监督指标）作图。辩手更有说服力时，相对错误答案，它们为正确答案辩护相对更好，优势变大。（右）在留出数据集上，用同一模型的副本跑辩论。有说服力的辩手优势大，在自我对弈中带来高裁判准确率。Claude2.1 为评分零点。

To generate a large set of different debater capability levels for our LLM judge experiments, we apply inference-time optimisation using best-of-$N$ (bo$N$) sampling. With bo$N$, models are sampled $N$ times, and a preference model is used to select the most persuasive arguments. The preference model is given a prompt to optimise for judge feedback. For a subset of models, we additionally use critique-and-refinement (c$N$) (Saunders et al. 2022), in which another model is used to produce $N$ critiques; a preference model then grades these and the most helpful is used to refine the answer. For all preference models, we use the log probabilities from GPT-4-Turbo. We document prompt iteration in Appendix D.3 and learning with examples in Appendix A.9.

为在 LLM 裁判实验中生成大量不同能力水平的辩手，我们用 best-of-$N$（bo$N$）采样做推理时优化：模型采样 $N$ 次，由偏好模型选出最具说服力的论证。偏好模型收到优化裁判反馈的提示。对部分模型，我们额外使用批评–refinement（c$N$）（Saunders et al. 2022）：另一模型产生 $N$ 条批评，偏好模型打分后用最有帮助者 refine 答案。全部偏好模型使用 GPT-4-Turbo 的对数概率。提示迭代见附录 D.3，示例学习见附录 A.9。

For LLM judging, we used GPT-4-Turbo, Mixtral-8x7B, GPT-3.5-Turbo and Claude 2.1. Prompts are provided in Appendix G.2. Similar to Zheng et al. 2023, we found strong biases with LLM judges. To mitigate positional biases, we run judgements over each debate twice - once with the original transcript and once with the debater order swapped. To mitigate verbosity biases, we apply strict word limits. We document judge biases in Appendix A.7 and mitigation techniques in Appendix D.6.

LLM 裁判使用 GPT-4-Turbo、Mixtral-8x7B、GPT-3.5-Turbo 与 Claude 2.1。提示见附录 G.2。类似 Zheng et al. 2023，我们发现 LLM 裁判有很强偏差。为缓解位置偏差，每场辩论裁判两次——一次原记录顺序，一次交换辩手顺序。为缓解冗长偏差，施加严格字数限制。裁判偏差见附录 A.7，缓解技术见附录 D.6。

Cross-play tournament — To investigate how debate scales with model capabilities, we require a way to compare debaters. We run cross-play debates, in which different debaters argue against one another. To compare a large set of debaters, we use a tournament to generate informative matches. For $n$ debaters, it is not feasible to run all possible matches as this requires $O(n^{2})$ matches. Instead, we use a Swiss-style tournament to decide matches, which produces rankings in $O(n\log n)$ matches (see Appendix D.4).

交叉对弈锦标赛 — 为研究辩论如何随模型能力扩展，需要比较辩手的方法。我们跑交叉对弈辩论，让不同辩手互相对阵。为比较大量辩手，用锦标赛生成信息量大的对阵。对 $n$ 名辩手，跑全部对阵需 $O(n^{2})$ 场，不可行；改用瑞士制锦标赛决定对阵，以 $O(n\log n)$ 场产生排名（见附录 D.4）。

Human judges --- We recruit 30 human judges via the referral-based annotator platform Pareto.[^2] Annotators judge debates via a web interface in which they are presented with the question, transcript and answers. The 30 annotators go through a training phase and are filtered down to 21 based on performance in a test phase. To avoid any optimisation based on ground truth labels (Perez et al. 2021), we filter annotators based on inter-annotator agreement and the quality of explanations they provide (see Appendix E.2).

人类裁判 — 我们通过基于推荐的标注平台 Pareto[^2] 招募 30 名人类裁判。标注者经网页界面裁判辩论，界面呈现问题、记录与答案。30 人经训练阶段后，按测试阶段表现筛至 21 人。为避免基于真值标签的优化（Perez et al. 2021），我们按标注者间一致性及其解释质量筛选（见附录 E.2）。

Additional base- and toplines — We introduce a naive judge and an expert judge to evaluate the worst and best-case performance. For the naive judge, we present only the question and possible answers to a non-expert model; this helps identify if there is any contamination of the dataset in the non-experts’ training. For the expert judge, we present the full text to demonstrate a reasonable upper bound for what a non-expert judge can hope to achieve. For human experiments, we did not run an expert judge baseline, but use Pang et al. 2022 who report annotators with access to the text achieve 89% accuracy on the hard subset. We filter for 100% untimed accuracy, and as such, expect 89% to be a lower bound for expert human performance.

额外底线与顶线 — 我们引入朴素裁判与专家裁判，以评估最差与最佳情形表现。朴素裁判只向非专家模型呈现问题与可能答案，用于识别非专家训练数据是否污染。专家裁判呈现全文，以展示非专家裁判可期望的合理上界。人类实验未跑专家裁判基线，而采用 Pang et al. 2022：有文本访问的标注者在困难子集上达 89% 准确率。我们过滤要求无计时准确率 100%，故预期 89% 是专家人类表现的下界。

2.3 Measuring Debater and Judge Performance

2.3 度量辩手与裁判表现

We are interested in oversight protocols with no access to labels, but existing work uses metrics that assume access to the ground truth. In this section, we introduce metrics to measure debater capabilities without this assumption. Given a dataset $\mathcal{D}$ consisting of $N$ questions ($q$) and candidate answers ($a_{1},a_{2}$), $\mathcal{D}=\{q_{i},a_{i1},a_{i2}\}_{i=1}^{N}$, a debate match is defined over a dataset by $(D_{1},D_{2},J)$, comprising two models $D_{1}$ and $D_{2}$, and a Judge $J$. The first argument refers to which model is assigned $a_{1}$ to support (in this case $D_{1}$), and the second argument refers to which model is assigned $a_{2}$. A consultancy match with a consultant $D$ is expressed as $(D,-,J)$ or $(-,D,J)$.

我们关心无标签可依的监督协议，但现有工作使用的指标假定可访问真值。本节引入不依赖该假定的辩手能力度量。给定由 $N$ 个问题（$q$）与候选答案（$a_{1},a_{2}$）构成的数据集 $\mathcal{D}=\{q_{i},a_{i1},a_{i2}\}_{i=1}^{N}$，一场辩论对阵在数据集上由 $(D_{1},D_{2},J)$ 定义，包含两模型 $D_{1}$、$D_{2}$ 与裁判 $J$。第一个参数指被分配支持 $a_{1}$ 的模型（此处为 $D_{1}$），第二个指被分配 $a_{2}$ 的模型。顾问为 $D$ 的咨询对阵记为 $(D,-,J)$ 或 $(-,D,J)$。

Figure 4: Stronger consultants result in worse judge accuracy. (Left) Win rate vs best-of-N for consultants by assignment. Incorrect consultants become relatively more persuasive than the correct consultants when under optimisation. (Middle) Judge Accuracy vs best-of-N: While optimising the consultants to be more persuasive, the overall accuracy of consultancy decreases. (Right) Consultancy accuracy against win rate. We take a larger set of consultants (including those with critique-and-refinement augmentations) and demonstrate that as consultants improve their win rate, the accuracy of the consultancy protocol degrades. For p-values less than 0.05, we fit a line of best fit augmented with an extra data point: 50% accuracy at a win rate of 100%. We use GPT-4-Turbo as a judge.

图 4：更强顾问导致更差裁判准确率。（左）顾问按分配相对 best-of-N 的胜率。优化下错误顾问相对正确顾问更有说服力。（中）裁判准确率相对 best-of-N：优化顾问更有说服力时，咨询总体准确率下降。（右）咨询准确率相对胜率。取更大顾问集合（含批评–refinement 增强），表明顾问胜率提高时咨询协议准确率下降。对 $p$ 值小于 0.05，拟合最佳直线并附加一点：胜率 100% 时准确率 50%。裁判为 GPT-4-Turbo。

Win rate — We define the win rate as the frequency with which a judge picks a specific debater’s answer. For the match $(D_{1},D_{2},J)$, the win rate, $\omega_{1}$, for Debater $D_{1}$ is:

胜率 — 胜率定义为裁判选中某辩手答案的频率。对对阵 $(D_{1},D_{2},J)$，辩手 $D_{1}$ 的胜率 $\omega_{1}$ 为：

$$\omega_{1}(D_{1},D_{2},J)=\frac{1}{N}\sum_{i=1}^{N}\mathds{1}\{J(q_{i},a_{i1},a_{i2})=a_{i1}\} \tag{1}$$

In practice, this win rate depends on the assignment of the answer choice, which gives an unfair advantage, as some assignments may be easier to argue for than others (e.g. some answers are easier to defend than others). To mitigate this, we flip assignments such that $D_{1}$ and $D_{2}$ argue for the opposite assignments and then take the average, $\bar{\omega}_{1}$. Assuming debaters perform equally well across all questions, we say $D_{1}$ is more persuasive than $D_{2}$ if:

实践中，该胜率依赖答案分配，会带来不公平优势，因为某些分配比其它更容易辩护（例如某些答案比其它更好守）。为缓解，我们翻转分配使 $D_{1}$ 与 $D_{2}$ 为相反分配辩护，再取平均 $\bar{\omega}_{1}$。假定辩手在所有问题上表现相当，若满足下式则称 $D_{1}$ 比 $D_{2}$ 更有说服力：

$$\bar{\omega}_{1}(D_{1},D_{2},J)>\frac{1}{2} \tag{2}$$

Elo rating — To capture a general notion of persuasiveness, we evaluate aggregate win rates for each debater across a population of co-debaters. We parameterise win rates by a latent skill, using the Elo ranking metric (Elo 1978). We calculate ratings by minimising predicted win rate error (see Appendix D.5). Aggregate rating is defined as the latent skill for a debater independent of assignment. Given two debaters, $D_{1}$ and $D_{2}$, and a judge $J$, we define each player’s aggregate rating, $E_{1}$ and $E_{2}$, as the solution to the equality:

Elo 评分 — 为刻画一般意义上的说服力，我们在共辩手群体上评估各辩手的总体胜率。用 Elo 排名指标（Elo 1978）以潜在技能参数化胜率。通过最小化预测胜率误差计算评分（见附录 D.5）。总体评分定义为与分配无关的辩手潜在技能。给定两辩手 $D_{1}$、$D_{2}$ 与裁判 $J$，各玩家总体评分 $E_{1}$、$E_{2}$ 定义为等式解：

$$\bar{\omega}_{1}(D_{1},D_{2},J)=\frac{1}{1+10^{(E_{2}-E_{1})/400}} \tag{3}$$

Figure 5: Different judges produce different ratings and accuracies for the same self-play debates. We find that GPT-4-Turbo is the strongest judge, achieving higher accuracy for all debates. Claude2.1 is set as the zero reference policy.

图 5：不同裁判对同一自我对弈辩论给出不同评分与准确率。 发现 GPT-4-Turbo 是最强裁判，对所有辩论准确率更高。Claude2.1 设为零点参考策略。

Win rate and aggregate rating are metrics that can be used to optimise oversight protocols without ground-truth labels.

胜率与总体评分可用于在无真值标签时优化监督协议。

To understand truthfulness in debates, we also consider the win rate $\omega_{C}$ conditioned on being assigned all correct answers (assuming ground truth labels). We calculate a players’ correct rating $E^{C}$ and incorrect rating $E^{I}$ as:

为理解辩论中的真实性，我们还考虑在被分配全部正确答案时的胜率 $\omega_{C}$（假定有真值标签）。玩家的正确评分 $E^{C}$ 与错误评分 $E^{I}$ 算为：

$$\omega_{C}(D_{1},D_{2},J)=\frac{1}{1+10^{(E^{I}_{2}-E^{C}_{1})/400}} \tag{4}$$

Figure 6: Human judges, using debate, achieve high accuracy and are well-calibrated. (Left) Judge accuracy of different protocols: Debate and Interactive Debate outperform their respective baselines. Human judges struggle when using incorrect consultants. (Middle) Selective accuracy vs coverage rate: If the cost of misclassification is high, we can threshold by judge confidence and reject judgements with low confidence. Over all rejection thresholds, debate protocols achieve a higher judge accuracy on the remaining test set than consultancy. (Right) Calibration plots for judge confidences: Judges in debate achieve lower Brier scores, indicating better calibration. Across debate protocols, human judges are under-confident in their answers. Error bars are presented as 95% C.I.

图 6：人类裁判使用辩论达到高准确率且校准良好。（左）不同协议的裁判准确率：辩论与交互式辩论优于各自基线。人类裁判在错误顾问下表现困难。（中）选择性准确率相对覆盖率：若错分代价高，可按裁判置信度阈值并拒绝对低置信判断。在所有拒答阈值上，辩论协议在剩余测试集上的裁判准确率均高于咨询。（右）裁判置信度校准图：辩论中裁判 Brier 分数更低，校准更好。各辩论协议中人类裁判对答案偏于不自信。误差条为 95% 置信区间。

Judge accuracy — In line with the original motivation for debate, we calculate judge accuracy, $\alpha$, as the accuracy over a match between copies of the same model; $\alpha(D,J)=\alpha(D,D,J)$. As the consultant’s intention is unknown to a judge, we take accuracy over both potential matches for consultancy $\alpha(D_{1},J)=\frac{1}{2}(\alpha(D_{1},-,J)+\alpha(-,D_{1},J))$.

裁判准确率 — 按辩论的原始动机，裁判准确率 $\alpha$ 定义为同一模型副本之间对阵的准确率；$\alpha(D,J)=\alpha(D,D,J)$。因顾问意图对裁判未知，咨询准确率取两种潜在对阵的平均 $\alpha(D_{1},J)=\frac{1}{2}(\alpha(D_{1},-,J)+\alpha(-,D_{1},J))$。

3 Results with Language Model Judges

3 语言模型裁判结果

Insight 1: Debate helps LLM judges answer hard questions better than consultancy. We perform hyper-parameter and prompt optimisation for debaters by optimising for win rate (see Appendix D.7). We evaluate each protocol on test sets ($T_{H}$, $T_{L}$), using GPT-4-Turbo and best-of-16 sampling for both debaters and consultants. To evaluate performance, we report judge accuracy along with standard error of the mean. We find that all protocols help non-experts answer questions better than the naive judge baseline (see Figure 1 and 9). Debate outperforms consultancy and, with stronger judges, reduces the gap with the expert judge (75.6% vs 92.5%), demonstrating promising performance at eliciting the truth. We provide sample transcripts in Appendix H.

洞见 1：辩论比咨询更能帮助 LLM 裁判回答难题。 我们通过优化胜率对辩手做超参与提示优化（见附录 D.7）。在测试集（$T_{H}$、$T_{L}$）上评估各协议，辩手与顾问均用 GPT-4-Turbo 与 best-of-16 采样。报告裁判准确率及均值标准误。发现所有协议都帮助非专家比朴素裁判基线答得更好（见图 1 与 9）。辩论优于咨询，且在更强裁判下缩小与专家裁判的差距（75.6% 对 92.5%），展示了引出真相的有希望表现。样本记录见附录 H。

Insight 2: Arguing for the correct answer provides an advantage to debaters, and this advantage increases with persuasiveness. In a cross-play tournament, we evaluate 20 different debaters against distinct opponents (e.g. no model plays against itself). Each debater is one of four different LLMs, augmented by a combination of critique-and-refinement or best-of-$N$. In each augmentation, we use a preference model without access to the text (like the judge model), thereby simulating optimising against the judge. Debaters play four matches each, with GPT-4-Turbo as a judge. Cross-play matches were generated on $D_{L}$, obtaining 80 win rates, which were used to generate Elo ratings (outcomes listed in Appendix F).

洞见 2：为正确答案辩护给辩手带来优势，且该优势随说服力增大。 在交叉对弈锦标赛中，我们评估 20 名不同辩手对阵不同对手（例如无模型自我对弈）。每名辩手来自四种 LLM 之一，并经批评–refinement 或 best-of-$N$ 组合增强。每次增强中，偏好模型无法访问文本（与裁判模型类似），从而模拟对裁判的优化。每名辩手打四场，裁判为 GPT-4-Turbo。交叉对弈在 $D_{L}$ 上生成，得到 80 个胜率并用于生成 Elo 评分（结果见附录 F）。

We present Elo ratings for each model in Figure 3-left and -middle. For a given model, we observe that models are more persuasive (obtain higher Elo) when assigned to the correct answer, and the advantage increases for models with higher aggregate Elo. This suggests that, for debates in QuALITY, telling the truth is easier than lying (a central assumption of Irving et al. 2018). We find this result holds across different judges (see Figure 5) and that the strongest indicator of persuasiveness is base LLM, with GPT-4-Turbo outperforming others. Moreover, models are less persuasive when using critique-and-refinement than when using best-of-$N$, suggesting generating useful critiques for refinement is more difficult than identifying good arguments.

图 3 左、中给出各模型 Elo。对给定模型，被分配正确答案时更有说服力（Elo 更高），且优势随总体 Elo 升高而增大。这表明在 QuALITY 辩论中，说真话比说谎更容易（Irving et al. 2018 的中心假设）。该结果对不同裁判成立（见图 5）；说服力最强指标是基座 LLM，GPT-4-Turbo 优于其它。此外，用批评–refinement 的模型不如 best-of-$N$ 有说服力，提示为 refinement 生成有用批评难于识别好论证。

Insight 3: Higher aggregate Elo rating leads to higher judge accuracy. Given a ranking of debaters, we next evaluate how aggregate Elo affects judge accuracy. We take each debater from the tournament and generate self-play debates (in which copies of the same model play against each other); we use GPT-4-Turbo to judge these debates. To ensure that our results generalise, we use separate questions for cross-play ($D_{L}$) and self-play ($T_{L}$) matches.

洞见 3：更高总体 Elo 带来更高裁判准确率。 有了辩手排名后，接着评估总体 Elo 如何影响裁判准确率。取锦标赛中每名辩手生成自我对弈辩论（同一模型副本互相对阵），用 GPT-4-Turbo 裁判。为确保结果可推广，交叉对弈（$D_{L}$）与自我对弈（$T_{L}$）使用不同问题。

We report judge accuracy against aggregate rating, noting a strong correlation between these two values (Figure 3-right). As debaters are optimised against the unsupervised metric of win rate (judge preference), judge accuracy improves, suggesting that training models to win debates may produce more truthful outcomes. This provides weak evidence that under optimal play debate produces truthful information (Irving et al. 2018), and suggests that with even stronger debaters, non-experts using debate can obtain higher accuracy. We analyse debater characteristics in Appendix A.4.

我们报告裁判准确率相对总体评分，二者强相关（图 3 右）。当辩手相对无监督指标胜率（裁判偏好）被优化时，裁判准确率提高，提示训练模型赢辩论可能产生更真实结果。这为「最优对弈下辩论产生真实信息」（Irving et al. 2018）提供弱证据，并提示有更强辩手时，用辩论的非专家可获更高准确率。辩手特征分析见附录 A.4。

Protocol	Interactive Judge	Final Judge
Static Debate	None	Human
Static Consultancy	GPT-4-Turbo	Human
Interactive Debate	Human	Human
Interactive Consultancy	Human	Human

Table 1: Human judge protocols. To control for human input, we consider cases with or without a human interactive judge.

协议	交互裁判	最终裁判
静态辩论	无	人类
静态咨询	GPT-4-Turbo	人类
交互式辩论	人类	人类
交互式咨询	人类	人类

表 1：人类裁判协议。为控制人类输入量，考虑有/无人类交互裁判的情形。

Insight 4: Higher consultant skill leads to worse accuracy. We also generate a large set of consultants using best-of-$N$ and critique-and-refinement augmentations. Here, we find that consultants get relatively more persuasive under optimisation when incorrect (see Figure 4-left). Since models are fine-tuned using RLHF, they tend to already perform well when advocating correct answers and poorly for incorrect answers. Optimisation might disproportionately improve consultants’ ability to advocate for incorrect answers as it provides an opportunity to explore deceptive approaches. This results in a degradation of judge performance, as the judge a priori does not know if a consultant is arguing the correct or incorrect answer (see Figure 4-middle and -right). We directly compare to debate in Appendix A.3.

洞见 4：顾问技能越高，准确率越差。 我们亦用 best-of-$N$ 与批评–refinement 增强生成大量顾问。此处发现：优化下错误顾问相对更有说服力（见图 4 左）。因模型经 RLHF 微调，为正确答案辩护时本已较好、为错误答案时较差；优化可能不成比例地提高顾问为错误答案辩护的能力，因其提供探索欺骗路径的机会。结果裁判表现下降——裁判先验不知顾问为正确还是错误答案辩护（见图 4 中、右）。与辩论的直接比较见附录 A.3。

Insight 5: More capable judges are better at identifying correct arguments across all debater strengths. To explore how different judge models affect debate performance, we re-run the same cross-play matches with Claude 2.1 and GPT-3.5-Turbo judges. Each judge produces different win rates, aggregate ratings and judge accuracy for each debate (see Figure 5). Strong judges generate a larger range of aggregate debater ratings than weak judges; they can distinguish between good arguments more easily, leading to higher accuracy across the full range of debater Elos. We find that even when preference and judge models are different LLMs, strong debaters improve debate accuracy. See approaches to improve judge performance in Appendix A.5.

洞见 5：更强裁判在所有辩手强度上更能识别正确论证。 为探索不同裁判模型如何影响辩论表现，我们用 Claude 2.1 与 GPT-3.5-Turbo 裁判重跑相同交叉对弈。各裁判对每场辩论给出不同胜率、总体评分与裁判准确率（见图 5）。强裁判产生的辩手总体评分范围大于弱裁判；它们更易区分好论证，从而在辩手 Elo 全范围内准确率更高。即便偏好模型与裁判模型是不同 LLM，强辩手仍提高辩论准确率。提高裁判表现的方法见附录 A.5。

4 Results with Human Judges

4 人类裁判结果

We are interested in understanding whether our LLM judge results generalise to human judges, whether human judges perform better than LLM judges, and any differences in how the protocols should be implemented when using humans. To control for the amount of human interaction required, we investigate two variants of each protocol (see Table 1). In static protocols, there is a fixed transcript with just a single human judge interaction at the end to choose the answer. In interactive protocols, the human judge interacts with the models throughout, e.g. by asking clarifying questions.

我们想了解 LLM 裁判结果是否推广到人类裁判、人类是否优于 LLM 裁判，以及使用人类时协议实现上有何差别。为控制所需人类交互量，我们研究各协议的两种变体（见表 1）。在静态协议中，记录固定，仅在最后有一次人类裁判交互以选定答案。在交互协议中，人类裁判全程与模型互动，例如提出澄清问题。

We conduct five weeks of iteration during which judges receive seven batches of questions (see Appendix 29). For the test phase, we release a batch of 47 questions per judge, which is completed over a period of four weeks. Answer position and assignment are randomised (see Appendix E). To reduce learning effects, judges are not given feedback after initial training. We collect 953 human judgements in the test phase, 2362 in the iteration phase, and 6476 in total.

我们进行五周迭代，裁判收到七批问题（见附录 29）。测试阶段每名裁判释放一批 47 题，四周内完成。答案位置与分配随机化（见附录 E）。为减少学习效应，初始训练后不给裁判反馈。测试阶段收集 953 条人类判断，迭代阶段 2362 条，合计 6476 条。

Insight 6: Debate protocols produce higher accuracy than consultancy protocols for human judges. We report judge accuracy over each protocol (Figure 6-left). We find that in both static and interactive settings, debate outperforms consultancy significantly ($p$=0.001, $p$=0.048, respectively). In consultancy, incorrect experts cause low judge accuracy, suggesting that it is easier to be misled in consultancy. Across all protocols, humans have higher judge accuracy than LLM judges (Figure 1).

洞见 6：对人类裁判，辩论协议准确率高于咨询协议。 报告各协议裁判准确率（图 6 左）。静态与交互设定下辩论均显著优于咨询（分别为 $p$=0.001、$p$=0.048）。咨询中错误专家导致低裁判准确率，提示咨询中更容易被误导。所有协议中人类裁判准确率均高于 LLM 裁判（图 1）。

Insight 7: More non-expert interaction does not improve accuracy. We find identical judge accuracy between static and interactive debate. This suggests that adding non-expert interactions does not help in information-asymmetric debates. This is surprising, as interaction allows judges to direct debates towards their key uncertainties.

洞见 7：更多非专家交互并不提高准确率。 静态与交互式辩论裁判准确率相同。这提示在信息不对称辩论中增加非专家交互并无帮助。令人惊讶，因为交互允许裁判把辩论导向其关键不确定点。

Insight 8: Highly confident debate judgements are reliable. Human judges give their answers as confidences between 5% and 95%. To assess the performance of human judges at varying confidence thresholds, we consider classification with rejection. For example, in static debate, rejecting judgements with confidences below 75% retains 65% of questions and achieves a judge accuracy of 94%. We find that across all rejection thresholds, debate protocols achieve a higher judge accuracy than consultancy protocols (Figure 6-middle). Human judges are also better calibrated using debate protocols than in consultancy protocols (Figure 6-right). We explore additional methods to improve human judge reliability in Appendix 27.

洞见 8：高置信的辩论判断可靠。 人类裁判以 5% 到 95% 的置信度给出答案。为评估不同置信阈值下表现，考虑带拒答的分类。例如静态辩论中，拒绝对置信度低于 75% 的判断可保留 65% 问题并达到 94% 裁判准确率。在所有拒答阈值上，辩论协议裁判准确率均高于咨询（图 6 中）。人类裁判在辩论协议下的校准亦优于咨询（图 6 右）。提高人类裁判可靠性的其它方法见附录 27。

Insight 9: LLM debaters are the bottleneck for higher judge accuracy. We conduct error analysis by reading transcripts and judge explanations from all cases where a human judge chooses the incorrect answer. For debate, the most common error is weak arguments from the correct debater, usually due to poor choice of quotes. This suggests that our debaters are far from optimal. For consultancy, the most common error is the judge failing to elicit key quotes from the incorrect consultant (see Appendix B.1).

洞见 9：LLM 辩手是更高裁判准确率的瓶颈。 我们对人类裁判选错答案的全部情形阅读记录与裁判解释做误差分析。辩论中最常见错误是正确辩手论证薄弱，通常因引文选择不佳。这提示我们的辩手远未最优。咨询中最常见错误是裁判未能从错误顾问处引出关键引文（见附录 B.1）。

5 Related Work

5 相关工作

Many approaches exist that attempt to supervise strong models (Christiano et al. 2018; Bowman et al. 2022). Similar to debate, methods attempt to exploit the fact that it is easier to identify a correct solution than it is to generate a correct solution (Christiano et al. 2017; Stiennon et al. 2020; Saunders et al. 2022). Other approaches encourage models to decompose their reasoning (Nye et al. 2022; Wei et al. 2022; Radhakrishnan et al. 2023; Yao et al. 2023), similar to how debate generates a transcript. Alternatively, we can develop inductive biases that allow stronger models to be supervised directly by weaker models (Burns et al. 2023).

存在许多尝试监督强模型的方法（Christiano et al. 2018；Bowman et al. 2022）。与辩论类似，一些方法试图利用「识别正确解比生成正确解更容易」（Christiano et al. 2017；Stiennon et al. 2020；Saunders et al. 2022）。另一些方法鼓励模型分解推理（Nye et al. 2022；Wei et al. 2022；Radhakrishnan et al. 2023；Yao et al. 2023），类似辩论生成记录。或者，可发展使较强模型直接被较弱模型监督的归纳偏置（Burns et al. 2023）。

Other approaches exist to augment human decision-making with human-AI teams. For example, combined teams can improve reasoning in credit risk prediction (Chromik et al. 2021). In comparison, we leverage more general LLMs, which can be applied over a series of tasks, e.g. learning how to generate jailbreaks (Nikola 2023). Human-AI teams have been shown to be overly confident in their suggestions (Bansal et al. 2021), whereas we find human judges in debate to consistently be underconfident.

亦有用人类–AI 团队增强人类决策的方法。例如联合团队可改进信用风险预测中的推理（Chromik et al. 2021）。相较之下，我们利用更通用的 LLM，可应用于一系列任务，例如学习如何生成越狱（Nikola 2023）。人类–AI 团队被证明对其建议过于自信（Bansal et al. 2021），而我们发现辩论中的人类裁判持续偏于不自信。

Irving et al. 2018 originally proposed the ‘debate game’ as a mechanism for training safe AI systems. Since then, a body of work has focused on verifying the usefulness for oversight (Barnes 2020; Parrish et al. 2022b; Parrish et al. 2022a; Michael et al. 2023). These studies are all conducted with human debaters, while in our work the debaters are LLMs. Using LLM debaters ensures that we can control for debater skill and investigate self-play debates.

Irving et al. 2018 最初提出「辩论游戏」作为训练安全 AI 系统的机制。此后一系列工作聚焦验证其监督效用（Barnes 2020；Parrish et al. 2022b；Parrish et al. 2022a；Michael et al. 2023）。这些研究均使用人类辩手，而我们的辩手是 LLM。使用 LLM 辩手确保我们能控制辩手技能并研究自我对弈辩论。

There is much previous work with LLM debaters (Perez et al. 2019; Michael et al. 2023; Radhakrishnan 2023; Du et al. 2023). In Perez et al. 2019, debates are conducted over comprehension, but debaters are limited to extracting relevant statements from a source text, not generating their own arguments. Michael et al. 2023 introduces the information-asymmetric debate setting for QuALITY but found no positive results when using LLM debaters. Their focus was primarily human debaters and, therefore, they did not consider cross-play win rates for improving debater capabilities. Concurrent to our work, Radhakrishnan 2023 conducts debaters in QuALITY; by contrast, their protocol is only a single turn of debate, focuses on training debaters via reinforcement learning, and does not use human judges.

已有许多使用 LLM 辩手的工作（Perez et al. 2019；Michael et al. 2023；Radhakrishnan 2023；Du et al. 2023）。Perez et al. 2019 在理解任务上辩论，但辩手仅限于从源文本抽取相关陈述，不生成自己的论证。Michael et al. 2023 为 QuALITY 引入信息不对称辩论设定，但使用 LLM 辩手时未发现正面结果；其重点主要是人类辩手，因此未考虑用交叉对弈胜率提升辩手能力。与我们同期，Radhakrishnan 2023 亦在 QuALITY 上跑辩手；相较之下其协议仅单轮辩论，聚焦用强化学习训练辩手，且未使用人类裁判。

6 Discussion, Limitations & Conclusion

6 讨论、局限与结论

In this work, we explore debate as a method to elicit more truthful answers from LLMs. We demonstrate that by allowing non-experts to judge a transcript between two experts, we can identify the correct answers to questions. Additionally, we show that this oversight mechanism can be automated with LLM judges. Although the original debate protocol that Irving et al. 2018 propose involves a stricter protocol in which only a sub-component has to be judged to validate the entire debate, our results show that judging over full debate transcripts is already useful for producing expert labels for data using only non-experts and untrustworthy experts. Our findings generalise to different base LLMs for both the expert debaters and non-expert judges, as well as to human judges. This indicates that the debate protocol is robust to variation in judge skill, which is important as models advance.

本文探索辩论作为从 LLM 引出更真实答案的方法。我们表明：让非专家裁判两名专家之间的记录，可识别问题的正确答案；此外该监督机制可用 LLM 裁判自动化。尽管 Irving et al. 2018 提出的原始辩论协议更严格——只需裁判子组件即可验证整场辩论——我们的结果显示，对完整辩论记录做裁判，已足以仅用非专家与不可信专家为数据产生专家级标签。发现可推广到专家辩手与非专家裁判的不同基座 LLM，以及人类裁判，表明辩论协议对裁判技能变化稳健，这对模型持续进步很重要。

Our work has important limitations. In our setup, the difference between strong and weak is only in access to information. In the future, stronger models may differ in reasoning ability or another skill. Furthermore, we evaluate models that have been fine-tuned with RLHF, which have a propensity for honesty; it is unclear if debate will be a suitable technique for deceptive models (Greenblatt et al. 2023; Hubinger et al. 2024).

工作有重要局限。我们的设定中，强弱差别仅在于信息访问。未来更强模型可能在推理能力或其它技能上不同。此外，我们评估的是经 RLHF 微调、倾向诚实的模型；辩论对欺骗性模型是否适用尚不清楚（Greenblatt et al. 2023；Hubinger et al. 2024）。

Finally, our results are limited to setups where the debaters can provide verified evidence to the judge (provided by the debater quote tool in our case). Without such a system, a debater arguing for the incorrect answer could simply create an alternative narrative in which their answer is correct (the judge, without access to the underlying story, would have no means to discover this). We expect that useful implementations of debate will be grounded in verifiable evidence, e.g. quoting trusted sources or data. Other domains have different notions of evidence, and debater tool-use will take different forms. For example, arguments in mathematical debates may require access to simulators, while physics debates can be grounded in experimental data. We posit that such tool-use capabilities will help judges to decide debates more quickly and accurately. Our work uses reading comprehension over short stories of approximately 8000 tokens. There is an open question of how well our approach will generalise to domains where a much larger set of evidence is required, e.g. quoting published literature when debating a science question. Stronger models may also possess ‘parametric knowledge’, which is not easily citeable or easily verifiable by any specific tool.

最后，结果限于辩手能向裁判提供经核验证据的设定（本例由辩手引用工具提供）。若无此系统，为错误答案辩护的辩手可干脆编造其答案正确的替代叙事（裁判无法访问原文故事，无从发现）。我们预期有用的辩论实现会锚定于可核验证据，例如引用可信来源或数据。其它领域有不同的证据概念，辩手工具使用形式也会不同。例如数学辩论中的论证可能需要模拟器，物理辩论可锚定于实验数据。我们主张此类工具使用能力将帮助裁判更快、更准地裁决辩论。本文在约 8000 token 短篇故事上做阅读理解。方法在需要大得多证据集的领域（例如辩论科学问题时引用已发表文献）表现如何，仍是开放问题。更强模型亦可能拥有不易被任何特定工具引用或核验的「参数知识」。

In our domain, we observe that debate becomes more truth-seeking with increased model persuasiveness. This finding is explained by models becoming relatively better at arguing for the correct answer compared to the incorrect answer when their outputs are optimised for judge approval. This indicates that optimising for persuasiveness can lead to more truthful models, paving the way for future research in fine-tuning LLMs via debate. Furthermore, we show how debate can be used to augment human judgements and generate accurate labels to questions beyond their knowledge. Overall, these results demonstrate that debate is a promising approach for scalable oversight.

在我们的领域中，观察到辩论随模型说服力增强而更趋向求真。该发现可解释为：当输出为裁判认可优化时，模型相对错误答案，为正确答案辩护变得相对更好。这表明优化说服力可导向更真实的模型，为未来经辩论微调 LLM 的研究铺路。此外我们展示辩论如何增强人类判断，并为超出其知识的问题生成准确标签。总体而言，这些结果说明辩论是可扩展监督的有希望途径。

The code we used is available at https://github.com/ucl-dark/llm_debate.

所用代码见 https://github.com/ucl-dark/llm_debate。

Impact Statement

影响声明

This work focuses on methods to allow humans to supervise models, particularly focusing on the open problem of supervision for superhuman models. We believe this work to be useful in this endeavour; however, the incorrect application of supervision can make detecting malicious behaviour harder (Hubinger et al. 2024), and as such, we recommend caution in its application.

本工作聚焦允许人类监督模型的方法，尤其关注超人类模型监督这一开放问题。我们相信本工作对此有用；但不正确应用监督可能使检测恶意行为更难（Hubinger et al. 2024），故建议审慎应用。

Contributions

贡献

AK led the project and was involved in all its components. JH ran LLM experiments and led infrastructure. DV ran human experiments and developed infrastructure. KS developed infrastructure. LR, AR, EG, SB, TR, and EP advised on research. LR led writing with EG, SB, EP, and TR as editors. EP generated a tremendous amount of experimental ideas.

AK 领导项目并参与全部组件。JH 跑 LLM 实验并主导基础设施。DV 跑人类实验并开发基础设施。KS 开发基础设施。LR、AR、EG、SB、TR 与 EP 提供研究顾问。LR 主导写作，EG、SB、EP 与 TR 为编辑。EP 贡献了大量实验想法。

Acknowledgements

致谢

We would like to thank Jack Hopkins, Julian Michael and David Rein for their valuable technical insights and discussions. We thank Rob Kirk, Henry Sleight, Timon Willi, Maxime Beau, Nat McAleese and Jesse Mu for their comments on early versions of the paper. We thank Rebecca Ward-Diorio for help writing the paper. We thank Phoebe Yao, Des Balisa, and Ignacio Mackenney for their help running the human trials, as well as the annotators who participated. AK is grateful to Nat McAleese, Richard Ngo and Beth Barnes for many formative discussions over the years. AK was supported by the EPSRC Grant EP/S021566/1, the Effective Ventures Fund and the UCL International Scholar Award for Doctoral Training Centres. JH and DV were funded by the MATS Program https://www.matsprogram.org/ for part of the project. LR is funded by the EPSRC Grant EP/S021566/1 and UCL International Scholar Award for Doctoral Training Centres. We also thank the OpenAI API Academic Access Program, the Claude Research Access Program, Anthropic and FAR AI for additional funding.

感谢 Jack Hopkins、Julian Michael 与 David Rein 的宝贵技术洞见与讨论。感谢 Rob Kirk、Henry Sleight、Timon Willi、Maxime Beau、Nat McAleese 与 Jesse Mu 对论文早期版本的意见。感谢 Rebecca Ward-Diorio 协助写作。感谢 Phoebe Yao、Des Balisa 与 Ignacio Mackenney 协助人类试验，以及参与的标注者。AK 感谢 Nat McAleese、Richard Ngo 与 Beth Barnes 多年来许多奠基性讨论。AK 受 EPSRC Grant EP/S021566/1、Effective Ventures Fund 与 UCL International Scholar Award for Doctoral Training Centres 支持。JH 与 DV 项目部分由 MATS Program（https://www.matsprogram.org/）资助。LR 受 EPSRC Grant EP/S021566/1 与 UCL International Scholar Award for Doctoral Training Centres 资助。亦感谢 OpenAI API Academic Access Program、Claude Research Access Program、Anthropic 与 FAR AI 的额外资助。

References

参考文献

Amodei et al. (2016) Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565 , 2016.

Bai et al. (2022) Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073 , 2022.

Bansal et al. (2021) Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., and Weld, D. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , pp. 1–16, 2021.

Barnes (2020) Barnes, B. Debate update: Obfuscated arguments problem. AI Alignment Forum , 2020.

Bowman et al. (2022) Bowman, S. R., Hyun, J., Perez, E., Chen, E., Pettit, C., Heiner, S., Lukosuite, K., Askell, A., Jones, A., Chen, A., et al. Measuring progress on scalable oversight for large language models. arXiv preprint arXiv:2211.03540 , 2022.

Burns et al. (2023) Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., Sutskever, I., and Wu, J. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390 , 2023.

Christiano et al. (2018) Christiano, P., Shlegeris, B., and Amodei, D. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575 , 2018.

Christiano et al. (2017) Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

Chromik et al. (2021) Chromik, M., Eiband, M., Buchner, F., Krüger, A., and Butz, A. I think i get your point, ai! the illusion of explanatory depth in explainable ai. In 26th International Conference on Intelligent User Interfaces , pp. 307–317, 2021.

Cotra (2021) Cotra, A. The case for aligning narrowly superhuman models. AI Alignment Forum , 2021.

Du et al. (2023) Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., and Mordatch, I. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325 , 2023.

Elo (1978) Elo, A. The Rating of Chessplayers: Past and Present . Ishi Press International, 1978. ISBN 9780923891275. URL https://books.google.com/books?id=syjcPQAACAAJ .

Gemini Team et al. (2023) Gemini Team, Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023.

Goodfellow et al. (2014) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural information processing systems , 27, 2014.

Goswami et al. (2023) Goswami, M., Sanil, V., Choudhry, A., Srinivasan, A., Udompanyawit, C., and Dubrawski, A. Aqua: A benchmarking tool for label quality assessment. arXiv preprint arXiv:2306.09467 , 2023.

Greenblatt et al. (2023) Greenblatt, R., Shlegeris, B., Sachan, K., and Roger, F. Ai control: Improving safety despite intentional subversion. arXiv preprint arXiv:2312.06942 , 2023.

Hendrycks et al. (2020) Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations , 2020.

Herbrich et al. (2006) Herbrich, R., Minka, T., and Graepel, T. Trueskill™: a bayesian skill rating system. Advances in neural information processing systems , 19, 2006.

Hubinger et al. (2024) Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., Lanham, T., Ziegler, D. M., Maxwell, T., Cheng, N., et al. Sleeper agents: Training deceptive llms that persist through safety training. arXiv preprint arXiv:2401.05566 , 2024.

Irving et al. (2018) Irving, G., Christiano, P., and Amodei, D. AI safety via debate. arXiv preprint arXiv:1805.00899 , 2018.

Jiang et al. (2024) Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 , 2024.

Lin et al. (2022) Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 3214–3252, 2022.

Madaan et al. (2023) Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., and Clark, P. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651 , 2023.

Menick et al. (2022) Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., Irving, G., et al. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147 , 2022.

Michael et al. (2023) Michael, J., Mahdi, S., Rein, D., Petty, J., Dirani, J., Padmakumar, V., and Bowman, S. R. Debate helps supervise unreliable experts. arXiv preprint arXiv:2311.08702 , 2023.

Nikola (2023) Nikola. Jailbreaking gpt-4’s code interpreter. LessWrong , July 2023. URL https://www.lesswrong.com/posts/KSroBnxCHodGmPPJ8/jailbreaking-gpt-4-s-code-interpreter .

Nye et al. (2022) Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. Show your work: Scratchpads for intermediate computation with language models. In Deep Learning for Code Workshop , 2022.

OpenAI (2023) OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Ouyang et al. (2022) Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730–27744, 2022.

Pang et al. (2022) Pang, R. Y., Parrish, A., Joshi, N., Nangia, N., Phang, J., Chen, A., Padmakumar, V., Ma, J., Thompson, J., He, H., et al. QuALITY: Question answering with long input texts, yes! In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 5336–5358, 2022.

Parrish et al. (2022a) Parrish, A., Trivedi, H., Nangia, N., Phang, J., Padmakumar, V., Saimbhi, A. S., and Bowman, S. R. Two-turn debate does not help humans answer hard reading comprehension questions. In NeurIPS ML Safety Workshop , 2022a.

Parrish et al. (2022b) Parrish, A., Trivedi, H., Perez, E., Chen, A., Nangia, N., Phang, J., and Bowman, S. Single-turn debate does not help humans answer hard reading-comprehension questions. In Proceedings of the First Workshop on Learning with Natural Language Supervision , pp. 17–28, 2022b.

Perez et al. (2019) Perez, E., Karamcheti, S., Fergus, R., Weston, J., Kiela, D., and Cho, K. Finding generalizable evidence by learning to convince Q&A models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pp. 2402–2411, 2019.

Perez et al. (2021) Perez, E., Kiela, D., and Cho, K. True few-shot learning with language models. Advances in neural information processing systems , 34:11054–11070, 2021.

Radhakrishnan (2023) Radhakrishnan, A. Anthropic fall 2023 debate progress update. AI Alignment Forum , 2023.

Radhakrishnan et al. (2023) Radhakrishnan, A., Nguyen, K., Chen, A., Chen, C., Denison, C., Hernandez, D., Durmus, E., Hubinger, E., Kernion, J., Lukošiūtė, K., et al. Question decomposition improves the faithfulness of model-generated reasoning. arXiv preprint arXiv:2307.11768 , 2023.

Rein et al. (2023) Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022 , 2023.

Saunders et al. (2022) Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, L., Ward, J., and Leike, J. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802 , 2022.

Stiennon et al. (2020) Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 33:3008–3021, 2020.

Wei et al. (2022) Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems , 35:24824–24837, 2022.

Yao et al. (2023) Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601 , 2023.

Zheng et al. (2023) Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging LLM-as-a-judge with MT-bench and chatbot arena. arXiv preprint arXiv:2306.05685 , 2023.

Appendix A Further Results with Language Model Judges

附录 A 语言模型裁判的进一步结果

A.1 Consultancy and debate best-of-N persuasiveness

A.1 咨询与辩论的 best-of-N 说服力

Leveraging the outcomes from the cross-play tournament depicted in Figure 3-Right and the consultancy findings shown in Figure 4-Right, we present a side-by-side comparison of judge accuracy and expert win rate for both protocols in Figure 7. The data presented include only Claude 2.1 and GPT-4-Turbo experts employing the best-of-N augmentation strategy. For simplicity, results from the critique-and-refinement protocol and GPT-3.5-Turbo have been excluded.

借助图 3 右所示交叉对弈锦标赛结果与图 4 右所示咨询发现，我们在图 7 并排比较两协议的裁判准确率与专家胜率。所呈数据仅含使用 best-of-N 增强策略的 Claude 2.1 与 GPT-4-Turbo 专家。为简洁起见，排除批评–refinement 协议与 GPT-3.5-Turbo 的结果。

The analysis reveals a direct correlation between the persuasiveness of debaters, as enhanced by best-of-N augmentation, and the improvement of judge accuracy. Conversely, in the consultancy protocol, an increase in persuasiveness leads to a reciprocal effect on judge accuracy.

分析揭示：经 best-of-N 增强的辩手说服力与裁判准确率提升直接相关。相反，在咨询协议中，说服力上升对裁判准确率产生反向效应。

Figure 7: More persuasive debaters results in better judge accuracy whereas the opposite is true for consultants. (Left) Judge accuracy decreases with aggregate win rate for the consultancy protocol. (Right) Judge accuracy increases with debater win rate for the debate protocol. Win rates for debate are calculated with the reference model of Claude 2.1 best-of-1 which has an aggregate Elo rating of 0.

图 7：更有说服力的辩手带来更高裁判准确率，顾问则相反。（左）咨询协议中裁判准确率随总体胜率下降。（右）辩论协议中裁判准确率随辩手胜率上升。辩论胜率以 Claude 2.1 best-of-1（总体 Elo 为 0）为参考模型计算。

A.2 Comparing protocol accuracy with more samples

A.2 用更多样本比较协议准确率

Figure 9 shows the performance of LLM judges across all our protocols with dataset $T_{L}$. This question set is approximately eight times larger than the question set $T_{H}$ used to compare protocols in Figure 1, which was tractable to run when not using human judges. In addition, we ran interactive debate with LLM interactive judges rather than humans, as previously presented in Figure 6. Figure 9 shows two main insights:

图 9 展示 LLM 裁判在数据集 $T_{L}$ 上各协议表现。该问题集约比图 1 比较协议所用的 $T_{H}$ 大八倍；不使用人类裁判时可操作。此外我们跑了 LLM 交互裁判的交互式辩论，而非图 6 先前呈现的人类。图 9 显示两点主要洞见：

1. We find that debate is still a stronger oversight protocol over a larger set of questions than consultancy for each non-expert judge.
2. Interactive debate does not improve the LLM judge accuracy when the model engages with the debaters. This is consistent across each judge. In contrast, for humans, interactive debate had the same accuracy as static debate.

1. 在更大问题集上，对每个非专家裁判，辩论仍是强于咨询的监督协议。
2. 当模型与辩手互动时，交互式辩论并不提高 LLM 裁判准确率；各裁判一致。相较之下，对人类交互式辩论与静态辩论准确率相同。

To analyse these results further, we provide the Performance Gap Recovered (PGR) metric (Burns et al. 2023) in Figure 9. To calculate this, we use the following formula:

为进一步分析，我们在图 9 提供 Performance Gap Recovered（PGR）指标（Burns et al. 2023）。计算公式为：

$$PGR=\frac{\text{protocol accuracy}-\text{Naive}}{\text{expert}-\text{Naive}}$$

The PGR metric measures the percentage of accuracy each oversight protocol recovers compared to the expert baseline, where the judge has access to the text. We find that debate recovers the most performance, and the advantage over other protocols increases as the judge’s capability increases. Conversely, in the consultancy protocol, we observe a decline in PGR performance correlating with enhancements in the base model’s capability.

PGR 度量各监督协议相对专家基线（裁判可访问文本）恢复的准确率百分比。我们发现辩论恢复最多，且优势随裁判能力增强而增大。相反，咨询协议中 PGR 表现随基座模型能力增强而下降。

Figure 8: Stronger judge base models lead to higher judge accuracy in debate. Consultancy shows minimal improvement in judge accuracy with better judge base models, but debate and baselines show a clear trend. We find that debate leads to significantly higher judge accuracy than either consultancy or the naive baseline. The introduction of judge-debater interaction in interactive debate results in a decrease in performance. All results are collected on a set of 400 questions ($T_{L}$), with identical transcripts produced by GPT-4-Turbo-bo16 experts and a GPT-4-Turbo interactive judge.

图 8：更强裁判基座模型在辩论中带来更高裁判准确率。 咨询随更好裁判基座改进很小，但辩论与基线趋势清晰。辩论裁判准确率显著高于咨询或朴素基线。交互式辩论中引入裁判–辩手交互导致表现下降。全部结果在 400 题（$T_{L}$）上收集，记录由 GPT-4-Turbo-bo16 专家与 GPT-4-Turbo 交互裁判产生，相同。

Figure 10: Best-of-N is more persuasive than Critique & Refinement. We analyse how different optimisation strategies affect judge accuracy in debate and consultancy, focusing on the impact of best-of-$N$ (bo$N$) and critique-and-refinement methods. bo$N$ shows higher persuasion skill, which increases with N. Debate shows the opposite trend compared to consultancy, where the judge accuracy increases with skill. This analysis shows detailed segments of plots from Figures 3 and 4, highlighting the specific optimisation strategies employed.

图 10：Best-of-N 比批评与 Refinement 更具说服力。 分析不同优化策略如何影响辩论与咨询中的裁判准确率，聚焦 best-of-$N$（bo$N$）与批评–refinement。bo$N$ 显示更高说服技能，且随 N 增大。辩论趋势与咨询相反：裁判准确率随技能上升。本分析展示图 3、4 的细节片段，突出所用具体优化策略。

A.3 Best-of-N versus critique-and-refinement

A.3 Best-of-N 对比批评–refinement

Figure 10 shows equivalent data as presented in Figures 3 and 4 and highlights the impact of augmenting the experts with best-of-$N$ (bo$N$) or critique-and-refinement for debate and consultancy.

图 10 展示与图 3、4 等价的数据，并突出用 best-of-$N$（bo$N$）或批评–refinement 增强专家对辩论与咨询的影响。

The consultancy aggregate win rate and debate aggregate Elo can both be considered as the skill of the expert in each protocol with respect to persuading the judge. The aggregate win rate is calculated after combining the win rate across the incorrect and correct assignments in consultancy. The aggregate Elo is calculated after 20 debaters competed against each other in a cross-play tournament (Appendix D.4).

咨询总体胜率与辩论总体 Elo 都可视为各协议中专家相对说服裁判的技能。总体胜率在合并咨询中错误与正确分配的胜率后计算。总体 Elo 在 20 名辩手交叉对弈锦标赛后计算（附录 D.4）。

We find the following:

我们发现：

• • Judge accuracy in debate improves with rising debater skill, yet diminishes in consultancy settings as consultant proficiency increases.
• • Critique-and-refinement improves skill compared to the vanilla model with no augmentation (Bo1). However, bo$N$ leads to more skilled debaters, which continue to improve as N increases.
• • We find that the skill plateaus as you go beyond Bo16 for GPT-4-Turbo. Bo32 uses a higher temperature of 1.0 compared to 0.8, which may have pushed the diversity of arguments too far to see further gains.
• • Combining both bo$N$ and critique-and-refinement is a competitive strategy but not as effective as bo$N$ alone when controlling for the number of inference calls to the expert models. Applying bo$N$ to critique-and-refinement involves sampling the initial arguments and the refinements bo$N$ times, leading to the number of inference calls being equal to 2*bo$N$ + c$N$. Therefore, Bo4 Co8, uses 16 calls which is equivalent to Bo16.

• 辩论中裁判准确率随辩手技能上升而提高，咨询设定中则随顾问熟练度上升而下降。
• 批评–refinement 相对无增强的 vanilla 模型（Bo1）提升技能；但 bo$N$ 带来更强辩手，且随 N 增大持续改进。
• GPT-4-Turbo 在 Bo16 之后技能趋于平台；Bo32 温度升至 1.0（相对 0.8），可能因论证多样性过大而未再增益。
• 同时使用 bo$N$ 与批评–refinement 有竞争力，但在控制专家模型推理调用次数时不如单独 bo$N$。对批评–refinement 施加 bo$N$ 需对初始论证与 refinement 各采样 bo$N$ 次，调用次数为 2*bo$N$ + c$N$。因此 Bo4 Co8 用 16 次调用，等价于 Bo16。

Figure 11: Stronger debaters use more verified quotes. (Left) We observe that stronger debaters in the same model class use more verified quotes, with GPT-4-Turbo showing a notably lower proportion of unverified quotes, indicating higher reliability in quoting the text. Claude, while using the highest number of quotes, has shorter ones, as detailed in Figure 12. (Right) We enforce a word count per argument between 60 and 150 words; GPT-4-Turbo consistently aligns near the 100-word target. Models with critique-and-refinement tend to have higher word counts, and there’s greater variability in other models within the specified limits.

图 11：更强辩手使用更多经核验引用。（左）同模型族中更强辩手使用更多经核验引用；GPT-4-Turbo 未核验引用比例明显更低，引用可靠性更高。Claude 引用数量最多但更短，见图 12。（右）我们强制每条论证 60–150 词；GPT-4-Turbo 稳定贴近 100 词目标。批评–refinement 模型词数偏高，其它模型在限内波动更大。

Figure 12: Stronger debaters prefer longer, verified quotes. We observe that weaker debaters tend to use fake or low n-gram-matching quotes, while stronger debaters prefer longer, verified quotes. This analysis highlights the correlation between quote authenticity and debater strength, with weaker debaters also showing a higher tendency for duplicate quotes.

图 12：更强辩手偏好更长且经核验的引用。弱辩手更常用伪造或低 n-gram 匹配引用，强辩手偏好长且经核验的引用。分析突出引用真实性与辩手强度的相关；弱辩手亦更倾向重复引用。

A.4 Analysis of debater argument statistics

A.4 辩手论证统计

After running the cross-play tournament across the 20 debaters denoted in Appendix D.4, we analysed the quote usage and word counts of a subset of these debaters. On the left of Figure 11, it shows the average number of quotes used by the debaters in their transcript across self-play runs on $T_{L}$. This is further broken down into verified, unverified and no quote tags. Verified refers to quotes that are direct matches, while unverified denotes those that are not. ”No quote tags” is when the debater does not use tags around their quotes, which means they also count as unverified by the judge. We find that GPT-4-Turbo debaters use predominantly verified quotes, which is likely one reason they have a higher debater skill than Claude 2.1 and GPT-3.5-Turbo. Furthermore, debaters enhanced by best-of-$N$ or critique-and-refinement use more verified quotes, on average, across all base models. Claude models tend to use more quotes in their arguments. However, they are shorter, as shown by Figure 12. As well as stronger debaters using more long verified quotes, we show a link with authenticity since GPT-4-Turbo uses fewer fake quotes compared to Claude and GPT-3.5-Turbo (Figure 12). Our heuristic for finding fake quotes is by normalising the quotes and the text, finding the n-gram similarity and choosing the quotes that have a similarity of 0 while being over 7 words long.

在附录 D.4 所列 20 名辩手交叉对弈锦标赛后，我们分析子集的引用使用与词数。图 11 左给出辩手在 $T_{L}$ 自我对弈记录中平均引用数，并拆分为经核验、未核验与无引用标签。经核验指直接匹配，未核验指不匹配。「无引用标签」指辩手未用 包裹引用，裁判亦计为未核验。GPT-4-Turbo 辩手以经核验引用为主，这或是其技能高于 Claude 2.1 与 GPT-3.5-Turbo 的原因之一。经 best-of-$N$ 或批评–refinement 增强的辩手在各基座上平均使用更多经核验引用。Claude 模型论证中引用更多，但更短（图 12）。除更强辩手使用更多长经核验引用外，我们亦展示真实性关联：相对 Claude 与 GPT-3.5-Turbo，GPT-4-Turbo 伪造引用更少（图 12）。伪造引用启发式：规范化引用与文本，计算 n-gram 相似度，取相似度为 0 且超过 7 词的引用。

The distribution of the number of words per argument (Figure 11-Right), shows the impact of applying our strict word filter. This filter only allows arguments between 70 and 150 words. We use this to mitigate the judge verbosity bias with the methodology explained in Appendix D.6. We also observe that GPT-4-Turbo has a tighter distribution around the target word limit of 100 than other base models.

每论证词数分布（图 11 右）显示严格词数过滤的影响。过滤仅允许 70–150 词，以缓解裁判冗长偏好，方法见附录 D.6。亦观察到 GPT-4-Turbo 相对其它基座更紧贴 100 词目标。

Figure 13: Majority voting improves judge accuracy in debate. We observe a trend where increasing votes enhances accuracy in debate but reduces it in consultancy. A setup with 0.8 temperature and three votes yields a 19% error reduction when using debate. Error bars denote the standard error of the mean, based on 400 questions.

图 13：多数投票提升辩论裁判准确率。增加票数在辩论中提高准确率，在咨询中降低。温度 0.8、三票设定下辩论误差相对下降 19%。误差棒为基于 400 题的均值标准误。

Figure 14: Fine-tuning and few-shot on gold labels improves judge accuracy, but fine-tuning on human reasoning does not. (Left) Fine-tuning GPT-3.5-Turbo shows improved performance when fine-tuned on gold labels, achieving parity with Claude 2.1 0-shot, but fine-tuning on the human chain-of-thought does not help. (Right) Few-shot learning yields diminishing returns beyond 4 samples; performance declines when challenging samples that human judges get wrong are used or using GPT-3.5-Turbo (16k).

图 14：在金标签上微调与少样本学习提升裁判准确率，在人类推理上微调则否。（左）在金标签上微调 GPT-3.5-Turbo 提升表现，达 Claude 2.1 零样本水平；模仿人类思维链则无助。（右）少样本超过 4 例收益递减；使用人类裁判做错的难题或 GPT-3.5-Turbo（16k）时表现下降。

A.5 Improving LLM judge performance

A.5 提升 LLM 裁判表现

Majority voting improves judge performance with debate. When running multiple judgements with a GPT-4-Turbo judge, aggregating the outcomes and taking the majority vote leads to a 4% absolute gain in accuracy (19% relative reduction in error). We sweep the temperature and show in Figure 13 that using a higher temperature of 0.8 and a judge prompt that uses Chain of Thought (Wei et al. 2022) leads to a stable performance when using as few as 3 votes. Performance does not increase when using more votes. See Figure 27 for a similar analysis with human judges.

多数投票在辩论中提升裁判表现。 用 GPT-4-Turbo 裁判多次评判后取多数，绝对准确率提升 4%（误差相对下降 19%）。扫描温度后图 13 显示：温度 0.8 且裁判提示使用思维链（Wei et al. 2022）时，少至 3 票即可稳定；更多票不再提升。人类裁判类似分析见图 27。

Majority voting degrades the judge’s accuracy with consultancy. The same experiment was performed with consultancy, and Figure 13 shows a small degradation of up to 0.5% when increasing the number of votes. This shows more evidence that as the judge gets stronger, performance with consultancy decreases.

多数投票在咨询中损害裁判准确率。 咨询上同实验显示，增加票数最多下降约 0.5%（图 13）。进一步表明裁判变强时咨询表现下降。

Fine-tuning improves the judge but not if using Chain of Thought. We fine-tuned GPT-3.5-turbo on 569 samples in the human-to-human debate dataset provided by Michael et al. 2023. We removed the final interactive judge question to stop bias towards the correct answer (since often the human would say, “I’m leaning towards A”). The transcripts were inserted into an identical judge prompt, and an assistant message was added with the correct letter “A” or “B”.

微调提升裁判，但基于思维链则否。 我们在 Michael et al. 2023 人–人辩论数据 569 样本上微调 GPT-3.5-turbo。去掉最后的交互裁判问题以免偏向正确答案（人类常说「我更倾向 A」）。记录填入相同裁判提示，并加助手消息给出正确字母「A」或「B」。

We found gains in judge accuracy when fine-tuning to predict only the ground truth label (e.g. A or B) in the LLM’s completion (denoted Gold Label in Figure 14). For example, the training data would be of the form: User: , Assistant: A. When we fine-tuned the judge to imitate the human chain of thought reasoning of the answer, we saw a degradation in accuracy. We believe this degradation is because deliberation is difficult for LLMs, and learning to imitate human reasoning prevents the model from learning deliberation efficiently.

仅微调预测真实标签（如 A/B）时裁判准确率上升（图 14「金标签」）。训练数据形如：用户: <裁判提示>，助手: A。微调模仿人类思维链推理时准确率下降。我们推测因 LLM 审议困难，模仿人类推理阻碍高效学习审议。

Few-shot learning with 4 examples improves the judge. We collected data in the same way described for fine-tuning but additionally filtered the data by only keeping debates with 3 rounds and less than 1600 tokens in the transcript. We ensured an equal split between swap and non-swap assignments of answers to A and B (see how we define swapping answers in Appendix D.6). These examples are randomly sampled from and inserted into the start of the judge prompt. We ended up with a list of messages alternating between user and assistant messages, where the assistant messages have the ground truth answer letter. We also removed all interactive judge messages.

用 4 例少样本学习提升裁判。 数据收集同微调，另过滤仅保留 3 轮且记录少于 1600 token 的辩论；保证答案在 A/B 互换与非互换分配均等（互换定义见附录 D.6）。样例随机采样插入裁判提示开头，形成用户–助手交替消息列表，助手消息为真实答案字母；去掉全部交互裁判消息。

We swept the number of few-shot samples used by the judge in Figure 14-Right and observed that performance improves for stronger judges when using 2-4 few-shot examples (6% absolute boost for GPT-4-Turbo which is similar compared to majority voting in Figure 13). GPT-3.5-Turbo (16k context window) does not benefit from any few-shot samples, as shown by its accuracy degrading as we added more. Furthermore, the performance across all judges either plateaued or degraded with more samples depending on the data split used.

图 14 右扫描少样本数量：更强裁判用 2–4 例时表现提升（GPT-4-Turbo 绝对提升 6%，与图 13 多数投票相近）。GPT-3.5-Turbo（16k 上下文）不受益，加样反降。再增加样本时，依数据划分不同，各裁判表现平台或下降。

Lastly, we hypothesised that few shot learning would be higher quality when using only samples which human judges judged incorrectly. In Figure 14-Right, we observe that using those samples leads to equivalent performance up to 4-shots. Beyond this point, the accuracy of few-shot learning using the ”human incorrect” starts to degrade for GPT-4-Turbo.

最后我们假设：仅用人裁判做错的样本做少样本质量更高。图 14 右显示至 4-shot 表现相当；此后「人类错误」划分上 GPT-4-Turbo 少样本准确率开始下降。

A.6 Impact on using and not using the quote system

A.6 引用系统有无的影响

We conduct an ablation on the usefulness of the quote verification system for our debate setting. We consider three variants of the QuALITY setting, in which models can 1) only use quotes, 2) can only generate arguments, and 3) use both quotes and arguments. We run both debate and consultancy with the expert as GPT-4-Turbo with best-of-16 optimisation and non-expert as GPT-4-Turbo. We found that across both protocols, arguments-only settings result in lower judge accuracy, as experts arguing for the incorrect answer can lie more easily (see Figure 15). Conversely, using quotes only strongly benefits the expert arguing the correct answer, demonstrated by the much higher accuracy over quotes and arguments. We choose to use both arguments as this is a natural setting for human readability.

我们对辩论设定中引用核验系统的效用做消融。考虑 QuALITY 三变体：模型 1）仅用引用；2）仅生成论证；3）引用与论证兼用。专家为 GPT-4-Turbo best-of-16，非专家为 GPT-4-Turbo，跑辩论与咨询。两协议中，仅论证设定裁判准确率更低——为错误答案辩护的专家更易撒谎（图 15）。相反，仅引用强力惠及为正确答案辩护的专家，准确率远高于引用+论证。我们选择兼用二者，因其更自然、便于人类阅读。

Figure 15: LLM judges benefit more from quotes than arguments. We find that without the quote system, using only arguments (Arguments) results in worse results across protocols. We also find, by comparing Quotes with Quotes+Arguments, that using any form of natural language helps incorrect debaters/consultants be more persuasive, resulting in decreased judge accuracy. We choose to use both Quotes+Arguments as it’s a more natural setting for human judges and is closer to real-world usage of LLMs.

图 15：LLM 裁判从引用受益多于论证。无引用系统、仅论证时各协议结果更差。对比「引用」与「引用+论证」可见，自然语言论证帮助错误辩手/顾问更有说服力，从而降低裁判准确率。我们选引用+论证，因其对人类裁判更自然、更贴近真实 LLM 使用。

Figure 16: Longer arguments are more persuasive to LLM judges. (Left) Increasing word limits results in higher aggregate Elo, influenced by the judge’s verbosity bias. A Swiss tournament with six players over 291 QuALITY questions illustrates this effect. (Right) Word count distributions show GPT-4-Turbo’s precision and lower variance compared to Claude 2.1 when adhering to word limit targets. We use Claude 2 as the zero reference policy.

图 16：更长论证对 LLM 裁判更有说服力。（左）提高词数上限提升总体 Elo，受裁判冗长偏好影响。六名选手、291 道 QuALITY 题的瑞士制锦标赛说明此效应。（右）词数分布显示 GPT-4-Turbo 相对 Claude 2.1 更精准、方差更低。以 Claude 2 为零参考策略。

Figure 17: LLM judges exhibit positional bias. Here we show judge accuracy on the $T_{H}$ set of 47 questions, with error bars indicating positional bias. We demonstrate that human judges exhibit minimal positional bias in contrast to LLM judges, with GPT-3.5-Turbo showing the largest bias across all protocols. Positional bias is notably higher in the naive baseline case with human judges.

图 17：LLM 裁判存在位置偏见。在 $T_{H}$ 47 题上报告裁判准确率，误差棒表示位置偏见。人类裁判位置偏见极小；LLM 裁判中 GPT-3.5-Turbo 跨协议偏见最大。人类裁判在朴素基线中位置偏见明显更高。

A.7 Understanding LLM judge biases

A.7 理解 LLM 裁判偏见

We found four types of bias in our LLM judges:

我们发现 LLM 裁判有四类偏见：

• • Verbosity — judges favour longer arguments, even lower quality and less persuasive arguments.
• • Positional — judges favour answers in certain positions over others. For example, in the debate case, this manifests as choosing A instead of B, even when arguments are swapped.
• • Overconfidence — judges assign miscalibrated probabilities to their chosen answers and tend to be overconfident in their judgements.
• • Sycophancy — judges sometimes believe unverified quotes and give them credence when they are told they should not trust them.

• 冗长偏好 — 裁判偏好更长论证，即使质量与说服力更低。
• 位置偏见 — 裁判偏好某些位置的答案。例如辩论中即使论证互换仍倾向选 A 而非 B。
• 过度自信 — 裁判对所选答案概率校准不良，倾向过度自信。
• 谄媚 — 裁判有时相信未核验引用，并在本不该信任时仍给予分量。

Self-enhancement bias is when an LLM favours completions generated by itself compared to other models. Other work has shown that this is another bias that LMs as judges have (Zheng et al. 2023). This is not an issue for our self-play debates where the same instance of the model acts as both the incorrect and correct debater. Using different judges for our cross-play tournament does alter the final ranking of some of the 20 debaters (see Appendix F). However, there is no clear sign that the judges rank debaters of the same base model more highly.

自我增强偏见指 LLM 相对其它模型更偏好自身生成的补全。其它工作表明这是 LM 作裁判的另一偏见（Zheng et al. 2023）。对我们自我对弈辩论不成问题——同一模型实例同时作错误与正确辩手。交叉对弈锦标赛换用不同裁判会改变 20 名辩手部分最终排名（附录 F），但无清晰迹象表明裁判更抬高同基座辩手。

See how we address the verbosity and positional bias in Appendix D.6. See how we iterated on our prompts to reduce the impact of sycophancy bias in Appendix D.8.

冗长与位置偏见的处理见附录 D.6；提示迭代以降低谄媚影响见附录 D.8。

We find a self-defeating bias in our debaters and consultants, where they will often concede when arguing for the incorrect answer. We use the next best ”distractor” answer (see Appendix D.1), which alleviates issues that arise from the self-defeating bias since the story can be ambiguous enough that LLMs tend not to refuse to argue for the wrong side. See our prompt iteration method to reduce the bias further in Appendix D.7.

我们在辩手与顾问中发现自我挫败偏见：为错误答案辩护时经常让步。我们使用次优「干扰」答案（附录 D.1），因故事足够含混，LLM 较少拒绝为错误方辩护，从而缓解该偏见。进一步降低偏见的提示迭代见附录 D.7。

Figure 19: LLM judges are poorly calibrated and overconfident. We investigate LLM judge calibration and confidence using GPT-4-Turbo log probabilities on the same transcripts seen by the human judges. (Left) Calibration plots for different oversight protocols reveal consistent overconfidence in LLM judge responses, contrasting with human judge patterns where they are under-confident. Error bars represent 95% confidence intervals. (Right) Selective accuracy vs coverage rate: when thresholding by judge confidence and rejecting judgements with low confidence, debate outperforms consultancy at all coverage rates. Since LLMs are overconfident, especially in consultancy, the accuracy is not high even for low coverage rates.

图 19：LLM 裁判校准差且过度自信。用 GPT-4-Turbo 对数概率在人类裁判所见相同记录上考察校准与置信。（左）不同监督协议校准图显示 LLM 裁判一致过度自信，人类裁判则偏不自信。误差棒为 95% 置信区间。（右）选择性准确率对覆盖率：按裁判置信阈值剔除低置信判断时，辩论在全部覆盖率上优于咨询。因 LLM 过度自信（咨询尤甚），即便低覆盖率准确率也不高。

Verbosity bias — In initial experiments, we faced challenges in determining whether enhancements in protocols or prompts were genuine or merely a result of the judge’s preference for lengthier arguments. For instance, the critique-and-refinement strategy often resulted in significantly longer arguments after integrating the critic’s feedback. Despite showing a higher win rate than the baseline, we remained uncertain if this win rate was due to the actual strength of the arguments.

冗长偏见 — 初期实验中，我们难以判断协议或提示的改进是真实的，还是仅因裁判偏好更长论证。例如批评–refinement 整合批评反馈后论证显著变长；尽管胜率高于基线，我们仍不确定是否来自论证真实强度。

We investigated verbosity bias by computing the Elo ratings of debaters from a Swiss tournament where debaters were instructed to use arguments of 100, 200, and 300 words (featuring Claude 2.1 and GPT-4-Turbo) . As depicted in Figure 16, a notable rise in aggregate Elo was observed with higher word counts, indicating a judging preference by GPT-4-Turbo towards lengthier arguments over shorter, more truthful responses.

我们通过瑞士制锦标赛计算辩手 Elo，指令要求论证分别为 100、200、300 词（含 Claude 2.1 与 GPT-4-Turbo），以考察冗长偏见。如图 16，词数更高时总体 Elo 显著上升，表明 GPT-4-Turbo 裁判偏好更长论证，而非更短但更真实的回应。

In addition, Figure 16 shows the word count distribution when you ask the model to produce an argument with a target number of words. The distribution is often centred on the words specified but has a significant variance. Sometimes, the Claude API will write 70 words greater than what was asked for in the prompt. The distribution differs between Claude and GPT-4-Turbo and shows that GPT-4-Turbo is more accurate with a smaller variance. Based on these distributions, we chose to ask the models for 100 words and reject any that lie outside 70 and 150.

此外图 16 显示：要求模型按目标词数写论证时，分布常以指定词数为中心但方差显著。Claude API 有时比提示所要求多写约 70 词。Claude 与 GPT-4-Turbo 分布不同，后者更准、方差更小。据此我们要求 100 词，并拒绝落在 70–150 之外者。

Positional bias — LLMs, as judges, have a positional bias as shown by the error bars in Figure 18. This means that judges’ accuracy can change dramatically when judges are provided with the same set of arguments but the answers corresponding to A and B are swapped (illustrated in Table 7). GPT-4-Turbo and humans have a similar positional bias but it is higher for humans in the naive case.

位置偏见 — LLM 作裁判有位置偏见，见图 18 误差棒。同一组论证但 A/B 答案互换时，裁判准确率可剧烈变化（表 7 示意）。GPT-4-Turbo 与人类位置偏见相近，但人类在朴素情形更高。

We observe that positional bias is significantly more pronounced in less capable models like GPT-3.5-Turbo, which lack deliberation capabilities, suggesting that these models might resort to guesswork. We hypothesised that GPT-4-Turbo’s confidence in its answers, as extracted from the log probabilities, would hover around 0.5 when the judgement outcome is altered by swapping the positions of answer choices. This was partially confirmed by the reduced average confidence for different outcomes, as illustrated in Figure 18. We noted 25% of the total cases where the model remained over 98% confident even when it reversed its decision due to the positioning of answers.

位置偏见在能力较弱、缺乏审议能力的模型（如 GPT-3.5-Turbo）上更显著，暗示其或诉诸猜测。我们假设：答案位置互换导致判断结果改变时，从对数概率提取的 GPT-4-Turbo 置信应约在 0.5。图 18 中不同结果的平均置信下降部分证实此点。亦观察到 25% 案例中，模型因位置互换改判后仍保持超过 98% 置信。

We extracted log probabilities of the chosen answer by running GPT-4-Turbo with the best-of-$N$ preference prompt provided in Appendix G.4. We then converted these to probabilities by taking the softmax of log probabilities of the tokens for A and B. Using GPT-4-Base would have provided more calibrated confidence. However, we found that the positional bias was so strong that the model would always predict choice A.

我们用附录 G.4 的 best-of-$N$ 偏好提示跑 GPT-4-Turbo，提取所选答案对数概率，并对 A、B token 对数概率做 softmax 转为概率。用 GPT-4-Base 本可得到更校准的置信，但其位置偏见过强，总是预测 A。

Overconfidence bias — We used the same method to find overconfidence bias as when finding the positional bias confidence. We extracted GPT-4-Turbo log probabilities for the predicted answer choice given the transcripts in each protocol used in the human trials: static consultancy, interactive consultancy, debate and interactive debate. Figure 19 plots calibration curves for each protocol and the selective accuracy curve in the same way as presented for human judges in Figure 6.

过度自信偏见 — 方法同位置偏见置信提取。对人类试验所用各协议记录（静态咨询、交互咨询、辩论、交互辩论）提取 GPT-4-Turbo 预测选项对数概率。图 19 按各协议绘制校准曲线与选择性准确率曲线，呈现方式同图 6 人类裁判。

The calibration plots highlight that LLMs are consistently overconfident which is a difference from human judges. There is little difference between static and interactive consultancy. Interactive debate becomes more calibrated like for the human judges but this is likely because the LLM can read the human’s interactive messages.

校准图显示 LLM 一致过度自信，与人类裁判不同。静态与交互咨询差别很小。交互辩论校准更好，与人类裁判类似，或因 LLM 可读人类交互消息。

Figure 20: Additional rounds help consultancy but not debate. We observe that weaker judges show a decrease in debate accuracy with more rounds, while strong judges like GPT-4-Turbo maintain high accuracy. In contrast, consultancy accuracy improves with additional rounds, possibly due to interactive judges reducing sycophantic tendencies towards consultants. The analysis is based on standard error of the mean across 400 QuALITY questions.

图 20：增加轮次帮助咨询、无助辩论。弱裁判辩论准确率随轮次增加下降；GPT-4-Turbo 等强裁判保持高准确。咨询准确率随轮次上升，或因交互裁判降低对顾问的谄媚。分析基于 400 道 QuALITY 题均值标准误。

A.8 Impact of number of rounds on accuracy

A.8 轮次数对准确率的影响

Using GPT-4-Turbo debaters with best-of-16, we run debate and consultancy for up to 6 rounds (see Figure 21). We observe that when using the debate protocol, weaker judges such as GPT-3.5-Turbo suffer from the inability to effectively deliberate as rounds continue, as shown by a significant decrease in judge accuracy after 2 rounds. Interestingly, we get the best LLM judge accuracy after only 1 or 2 rounds. We used 3 rounds for all experiments in this work because, qualitatively, it was the minimum we needed for a reasonable amount of interaction between debaters. For instance, they can critique their opponent in round 2 and defend in round 3. When considering interactive protocols, we decided it was appropriate to have two human judge interactions, which is possible in a 3 round debate. Furthermore, Michael et al. 2023 find that human judges have a failure mode of ending the debate or consultancy too early, which we wanted to avoid.

用 GPT-4-Turbo 辩手 best-of-16，辩论与咨询最多跑 6 轮（图 21）。辩论协议下，GPT-3.5-Turbo 等弱裁判随轮次继续无法有效审议，2 轮后裁判准确率显著下降。有趣的是，LLM 裁判最佳准确率仅出现在 1 或 2 轮后。本文全部实验用 3 轮，因定性上这是辩手间合理交互的最低轮数：第 2 轮可批评对手、第 3 轮可辩护。交互协议中我们希望人类裁判有两次交互，3 轮辩论可实现。此外 Michael et al. 2023 发现人类裁判有过早结束辩论或咨询的失败模式，我们希望避免。

Because more rounds do not increase judge accuracy (Figure 21), it suggests the incorrect debater becomes more persuasive to LLM judges as rounds continue. As discussed in Appendix B.1, we find a common source of error was poor quote selection from the correct debater, and we believe that improving the recall of the best quotes for the correct answer would mitigate the judge accuracy decreasing with a number of rounds.

因更多轮次不提升裁判准确率（图 21），暗示错误辩手随轮次继续对 LLM 裁判更有说服力。附录 B.1 讨论常见错误源是正确辩手引用选择差；我们认为提升正确答案最佳引用的召回，可缓解裁判准确率随轮次下降。

For consultancy, on the other hand, the accuracy increases as rounds continue. We hypothesise that this is due to the judge becoming less sycophantic towards the consultant as the interactive judge asks more questions. This behaviour can be clearly seen in Figure 21 where the incorrect and correct consultant both become less persuasive to the judge as rounds continue.

咨询方面，准确率随轮次继续上升。我们假设因交互裁判提问增多，裁判对顾问谄媚减弱。图 21 清晰可见：错误与正确顾问随轮次继续对裁判说服力都下降。

A.9 Improving the debaters with fine-tuning and few-shot learning

A.9 用微调与少样本学习提升辩手

Fine-tuning helps but not significantly due to context length limitations. — Using the methodology in Appendix D.9, we fine-tune GPT-3.5-Turbo on arguments provided in the human-human debate dataset (Michael et al. 2023). This method involves truncating the Gutenberg story to fit in the context window, and running cross-play matches against Claude 2.0 on Slate questions in the QuALITY dataset to find the optimal model. An example Slate questions is “Why does the author tell a story about his vehicle?” and the correct answer choice is “To make a point about what has the most impact on the economy”. Unlike Gutenberg stories, slate questions were used since they fit into the context window.

微调有帮助，但因上下文长度限制不显著。 — 按附录 D.9 方法，在人–人辩论数据（Michael et al. 2023）论证上微调 GPT-3.5-Turbo。方法包括截断 Gutenberg 故事以适配上下文窗口，并在 QuALITY 的 Slate 题上对 Claude 2.0 做交叉对弈以选最优模型。Slate 题例：「作者为何讲关于其车辆的故事？」正确答案为「为说明什么对经济影响最大」。相对 Gutenberg 故事，Slate 题可放入上下文窗口故被采用。

Figure 22 shows that the fine-tuned model gets an Elo that sits between GPT-3.5-Turbo and Claude 2.0. This shows fine-tuning helps but does not boost debater skill as much as we would have liked. In future work, we’re excited to apply fine-tuning to stronger base models with longer context windows, allowing us to run debates on Gutenberg stories and fine-tune using stories that are not truncated.

图 22 显示微调模型 Elo 介于 GPT-3.5-Turbo 与 Claude 2.0 之间。说明微调有帮助，但辩手技能提升不及预期。未来工作中我们期待对更强、更长上下文的基座微调，以便在 Gutenberg 故事上辩论并用未截断故事微调。

Few shot learning does very little to increase debater skill. — We construct few-shot samples by adding the transcript between tags. To indicate which behaviour to imitate, we mark the name of the debater who wins in the transcript as DebaterWinner and the other as DebaterLoser. These example transcripts are added at the start of the first user message in the debater prompt found in G.1. We balance examples of incorrect and correct debaters winning and also the swap position.

少样本学习对提升辩手技能几乎无助。 — 我们在 标签间加入记录构造少样本。为指示应模仿的行为，将记录中胜方标为 DebaterWinner、另一方为 DebaterLoser。样例记录加在 G.1 辩手提示首条用户消息开头。平衡错误与正确辩手获胜样例，以及互换位置。

Next, we ran a Swiss tournament with 4 Claude 2.1 debaters (vanilla, 8-shot, 32-shot and best-of-8). The aggregate Elo indicates that few-shot learning does little to help debater skill, and increasing to 32 samples leads to degradation in skill compared to the vanilla baseline (see plot on the right of Figure 22). Furthermore, best-of-8 significantly outperforms few-shot learning, so we did not use few-shot strategies in our larger Swiss tournament presented in the main paper.

随后用 4 名 Claude 2.1 辩手（vanilla、8-shot、32-shot、best-of-8）跑瑞士制锦标赛。总体 Elo 显示少样本几乎无助辩手技能；增至 32 样本相对 vanilla 基线技能下降（图 22 右）。best-of-8 显著优于少样本，故主文更大瑞士制锦标赛未用少样本策略。

Figure 22: Fine-tuning improves debater performance but few-shot learning does not. (Left) We observe that while fine-tuning GPT-3.5-Turbo improves debater skill, it fails to outperform Claude 2.0 in a round-robin tournament. Slate articles were used to stay within a 4k context window. (Right) We find that 8-shot learning slightly enhances Elo ratings, but increasing to 32 shots diminishes performance compared to zero-shot. Both few-shot methods are significantly less effective than the best-of-$N$ approach. Few-shot examples exclude the story component and focus on debate transcripts with winners clearly identified. For each tournament, we use the Claude (2.0/2.1) model as the zero-policy respectively.

图 22：微调提升辩手表现，少样本学习则否。（左）微调 GPT-3.5-Turbo 提升辩手技能，但循环赛中未能超过 Claude 2.0。用 Slate 文章以适配 4k 上下文。（右）8-shot 略升 Elo，增至 32-shot 相对零样本下降。两种少样本均远不及 best-of-$N$。少样本样例不含故事部分，聚焦标明胜方的辩论记录。各锦标赛分别以 Claude（2.0/2.1）为零策略。

A.10 Analysis of different preference models

A.10 不同偏好模型分析

Using a different preference model than the judge decreases accuracy with best-of-$N$. — To understand the impact of the preference model on accuracy, we ran the same set of best-of-$N$ and critique-and-refinement strategies used in the cross-play tournament (listed in Appendix D.4) but using GPT-4-Base. As shown in Figure 23, the self-play accuracy with a GPT-4-Turbo judge decreases. This can be explained due to the different preferences of these two models. The ranking of the arguments generated with best-of-$N$ will persuade the two models differently. Therefore, we match the preference and judge models to maximise debate accuracy. Having a disjoint preference model and judge does not invalidate the claim made that judge accuracy increases with debater strength because Figure 5 shows increasing linear trends for 3 different judges.

偏好模型与裁判不同会降低 best-of-$N$ 准确率。 — 为理解偏好模型对准确率影响，我们用 GPT-4-Base 重跑交叉对弈锦标赛中的同一组 best-of-$N$ 与批评–refinement 策略（附录 D.4 所列）。如图 23，用 GPT-4-Turbo 裁判的自我对弈准确率下降。可解释为两模型偏好不同：best-of-$N$ 生成论证的排序对两模型说服力不同。故匹配偏好与裁判模型以最大化辩论准确率。偏好与裁判分离并不否定「裁判准确率随辩手强度上升」——图 5 对 3 名不同裁判呈上升线性趋势。

The preference model’s selection probability for the incorrect debater decreases with rounds. — Figure 24 indicates that it becomes harder for the incorrect debater to convince the GPT-4-Turbo judge as the rounds go on and the opposite for the correct debater. The variance in the probabilities is much higher for the incorrect debater, too, as shown by the absolute difference between arguments rated the highest and the lowest in the best-of-16 debater. The larger variance suggests there is a variety of poor answers that the incorrect debater can use, and best-of-$N$ does the job of discarding the unpersuasive ones.

偏好模型对错误辩手的选择概率随轮次下降。 — 图 24 表明：随轮次推进，错误辩手更难说服 GPT-4-Turbo 裁判，正确辩手则相反。best-of-16 辩手最高与最低评分论证的绝对差显示，错误辩手概率方差也大得多。更大方差暗示错误辩手可用的差劲答案多样，best-of-$N$ 负责丢掉不具说服力者。

GPT-4-Base has a big shift in selection probability distribution between the correct and incorrect debaters. From Figure 24, you can observe a big difference in distribution between GPT-4-Turbo and GPT-4-Base. GPT-4-Base has a big positional bias, which may partially explain the shift in the distribution between correct and incorrect.

GPT-4-Base 在正确与错误辩手间选择概率分布有大偏移。 图 24 可见 GPT-4-Turbo 与 GPT-4-Base 分布差别大。GPT-4-Base 位置偏见强，或部分解释正确与错误间分布偏移。

Figure 23: Using a preference model which matches the judge model results in better scaling of judge accuracy as best-of-$N$ optimisation increases. We observe higher judge accuracy when the preference model aligns with the judge (GPT-4-Turbo). However, with GPT-4-Base, the correlation between debater skill and accuracy is less pronounced. One limitation is the assumption of consistent debater Elos from Figure 3, overlooking potential skill variations when altering the preference model.

图 23：偏好模型与裁判模型匹配时，best-of-$N$ 优化增强下裁判准确率扩展更好。偏好与裁判对齐（GPT-4-Turbo）时裁判准确率更高；用 GPT-4-Base 时辩手技能与准确率相关较弱。局限之一是假设图 3 辩手 Elo 不变，忽略改偏好模型时可能的技能变化。

Figure 24: Correct debaters become more persuasive as rounds continue for GPT-4-Turbo preference models. (Top) Our analysis reveals a decreasing selection probability for the incorrect debater over successive rounds, suggesting increased difficulty in persuading the judge. GPT-4-Turbo has an expanded probability range among the 16 samples from the incorrect debater. In contrast, GPT-4-Base consistently assigns lower probabilities to the incorrect debater, hinting at a significant positional bias. (Bottom) The rating histograms exhibit distinct distribution patterns: GPT-4-Turbo displays a broad probability spread with a pronounced tail, likely influenced by RLHF training, while GPT-4-Base reveals a noticeable shift in distribution between correct and incorrect debaters.

图 24：对 GPT-4-Turbo 偏好模型，正确辩手随轮次继续更有说服力。（上）分析显示错误辩手选择概率随轮次递减，暗示说服裁判更难。GPT-4-Turbo 在错误辩手 16 样本间概率范围更宽；GPT-4-Base 持续给错误辩手更低概率，暗示显著位置偏见。（下）评分直方图模式不同：GPT-4-Turbo 概率散布宽、尾部明显，或受 RLHF 训练影响；GPT-4-Base 在正确与错误辩手间分布明显偏移。

Appendix B Further Results with Human Judges

附录 B 人类裁判的进一步结果

B.1 Error Analysis

B.1 错误分析

Human judges chose the wrong answer 161 times in our test set, out of 953 total annotations. Judges chose the wrong answer in 22% of consultancies and 12% of debates. We investigate each of these judgements and label them based on the errors which contributed to the wrong answer. Each incorrect judgement can be caused by multiple errors. To perform this investigation, we read the transcripts, judge explanations, and relevant portions of the underlying texts.

测试集共 953 条标注中，人类裁判选错 161 次。咨询错误率 22%，辩论 12%。我们逐条审查这些判断，按导致错误的因素打标签；一次错误判断可对应多种错误。审查时阅读记录、裁判解释与原文相关部分。

Error	Description	Consultancies (%)	Debates (%)
Judge Errors
Cannot Extract Key Evidence	Judge cannot extract key evidence from the incorrect consultant	74	-
Judge Too Trusting	Judge goes along with incorrect consultant despite no plausible evidence	14	-
Judge Inattentiveness	Judge misreads the debate or fails to notice key evidence	11	25
Accident	Judge intended to choose one answer (based on explanation), but accidentally chose the other one	3	4
Invalid Logic	Judge made an invalid leap of logic	4	2
Misled by Intuition	Judge relied on incorrect intuition or priors	3	5
Debater Errors
Suboptimal Evidence Use	The correct debater made poor quote choices	13	67
Lack of Clash	The correct debater did not respond adequately to their opponents arguments	-	26
Debate is Confusing	Debate is difficult to follow, even for attentive judges	1	9
Judge Concern Not Addressed	Correct debater does not address questions from the judge	3	5
Uses Unverified Quotes	Honest debater uses unverified quotes	3	0
Subtle Errors
Difficult to Demonstrate via Quotes	The correct answer is difficult to demonstrate with quotes	5	26
Correct Answer is Implicit	Correct answer requires large inferences or guesses beyond the text	4	14

Table 2: Causes of incorrect judgements in debate and consultancy. Static and interactive protocols are combined here.

错误	描述	咨询（%）	辩论（%）
裁判错误
无法抽取关键证据	裁判无法从错误顾问处抽出关键证据	74	-
裁判过于信任	裁判在无合理证据时仍跟随错误顾问	14	-
裁判不专注	裁判误读辩论或未注意到关键证据	11	25
操作失误	裁判解释本欲选一方，却误选另一方	3	4
无效逻辑	裁判逻辑跳跃无效	4	2
被直觉误导	裁判依赖错误直觉或先验	3	5
辩手错误
证据使用欠佳	正确辩手引用选择差	13	67
缺乏交锋	正确辩手未充分回应对手论证	-	26
辩论令人困惑	即便专注裁判也难跟随	1	9
未回应裁判关切	正确辩手未回答裁判问题	3	5
使用未核验引用	诚实辩手使用未核验引用	3	0
微妙错误
难以用引用证明	正确答案难以用引用展示	5	26
正确答案是隐含的	正确答案需远超文本的推断或猜测	4	14

表 2：辩论与咨询中错误判断的原因。此处合并静态与交互协议。

Errors in Debate — Most incorrect judgements in debate were attributable to poor performance by the correct debater, primarily due to poor quote selection. Often the story would contain a quote that either clearly demonstrated the correct answer, or clearly disproved a claim made by the incorrect debater, and the correct debater would fail to choose this quote. Another common problem was repeating quotes that had already been used in earlier rounds, rather than presenting new evidence.

辩论中的错误 — 辩论中多数错误判断可归因于正确辩手表现差，主要因引用选择不佳。故事中常有清晰证明正确答案、或明确反驳错误辩手主张的引用，而正确辩手未能选用。另一常见问题是重复早轮已用引用，而非提出新证据。

Errors in Consultancy — By far the most common error in consultancy is an inability of the judge to extract key evidence from the incorrect consultant. Often, the incorrect consultant was able to strategically quote from the text to present evidence that seemed to plausibly support its’ answer, while avoiding important quotes that would prove it wrong. The judge attempts to elicit this contrary evidence with questions, but is often unsuccessful. It is then very difficult for the judge to know if no contrary evidence is shown because they are dealing with an honest consultant and no such evidence exists, or because they are dealing with a dishonest consultant who is hiding it.

咨询中的错误 — 咨询中最常见错误是裁判无法从错误顾问处抽出关键证据。错误顾问常策略性引用，呈现看似合理支持其答案的证据，同时回避会证伪的重要引用。裁判试图用提问引出反证，但常不成功。随后裁判很难判断：未见反证是因为诚实顾问、确实无此证据，还是不诚实顾问在隐藏。

B.2 Judge Variation

B.2 裁判差异

Figure 25: Variation among human judge performance is smaller in debate. (Top Left) The Proportion of correct to incorrect consultancies assigned to each judge. Consultant assigned answer was randomised per-consultancy, resulting in uneven distribution across judges. (Top Right) Distribution of judge accuracy per protocol. Each point is the mean accuracy of a single judge in that protocol. The overall consultancy accuracy for each judge is $\frac{1}{2}(\text{correct\_mean\_accuracy}+\text{incorrect\_mean\_accuracy})$. (Bottom Left) Distribution of judge confidence per protocol. Each point is the mean confidence a judge assigned their chosen answers. Each confidence rating ranges from 55%-95%. (Bottom Right) Distribution of mean judge Brier score per protocol. Consultancy results are combined in the same way as with accuracy. Brier score ranges from 0 to 1, with lower numbers representing better calibration.

图 25：辩论中人类裁判表现差异更小。（左上）各裁判被分配到正确/错误咨询的比例。顾问答案按次随机，故分布不均。（右上）各协议裁判准确率分布；每点为单裁判在该协议均值准确率。各裁判总体咨询准确率为 $\frac{1}{2}(\text{correct\_mean\_accuracy}+\text{incorrect\_mean\_accuracy})$。（左下）各协议裁判置信分布；每点为该裁判所选答案平均置信，范围 55%–95%。（右下）各协议平均 Brier 分数分布。咨询合并方式同准确率。Brier 0–1，越低校准越好。

We used 21 human judges, who were chosen based on their inter-annotator agreement and quality of their explanations (see Appendix E.2). Figure 25 shows the variance in accuracy, confidence and calibration between judges in consultancy and debate. Calibration is measured with a Brier score. Each judge performed an approximately equal number of annotations in each protocol. In consultancy protocols, answer assignment was randomised per-annotation. This means there is no guarantee that each judge received an equal number of correct and incorrect consultancies. For this reason, we calculate the combined consultancy accuracy for each judge as $\frac{1}{2}(\text{correct\_mean\_accuracy}+\text{incorrect\_mean\_accuracy})$. We combine static and interactive protocols for this analysis, as we do not observe significant differences between them. Debate has a lower variance in accuracy and calibration compared to consultancy. This is due to the very high variance in incorrect consultancy. Correct consultancy is quite similar to debate. Judges were also slightly more confident in their answers when judging debates.

我们使用 21 名人类裁判，依据标注者间一致性与解释质量选出（附录 E.2）。图 25 展示咨询与辩论中准确率、置信与校准的裁判间方差。校准用 Brier 分数。各裁判在各协议标注数大致相等。咨询协议中答案分配按次随机，故不保证每裁判收到等量正确/错误咨询；因此各裁判合并咨询准确率取正确与错误均值之半。静态与交互协议合并分析，因二者差别不显著。辩论在准确率与校准上方差低于咨询，主因错误咨询方差极高；正确咨询与辩论相近。裁判对辩论答案置信亦略高。

B.3 Question Variation

B.3 问题差异

We use 47 questions for the human test set. Question selection is described in Appendix D.1. Each question received an approximately equal number of annotations in each protocol. Figure 26 shows the variance in accuracy, confidence and judge calibration between questions in consultancy and debate. As with judges, the ratio of correct to incorrect consultancies per-question is not guaranteed to be equal. We combine static and interactive protocols for this analysis. The variance difference between debate and consultancy is less pronounced for questions than for judges. Correct consultancies are still much lower variance than incorrect consultancies.

人类测试集用 47 题，选题见附录 D.1。各题在各协议标注数大致相等。图 26 展示咨询与辩论中准确率、置信与裁判校准的问题间方差。与裁判类似，每题正确/错误咨询比例不保证相等。静态与交互合并分析。辩论与咨询对方差差别在问题上不如在裁判上显著。正确咨询方差仍远低于错误咨询。

Figure 26: Variation of per-question performance with human judges is lower in debate. (Top Left) The proportion of correct to incorrect consultancies assigned to each question. (Top Right) Distribution of question accuracy per protocol. (Bottom Left) Distribution of mean confidence in chosen answer per protocol. (Bottom Right) Distribution of mean question Brier score per protocol.

图 26：人类裁判下每题表现差异在辩论中更低。（左上）各题正确/错误咨询比例。（右上）各协议问题准确率分布。（左下）各协议所选答案平均置信分布。（右下）各协议平均问题 Brier 分数分布。

B.4 Human Judge Ensembles

B.4 人类裁判集成

Figure 27: Ensemble methods significantly improve human judge performance. Confidence-weighted majorities perform best in all protocols. Accuracy in all protocols are boosted by up to 10% by ensemble methods. Debate protocols still have higher accuracy after ensembling, achieving up to 98% accuracy with 6 votes.

图 27：集成方法显著提升人类裁判表现。置信加权多数在各协议最佳。各协议准确率因集成最多提升约 10%。集成后辩论仍更高，6 票可达约 98%。

We explore a variety of ensemble methods in an attempt to boost the accuracy of our oversight protocols. In our human experiments, each question was answered by several different judges under each protocol. We report mean accuracy in Figure 6, but here we consider the accuracy of different ensemble methods if the judgements for a question are pooled together for each protocol.

我们探索多种集成方法以提升监督协议准确率。人类实验中每题在各协议由多名裁判作答。图 6 报告均值准确率；此处考虑将同题同协议判断汇总后不同集成的准确率。

We consider the following ensemble methods:

我们考虑以下集成方法：

1. Most confident — From $N$ judgements, choose the one where the judge gave the highest confidence. If several are tied for most confident, choose randomly among them.
2. Strict majority — Use the answer that the majority of judges chose. If there is a tie, choose randomly.
3. Confidence-weighted majority — Sum the confidences assigned to each answer and choose the answer with the highest aggregate confidence.
4. Squared confidence-weighted majority — Similar to confidence-weighted majority, but square the confidences first. This gives much more weight to highly confident judgements.

1. 最高置信 — 从 $N$ 次判断中选裁判置信最高者；并列则随机。
2. 严格多数 — 取多数裁判所选答案；平局随机。
3. 置信加权多数 — 对各答案置信求和，取聚合置信最高者。
4. 平方置信加权多数 — 类似置信加权，但先平方置信，更重高置信判断。

For each ensemble method, we are interested in seeing how accuracy changes as more annotations are used. Given that we have already collected $N$ judgements for a particular question and protocol, we calculate the accuracy of an ensemble function $f$ for $k<=N$ annotations as the mean accuracy when $f$ is applied to all combinations $C_{i}$ of $N$ choose $k$.

对每种集成，我们关注随标注增多准确率如何变化。对已收集某题某协议 $N$ 次判断，集成函数 $f$ 在 $k<=N$ 标注上的准确率，取对所有 $N$ 选 $k$ 组合 $C_{i}$ 应用 $f$ 的均值准确率。

The number of annotations collected for each question/protocol combination varies from 4 to 8. In cases where $k>N$, we use the value for $f(N)$ instead.

各题/协议组合标注数从 4 到 8 不等。若 $k>N$，则用 $f(N)$ 的值。

Figure 27 shows that all ensemble methods increase accuracy for all protocols. The confidence-weighted protocols are the most successful, boosting debate accuracy from 88% to 95-98%.

图 27 显示全部集成方法提升全部协议准确率。置信加权协议最成功，将辩论准确率从 88% 提至 95–98%。

An open question is whether the benefits of ensembling are mainly due to variations in judge performance (i.e. reducing the impact of judge mistakes), or due to variations in debater skill (i.e. reducing the impact of honest debater mistakes).

开放问题是：集成收益主要来自裁判表现差异（降低裁判失误影响），还是辩手技能差异（降低诚实辩手失误影响）。

B.5 Human Judge Iteration

B.5 人类裁判迭代

Figure 28: Debate achieves high accuracy with human judges across wide variation in debater models and experimental setup. Each group here is a different experiment. Between experiments there were variations in prompting strategy, degree of optimisation, debater model, preference model, and word limits. Each experiment was done on a different set of questions so some variation between experiments is expected. Judge accuracy in static debate ranges from 86% to 92% across all experiments. Within each experiment, debate always outperforms consultancy. Error bars are standard error of the mean.

图 28：在辩手模型与实验设置大幅变化下，辩论对人类裁判仍达高准确率。各组为不同实验；实验间提示策略、优化程度、辩手模型、偏好模型与词数上限有差异。各实验用不同题集，故预期有差异。静态辩论裁判准确率跨实验在 86%–92%。各实验内辩论始终优于咨询。误差棒为均值标准误。

Before our main human experiment on the $D_{H}$ test set, we spent five weeks iterating on our experimental setup with humans, conducting seven smaller experiments. The overall goal of this iteration phase was to validate and refine our human experiment protocol, and to validate new debater models that seemed promising in our LLM-judged experiments. The experimental variations are described in the list below, and results are shown in Figure 28. Note that each experiment uses a different set of questions, because human judges cannot be given the same question multiple times. All experiments use the question selection criteria described in Appendix D.1, but there may still be variance in the difficulty of different question batches. Figure 29 shows the same plots as Figure 6, but using combined results from the test set and all iteration experiments.

主人类实验（$D_{H}$ 测试集）前，我们花五周与人类迭代实验设置，做了七个较小实验。目标是验证与 refining 人类实验协议，并验证 LLM 裁判实验中看起来有希望的新辩手模型。实验变体见下表，结果见图 28。注意各实验用不同题集，因人类裁判不能多次做同题。全部实验用附录 D.1 选题标准，但不同批次难度仍可能有方差。图 29 同图 6，但合并测试集与全部迭代实验。

Human judge iteration experiments - experiments were run between Nov 9th 2023 and Jan 26th 2024 (start dates given below):

人类裁判迭代实验——2023 年 11 月 9 日至 2024 年 1 月 26 日（下列为开始日期）：

1. Experiment 1 (Nov 9th) — 32 questions, 567 judgements. Static and interactive protocols. GPT-4-32k models, no inference-time optimisation (bo$N$ or c$N$). Motivation: Initial test of our experimental setup with humans.
2. Experiment 2 (Nov 15th) — 16 questions, 152 judgements. Static only. Changed model to GPT-4-Turbo. Using bo4 with a GPT-3.5-Turbo-Instruct preference model. Motivation: Test the impact of GPT-4-Turbo and inference-time optimisation in humans.
3. Experiment 3 (Nov 16th) — 16 questions, 170 judgements. Static only. bo4. Using a new prompt that encourages more conflict between debaters. Motivation: Evaluate new prompt variant.
4. Experiment 4 (Nov 24th) — 16 questions, 336 judgements. Static only. Only debates, no consultancies. First test of Critique & Refinement in humans. bo1, c8. Motivation: Test Critique and Refinement in humans.
5. Experiment 5 (Nov 30th) — 32 questions, 578 judgements. Static only. bo4. Change preference model to GPT-4-base. New prompts. Two experimental conditions - short and long. Short enforces 80-110 words per argument. Long enforces 300-350 words per argument. Motivation: Test the effect of argument length in humans (we had not previously enforced any word limits).
6. Experiment 6 (Dec 7th) — 16 questions, 318 judgements. Static only. Controlling words per round for the first time by giving consultants twice the word count of debaters. Debater arguments limited to 150 words, consultants limited to 300. bo4, c8. New prompt for debaters. Motivation: Test the effect of controlling for transcript length between protocols - rule out hypothesis that debate wins because transcripts are longer.
7. Experiment 7 (Dec 12th) — 16 questions, 241 judgements. Interactive only. New consultant prompt. bo16 for consultants, debaters use bo4 c6. Motivation: Test interactive protocols with our optimised models.
8. Experiment 8 (Dec 20th) — 47 questions, 953 judgements. Static and interactive protocols. bo16 for all models. Changed preference model to GPT-4-Turbo. Motivation: Get statistically significant results across all protocols using our most optimised models.
9. Experiment 9 (Jan 10th) — 24 questions, 480 judgements. Naive judge baseline in which humans were not provided with any transcript.
10. Experiment 10 (Jan 12th) — 32 questions, 519 judgements. Low Elo experiment using Claude-2.1 and GPT-3.5 models.

1. 实验 1（11 月 9 日）— 32 题，567 次判断。静态与交互。GPT-4-32k，无推理时优化（bo$N$ 或 c$N$）。动机：人类实验设置初测。
2. 实验 2（11 月 15 日）— 16 题，152 次。仅静态。改 GPT-4-Turbo；bo4，偏好模型 GPT-3.5-Turbo-Instruct。动机：测 GPT-4-Turbo 与推理时优化对人类影响。
3. 实验 3（11 月 16 日）— 16 题，170 次。仅静态。bo4。新提示鼓励辩手更多冲突。动机：评估新提示变体。
4. 实验 4（11 月 24 日）— 16 题，336 次。仅静态。仅辩论无咨询。人类首次测批评与 refinement。bo1，c8。动机：人类测批评与 refinement。
5. 实验 5（11 月 30 日）— 32 题，578 次。仅静态。bo4。偏好改 GPT-4-base。新提示。两条件：短（每论证 80–110 词）与长（300–350 词）。动机：测人类论证长度效应（此前未强制词限）。
6. 实验 6（12 月 7 日）— 16 题，318 次。仅静态。首次按轮控制词数：顾问词数两倍于辩手。辩手 150、顾问 300。bo4，c8。新辩手提示。动机：控制协议间记录长度，排除「辩论因记录更长而胜」假设。
7. 实验 7（12 月 12 日）— 16 题，241 次。仅交互。新顾问提示。顾问 bo16，辩手 bo4 c6。动机：用优化模型测交互协议。
8. 实验 8（12 月 20 日）— 47 题，953 次。静态与交互。全部模型 bo16。偏好改 GPT-4-Turbo。动机：用最优化模型在各协议获统计显著结果。
9. 实验 9（1 月 10 日）— 24 题，480 次。朴素裁判基线，人类不提供任何记录。
10. 实验 10（1 月 12 日）— 32 题，519 次。低 Elo 实验，用 Claude-2.1 与 GPT-3.5。

Figure 29: Aggregated experiment results are comparable to test set results. Here we show the same plots as in Figure 6, but with the 3315 judgements from experiments 1-8, rather than just the test set results (Experiment 8). (Left) Judge accuracy of different protocols (Middle) Receiver Operator Curve for reported judge confidences (Right) Calibration plots for judge confidences

图 29：聚合实验结果与测试集可比。同图 6，但用实验 1–8 的 3315 次判断，而非仅测试集（实验 8）。（左）各协议裁判准确率（中）报告置信的 ROC（右）裁判置信校准图

B.6 Low Elo debaters

B.6 低 Elo 辩手

Figure 30: Low Elo debaters achieve lower judge accuracy in debate with human judges. We choose two debater models that scored much lower Elo in our LLM-judged Swiss tournament, and evaluate them with human judges. Both consultants are much less persuasive than GPT-4-Turbo, as seen in a smaller judge accuracy gap between correct and incorrect consultants. We also seen lower debate performance compared to all of our experiments with GPT-4 (Figure 28). 519 judgements. Error bars are standard error of the mean.

图 30：低 Elo 辩手在人类裁判辩论中准确率更低。选取 LLM 裁判瑞士制锦标赛 Elo 明显更低的两名辩手模型，用人裁判评估。两顾问相对 GPT-4-Turbo 说服力弱得多，正确与错误顾问间裁判准确率差距更小。相对全部 GPT-4 实验（图 28）辩论表现亦更低。519 次判断。误差棒为均值标准误。

We investigated how low-Elo models perform when judged by humans (Figure 30). Our test set experiments used GPT-4-Turbo-bo16, which had an Elo rating of 141 across all of our LLM judges. For the low-Elo experiment, we use Claude-2.1-bo4 (Elo 79) and GPT-3.5-bo16 (Elo -60). Elo ratings use Claude-2.1-bo1 as the 0-point. In all cases we use GPT-4-Turbo as the preference model, and the intermediary judge in static consultancy uses the same model as the consultant.

我们考察低 Elo 模型在人类裁判下的表现（图 30）。测试集实验用 GPT-4-Turbo-bo16，跨全部 LLM 裁判 Elo 为 141。低 Elo 实验用 Claude-2.1-bo4（Elo 79）与 GPT-3.5-bo16（Elo -60）。Elo 以 Claude-2.1-bo1 为零点。全部用 GPT-4-Turbo 作偏好模型；静态咨询中介裁判与顾问同模型。

We see from the consultancy results that both low-Elo models are significantly less persuasive to humans than GPT-4-Turbo-bo16. We also see that this leads to significantly worse performance in debate (compare with Figure 6). Interestingly, the results from both low-Elo models are very similar, despite the Elo gap between them being larger than the gap between Claude-2.1-bo4 and GPT-4-Turbo-bo16. This suggests that the Elo rankings for human judges are different than those for LLM judges, indicating that humans and LLMs differ in what they are persuaded by.

咨询结果显示两低 Elo 模型对人说服力显著弱于 GPT-4-Turbo-bo16，并导致辩论表现显著更差（对比图 6）。有趣的是两低 Elo 结果很相近，尽管其间 Elo 差大于 Claude-2.1-bo4 与 GPT-4-Turbo-bo16 之差。暗示人类裁判 Elo 排名与 LLM 裁判不同，说明人类与 LLM 被说服的因素不同。

B.7 Human judge feedback

B.7 人类裁判反馈

After Experiment 4, we collected annotator feedback via a Google form. 16 out of 21 annotators completed it. Key insights are given here, and the full data will be released alongside the paper. The full data has numerous responses to qualitative questions about the protocols, efficacy of the models, and judging strategies.

1. 75% of judges said the found it easier to judge debates than consultancies, while 19% said they were the same.
2. 56% said that it is easier to judge longer debate arguments, while 38% said they’re the same.
3. No judges said that they usually decide their answer after the first round.
4. 44% said it would be useful if debates had more than 3 rounds.
5. 81% said that they mostly rely on quotes when making their decision.

1. 75% 裁判称辩论比咨询更好判，19% 称相同。
2. 56% 称更长辩论论证更好判，38% 称相同。
3. 无裁判称通常第一轮后就决定答案。
4. 44% 称辩论超过 3 轮会有用。
5. 81% 称做决定时主要依赖引用。

Appendix C Recommendations for working on debate

附录 C 辩论研究建议

This section explains three different task setups for debate where information or capability differ between the debaters and judges. With current model capabilities up to GPT-4, we then motivate why information asymmetry and the QuALITY dataset is a good test bed for debate. We then provide iteration and implementation recommendations to others who want to work on debate in the future.

本节说明三种辩手与裁判间信息或能力不同的辩论任务设定。在当前至 GPT-4 的模型能力下，论证为何信息不对称与 QuALITY 数据集是辩论的良好试验床。随后给出未来从事辩论工作者的迭代与实现建议。

C.1 Task setup variations

C.1 任务设定变体

We considered three different regimes for debate:

我们考虑辩论的三种体制：

1. Capability asymmetric — the debaters are stronger than the judge with respect to base model capability. For example, using GPT-4-Turbo debaters and a GPT-3.5-Turbo judge. Here, the baseline to compare to is the GPT-3.5-Turbo judge’s few-shot or chain of thought performance.
2. Information asymmetric — the debaters have access to privileged information about the question, which the judges do not. For example, all agents are GPT-4-Turbo, but the judge cannot access the necessary text to answer the question. The baseline to compare to in this setting is consultancy, where the judge engages with only one copy of the model.
3. Symmetric — the debaters and judges have equal capability and information. In practice, this means all agents in the debate game use the same base LLM, such as GPT-4-Turbo. The baseline to compare here is the accuracy of GPT-4-Turbo by using few-shot examples or chain-of-thought reasoning.

1. 能力不对称 — 辩手在基座能力上强于裁判。例如 GPT-4-Turbo 辩手与 GPT-3.5-Turbo 裁判。比较基线为 GPT-3.5-Turbo 裁判的少样本或思维链表现。
2. 信息不对称 — 辩手掌握裁判所无的问题特权信息。例如全部智能体为 GPT-4-Turbo，但裁判无法访问答题所需文本。比较基线为咨询：裁判只与模型的一份副本交互。
3. 对称 — 辩手与裁判能力与信息相等。实践中辩论游戏全部智能体用同一基座 LLM，如 GPT-4-Turbo。比较基线为 GPT-4-Turbo 少样本或思维链准确率。

We call a protocol an inference-time protocol if it uses models which are not trained or fine-tuned for the task. Augmentation occurs when the protocol runs in deployment using methods such as few-shot learning, chain of thought or best-of-$N$ (where the most persuasive argument is chosen from many completions by a preference model). Protocols may be used for self-improvement if multiple copies of the same model are used to improve accuracy on a task. An example is self-refine (Madaan et al. 2023), where models give feedback on completions before further refining their answers.

若协议所用模型未经针对任务的训练或微调，我们称其为推理时协议。增强指协议在部署中用少样本、思维链或 best-of-$N$（由偏好模型从多份补全选最具说服力论证）等方法。若用同一模型多份副本提升任务准确率，协议可用于自我改进。例为 self-refine（Madaan et al. 2023）：模型先对补全给反馈再 refining 答案。

We discuss in the next section why debate does not perform well for self-improvement at inference time in the symmetric regime for current models.

下一节讨论：为何对当前模型，辩论在对称体制下作为推理时的自我改进表现不佳。

C.2 Task setup findings

C.2 任务设定发现

Inference-time debate is ineffective for self-improvement in symmetric regimes with current LLMs. In our prior research, we examined the efficacy of debate as a self-improvement protocol in symmetric regimes. This approach involved leveraging debate in an attempt to enhance an LLM’s reasoning and factual accuracy. We used GPT-4 to argue between two potential answers to various questions in datasets such as MMLU (Hendrycks et al. 2020) before scoring the accuracy of a GPT-4 judge that deliberated between debater arguments. Our findings indicate that models using debate produce lower test accuracy compared to 0-shot or chain-of-thought approaches, as evidenced by the accuracy observed across 10 tasks in Figure 31. Notably, only in TruthfulQA (Lin et al. 2022) does Debate outperform zero-shot, 5-shot, and chain of thought methodologies.

推理时辩论在对称体制下对当前 LLM 的自我改进无效。 先前研究中我们考察辩论作为对称体制自我改进协议的效力：试图用辩论增强 LLM 推理与事实准确率。用 GPT-4 在 MMLU（Hendrycks et al. 2020）等数据集两候选答案间辩论，再评分审议辩手论证的 GPT-4 裁判准确率。发现相对 0-shot 或思维链，辩论测试准确率更低，见图 31 跨 10 任务结果。仅 TruthfulQA（Lin et al. 2022）上辩论优于零样本、5-shot 与思维链。

Figure 31: Debate is not effective for self-improvement in symmetric regimes. Accuracy of debate vs single and multi-model (Du et al. 2023) baselines across 10 open-source benchmarks. With the exception of TruthfulQA, we find that debate has worse performance than ”Single-Model”, which encapsulates the best result between zero-shot, 5-shot, chain-of-thought and plan&solve. In this regime, the debaters and judges are the same models, so there is no knowledge asymmetry. Thus, inference-time debate does not help with self-improvement. We believe training models and using harder datasets could still be promising for future work.

图 31：对称体制下辩论对自我改进无效。跨 10 个开源基准，辩论对单模型与多模型（Du et al. 2023）基线的准确率。除 TruthfulQA 外，辩论差于「单模型」（零样本、5-shot、思维链与 plan&solve 中最佳）。此体制辩手与裁判同模型，无知识不对称，故推理时辩论无助自我改进。我们认为训练模型并用更难数据集对未来工作仍有希望。

We hypothesised that debate might perform better on datasets where the judge model has low 0-shot accuracy (all of our prior datasets had a 0-shot accuracy above 70%). GPQA (Rein et al. 2023) is a benchmark written by biology, physics, and chemistry domain experts. It has questions that are hard to find the answer to on the internet, and GPT-4 has an accuracy of less than 40% with few-shot chain of thought. Anecdotally, when applying debate we found debaters lacked the knowledge to build a persuasive argument, thereby not assisting the judge in choosing the correct answer.

我们假设：在裁判 0-shot 准确率低的数据集上辩论可能更好（先前数据集 0-shot 均超 70%）。GPQA（Rein et al. 2023）由生物、物理、化学领域专家撰写；答案难在网上找到，GPT-4 少样本思维链准确率低于 40%。轶事上，应用辩论时辩手缺乏构建有说服力论证的知识，因而无助裁判选对。

In conclusion, debate in our inference time setup does not lead to truth-seeking behaviour in situations where the judge has the same information and capability as the debaters. However, we think that work on self-improvement with debate is still promising, especially when training models (which we did not do) and using harder datasets.

结论：在裁判与辩手信息与能力相同的情形下，我们的推理时辩论设定未导向求真行为。但我们认为用辩论做自我改进仍有希望，尤其在训练模型（我们未做）并用更难数据集时。

Inference-time debate is ineffective for regimes with capability asymmetry for model capabilities up to Claude-2. In a sandwiching setup, we use a less capable judge, Claude 2, with more proficient debaters like GPT-4. Capability asymmetry arises due to the different reasoning capabilities of the models. Despite this asymmetry, we found no improvement to judge’s performance when running our debate protocol. For instance, Claude’s performance on tasks such as MMLU did not surpass zero-shot despite access to arguments by duelling GPT-4 debaters.

至 Claude-2 能力下，能力不对称体制的推理时辩论无效。 在夹心设定中，用较弱裁判 Claude 2 与更强辩手如 GPT-4。不对称来自推理能力差异。尽管如此，辩论协议未改进裁判表现。例如 Claude 在 MMLU 等任务上即便可访问对决 GPT-4 辩手的论证，也未超过零样本。

Our hypothesis was that the weaker judge’s deliberation ability was the bottleneck for getting this setting to work. Once the next generation of models is released, the sandwiching setup is worth revisiting, especially on difficult domains such as GPQA.

我们假设弱裁判的审议能力是瓶颈。下一代模型发布后，夹心设定值得重访，尤其在 GPQA 等困难领域。

Inference-time debate is effective at increasing judge accuracy in a task that has information asymmetry. Our findings indicate that information asymmetry is important when testing the effectiveness of debate with LLM capability up to GPT-4-Turbo. Information asymmetry forces the judge to rely on the debaters for problem resolution. In the QuALITY setup, the asymmetry stems from the judge’s lack of access to the story necessary for answering the reading comprehension question. By contrast, in knowledge-symmetric scenarios, the judge can draw upon their own expertise to address the problem and disregard the debate transcript entirely. Another advantage of information asymmetry over capability asymmetry is being able to use the most capable models as the judge. At the time of writing, this is GPT-4-Turbo, and we find the improved deliberation capability crucial for reliably iterating on debate protocols since the aggregate ELOs are spread out over a larger range (as shown by Figure 25).

信息不对称任务上，推理时辩论有效提升裁判准确率。 发现表明：在测试至 GPT-4-Turbo 能力的辩论效力时，信息不对称很重要。信息不对称迫使裁判依赖辩手解题。QuALITY 设定中，不对称来自裁判无法访问阅读理解所需故事。相较知识对称场景，裁判可凭自身专长解题、完全无视辩论记录。信息不对称相对能力不对称的另一优点是可用最强模型作裁判。撰写时为 GPT-4-Turbo；我们发现更好的审议能力对可靠迭代辩论协议至关重要，因总体 Elo 分布区间更大（见图 25）。

C.3 Dataset recommendations

C.3 数据集建议

We recommend QuALITY as a dataset to test debate. The QuALITY dataset is an excellent resource for testing debate, primarily due to the ability to establish information asymmetry and the rarity of self-defeating behaviour in LLM debaters. The latter is largely attributed to its questions, which are inherently easier to argue for either side due to being able to select quotes from lengthy narratives.

我们推荐 QuALITY 作为测试辩论的数据集。 QuALITY 适合测试辩论，主因能建立信息不对称，且 LLM 辩手自我挫败行为稀少。后者 largely 因其问题可从长叙事选引用，任一方都较易辩护。

Advantages of QuALITY:

QuALITY 优点：

• • QuALITY’s stories average 7k tokens in length, and are used to create the asymmetry between the debaters and judges. Each story provides plenty of material for debaters to analyse and discuss over three debate rounds. This breadth of context enables the strategic use of quotes, potentially in a deceptive manner.
• • Extensive metadata accompanying each question is available, such as the proportion of dataset annotators agreeing that the question is answerable. Filtering for high answerability ensures the reliability of the question set for debate experiments, safeguarding against debates on unresolvable topics.
• • An option to use a highly plausible but incorrect answer option known as the “best distractor”. Using the best distractor reduces the likelihood that the debater assigned to that answer uses self-defeating behaviour (such as conceding to the opposing side). One reason for the reduced likelihood is that there are many quotes can make it seem like the best distractor answer is true, so we avoid the issue of RLHF trained models refusing to argue for a falsehood.

• QuALITY 故事平均约 7k token，用于制造辩手与裁判不对称。每故事为三轮辩论提供充足材料；语境广度支持策略性引用，甚至欺骗性使用。
• 每题有丰富元数据，如标注者认为可答的比例。过滤高可答性确保辩论实验问题集可靠，避免不可裁决议题。
• 可选高度合理但错误的「最佳干扰项」。用最佳干扰项降低被分配该答案的辩手自我挫败（如向对方让步）概率；原因之一是许多引用可使干扰项看似为真，从而避免 RLHF 训练模型拒绝为虚假辩护。

Many other datasets such as MMLU and AQuA (Goswami et al. 2023) are less suitable to test debate with current models because it can be very challenging to craft a proof for the incorrect answer in a persuasive way. If unable to formulate a convincing argument, the LLM debater might often concede or acknowledge subtle errors introduced in their solution to align with the assigned answer. Self-defeating behaviour is not a surprise, given that RLHF training promotes honesty. RLHF training moderately promotes answers that look good to humans which can often be linked with correctness, but in debate, we need to generate plausible arguments for incorrect answers. Our attempts to mitigate self-defeating behaviour via prompt engineering were unsuccessful. We believe helpful-only models would fare better as debaters in these datasets. We fine-tuned GPT-3.5-Turbo for debate, but found that it could not produce compelling debates or adhere to intricate prompts. We’re excited for future work fine-tuning GPT-4-Turbo for debate.

MMLU、AQuA（Goswami et al. 2023）等许多数据集较不适合测当前模型辩论，因很难为错误答案写出有说服力的「证明」。若无法形成令人信服的论证，LLM 辩手常会让步或承认解法中引入的细微错误，以对齐所分配答案。鉴于 RLHF 促进诚实，自我挫败并不意外。RLHF 适度促进看起来对人类好的答案，常与正确相关；但辩论需要为错误答案生成看似合理的论证。我们用提示工程缓解自我挫败未成功。相信仅 helpful 模型在这些数据上作辩手更好。我们微调 GPT-3.5-Turbo 做辩论，但无法产生有力辩论或遵循复杂提示。期待未来微调 GPT-4-Turbo 做辩论。

C.4 Iteration recommendations

C.4 迭代建议

Optimise for Elo rating rather than judge accuracy. Ultimately, we want debate to improve the judge accuracy on a task. However, it is important not to optimise for this directly when improving debaters. In competitive debate settings, rewarding debaters for persuasiveness to the judge is crucial for developing the skills required to win, making Elo a great metric. The calculated Elo should consider the aggregate win rate when the debater gets a chance to argue for both sides of the argument. Therefore, the debater with the best Elo will be the most persuasive when arguing for correct and incorrect answers. Elo also has the advantage of being an unsupervised metric. An example of where optimising for accuracy can go wrong is when debaters have self-defeating behaviour. Incorrect debaters are much more likely to concede, leading to inflated accuracy since the judge can easily choose the correct answer.

优化 Elo 评分而非裁判准确率。 最终希望辩论提升任务上裁判准确率；但改进辩手时不宜直接优化它。竞技辩论中，奖励对裁判的说服力对培养获胜技能至关重要，故 Elo 是好指标。计算 Elo 应考虑辩手有机会为双方论证时的总体胜率；因此最佳 Elo 辩手在为正确与错误答案辩护时都最有说服力。Elo 亦为无监督指标。直接优化准确率出错的例子：辩手有自我挫败行为时，错误辩手更易让步，裁判易选对，准确率虚高。

Test prompt improvements against each base model family. Testing each prompt is tedious, but we found small prompt changes can lead to unexpected behaviour. For instance, we reworded a small portion of our consultant prompt, and an extra 3% of the total questions became affected by concessions. Furthermore, it is important to test with multiple LLMs since we found that some prompt changes did not transfer from the family of Claude models to GPT models.

对每个基座模型族测试提示改进。 逐提示测试繁琐，但我们发现小改可导致意外行为。例如微调顾问提示一小段后，额外 3% 题目受让步影响。亦须用多个 LLM 测试，因部分提示改动无法从 Claude 族迁移到 GPT 族。

Try out interactive judging yourself while iterating on protocols. We used the same annotation platform (see Appendix E.1) provided in our human trial as a tool to read transcripts and judge debate questions interactively. We learned a lot about our debaters’ argument quality when judging questions. Acting as the judge without knowing the answer allows you to learn how persuasive the incorrect debater is compared to the correct one. In addition, it also allows you to find self-defeating behaviour and subsequently change the prompt to stop it from happening. Duplicate quoting, lack of engagement with the opponent or interactive judge and inability to use quote tags correctly were other examples of failure modes we quickly picked up on by reading lots of transcripts.

迭代协议时亲自做交互裁判。 我们用人试验同一标注平台（附录 E.1）阅读记录并交互裁判辩论题。作裁判时学到很多辩手论证质量。在不知答案时作裁判，可体会错误辩手相对正确辩手的说服力；亦可发现自我挫败并改提示阻止。重复引用、不交锋对手或交互裁判、无法正确用引用标签等失败模式，亦通过大量阅读记录迅速察觉。

C.5 Implementation recommendations

C.5 实现建议

Use a Swiss-style tournament when calculating Elos for multiple debaters. Running cross-play tournaments is compute-intensive, especially if running a round-robin tournament where each debater variant plays each other. Implementing a Swiss-style tournament was vital for fast iteration and reducing costs. A Swiss tournament assumes you start with an initial seed ranking of your players, then make pairings based on the nearest neighbour in the list. This list is then sorted after each round based on the total number of matchups won. This means it has $O(n\log n)$ complexity rather than $O(N^{2})$. A helpful framing is that the tournament implements an adaptive sorting of the initial list to ultimately find the true final ranking of the players. Once the tournament is complete, we use the fine-grain win rates over the question set the debaters used for each match-up and use the method in Appendix D.5 to find Elo ratings for each player.

计算多名辩手 Elo 时用瑞士制锦标赛。 交叉对弈很耗算力，尤其循环赛每变体互打。瑞士制对快速迭代与降本至关重要：假设有初始种子排名，按列表近邻配对，每轮后按累计胜场排序；复杂度 $O(n\log n)$ 而非 $O(N^{2})$。可理解为对初始列表做自适应排序以找真实最终排名。赛后用各对阵在问题集上的细粒度胜率，按附录 D.5 方法求各选手 Elo。

Judge debates with both answer permutations to mitigate LLM positional bias Positional bias is evident when LLM judges achieve different accuracy levels if answers A and B are swapped. It is especially common in less capable models such as GPT-3.5-Turbo. This bias is less pronounced but still present in more sophisticated models like GPT-4-Turbo. To mitigate this, we encourage running judgements with both permutations of answer position for every debate transcript (see Table 7).

用两种答案排列裁判辩论以缓解 LLM 位置偏见。 A/B 互换时 LLM 裁判准确率不同即位置偏见；GPT-3.5-Turbo 等弱模型尤甚；GPT-4-Turbo 等更强模型仍存在但较弱。为缓解，建议对每场辩论记录用两种答案位置排列都做裁判（见表 7）。

Apply strict word limit filters or truncation to mitigate verbosity bias in LLM judges. Verbosity bias is when the judge favours longer arguments, which we found to be the case with a significant rise in Elo rating as the argument word count increased. Decoupling this bias and argument quality is difficult, and we found qualitatively that this preference can falsely attribute skill to lengthier arguments. To mitigate this, we recommend using a strict word limit with rejection sampling as in Appendix D.6.

用严格词限过滤或截断缓解 LLM 裁判冗长偏见。 冗长偏见指裁判偏好更长论证；我们发现论证词数上升时 Elo 显著上升。将此偏见与论证质量解耦困难；定性上该偏好可把技能错误归因于更长论证。建议如附录 D.6 用严格词限与拒绝采样。

LLMs are poor at deliberation, so use at least GPT-4-Turbo capability for a judging model. Through our experience of protocol iteration, we often found confusing results after judging with GPT-3.5-Turbo and Claude, particularly with a smaller sample size. Weaker judges are more prone to being misled by trivial aspects of one side of an argument. GPT-4’s superiority as a judge is evidenced by a notably wider performance gap recovered, as shown in Figure 9, and it also has a larger area under the curve in Figure 5.

LLM 审议能力弱，裁判模型至少用 GPT-4-Turbo 级。 协议迭代中用 GPT-3.5-Turbo 与 Claude 裁判常得令人困惑的结果，尤其样本较小时。弱裁判更易被论证琐碎方面误导。GPT-4 作裁判优势见更大恢复表现差距（图 9）与图 5 更大曲线下面积。

Use the same prompts for both the correct and incorrect debater. It might be tempting to use a different prompt for the incorrect debater, perhaps explaining deceptive strategies to improve performance. There are several reasons why this is ill-advised:

正确与错误辩手用相同提示。 或想给错误辩手不同提示、讲解欺骗策略以提升表现；但有若干理由不建议：

1. Admitting to the model that it is arguing the incorrect answer or it should use deceptive strategies is more likely to lead to self-defeating behaviour in current RLHF-based models, thus invalidating the debate. If a model gives up every time on the incorrect side, the judge accuracy is 100%, but the protocol is useless when you don’t have labels and the model does not know the correct answer.
2. The behaviour between debaters could be subtly different, leading to information leakage to the judge.
3. In a reinforcement learning setup where debaters are rewarded for convincing the judge, the copies of the model should operate with identical prompts.
4. At inference time, you may not have labels for the correctness of the answer choices, so you would not know which answer to assign to which debater.

1. 向模型承认其在为错误答案辩护或应使用欺骗策略，更易导致当前 RLHF 模型自我挫败，从而使辩论失效。若错误方每次放弃，裁判准确率 100%，但无标签且模型不知正确答案时协议无用。
2. 辩手间行为可微妙不同，向裁判泄漏信息。
3. 强化学习设定中辩手因说服裁判获奖励时，模型副本应使用相同提示。
4. 推理时可能无答案正确性标签，不知应将哪一答案分给哪一辩手。

Appendix D Implementation Details

附录 D 实现细节

D.1 Filtering Questions

D.1 过滤问题

Dataset — QuALITY (Pang et al. 2022) is a multiple-choice Q&A dataset for long-document comprehension. It contains documents from a number of sources, such as Slate articles or project Gutenberg short stories. Each document has a set of comprehension questions (with 4 possible answers) written by crowdworkers who have first read the entire document. Different sets of crowdworkers then attempt to answer the questions under 2 possible conditions: 1) Untimed annotators can take as much time as they want to read the document when answering the question; 2) Speed annotators are only allowed to read the document for 45 seconds before answering. The annotators also provide feedback on each question, giving ratings on 1) if the question is answerable and unambiguous 2) How much context from the passage is required to answer 3) Which unchosen answer is the best ”distractor” answer (question writers were encouraged to write difficult distractor answers).

数据集 — QuALITY（Pang et al. 2022）是长文档理解多选问答数据集，含 Slate 文章、Project Gutenberg 短篇故事等来源。每文档有一批理解题（4 选项），由先通读全文的众包撰写。另有两批标注者在两种条件下答题：1）不计时标注者可随意时长阅读；2）限时标注者仅允许读 45 秒。标注者亦对每题反馈：1）是否可答且无歧义；2）需多少篇章语境；3）未选答案中最佳「干扰项」（出题者被鼓励写难干扰项）。

Question Selection — We use only questions from the project Gutenberg short science fiction stories to ensure that judges have no prior real-world knowledge to influence their answers. Most of the stories are from the 1950s-1970s, making it unlikely that our human judges have read them. We wanted to select questions that were difficult enough to generate non-trivial debates, while still having clear and unambiguous correct answers. To this end, we applied the following filtering to the QuALITY dataset:

选题 — 仅用 Project Gutenberg 科幻短篇问题，确保裁判无真实世界先验影响答案。故事多来自 1950–1970 年代，人类裁判读过的可能性低。希望题足够难以产生非平凡辩论，同时正确答案清晰无歧义。故对 QuALITY 施加如下过滤：

1. 100% of untimed annotators chose the correct answer
2. Less than 50% of timed annotators chose the correct answer
3. All untimed annotators agree that the question is answerable and unambiguous
4. Average ”context required” rating from untimed annotators is at least 1.5
5. Writer label matches the gold label (the answer voted as the correct answer by annotators matches what the question writer labelled as the correct answer)

1. 100% 不计时标注者选对
2. 少于 50% 限时标注者选对
3. 全部不计时标注者同意题目可答且无歧义
4. 不计时标注者平均「所需语境」评分至少 1.5
5. 作者标签与金标签一致（标注者投票正确答案与出题者标注一致）

This is essentially a stricter subset of the HARD QuALITY questions.

本质上是 HARD QuALITY 问题的更严子集。

QuALITY questions have 4 answers, however for our debates we used only 2 answers. For each question, we used the correct answer and the best ”distractor” answer. For the human experiments, we manually reviewed each question and pair of answers to ensure they made sense. We removed questions that were incompatible with our 2-answer requirement, e.g. questions where one answer was ”all of the above”, ”none of the above”, etc.

QuALITY 每题 4 答案，辩论仅用 2 个：正确答案与最佳「干扰项」。人类实验中人工审阅每题与答案对，去掉与双答案要求不符者，如「以上皆是」「以上皆非」等。

Human vs LLM judge considerations — For our LLM judges, we could use the same set of questions for each experiment. However, since humans remember past experiments, unlike LLM judges, we had to apply the following additional criteria when selecting questions for each human experiment: 1) This question has not been seen by human judges before 2) No other question in this story has been seen by human judges before (to avoid leaking any prior knowledge). This left us with a total of 200 usable questions for the human judge experiments.

人类与 LLM 裁判考量 — LLM 裁判各实验可用同一题集；人类会记住既往实验，故每人实验选题另加：1）人类裁判未见过该题；2）该故事其它题亦未见过（避免先验泄漏）。最终人类裁判实验共约 200 道可用题。

D.2 Quote Verification System

D.2 引用核验系统

Having access to the text, unlike the judge, the debater’s best strategy for winning a debate is to choose compelling evidence from the story that supports their assigned answer. Debaters are told in their prompt that any quotes they use should be wrapped in XML tags. Here is a snippet from their prompt (see Appendix G.1) explaining the quote system to the debater.

辩手可访问文本而裁判不可，其最佳获胜策略是从故事中选取支持所分配答案的有力证据。提示要求引用用 XML 标签包裹。以下为提示片段（完整见附录 G.1）向辩手说明引用系统。

Our quote verification system is a simple direct string match after normalisation has been applied to both the quote and the story text. Normalisation involves stripping punctuation and making the text lowercase.

引用核验系统在对引用与故事文本规范化后做简单直接字符串匹配。规范化包括去标点并转小写。

There are three possible cases after a debater uses a quote:

辩手使用引用后有三种情形：

• • If it is a direct string match, then the quote is wrapped in v_quote XML tags to indicate it has been verified.
• • If it fails the string match, then it is wrapped in u_quote XML tags to indicate it is unverified.
• • If the debater does not put quote tags around the quote, then it will not be processed by the quote verification system and will remain as is.

• 若直接字符串匹配，用 v_quote XML 标签包裹表示已核验。
• 若匹配失败，用 u_quote XML 标签包裹表示未核验。
• 若辩手未给引用加标签，则不被核验系统处理，原样保留。

D.3 Optimising Debaters

D.3 优化辩手

Best-of-N — At a high level, best-of-$N$ (bo$N$) is a mechanism that allows an LLM to generate many completions, followed by having a preference model choose one response that does well on a specific set of criteria. The preference model used is often another LLM prompted to assign a reward based on the criteria provided. The temperature of the LLM can be raised as the number of completions increases to encourage diverse samples.

Best-of-N — 高层上，best-of-$N$（bo$N$）让 LLM 生成多份补全，再由偏好模型按特定准则选一份。偏好模型常为另一 LLM，按所给准则赋奖励。随补全数增加可提高温度以鼓励多样采样。

In our case, we use bo$N$ to select the most persuasive argument a debater could use during the debate round, according to a judge. See how bo$N$ works alongside critique-and-refinement in Algorithm 1.

我们用 bo$N$ 按裁判视角选出辩论轮次中最具说服力的论证。bo$N$ 与批评–refinement 如何配合见算法 1。

The first stage of bo$N$ is to generate $N$ completions from the debater in the current round of the debate, followed by extracting the argument in tags (since we do not want the judge preference model to see the hidden scratchpad). Next, we apply the quote verification system as explained in Appendix D.2.

bo$N$ 第一阶段：在当前辩论轮从辩手生成 $N$ 份补全，再从 标签提取论证（因不想让裁判偏好模型看到隐藏草稿）。接着按附录 D.2 施加引用核验。

The second stage is getting a rating from the preference model (which does not have access to the the story like the judge) on how persuasive the argument is. As we use simultaneous debate, the correct and incorrect debaters concurrently generate responses for a given round. Therefore, we provide a dummy opponent response when appending the candidate arguments to the transcript. We set this to "My answer is the best choice, and my opponent is wrong.".

第二阶段：偏好模型（同裁判一样无法访问故事）对论证说服力评分。因用同时辩论，正确与错误辩手并行生成该轮回应；故将候选论证追加到记录时提供虚拟对手回应：「我的答案是最佳选择，对手是错的。」

The preference model has a similar prompt to the judge but specific instructions to output A and B only (see Appendix G.4). We sample a single token from GPT-4-Turbo and extract the top 5 log probabilities to provide a score for tokens A and B. If the debater is the correct debater and defending assignment A, then the score used will be the log probability for A. On the rare occasion that the log probability for A or B is not present in the top 5, we assign it a value of -100, which will reject that sample compared to others.

偏好模型提示类似裁判，但明确指示只输出 A 与 B（附录 G.4）。我们从 GPT-4-Turbo 采样单 token，取前 5 对数概率为 A、B 打分。若辩手为正确方且辩护 A，则用 A 的对数概率。若 A 或 B 对数概率不在前 5（少见），赋 -100，相对其它样本拒绝该样本。

Once all N arguments are rated, the one with the greatest log probability is chosen to use in the transcript, while the others are discarded. See an example in Table 3 that shows how the judge sees a potential argument alongside the dummy answer and the log probability given for A being chosen by the preference model judge.

全部 N 条论证评分后，选对数概率最大者写入记录，其余丢弃。表 3 示例展示裁判如何看到潜在论证与虚拟答案，以及偏好模型裁判给出选 A 的对数概率。

We found that using the same preference model as the judge model led to the largest gains in aggregate Elo and judge accuracy (as shown in Figure 23).

我们发现偏好模型与裁判模型相同时，总体 Elo 与裁判准确率增益最大（图 23）。

If using a completion model instead of chat, it can be helpful to append an assistant message such as “Assistant: Based on the provided arguments, the correct answer is most likely to be (”. This trick ensures that the model is biased towards giving logits for A or B.

若用补全模型而非聊天模型，可追加助手消息如「助手：基于所提供论证，正确答案最可能是（」。此技巧使模型偏向给出 A 或 B 的 logits。

Round Arguments	Rating for A
Debater A: The narrator, J.F. Corrigan, exhibits a skeptical and dismissive attitude toward Ildwar Gorb. Initially, when Gorb claims to be from an Earth-like planet, Corrigan immediately suspects a scam, stating, ”I don’t mind conning the public from time to time, but I draw the line at getting bilked myself.” Furthermore, when Gorb persists, Corrigan bluntly retorts, ”Good-by, Mr. Gorb.”, signifying a definitive lack of faith or admiration for Gorb’s deceit. Corrigan’s consistent dismissal of Gorb, both as a phony alien and potential hero, is palpable throughout the encounter.
Debater B: My answer is the best choice and my opponent is wrong.	-0.00015490896
Debater A: The narrator’s persistent suspicion and lack of respect for Gorb’s claims clearly indicate that he does not see Gorb as a hero. This is evidenced when he refers to Gorb’s proposition as ”the best cN switch yet—an Earthman posing as an alien to get a job” and expresses direct annoyance, ”I wasn’t buying it, even if I could appreciate his cleverness intellectually.” Nowhere does the narrator express admiration or attribute noble qualities to Gorb, which would be necessary for the label of ”hero”.
Debater B: My answer is the best choice and my opponent is wrong.	-0.047604296

Table 3: Debate best-of-$N$ example where one candidate answer from Debater A is put in the transcript alongside a dummy argument for Debater B. It also shows the log probabilities from GPT-4-Turbo which indicate the first sample is more persuasive.

轮次论证	A 的评分
辩手 A：叙述者 J.F. Corrigan 对 Ildwar Gorb 持怀疑与轻蔑态度。……
辩手 B：我的答案是最佳选择，对手是错的。	-0.00015490896
辩手 A：叙述者持续怀疑且不尊重 Gorb 的主张，清楚表明他不视 Gorb 为英雄。……
辩手 B：我的答案是最佳选择，对手是错的。	-0.047604296

表 3：辩论 best-of-$N$ 示例：辩手 A 一份候选答案与辩手 B 虚拟论证一并放入记录；亦展示 GPT-4-Turbo 对数概率，表明第一份样本更有说服力。

Critique and Refinement Critique-and-refinement is a technique to improve the initial output of an LLM on a specific task. Other work, such as Self-Refine (Madaan et al. 2023), uses iterative feedback and refinement steps using the same model to improve its initial output. This can be extended by generating many critiques and choosing the most helpful one (Saunders et al. 2022).

批评与 Refinement 批评–refinement 用于改进 LLM 在特定任务上的初始输出。其它工作如 Self-Refine（Madaan et al. 2023）用同一模型迭代反馈与 refining。可扩展为生成多条批评并选最有帮助者（Saunders et al. 2022）。

In our debate protocol, we ask a critic model that has access to the story to produce $N_{C}$ critiques of the argument after it is extracted and processed by the quote verification system. A preference model then gives a rating to each of the critiques based on which is most helpful for the debater to improve their argument. Helpful is described as being contextually relevant and directly applicable to strengthening the debater’s assigned answer. The critique preference model is asked on the final line of the prompt if the critique is helpful Y/N, and the critique rating is assigned to the log probability corresponding to “Y”. Only a single token needs to be sampled.

辩论协议中，我们请可访问故事的批评模型在论证经提取与引用核验后生成 $N_{C}$ 条批评。偏好模型再按「对辩手改进论证最有帮助」评分；「有帮助」指情境相关且可直接用于强化所分配答案。批评偏好模型在提示末行问批评是否有帮助 Y/N，评分取「Y」对应对数概率；只需采样单 token。

In a follow-up message to their original response, the debater is provided with the critique with the highest log probability from the preference model and asked to refine their argument. See prompts in Appendix G.5. The critic and preference model prompts differ slightly between debate and consultancy, but the refinement prompt is identical.

在对原回应的后续消息中，辩手收到偏好模型对数概率最高的批评，并被要求 refining 论证。提示见附录 G.5。批评与偏好模型提示在辩论与咨询间略有不同，refinement 提示相同。

If best-of-$N$ sampling is also being used, then the $N_{A}$ refinements will be generated and chosen by the judge preference model. When $N_{C}=1$, this protocol is equivalent to one turn of self-refine. This is illustrated in Algorithm 1.

若同时用 best-of-$N$ 采样，则生成 $N_{A}$ 份 refinement 并由裁判偏好模型选择。当 $N_{C}=1$ 时，该协议等价于一轮 self-refine。见算法 1。

Input: Transcript $T$, Debater $\mathcal{M}_{\text{debater}}$, JudgePreferenceModel $\mathcal{P}_{\text{judge}}$, Critic $\mathcal{M}_{\text{critic}}$, CriticPreferenceModel $\mathcal{P}_{\text{critic}}$, Number of Arguments $N_{A}$, Number of Critiques $N_{C}$
Output: Final Argument $a^{*}$
$\mathcal{A}\leftarrow$ GenerateArguments($\mathcal{M}_{\text{debater}}$, $T$, $N_{A}$)
$a^{*}\leftarrow$ SelectMostPersuasiveArgument($\mathcal{A}$, $\mathcal{P}_{\text{judge}}$, $T$)
if $N_{C}>0$ then
$\mathcal{C}\leftarrow$ GenerateCritiques($a^{*}$, $\mathcal{M}_{\text{critic}}$, $T$, $N_{C}$)
$c^{*}\leftarrow$ SelectMostHelpfulCritique($\mathcal{C}$, $\mathcal{P}_{\text{critic}}$, $T$)
$R\leftarrow$ GenerateRefinements($a^{*}$, $c^{*}$, $\mathcal{M}_{\text{debater}}$, $T$, $N_{A}$)
$a^{*}\leftarrow$ SelectMostPersuasiveArgument($R$, $\mathcal{P}_{\text{judge}}$, $T$)
end if

Algorithm 1 Best-of-N Sampling and critique-and-refinement in Debate Protocol

输入： 记录 $T$，辩手 $\mathcal{M}_{\text{debater}}$，裁判偏好模型 $\mathcal{P}_{\text{judge}}$，批评 $\mathcal{M}_{\text{critic}}$，批评偏好模型 $\mathcal{P}_{\text{critic}}$，论证数 $N_{A}$，批评数 $N_{C}$
输出： 最终论证 $a^{*}$
$\mathcal{A}\leftarrow$ GenerateArguments($\mathcal{M}_{\text{debater}}$, $T$, $N_{A}$)
$a^{*}\leftarrow$ SelectMostPersuasiveArgument($\mathcal{A}$, $\mathcal{P}_{\text{judge}}$, $T$)
若 $N_{C}>0$ 则
$\mathcal{C}\leftarrow$ GenerateCritiques($a^{*}$, $\mathcal{M}_{\text{critic}}$, $T$, $N_{C}$)
$c^{*}\leftarrow$ SelectMostHelpfulCritique($\mathcal{C}$, $\mathcal{P}_{\text{critic}}$, $T$)
$R\leftarrow$ GenerateRefinements($a^{*}$, $c^{*}$, $\mathcal{M}_{\text{debater}}$, $T$, $N_{A}$)
$a^{*}\leftarrow$ SelectMostPersuasiveArgument($R$, $\mathcal{P}_{\text{judge}}$, $T$)
结束若

算法 1 辩论协议中的 Best-of-N 采样与批评–refinement

Common pitfalls in the arguments generated by the LLM debaters include the repetition of quotes, the use of unverified quotes, inadvertently supporting the opposition’s stance, and overlooking quotes that could bolster the argument. The critic addresses these issues by providing targeted, constructive feedback. This feedback is specifically designed to identify and rectify weaknesses in the arguments, such as pointing out where quotes may be repetitive or unverified or where they inadvertently lend credence to the opposing side. A crucial aspect of the critic’s role is ensuring that quotes are direct string matches and are properly encapsulated within tags. By highlighting these areas and suggesting more persuasive and accurately sourced quotes, the critic plays a pivotal role in refining the debater’s presentation. Ensuring it is manageable for the debater to implement the feedback within their assigned word limit, the critique length is limited to 150 words and two bullet points (and double that for consultancy). We found when testing with five bullet points that debaters would implement everything in their refinement and stop listening to the word limit provided.

LLM 辩手常见陷阱包括：重复引用、使用未核验引用、无意支持对方立场、忽略可强化论证的引用。批评通过有针对性的建设性反馈处理这些问题：指出重复或未核验引用、无意助对方等弱点。批评关键职责之一是确保引用为直接字符串匹配并正确包在 标签内。通过指出这些并建议更有说服力、来源准确的引用，批评在 refining 辩手呈现中起枢纽作用。为保证辩手能在词限内落实反馈，批评限 150 词与两条要点（咨询翻倍）。用五条要点测试时，辩手会在 refinement 中全部落实并停止遵守词限。

Concessions occur more often when using critique-and-refinement since the critic can provide tough or impossible feedback for the incorrect debater to implement. Lots of iteration was needed to reduce the frequency of concessions, especially in Consultancy, where minor edits sometimes lead to a 2-3% increase of the total questions being impacted. We found the following to help reduce concessions:

使用批评–refinement 时让步更常见，因批评可给错误辩手难以或无法落实的反馈。需大量迭代降低让步频率，尤其咨询中小改有时使受影响题增 2–3%。我们发现以下有助于减让步：

• • Write the answer they should be supporting again in the refinement prompt message.
• • Explicitly tell them not to concede
• • Tell the debater they don’t have to include all the items in the critique if doing so would compromise the argument.
• • Tell the critic they should provide constructive criticism to help the debater continue arguing for their assigned side. Without supporting quotes, the critic is told to help the debater construct a strategic argument without relying on quotes.

• 在 refinement 提示消息中再次写明应支持的答案。
• 明确告知不要让步。
• 告知辩手若落实全部批评项会损害论证，可不全部纳入。
• 告知批评应提供建设性批评，帮助辩手继续为所分配方辩护；无支持引用时，批评应帮助辩手构建不依赖引用的策略性论证。

Other issues included ensuring the debater did not mention that it had implemented feedback from a critique. We prompted the model not to include the phrase “The critique suggests”. Empirically, we noticed that the incorrect debater was more likely to mention the critique, so we decided to also use rejection sampling if ”critique” was mentioned in the response. Occasionally, Claude 2.1 refuses to refine the argument for ethical reasons. When this occurred, we used the original argument before refinement. An area for improvement is ensuring the critic does not misquote the story when suggesting quotes.

其它问题包括确保辩手不提及已落实批评反馈。我们提示模型不要写「批评建议」。经验上错误辩手更常提及批评，故若回应出现「critique」亦用拒绝采样。偶有 Claude 2.1 以伦理理由拒绝 refining；此时用 refining 前原论证。改进方向之一是确保批评建议引用时不误引故事。

D.4 Cross-Play Tournament

D.4 交叉对弈锦标赛

Tournament Details We use a Swiss-style tournament system for our cross-play tournament, allowing us to evaluate a large number of players ($N$) efficiently. Running a round-robin tournament where each player plays everyone would require $O(N^{2})$ matches. The Swiss tournament has $O(n\log n)$ complexity, allowing each player ample opportunities to compete against similarly ranked players, leading to accurate final rankings.

锦标赛细节 交叉对弈用瑞士制，以便高效评估大量选手（$N$）。循环赛每人互打需 $O(N^{2})$ 场；瑞士制 $O(n\log n)$，使每人对实力相近对手有充分机会，最终排名更准。

Players should be seeded initially so the best pairings happen between similarly matched players in early rounds. The number of rounds is determined by the ceiling of the $\log 2$ of the total number of players, ensuring a balanced and manageable competition structure.

选手应先种子排序，使早期轮次实力相近者对阵。轮次数取选手总数 $\log 2$ 上取整，以保证结构均衡可管理。

Pairings for each round are generated based on the closest neighbour that has not played yet. We avoid repeat matchups. In cases with an odd number of participants, a player who does not find a match is granted a bye. This process ensures equitable distribution of opponents that are closest in strength.

每轮按尚未交手的最近邻配对，避免重复对阵。奇数人时未配对者轮空。此过程保证对手实力分布尽可能接近。

Each match is conducted, and the winner is determined based on the player’s number of wins in both correct and incorrect debater permutations. Following each match, scores are updated: winners receive a point, while losers do not. This scoring system allows us to dynamically re-ranking players after each tournament round, reflecting their performance in the tournament.

每场比赛后，按选手在正确与错误辩手排列下的胜场数定胜负。随后更新积分：胜者加分、负者不加。该计分使我们能在每轮后动态重排，反映锦标赛表现。

After the tournament, participants are ranked based on their total score. We then use game history to calculate aggregate Elos based on the win rates in each match. The pseudo-code for the Swiss tournament is given in Algorithm 2.

赛后按总分排名。再用对局历史与各场胜率计算总体 Elo。瑞士制伪代码见算法 2。

We use 291 questions from the quality development set (denoted $D_{L}$) for all cross-play matches and use 400 questions from the train set (denoted $T_{L}$ when running self-play matches to get judge accuracy. The filtering of these sets is equivalent to the method described in Appendix D.1, and we allow a maximum of 5 questions per story.

全部交叉对弈用 QuALITY 开发集 291 题（记 $D_{L}$）；自我对弈求裁判准确率用训练集 400 题（记 $T_{L}$）。过滤等价于附录 D.1，每故事最多 5 题。

Input: List of players $P$ sorted by seed
Output: Final ranking, scores
Initialise $scores$ to 0 for all players in $P$
Determine number of rounds $R=Ceil(\log_{2}(len(P)))$
Initialise $previous\_matchups$ as empty for each player
for $i=1$ to $R$ do
Sort $P$ based on scores
$pairings\leftarrow$ pair_players_without_rematch($P$, $previous\_matchups$)
for $p$ in $pairings$ do
$results\leftarrow$ play_round($p[0]$, $p[1]$)
$results\leftarrow$ play_round($p[1]$, $p[0]$)
update_scores($scores$, $results$)
end for
end for
$final\_ranking\leftarrow$ sort $P$ based on final scores

Algorithm 2 Swiss Tournament Logic

输入： 按种子排序的选手列表 $P$
输出： 最终排名、积分
对 $P$ 中全部选手初始化 $scores$ 为 0
轮次数 $R=Ceil(\log_{2}(len(P)))$
为每选手初始化空的 $previous\_matchups$
for $i=1$ to $R$ do
按积分排序 $P$
$pairings\leftarrow$ pair_players_without_rematch($P$, $previous\_matchups$)
for $p$ in $pairings$ do
$results\leftarrow$ play_round($p[0]$, $p[1]$)
$results\leftarrow$ play_round($p[1]$, $p[0]$)
update_scores($scores$, $results$)
end for
end for
$final\_ranking\leftarrow$ 按最终积分排序 $P$

算法 2 瑞士制锦标赛逻辑

Models and Hyper-Parameters We use 20 players in our Swiss tournament and choose various best-of-$N$ and critique variations across four base models to get an extensive range of skill levels. The initial ranking for each player is given in Table 4 and corresponding hyperparameters in Table 5 and Table 6. The initial ranking was determined by preliminary experiments where we ran smaller round-robin tournaments between Claude and GPT models. Combining best-of-$N$ with critique-and-refinement in smaller round-robin tournaments boosted aggregate Elo more for Claude than GPT-4-Turbo models. It was assumed that all GPT-4-Turbo models would beat Claude variations, even though that was not true after running the tournament. For best-of-$N$, we did temperature tuning using smaller tournaments of 4 players, finding 0.8 led to robust improvements across all $N$.

模型与超参 瑞士制用 20 名选手，在四个基座上选多种 best-of-$N$ 与批评变体以覆盖广技能范围。初始排名见表 4，对应超参见表 5、6。初始排名由 Claude 与 GPT 间较小循环赛初步实验决定。小循环赛中 best-of-$N$ 与批评–refinement 结合对 Claude 总体 Elo 提升多于 GPT-4-Turbo。曾假设全部 GPT-4-Turbo 会击败 Claude 变体，赛后并非如此。best-of-$N$ 用 4 人小赛调温度，发现 0.8 在各 $N$ 上稳健改进。

We finished the tournament after 4 rounds rather than 5 because we found the resulting Elos were representative, and running a 5th round would not be a good use of API credits.

我们在 4 轮后结束而非 5 轮，因所得 Elo 已有代表性，第 5 轮不划算消耗 API 额度。

Comprehensive results for debater win rates across all 40 matches and with 3 judges are provided in Appendix F. The final ranking order for each judge is also provided.

全部 40 场、3 名裁判下的辩手胜率综合结果见附录 F，亦给出各裁判最终排名。

Initial Seed	Player Name
1	GPT-4-Turbo bo32
2	GPT-4-Turbo bo16
3	GPT-4-Turbo bo8
4	GPT-4-Turbo bo4 c8
5	GPT-4-Turbo c16
6	GPT-4-Turbo bo4
7	GPT-4-Turbo bo1
8	claude-2.1 bo4 c8
9	claude-2.1 bo16
10	claude-2.1 bo8
11	claude-2.1 bo4
12	claude-2.1 c16
13	claude-2.1 c2
14	claude-2.1 bo1
15	claude-v1.3 bo1
16	gpt-3.5-turbo-16k bo16
17	gpt-3.5-turbo-16k bo8
18	gpt-3.5-turbo-16k c2
19	gpt-3.5-turbo-16k bo4
20	gpt-3.5-turbo-16k bo1

Table 4: Debater models with best-of-$N$ (bo$N$) and critique-and-refinement (c$N$) variations along with initial seeds for each player in the cross-play tournament

初始种子	选手名
1	GPT-4-Turbo bo32
2	GPT-4-Turbo bo16
3	GPT-4-Turbo bo8
4	GPT-4-Turbo bo4 c8
5	GPT-4-Turbo c16
6	GPT-4-Turbo bo4
7	GPT-4-Turbo bo1
8	claude-2.1 bo4 c8
9	claude-2.1 bo16
10	claude-2.1 bo8
11	claude-2.1 bo4
12	claude-2.1 c16
13	claude-2.1 c2
14	claude-2.1 bo1
15	claude-v1.3 bo1
16	gpt-3.5-turbo-16k bo16
17	gpt-3.5-turbo-16k bo8
18	gpt-3.5-turbo-16k c2
19	gpt-3.5-turbo-16k bo4
20	gpt-3.5-turbo-16k bo1

表 4：交叉对弈锦标赛辩手模型及其 best-of-$N$（bo$N$）、批评–refinement（c$N$）变体与初始种子

Model	Temperature
Debater	\begin{aligned} f(\text{temperature})=\begin{cases}0.4&\text{if }\text{boN}=1,\\ 0.8&\text{if }1<\text{boN}\leq 16,\\ 1.0&\text{if }\text{boN}>16.\end{cases}\end{aligned}
Consultant	\begin{aligned} f(\text{temperature})=\begin{cases}0.4&\text{if }\text{boN}=1,\\ 0.8&\text{if }\text{boN}>1.\end{cases}\end{aligned}
Interactive Judge	0.4
Bo$N$ Preference Model	0
Judge	0
Critic	$\begin{aligned} f(\text{temperature})=\begin{cases}0.6&\text{if }1<\text{cN}\leq 16,\\ 0.8&\text{if }\text{cN}\geq 16.\end{cases}\end{aligned}$
Critique Preference model	0

Table 5: The temperature used for each model as a function of best-of-$N$ (bo$N$) or critique-of-N (c$N$).

模型	温度
辩手	见原文分段函数：bo$N$=1 时 0.4；1<bo$N$≤16 时 0.8；bo$N$>16 时 1.0
顾问	bo$N$=1 时 0.4；bo$N$>1 时 0.8
交互裁判	0.4
Bo$N$ 偏好模型	0
裁判	0
批评	1<cN≤16 时 0.6；cN≥16 时 0.8
批评偏好模型	0

表 5：各模型温度随 best-of-$N$（bo$N$）或 critique-of-N（c$N$）的函数

Model	Word Target	Min words	Max words
Debater	100	70	150
Consultant	200	140	300
Critic	150	-	-

Table 6: The word target is included in the prompts for each model. The min and max words are used to reject any arguments which lie outside this range.

模型	词数目标	最少词	最多词
辩手	100	70	150
顾问	200	140	300
批评	150	-	-

表 6：各模型提示中的词数目标；最少/最多词用于拒绝范围外论证

D.5 Calculating Elo Ranking

D.5 计算 Elo 排名

Elo ratings, initially developed for chess, offer a simple yet robust method for estimating the relative skill levels of players in competitor-versus-competitor games. The algorithm we employ for Elo rating calculation is based on the assumption that the performance of each player is a normally distributed random variable. The expected score of a player is modelled as a logistic function of the difference in ratings between the player and their opponent.

Elo 最初为国际象棋设计，是估计对局游戏相对技能的简单稳健方法。我们采用的算法假设每名选手表现服从正态分布；期望得分建模为选手与对手评分差的 logistic 函数。

Expected Win Rate: The expected win rate for a player $P_{1}$ against another player $P_{2}$ with Elo ratings $E_{1}$ and $E_{2}$, respectively, is given by the formula:

期望胜率： 选手 $P_{1}$ 对 $P_{2}$（Elo 分别为 $E_{1}$、$E_{2}$）的期望胜率为：

$$\hat{\omega}_{P_{1}}=\frac{1}{1+10^{((E_{2}-E_{1})/400)}}$$

This function yields a value between 0 and 1, representing the expected probability of $P_{1}$ winning against $P_{2}$.

该函数取值 0 到 1，表示 $P_{1}$ 战胜 $P_{2}$ 的期望概率。

Cost Function for Elo Rating: The cost function used in Elo rating optimisation is the squared error between the predicted win rate, based on Elo ratings, and the actual win rate observed in historical data. Given a set of player pairs with observed win rates and match counts, the cost function is defined as:

Elo 评分代价函数： 优化用代价函数为基于 Elo 的预测胜率与历史观测胜率的平方误差。给定带观测胜率与对局数的选手对集合，代价定义为：

$$\text{Cost}=\sum_{(P_{i},P_{j})}\left(\hat{\omega}_{P_{i}}-\omega_{P_{i},P_{j}}\right)^{2}$$

where $\omega_{P_{i},P_{j}}$ is the actual win rate of player $P_{i}$ against player $P_{j}$ in the dataset.

其中 $\omega_{P_{i},P_{j}}$ 为数据中 $P_{i}$ 对 $P_{j}$ 的实际胜率。

Optimisation: To compute the optimal Elo ratings for a given dataset, we estimate the first-order gradients and apply a simple optimisation technique. The player ratings are first initialised at the zero references of a specific policy (in most cases, we choose the Claude 2.1 model). We estimated ratings using Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm. To estimate confidence intervals we use statistical bootstrapping over 1000 seeds.

优化： 为计算给定数据集最优 Elo，估计一阶梯度并施简单优化。选手评分先初始化为某策略零参考（多数情况选 Claude 2.1）。用 BFGS 算法估计评分；置信区间用 1000 个种子统计自助法。

Judge	PearsonR	P-value
Claude-2.1	0.997	1.5e-21
GPT-3.5-turbo	0.989	1.6e-16
GPT-4-turbo	0.998	6.7e-24
all	0.991	3.8e-52

裁判	PearsonR	P 值
Claude-2.1	0.997	1.5e-21
GPT-3.5-turbo	0.989	1.6e-16
GPT-4-turbo	0.998	6.7e-24
all	0.991	3.8e-52

Comparison to using Trueskill: We also calculated persuasiveness using Trueskill, a bayesian message passing based system (Herbrich et al. 2006). We found high agreement with Elo ratings, and found both metrics to be suitable for parameterising persuasiveness. In Table D.5 we show the agreement between persuasiveness scores calculated from both Elo and Trueskill for the cross-play tournament. Under either method - rankings did not change. Whilst Trueskill explicitly provides uncertainty, we report Elo as it’s more prevalent in the LLM community at the time of writing.

与 Trueskill 比较： 我们亦用基于贝叶斯消息传递的 Trueskill（Herbrich et al. 2006）计算说服力。与 Elo 高度一致，两种指标均适合参数化说服力。表 D.5 展示交叉对弈锦标赛 Elo 与 Trueskill 说服力分数一致性；两种方法排名不变。Trueskill 显式给不确定性，但报告 Elo 因其在撰写时 LLM 社区更常见。

D.6 Mitigating LLM bias

D.6 缓解 LLM 偏见

Strict word limit — We use a strict word limit with rejection sampling to mitigate the verbosity bias of LLM judges (see the impact in Appendix A.7. Our methodology is provided in Algorithm 3, which illustrates generating a pool of candidate responses from the LLM and subsequently filtering them based on criteria in an is_valid. We specify the LLM response should contain an tag, and the argument within those tags must be within the word limit boundaries in Table 6. We generate 3 candidates per completion required for bo$N$, which, empirically, we find maximises the chance of obtaining a sufficient number of valid responses.

严格词限 — 用严格词限与拒绝采样缓解 LLM 裁判冗长偏见（影响见附录 A.7）。方法见算法 3：从 LLM 生成候选池，再按 is_valid 准则过滤。规定回应须含 标签，且标签内论证落在表 6 词限。每 bo$N$ 所需补全生成 3 候选，经验上最大化获得足够有效回应的机会。

When we don’t have enough responses after rejection sampling, we pad the valid responses with invalid responses. This strategy guarantees that the total number of responses aligns with the requirement of best-of-$N$. On the occasions where pad the responses, we truncate the argument if they were over the specific word limit. Our truncation function carefully adjusts responses, particularly paying attention to not disrupt structural elements like quotes. If truncation slices in the middle of a quote, we add a closing tag at the end so a partial quote can still be sent for verification. If we truncate, we append ... to the end of the argument.

拒绝采样后若回应不够，用无效回应填充有效列表，以保证总数对齐 best-of-$N$。填充时对超词限论证截断；截断函数注意不破坏引用等结构。若截断切在引用中间，末尾加 以便部分引用仍可核验。截断后在论证末追加 ...。

Input: Model $M$, Prompt $prompt$, Number of completions $n$,
Candidates per completion $num\_candidates\_per\_completion$,
Validation function $is\_valid$
Output: List of valid LLM responses
$num\_candidates\leftarrow n\times num\_candidates\_per\_completion$
$candidate\_responses\leftarrow$ get_llm_response($M$, $prompt$, $num\_candidates$)
$valid\_responses$, $invalid\_responses\leftarrow$ filter $candidate\_responses$ using $is\_valid$
$num\_valid\leftarrow$ length of $valid\_responses$
if $num\_valid<n$ then
$invalids\_needed\leftarrow n-num\_valid$
$truncated\_responses\leftarrow$ apply $truncate$ to each element of $invalid\_responses$
Append first $invalids\_needed$ elements of $invalid\_responses$ to $valid\_responses$
end if

Algorithm 3 Rejection Sampling on LLM Outputs

输入： 模型 $M$，提示 $prompt$，补全数 $n$，每补全候选数 $num\_candidates\_per\_completion$，校验函数 $is\_valid$
输出： 有效 LLM 回应列表
$num\_candidates\leftarrow n\times num\_candidates\_per\_completion$
$candidate\_responses\leftarrow$ get_llm_response($M$, $prompt$, $num\_candidates$)
用 $is\_valid$ 过滤得 $valid\_responses$、$invalid\_responses$
$num\_valid\leftarrow$ length($valid\_responses$)
若 $num\_valid<n$ 则
$invalids\_needed\leftarrow n-num\_valid$
对 $invalid\_responses$ 各元素应用 $truncate$
将前 $invalids\_needed$ 个无效回应追加到 $valid\_responses$
结束若

算法 3 对 LLM 输出的拒绝采样