JEV 复测七项政治学任务：更快，批处理并不更便宜

JEV versus LLMs: Accuracy, Cost and Calibration on Seven Political Science Replications

JEV 对比大语言模型：七项政治学复现中的准确率、成本与校准

Steven Denney † † thanks: Leiden Institute for Area Studies, Leiden University. s.c.denney@hum.leidenuniv.nl Matthew DiGiuseppe † † thanks: Institute of Political Science, Leiden University. m.r.di.giuseppe@fsw.leidenuniv.nl

Abstract

摘要

Large language models (LLMs) annotate and scale political text or constructs by generating text tokens. A new class of models, which TypeSafe markets as “System One” models, instead returns decisions and probability distributions across a user-supplied fixed answer set. A commercial model, JEV, is advertised as having a dramatic cost and speed advantage over traditional LLMs along with better calibrated decisions. As such, it might be useful for social scientists looking to quickly and cost-effectively annotate or scale large corpora of text and have a reliable indicator of a classifier’s uncertainty. Yet, the accuracy of these claims and the broader model accuracy in social science text-based tasks are not yet established. In this paper, we do just that and hope to establish the suitability of JEV for social science tasks. We compare JEV with LLMs and human coders from published research, and with a current mid-tier commercial LLM (GPT-6 Luna) and an open-weight alternative (Qwen3.8-27B). We find that JEV matches, or comes close to, the capabilities of both LLMs in a variety of tasks. However, we find no cost advantage over GPT-6 Luna at OpenAI’s batch prices. Further, we find that, when each question is asked once, JEV’s probabilities are better calibrated than GPT-6 Luna’s token probabilities, but not consistently better than Qwen3.8-27B’s. We conclude that unless researchers have a need for speed, JEV’s only obvious advantage is ease of parsing the underlying choice probabilities.

大语言模型（LLM）通过生成文本 token 来标注与缩放政治文本或构念。一类新型模型被 TypeSafe 以「System One」之名推向市场：它不生成文本，而是在用户给定的固定答案集上返回决策与概率分布。商业模型 JEV 被宣传为相对传统 LLM 有巨大的成本与速度优势，且决策校准更好。因此，它可能适合希望快速、低成本地标注或缩放大规模文本语料、并可靠获知分类器不确定性的社会科学研究者。然而，这些主张是否成立，以及该模型在社会科学文本任务上的更广准确率，尚未得到检验。本文正做此事，以期确立 JEV 对社会科学任务的适用性。我们将 JEV 与已发表研究中的 LLM 与人工编码者对照，并与当前中档商业 LLM（GPT-6 Luna）以及开源权重模型（Qwen3.8-27B）比较。结果表明，JEV 在多种任务上与这两种 LLM 能力相当或接近。但在 OpenAI 的批处理价格下，它相对 GPT-6 Luna 并无成本优势。进一步看，当每个问题只问一次时，JEV 的概率比 GPT-6 Luna 的 token 概率校准更好，但并不一贯优于 Qwen3.8-27B。我们的结论是：除非研究者需要速度，JEV 唯一明显的优势在于底层选择概率易于解析。

Keywords: large language models, text as data, measurement, annotation, scaling, pairwise comparison, Bradley-Terry

关键词 ： 大语言模型，文本即数据，测量，标注，缩放，成对比较，Bradley-Terry

LLMs have democratised text-as-data tools in the social sciences. Prior to LLMs, most scholars had to invest substantial time in learning text-as-data techniques or have the resources to hire crowd-sourced workers (Benoit et al., 2016) or research assistants to code a training dataset. Now scholars can use R packages (Maerz and Benoit, 2026) to process text data with common statistical software and a shallow learning curve. Still, LLMs are not costless. For large datasets and for frontier capabilities, LLMs do carry a substantial (but declining) token cost. Further, scaling tasks that rely on pairwise comparisons across a corpus require about 20 times as many calls as rating each text once and can quickly get expensive (DiGiuseppe and Flynn, 2026; Licht et al., 2025).

LLM 降低了社会科学中「文本即数据」工具的使用门槛。在 LLM 出现之前，多数学者要么投入大量时间学习文本分析方法，要么有资源雇佣众包工人（Benoit 等，2016）或研究助理来编码训练集。现在学者可以用 R 包（Maerz 与 Benoit，2026）在常用统计软件中处理文本，学习曲线平缓。但 LLM 并非零成本。面对大数据集与前沿能力，LLM 仍带来可观（虽在下降）的 token 费用。此外，依赖语料上成对比较的缩放任务大约需要单次评分的 20 倍调用，费用很快上去（DiGiuseppe 与 Flynn，2026；Licht 等，2025）。

A new class of models, which TypeSafe markets as “System One” models, is marketed as a cheaper and quicker alternative that could further democratise text analysis in the social sciences. The name borrows from the distinction between fast, intuitive “System 1” and slow, deliberate “System 2” thinking popularised by Kahneman (2011). The analogy implies that the model makes fast, intuitive decisions rather than slow, well-reasoned ones. TypeSafe presents its JEV (1.13), released in September 2026, as its first public model in this class, and says it works differently from a generative LLM. A generative model answers by writing text one token at a time, and each token needs another pass through the model. The researcher then reads a label or a number out of what the model wrote. JEV returns no text. The user sends a piece of text, which TypeSafe calls “the state”, and one or more questions with a fixed set of answers. According to TypeSafe, the model reads the state once and judges every question against it in parallel, returning all of its answers in one response (TypeSafe, 2026). TypeSafe has not published the architecture, so we cannot check this account (Appendix E). JEV offers three output types. They are a choice from a user-provided list of categories (Choice), an item on a scale with worded levels (Score), and the probability of a yes answer to a yes/no question (Noul). For all output types, the model reports the probability it attaches to each potential option. Some LLM APIs also report token probabilities, the probability the model gave to each candidate token at each position of its reply, which can serve as a measure of uncertainty. Most report only the 5 to 20 most probable candidates at each position, and some report none. Further, it is not clear that these probabilities are informative and well calibrated, since post-training with human feedback tends to make them overconfident (OpenAI, 2023; Tian et al., 2023).

一类新型模型被 TypeSafe 以「System One」之名营销为更便宜、更快的替代方案，有望进一步降低社会科学文本分析的门槛。名称借用 Kahneman（2011）普及的快直觉「系统 1」与慢深思「系统 2」之分。这一类比意味着模型做的是快速直觉式决策，而非缓慢、充分推理的决策。TypeSafe 将其于 2026 年 9 月发布的 JEV（1.13）呈现为该类中的首个公开模型，并称其工作方式不同于生成式 LLM。生成模型逐 token 写文本作答，每个 token 都要再过一遍模型；研究者再从模型写出的内容里读出标签或数字。JEV 不返回文本。用户发送一段 TypeSafe 称为「状态」（state）的文本，以及一个或多个带固定答案集的问题。按 TypeSafe 的说法，模型读一次状态，再并行对照每个问题作判断，并在一次响应中返回全部答案（TypeSafe，2026）。TypeSafe 未公布架构，我们无法核验这一说法（附录 E）。JEV 提供三类输出：从用户提供的类别列表中选择（Choice）、在带文字等级的量表上给出一项（Score），以及对是否题给出「是」的概率（Noul）。对所有输出类型，模型都会报告其赋给每个潜在选项的概率。部分 LLM API 也会报告 token 概率，即模型在回复每一位置赋给各候选 token 的概率，可用作不确定性度量。多数只报告每个位置概率最高的 5 到 20 个候选，有的则完全不报告。此外，这些概率是否有信息量且校准良好并不清楚，因为经人类反馈的后训练往往使它们过度自信（OpenAI，2023；Tian 等，2023）。

TypeSafe charges only for input tokens and output is free. Output tokens cost more per token than input tokens, and reasoning models can write hundreds or thousands of them before they answer, so TypeSafe advertises large cost and speed advantages (TypeSafe, 2026). For classification with short labels, however, most of the bill is input. In our tasks GPT-6 Luna wrote about four tokens per answer, while JEV counts the question and every answer option as input and billed 1.2 to 3.3 times as many input tokens per decision as Luna. At the time of writing, the details of JEV’s architecture are not public. As such, researchers cannot verify why the model is cheap and fast or what that implies for its output.

TypeSafe 只对输入 token 计费，输出免费。输出 token 单价高于输入，推理模型在作答前还可能写出成百上千个输出 token，因此 TypeSafe 宣传巨大的成本与速度优势（TypeSafe，2026）。但对短标签分类而言，账单主要来自输入。在我们的任务里，GPT-6 Luna 每次回答大约写四个 token，而 JEV 把问题与全部答案选项计入输入，每个决策计费的输入 token 是 Luna 的 1.2 到 3.3 倍。截至写作时，JEV 架构细节未公开。因此研究者无法核验模型为何便宜、为何快，以及这对输出意味着什么。

Despite this opacity, the model’s efficiency potentially holds many benefits for researchers analysing text, or wishing to exploit the knowledge embedded in such models. Yet, we have little existing evidence that JEV or other “System One” models are suitable for social science applications. In this paper, we ask whether the cost and speed advantages of JEV (1.13) over LLMs require a sacrifice in performance. Further, we examine TypeSafe’s claim of improved calibration, which it attributes to its training method, reinforcement learning for calibrated decisions (RLCD). We replicate seven published studies that evaluated LLMs for annotation, scaling or data generation. We score JEV against each study’s human benchmark and compare it with the published models, with GPT-6 Luna (a current low-cost commercial LLM) and with Qwen3.8-27B (an open-weight alternative).

尽管不透明，该模型的效率对分析文本、或希望利用此类模型所嵌入知识的研究者仍可能带来诸多益处。但我们几乎没有现成证据表明 JEV 或其他「System One」模型适合社会科学应用。本文追问：JEV（1.13）相对 LLM 的成本与速度优势，是否要以牺牲性能为代价。我们还检验 TypeSafe 关于校准改进的主张——该公司将其归因于训练方法「校准决策强化学习」（RLCD）。我们复现七项已发表、用 LLM 做标注、缩放或数据生成评估的研究。以各研究的人工基准为标尺给 JEV 打分，并与已发表模型、GPT-6 Luna（当前低成本商业 LLM）以及 Qwen3.8-27B（开源权重替代）比较。

Our analysis finds that JEV performs on par with its contemporary commercial and open-weight alternatives when using human coding as a benchmark. Further, we find little cost advantage compared to (a) GPT-6 Luna called with batch processing or (b) open-weight models run locally on commercial hardware. JEV does, however, have a clear speed advantage. We also find that JEV’s probabilities are better calibrated than the token probabilities GPT-6 Luna returns when each question is asked once, on all eight tasks where we compare them. Post-training for chat tends to make LLMs’ answer probabilities overconfident (OpenAI, 2023; Tian et al., 2023), and reasoning models often state high confidence in answers that are wrong (Mei et al., 2025). GPT-6 Luna fits this pattern: 74% of its probabilities on the party comparisons are below 0.01 or above 0.99, and OpenAI returns none when it reasons.[^1] Qwen3.8-27B does not (16%), and its calibration is similar to JEV’s when human coding is the benchmark.

分析表明，以人工编码为基准时，JEV 与同期商业与开源权重替代表现相当。相对（a）以批处理调用的 GPT-6 Luna，或（b）在商用硬件本地运行的开源权重模型，我们也几乎看不到成本优势。不过 JEV 确有清晰的速度优势。我们还发现：当每个问题只问一次时，在全部八项可比任务上，JEV 的概率都比 GPT-6 Luna 返回的 token 概率校准更好。面向聊天的后训练往往使 LLM 的答案概率过度自信（OpenAI，2023；Tian 等，2023），推理模型也常对错误答案给出高置信度（Mei 等，2025）。GPT-6 Luna 符合这一模式：在政党比较上，其 74% 的概率低于 0.01 或高于 0.99；而在推理时 OpenAI 不返回任何概率。[^1] Qwen3.8-27B 则不然（16%），以人工编码为基准时其校准与 JEV 相近。

In all, we find that in the analysis of social science text, JEV has only minor advantages over the existing options we tested. It is clearly faster and it has bounded choice probabilities that are easy to parse.

总的来说，在社会科学文本分析中，相对我们测试过的现有选项，JEV 只有次要优势。它明显更快，并且有易于解析的有界选择概率。

What JEV returns

JEV 返回什么

A JEV request contains a state and typed questions. The “state” is the text to be judged, while each question specifies what the model should decide and supplies the answer options against which it should evaluate that text (TypeSafe, 2026). Users can pick one of three output types: Choice, Score or Noul. “Choice” returns a named option and probabilities over user-supplied options. “Score” accepts two to ten worded levels and returns their probability-weighted mean, with the probability assigned to each level. “Noul” returns the probability of a positive answer to a binary question. Researchers therefore receive an answer distribution directly, without parsing a generated explanation or extracting a category from free text. Beyond these probabilities, a Choice answer also carries a “confidence” value. In our outputs it is almost exactly a rescaling of the probability of the chosen option (correlation 0.9997), so we treat it as adding nothing beyond the probabilities and do not analyse it separately.

一次 JEV 请求包含一个状态与若干类型化问题。「状态」是待判断的文本；每个问题规定模型应决定什么，并给出评判该文本所对照的答案选项（TypeSafe，2026）。用户可在三类输出中选一：Choice、Score 或 Noul。「Choice」返回一个具名选项以及用户所供选项上的概率。「Score」接受二到十个带文字的等级，返回其概率加权均值，并给出各等级概率。「Noul」返回对二元问题给出肯定回答的概率。研究者因此直接得到答案分布，无需解析生成式解释，也无需从自由文本中抽取类别。除这些概率外，Choice 答案还带一个「confidence」值。在我们的输出里，它几乎就是所选选项概率的重缩放（相关 0.9997），因此我们视其未超出概率本身提供新信息，不单独分析。

These probabilities resemble the token probabilities (log probabilities) that many LLM APIs report for their output. In this case, however, the probabilities are defined over the answer options rather than over possible tokens. A log probability describes the next token, so the researcher must map tokens to answer labels, discard mass on tokens that are not labels, and renormalise. Closed APIs also truncate what they report, or report no probabilities at all. For GPT-6 Luna, OpenAI returns at most the five most probable tokens at each position, so on a nine-point scale some levels receive no probability at all, and it returns none when the model reasons before answering. Anthropic does not report token probabilities at all. Even open-weight models accessed through an inference service will often limit the available probabilities. JEV returns a probability for every option the researcher supplies, rounded to two decimal places, regardless of the number of options.

这些概率类似许多 LLM API 为其输出报告的 token 概率（对数概率）。但此处概率定义在答案选项上，而非可能的 token 上。对数概率描述的是下一个 token，研究者必须把 token 映射到答案标签、丢掉非标签 token 上的质量并重新归一化。封闭 API 还会截断报告，或完全不报告概率。对 GPT-6 Luna，OpenAI 每个位置最多返回五个最可能 token，因此在九点量表上有些等级完全得不到概率；模型在作答前推理时则不返回任何概率。Anthropic 完全不报告 token 概率。即便通过推理服务访问的开源权重模型，也常限制可用概率。JEV 对研究者提供的每个选项都返回概率，四舍五入到小数点后两位，与选项数量无关。

TypeSafe calls its post-training method reinforcement learning for calibrated decisions (RLCD), which it says optimizes for “epistemically honest probabilities”, so that higher confidence means higher accuracy (TypeSafe, 2026). A calibrated model’s answers given probability 0.8 should be correct about 80% of the time across many answers. TypeSafe does not publish the training data, the reward or the architecture, so we cannot check the method directly. We can, however, check its implication against “gold standard” human benchmarks. Whatever calibration RLCD achieves, it achieves on TypeSafe’s training data. Calibration depends on the data as well as the model, so probabilities calibrated on one set of texts need not be calibrated on another. As such, it is no obvious that the RLCD would translate to honest calibration across all uses cases.

TypeSafe 称其后训练方法为「校准决策强化学习」（RLCD），并说其优化目标是「认知上诚实的概率」，从而使更高置信度对应更高准确率（TypeSafe，2026）。校准良好的模型在给出概率 0.8 时，跨大量答案应约有 80% 正确。TypeSafe 不公布训练数据、奖励或架构，我们无法直接核验该方法。但我们可以对照「金标准」人工基准检验其含义。无论 RLCD 达到何种校准，都是在 TypeSafe 的训练数据上达到的。校准既依赖模型也依赖数据，因此在一类文本上校准过的概率，未必在另一类上仍校准。因此，并不能显然认为 RLCD 会在所有用例上转化为诚实的校准。

Still, the underlying choice probabilities produced by JEV are potentially its most important contribution. As we mentioned, LLM token probabilities are limited and not always available. Research suggests that the token probabilities of chat-tuned LLMs are often a poor guide to how often an answer is “right”. The largest pretrained models are reasonably well calibrated on multiple-choice questions whose options are shown to them (Kadavath et al., 2022). However, post-training (reinforcement learning) for chat undoes much of this, although a simple temperature adjustment largely restores it (Kadavath et al., 2022). On a subset of the MMLU benchmark, GPT-4’s expected calibration error on its answer probabilities rose from 0.007 before post-training to 0.074 after post-training (OpenAI, 2023). For GPT-3.5-turbo, GPT-4 and Claude 1 and 2, answer probabilities estimated from repeated samples are poorly calibrated, and the confidence a model states when asked is often better calibrated, in some cases halving the calibration error (Tian et al., 2023). One proposed contributor is that the reward models used in this training favor responses that state high confidence regardless of their quality (Leng et al., 2025). Reasoning models, whose token probabilities commercial APIs usually do not return, often state confidence above 85% even on answers they get wrong (Mei et al., 2025). Below, we compare JEV with GPT-6 Luna and Qwen3.8-27B to see whether its probabilities track correctness better than theirs do.

即便如此，JEV 产生的底层选择概率仍可能是其最重要的贡献。如前所述，LLM 的 token 概率有限且并非总是可得。研究表明，经聊天调优的 LLM 的 token 概率，往往并不能很好反映答案「正确」的频率。最大的预训练模型在选项被展示的选择题上校准相当不错（Kadavath 等，2022）。但面向聊天的后训练（强化学习）会大幅破坏这一点，尽管简单的温度调节大体可以恢复（Kadavath 等，2022）。在 MMLU 基准的一个子集上，GPT-4 答案概率的期望校准误差在后训练前为 0.007，后训练后升至 0.074（OpenAI，2023）。对 GPT-3.5-turbo、GPT-4 以及 Claude 1 与 2，由重复采样估计的答案概率校准较差；而直接询问模型给出的置信度往往校准更好，有时可使校准误差减半（Tian 等，2023）。一个被提出的原因是：该训练所用的奖励模型偏好陈述高置信度的回复，而不论其质量（Leng 等，2025）。推理模型——商业 API 通常不返回其 token 概率——即便答错也常声称置信度高于 85%（Mei 等，2025）。下文我们将 JEV 与 GPT-6 Luna、Qwen3.8-27B 比较，看其概率是否比后两者更紧跟正确性。

JEV’s exact architecture also remains unknown. TypeSafe states that JEV reads the state once and evaluates the questions in parallel, producing its outputs in one query, but provides no technical account from which we can verify how those evaluations are performed (TypeSafe, 2026).

JEV 的精确架构同样未知。TypeSafe 称 JEV 读一次状态并并行评估问题、在一次查询中产生输出，但未提供可核验这些评估如何执行的技术说明（TypeSafe，2026）。

How does JEV compare to LLMs and humans?

JEV 与 LLM、人类相比如何？

There have been a variety of papers published in political science and the social sciences more broadly to assess the ability of LLMs to complete tasks assigned to human coders and encoder-only models like BERT. Here we replicate several of these analyses, using the validation benchmarks of these existing studies, to see how JEV compares with these benchmarks, previous-generation LLMs, and the two contemporary models we mentioned above. Table 1 identifies the papers and tasks within those papers that we replicate here with JEV called via OpenRouter (for V-Dem, TypeSafe’s own endpoint), GPT-6 Luna called directly from OpenAI servers, and Qwen3.8-27B called via a third-party inference provider (Together.ai).

政治学及更广的社会科学已有多篇论文评估 LLM 完成原属人工编码者与 BERT 一类仅编码器模型的任务的能力。这里我们复现其中若干分析，借助这些既有研究的验证基准，看 JEV 相对这些基准、上一代 LLM，以及上述两个当代模型表现如何。表 1 列出我们复现的论文与任务：JEV 经 OpenRouter 调用（V-Dem 任务则用 TypeSafe 自有端点），GPT-6 Luna 直接从 OpenAI 服务器调用，Qwen3.8-27B 经第三方推理提供商（Together.ai）调用。

Our replications include five reading (scaling and annotation) applications and two tasks which rely on the models’ embedded knowledge to generate new data. Regarding the latter, the first attempts to create an ideological scaling of European political parties (Di Leo et al., 2025) and the second reproduces V-Dem’s expert codes for 53 indicators in 2023 (Weidmann et al., 2026). We retain each published task and prompt wording where the interface permits, removing reply-format instructions that JEV cannot follow. Appendix A records the prompts, and Appendices C, F and G document changes to answer options and rating scales.

我们的复现包括五项阅读（缩放与标注）应用，以及两项依赖模型嵌入知识来生成新数据的任务。后者中，第一项试图对欧洲政党做意识形态缩放（Di Leo 等，2025），第二项复现 V-Dem 在 2023 年对 53 项指标的专家编码（Weidmann 等，2026）。在接口允许处我们保留各已发表任务与提示措辞，并去掉 JEV 无法遵循的回复格式指令。附录 A 记录提示词，附录 C、F、G 记录答案选项与评分量表的改动。

				JEV compared with
Study	Data	Test replicated	Human reference	published models	GPT-6 Luna	Qwen 27B
Annotation
Gilardi et al. (2023)	Tweets on content moderation, 2021 and 2023	Relevance (relevant or not), four runs; accuracy	Trained annotators; Mechanical Turk workers as a comparator	Better than ChatGPT	Worse	Slightly better (2021); worse (2023)a
Ornstein et al. (2025)	945 tweets on two Supreme Court decisions	Sentiment (positive, neutral, negative); correlation of the probability-based score	Mean of three expert ratings	Worse than GPT-3 and GPT-4; better than Twitter RoBERTa and Naive Bayes	Worse	Worse
Brandt et al. (2026)	37,709 Global Terrorism Database incidents; 322 BBC news articles	Attack type, nine categories, names only; accuracy and macro F1. BBC news: conflict F1	GTD coders’ attack type	Better than prompted LLMs; worse than fine-tuned ConfliBERT and ConflLlama. BBC: matches Llama 3.1, worse than ConfliBERT	Better acc.; slightly better F1. BBC: better	Better acc. and F1. BBC: better
Weidmann et al. (2026)	53 V-Dem indicators, 171 countries, 2023 (recall)	Ordinal indicator coded from the country name; correlation and exact agreement	V-Dem expert codes	Better than GPT-4o and Llama-3.1 70B	Better	Better
Scaling
Di Leo et al. (2025)	European parties in 9 EP election years, 1979–2019 (recall)	Pairwise “more right-wing” from party name; Bradley-Terry scale	CHES expert placements; CMP manifesto codes; TEV voter placements	Better than GPT-3.5	Slightly better	Better
DiGiuseppe and Flynn (2026)	1,402 answers on interest rates (Study 1); debt-ceiling answers (Study 2)	Pairwise knowledge and uncertainty; scale and expert-pair F1	Expert raters’ pairwise judgments	Within the range of six published and nine further generative scorers (mid-table)	Slightly worse	Slightly better
Le Mens and Gallego (2025)	18 UK manifestos; 36 EP speeches on coal subsidies	Sentence ratings averaged to documents; correlation	Expert coding (manifestos); crowd coding (speeches)	Within the range of the published models on manifestos; worse than GPT-4o on speeches	Worse	Slightly worse

				JEV 与下列对照
研究	数据	复现的检验	人类参照	已发表模型	GPT-6 Luna	Qwen 27B
标注
Gilardi 等 (2023)	内容审核相关推文，2021 与 2023	相关性（相关与否），四次运行；准确率	受训标注员；Mechanical Turk 工人作比较	优于 ChatGPT	更差	略优 (2021)；更差 (2023)a
Ornstein 等 (2025)	关于两项最高法院裁决的 945 条推文	情感（正、中、负）；基于概率的分数相关	三位专家评分均值	差于 GPT-3 与 GPT-4；优于 Twitter RoBERTa 与朴素贝叶斯	更差	更差
Brandt 等 (2026)	全球恐怖主义数据库 37,709 起事件；322 篇 BBC 新闻	袭击类型，九类，仅名称；准确率与宏 F1。BBC 新闻：冲突 F1	GTD 编码者的袭击类型	优于提示式 LLM；差于微调的 ConfliBERT 与 ConflLlama。BBC：与 Llama 3.1 持平，差于 ConfliBERT	准确率更好；F1 略好。BBC：更好	准确率与 F1 更好。BBC：更好
Weidmann 等 (2026)	53 项 V-Dem 指标，171 国，2023（回忆）	仅凭国名编码的序数指标；相关与精确一致	V-Dem 专家编码	优于 GPT-4o 与 Llama-3.1 70B	更好	更好
缩放
Di Leo 等 (2025)	1979–2019 共 9 个欧洲议会选举年的欧洲政党（回忆）	仅凭党名做成对「更右翼」判断；Bradley-Terry 量表	CHES 专家定位；CMP 宣言编码；TEV 选民定位	优于 GPT-3.5	略优	更好
DiGiuseppe 与 Flynn (2026)	利率相关答案 1,402 条（研究 1）；债务上限答案（研究 2）	成对知识与不确定性；量表与专家对 F1	专家评分者的成对判断	落在六项已发表与另九项生成式评分者区间内（中游）	略差	略优
Le Mens 与 Gallego (2025)	18 份英国党纲；36 篇关于煤炭补贴的欧洲议会演讲	句子评分汇总到文档；相关	专家编码（党纲）；众包编码（演讲）	党纲上落在已发表模型区间内；演讲上差于 GPT-4o	更差	略差

We first assess the speed and cost of JEV given that these are the advantages highlighted in its marketing. Table 2 compares cost and speed on the primary task of each application. JEV is not cheaper than GPT-6 Luna at OpenAI’s Batch prices. Luna costs less on six of the seven tasks and about the same (3% more) on the seventh. Against Luna’s Standard prices, which apply when answers are needed at once rather than within a day, JEV is cheaper on six of the seven tasks. We ran inference on Qwen through an external provider’s API (OpenRouter, served by a single host, Parasail, at 8-bit precision).[^2] In this case, Qwen was more expensive than both GPT-6 Luna and JEV, at 2.35 to 5.41 times JEV’s cost per decision. However, given the model’s small size, many scholars can run it on a local commercial machine or better yet on university supercomputer infrastructure at no cost to the researcher.

我们先评估 JEV 的速度与成本，因为这正是其营销所强调的优势。表 2 比较各应用主任务上的成本与速度。在 OpenAI 的 Batch 价格下，JEV 并不比 GPT-6 Luna 更便宜。七项任务中 Luna 有六项更低，第七项大致相当（约贵 3%）。相对 Luna 的标准价——适用于需要立刻而非一天内拿到答案的情形——JEV 在七项中有六项更便宜。我们通过外部提供商的 API 对 Qwen 做推理（OpenRouter，由单一主机 Parasail 以 8 比特精度提供）。[^2] 此时 Qwen 比 GPT-6 Luna 与 JEV 都贵，每个决策成本为 JEV 的 2.35 到 5.41 倍。不过鉴于模型体量不大，许多学者可在本地商用机器上运行，或更好是在大学超算设施上运行，对研究者零成本。

JEV is faster than the two alternatives. Sending one request at a time, it returned each decision in 0.09 to 0.27 seconds, against 0.75 to 1.04 seconds for Luna and 0.39 to 0.87 for Qwen.[^3] We now proceed to examine the accuracy of JEV in each of the annotation and scaling tasks.

JEV 快于两个对照。逐个发送请求时，每个决策返回时间为 0.09 到 0.27 秒，Luna 为 0.75 到 1.04 秒，Qwen 为 0.39 到 0.87 秒。[^3] 下面我们转而考察 JEV 在各项标注与缩放任务上的准确率。

$ per 1,000 decisions Seconds per decision Decisions per second Application JEV Luna (Batch) Luna (Std.) Qwen JEV Luna Qwen JEV Luna Qwen Gilardi et al. (2023) 0.017 0.017 0.034 0.085 0.09 0.77 0.44 1561831 Ornstein et al. (2025) 0.024 0.012 0.025 0.128 0.26 0.75 0.39 471830 Brandt et al. (2026) 0.027 0.018 0.035 0.139 0.27 1.04 0.87 511414 Weidmann et al. (2026) a 0.020 0.012 0.024 0.058 0.26 0.75 0.41 451731 Di Leo et al. (2025) 0.018 0.008 0.016 0.042 0.25 0.78 0.43 462028 DiGiuseppe and Flynn (2026) 0.012 0.012 0.024 0.058 0.13 0.78 0.44 931332 Le Mens and Gallego (2025) 0.014 0.009 0.019 0.058 0.13 0.87 0.49 951721 Table 2: Cost and speed of JEV, GPT-6 Luna and Qwen3.8-27B on the primary task of each application. A decision is one answer to one question about one item. Cost: dollars per 1,000 decisions; JEV and Qwen as billed, GPT-6 Luna at OpenAI’s posted Batch and Standard rates. Qwen ran through OpenRouter on one host (Parasail, FP8) with no batch rate, so its cost is that of this route, not of running the open weights oneself. Speed: median seconds per decision with one request at a time, and decisions per second with 16 at a time, from 200 timed items per application sent from the Netherlands on 29-30 September 2026. Throughput depends on the route and rate limits and is not a property of the models. a Costs for JEV and Luna are from the run’s totals.

	$ per 1,000 decisions	Seconds per decision	Decisions per second
Application	JEV	Luna (Batch)	Luna (Std.)	Qwen	JEV	Luna	Qwen	JEV	Luna	Qwen
Gilardi et al. (2023)	0.017	0.017	0.034	0.085	0.09	0.77	0.44	156	18	31
Ornstein et al. (2025)	0.024	0.012	0.025	0.128	0.26	0.75	0.39	47	18	30
Brandt et al. (2026)	0.027	0.018	0.035	0.139	0.27	1.04	0.87	51	14	14
Weidmann et al. (2026)a	0.020	0.012	0.024	0.058	0.26	0.75	0.41	45	17	31
Di Leo et al. (2025)	0.018	0.008	0.016	0.042	0.25	0.78	0.43	46	20	28
DiGiuseppe and Flynn (2026)	0.012	0.012	0.024	0.058	0.13	0.78	0.44	93	13	32
Le Mens and Gallego (2025)	0.014	0.009	0.019	0.058	0.13	0.87	0.49	95	17	21

每 1,000 决策美元 每决策秒数 每秒决策数 应用 JEV Luna (Batch) Luna (Std.) Qwen JEV Luna Qwen JEV Luna Qwen Gilardi 等 (2023) 0.017 0.017 0.034 0.085 0.09 0.77 0.44 1561831 Ornstein 等 (2025) 0.024 0.012 0.025 0.128 0.26 0.75 0.39 471830 Brandt 等 (2026) 0.027 0.018 0.035 0.139 0.27 1.04 0.87 511414 Weidmann 等 (2026) a 0.020 0.012 0.024 0.058 0.26 0.75 0.41 451731 Di Leo 等 (2025) 0.018 0.008 0.016 0.042 0.25 0.78 0.43 462028 DiGiuseppe 与 Flynn (2026) 0.012 0.012 0.024 0.058 0.13 0.78 0.44 931332 Le Mens 与 Gallego (2025) 0.014 0.009 0.019 0.058 0.13 0.87 0.49 951721 表 2：JEV、GPT-6 Luna 与 Qwen3.8-27B 在各应用主任务上的成本与速度。一个决策指对一个条目的一个问题给出一个答案。成本：每 1,000 决策的美元；JEV 与 Qwen 按实付账单，GPT-6 Luna 按 OpenAI 公布的 Batch 与标准费率。Qwen 经 OpenRouter 在单一主机（Parasail，FP8）上运行且无批处理折扣，故其成本是该路由的成本，而非自行跑开源权重的成本。速度：逐个请求时的中位每决策秒数，以及一次 16 路时的每秒决策数；来自 2026 年 9 月 29-30 日自荷兰发送的各应用 200 条计时样本。吞吐依赖路由与限速，并非模型固有属性。 a JEV 与 Luna 的成本来自该次运行合计。

	每 1,000 决策美元	每决策秒数	每秒决策数
应用	JEV	Luna (Batch)	Luna (Std.)	Qwen	JEV	Luna	Qwen	JEV	Luna	Qwen
Gilardi 等 (2023)	0.017	0.017	0.034	0.085	0.09	0.77	0.44	156	18	31
Ornstein 等 (2025)	0.024	0.012	0.025	0.128	0.26	0.75	0.39	47	18	30
Brandt 等 (2026)	0.027	0.018	0.035	0.139	0.27	1.04	0.87	51	14	14
Weidmann 等 (2026)a	0.020	0.012	0.024	0.058	0.26	0.75	0.41	45	17	31
Di Leo 等 (2025)	0.018	0.008	0.016	0.042	0.25	0.78	0.43	46	20	28
DiGiuseppe 与 Flynn (2026)	0.012	0.012	0.024	0.058	0.13	0.78	0.44	93	13	32
Le Mens 与 Gallego (2025)	0.014	0.009	0.019	0.058	0.13	0.87	0.49	95	17	21

Tweet relevance

推文相关性

Gilardi et al. (2023) were among the first to show that LLMs could provide high-quality annotations of a common task: annotating tweets. They use tweets from Alizadeh et al. (2022), who collected English-language tweets about content moderation through Twitter’s academic API from January 2020 to April 2021, about 2.6 million tweets after filtering. Gilardi et al. (2023) drew a random sample from this corpus and added a new sample of tweets from January 2023, collected the same way.[^4] The tasks we replicate are whether a tweet is relevant to content moderation and, separately, whether it frames content moderation as a problem or as a solution.

Gilardi 等（2023）较早表明 LLM 可对常见任务——标注推文——给出高质量标注。他们使用 Alizadeh 等（2022）的推文：后者经 Twitter 学术 API 收集了 2020 年 1 月至 2021 年 4 月关于内容审核的英语推文，过滤后约 260 万条。Gilardi 等（2023）从该语料随机抽样，并另加按同样方式收集的 2023 年 1 月新样本。[^4] 我们复现的任务是：一条推文是否与内容审核相关；以及分别判断它把内容审核框定为问题还是解决方案。

-0.8

-0.8

Tweet sentiment

推文情感

+0.025

+0.025

Terrorist attack type

恐袭类型

-0.034

-0.034

Open-ended survey responses

开放式调查回答

DiGiuseppe and Flynn (2026) show that scales built from LLM pairwise comparisons agree more closely across models than naive (0--10) point-wise ratings do. Their first study scales 1,402 explanations of how interest rates are determined as a proxy for economic knowledge. Their second study scales uncertainty in respondent-supplied likely consequences of a potential breach of the US debt ceiling in 2023.[^7] Both fit a Bayesian Bradley-Terry model (Bradley and Terry, 1952) to recover the scale from twenty comparisons per answer. We compare JEV with GPT-4o, the frontier model at the time of their study, and with the two contemporary models.

DiGiuseppe 与 Flynn（2026）表明，由 LLM 成对比较构建的量表，跨模型一致性高于朴素的（0--10）逐点评分。其第一项研究将 1,402 条关于利率如何决定的解释作为经济知识的代理加以缩放。第二项研究缩放受访者所给出的、2023 年美国债务上限可能被突破时可能后果中的不确定性。[^7] 两者都对每条答案做二十次比较，拟合贝叶斯 Bradley-Terry 模型（Bradley 与 Terry，1952）从比较中恢复量表。我们将 JEV 与当时前沿模型 GPT-4o，以及两个当代模型比较。

Each study compares several language models, of varying sizes, on three tasks. The models judge the answers in pairs (there are 28,040 pairs in Study 1 and 30,280 in Study 2, with each answer paired with twenty others selected at random), rate each answer on a scale of 0 to 10 and, in Study 1 only, judge the pairs that the experts also judged. For each of these tasks we use the same pairs of answers and the same wording as the original, except that for JEV we leave out the lines about how the reply should be formatted. JEV judges each pair in both orders, with the published order as the primary specification. GPT-6 Luna and Qwen judge pairs only in the published order, except for the pairs judged by the experts, which they judge in both orders. For the ratings, we ask JEV to give each answer a Score on a ten-level scale and rescale these to the range of the original ratings. We also add two blocks of requests that are not in the original. The first has every pair among a random selection of 80 answers per study, in order to test transitivity. In the second we repeat requests, in order to test reliability.

每项研究在三项任务上比较多种体量的语言模型。模型成对判断答案（研究 1 有 28,040 对，研究 2 有 30,280 对，每条答案与随机选取的另二十条配对），在 0 到 10 的量表上给每条答案打分，并仅在研究 1 中判断专家也判断过的那些对。对每项任务我们使用与原文相同的答案对与措辞，只是对 JEV 去掉关于回复应如何格式化的句子。JEV 以两种顺序判断每对，以已发表顺序为主要设定。GPT-6 Luna 与 Qwen 只按已发表顺序判断成对，但专家判断过的那些对则两种顺序都判断。评分方面，我们请 JEV 在十级量表上给每条答案一个 Score，再缩放到原评分的范围。我们还增加两块原文没有的请求。第一块对每项研究随机抽取的 80 条答案做全部成对，以检验传递性。第二块重复请求，以检验信度。

-0.004

-0.004

Manifestos and speeches

党纲与演讲

Le Mens and Gallego (2025) use LLMs to position political texts on policy and ideological dimensions. They prompt an LLM to score each tweet or sentence on a specified scale, allowing an “NA” response when the text lacks relevant political content. For longer documents, they average the numeric sentence scores to estimate the document’s position. We repeat this with JEV, GPT-6 Luna and Qwen3.8-27B for 18,263 sentences from 18 British party manifestos on economic and social dimensions, and 429 sentences from 36 European Parliament speeches on coal subsidies. The benchmarks are document-level judgments. The manifesto targets are expert coding and the speech targets are crowd coding from Benoit et al. (2016).

Le Mens 与 Gallego（2025）用 LLM 在政策与意识形态维度上定位政治文本。他们提示 LLM 在指定量表上给每条推文或句子打分，并在文本缺乏相关政治内容时允许回答「NA」。对较长文档，他们平均数值句子分数以估计文档位置。我们用 JEV、GPT-6 Luna 与 Qwen3.8-27B 重复：对 18 份英国政党党纲中 18,263 句做经济与社会维度，以及对 36 篇欧洲议会煤炭补贴演讲中 429 句。基准是文档级判断。党纲目标为专家编码，演讲目标为 Benoit 等（2016）的众包编码。

Both corpora are taken from Benoit et al. (2016). The manifesto corpus contains the manifestos of the Conservatives, Labour and the Liberal Democrats at the six general elections from 1987 to 2010, 18,263 sentences in total. Each sentence was coded by four to six political scientists (faculty members and doctoral students), which gives about seven ratings per sentence once both coding orders are counted. The benchmark is the scaled expert position of each manifesto, as calculated by Benoit et al. (2016). The speech corpus contains the 36 contributions to the European Parliament’s debate of 23 November 2010 on state aid to close uncompetitive coal mines, 429 sentences in their original languages. Crowd workers coded these in six language versions, and the benchmark is the average of the six crowd scores.

两个语料均取自 Benoit 等（2016）。党纲语料包含保守党、工党与自由民主党在 1987 至 2010 年六次大选的党纲，共 18,263 句。每句由四到六位政治学家（教员与博士生）编码；计两种编码顺序后约每句七个评分。基准是 Benoit 等（2016）计算的各党纲缩放专家位置。演讲语料为 2010 年 11 月 23 日欧洲议会关于向关闭不具竞争力煤矿提供国家援助辩论中的 36 篇发言，共 429 句，保持原语言。众包工人按六种语言版本编码，基准为六份众包分数的均值。

Le Mens and Gallego (2025) report four applications. These are tweets from members of the 118th US Congress, the positions of the senators of the 117th Congress based on their tweets, the British manifestos and the coal debate. We replicate the last two, using the same specifications as their main results. The manifestos are used without a description of the dimension, and the speeches in their original languages with a description. For JEV, instead of the 0-100 scale we use nine worded Score levels, and instead of the instruction to reply NA we add a separate yes/no question asking whether the sentence is relevant. We fixed the primary inclusion threshold at 0.5 before the run. For GPT-6 Luna and Qwen we use the published prompts, including the scale and the NA option.

Le Mens 与 Gallego（2025）报告四项应用：第 118 届美国国会成员的推文、基于推文的第 117 届国会参议员位置、英国党纲，以及煤炭辩论。我们复现后两项，规格与其主要结果相同。党纲不提供维度描述；演讲保持原语言并提供描述。对 JEV，我们不用 0-100 量表而用九个带文字的 Score 等级，并用单独的是否题询问句子是否相关，以替代「回答 NA」的指令。我们在运行前将主要纳入阈值固定为 0.5。对 GPT-6 Luna 与 Qwen 使用已发表提示，含量表与 NA 选项。

JEV’s expected scores, averaged over included sentences, correlate 0.97 with the economic benchmark and 0.92 with the social benchmark, within the published models’ range but below the published GPT-4o results of 0.98 and 0.94, respectively. The speeches are harder. JEV reaches 0.81 on the 35 speeches with at least one sentence it judges relevant, below GPT-4o’s 0.93 (Table 14). GPT-6 Luna is above JEV on all three benchmarks (correlations of 0.977, 0.952 and 0.947), while Qwen3.8-27B is level with JEV on all three (0.974, 0.940 and 0.905, judged level by a paired bootstrap over documents).

JEV 的期望分数在纳入句子上平均后，与经济基准相关 0.97、与社会基准相关 0.92，落在已发表模型区间内，但分别低于已发表的 GPT-4o 结果 0.98 与 0.94。演讲更难。JEV 在至少有一句被其判为相关的 35 篇演讲上达到 0.81，低于 GPT-4o 的 0.93（表 14）。GPT-6 Luna 在三项基准上均高于 JEV（相关 0.977、0.952 与 0.947），而 Qwen3.8-27B 在三项上均与 JEV 持平（0.974、0.940 与 0.905，以文档上的成对自助法判定持平）。

Party positions from party names

从党名推断政党位置

Di Leo et al. (2025) test whether an LLM can place European parties on a left-right scale from its own embedded knowledge. This allows for scaling of parties that have no common sources of votes or corpora. In the application, the model sees only two party names and must decide which is more right-wing. A Bradley-Terry model turns these pairwise answers into a scale, which is compared with Chapel Hill expert placements (Jolly et al., 2022).

Di Leo 等（2025）检验 LLM 能否仅凭自身嵌入知识，把欧洲政党放在左右量表上。这使得对没有共同选票来源或语料的政党也能缩放。应用中，模型只看到两个党名，必须判定哪个更右翼。Bradley-Terry 模型把这些成对答案变成量表，再与 Chapel Hill 专家定位比较（Jolly 等，2022）。

The party names and benchmarks are from three sources. These are the Manifesto Project dataset (Lehmann et al., 2024), the Chapel Hill Expert Survey (Jolly et al., 2022), extended back to 1984 with the Ray-Marks-Steenbergen file (Ray, 1999; Steenbergen and Marks, 2007), whose left-right values come from earlier expert surveys for 1984 and 1996 and are interpolated for 1988 and 1992, and voters’ placements of parties in the harmonised True European Voter surveys (Schmitt, 2021). Placements from the Comparative Study of Electoral Systems (The Comparative Study of Electoral Systems, 2024) provide an alternative benchmark. We have one sample per European Parliament election between 1979 and 2019. Each sample includes the European parties that ran in a national election in the four years before that European Parliament election. The samples contain from 112 to 362 parties from 18 to 39 countries. Each party is compared with every other party in its sample, which gives 347,801 dyads in total. From 1984 on there are such placements for between 74 and 239 parties a year (those used for 1989 are mostly interpolations), and for 1979 there are none.

党名与基准来自三个来源：Manifesto Project 数据集（Lehmann 等，2024）；Chapel Hill 专家调查（Jolly 等，2022），并用 Ray-Marks-Steenbergen 文件（Ray，1999；Steenbergen 与 Marks，2007）回溯至 1984——其左右值来自 1984 与 1996 年更早的专家调查，1988 与 1992 年为插值；以及协调后的 True European Voter 调查中选民对政党的定位（Schmitt，2021）。比较选举制度研究（The Comparative Study of Electoral Systems，2024）的定位提供另一基准。我们对 1979 至 2019 年间每一次欧洲议会选举各有一个样本。每个样本纳入该次欧洲议会选举前四年内参加过全国选举的欧洲政党。样本含 18 到 39 国、112 到 362 个政党。样本内每个政党与其他每个政党比较，合计 347,801 个二元组。自 1984 年起，每年有此类定位的政党在 74 到 239 个之间（1989 年所用多为插值），1979 年则没有。

The task covers the whole of the main analysis in the original, which concerns the left-right dimension only, with every dyad in every year. Di Leo et al. (2025) take the most common answer of seven runs. We use only one run. For JEV we remove the closing instruction to reply with exactly “Party 1” or “Party 2”, but we keep the original order of presentation. We do not repeat the robustness analyses in their supplementary material, such as pooled years, few-shot prompts or other framings.

任务覆盖原文主分析的全部内容，仅关左右维度，且含每一年的每个二元组。Di Leo 等（2025）取七次运行中最常见的答案。我们只用一次运行。对 JEV，我们去掉结尾「恰好回答 Party 1 或 Party 2」的指令，但保留原始呈现顺序。我们不重复其补充材料中的稳健性分析，如合并年份、少样本提示或其他框定方式。

Across the 8 election years, JEV’s scale correlates on average (Table 13) 0.877 with the experts, above the 0.772 of the GPT-3.5 model in the original study, although their supplement reports that GPT-4o and Llama-3.1 70B did better than GPT-3.5. GPT-6 Luna averages 0.844, and its difference from JEV is indistinguishable from zero in all 8 years. Qwen3.8-27B averages 0.755 and is below JEV in all eight years, with differences of between 0.09 and 0.15.

在 8 个选举年上，JEV 的量表与专家平均相关（表 13）为 0.877，高于原研究中 GPT-3.5 的 0.772，尽管其补充材料报告 GPT-4o 与 Llama-3.1 70B 优于 GPT-3.5。GPT-6 Luna 平均 0.844，与 JEV 的差异在全部 8 年都无法与零区分。Qwen3.8-27B 平均 0.755，八年皆低于 JEV，差在 0.09 到 0.15 之间。

Democracy indicators from country names

从国名推断民主指标

Weidmann et al. (2026) ask whether LLMs can reproduce V-Dem expert codes. The model receives a country’s name, the year and a V-Dem question with its answer scale, and returns a code. We repeat this for 53 indicators in 171 countries in 2023. JEV’s codes correlate 0.68 with V-Dem’s on average (Table 12), above GPT-4o (0.63) and Llama-3.1 70B (0.50), recomputed from the original study’s published codes (the article reports 0.64 for GPT-4o), and above GPT-6 Luna (0.63) and Qwen3.8-27B (0.59). All models do worse on the codes that changed from the previous year.

Weidmann 等（2026）追问 LLM 能否复现 V-Dem 专家编码。模型收到国名、年份以及带答案量表的 V-Dem 问题，并返回一个编码。我们对 2023 年 171 国的 53 项指标重复此事。JEV 的编码与 V-Dem 平均相关 0.68（表 12），高于由原研究已发表编码重算的 GPT-4o（0.63）与 Llama-3.1 70B（0.50）（文章对 GPT-4o 报告 0.64），也高于 GPT-6 Luna（0.63）与 Qwen3.8-27B（0.59）。所有模型在相对上一年发生变化的编码上都表现更差。

We use version 14 of V-Dem (Coppedge et al., 2024c; Coppedge et al., 2024a), coded for 2023, as the benchmark. The 53 indicators are the ordinal, expert-coded indicators that Weidmann et al. (2026) selected from the components of V-Dem’s five high-level democracy indices; indicators coded only in election years are not among them. Each code is based on the ratings of several country experts (a median of six per code in V-Dem’s coder-level data, Coppedge et al. 2024b) and is computed with V-Dem’s measurement model, which adjusts for differences between experts in how they use the scale (Pemstein et al., 2024). Of the 9,063 country-indicator pairs, 9,041 have a V-Dem code and are scored.

我们以 V-Dem 第 14 版（Coppedge 等，2024c；Coppedge 等，2024a）对 2023 年的编码为基准。53 项指标是 Weidmann 等（2026）从 V-Dem 五个高层民主指数的组成部分中选出的序数、专家编码指标；仅在选举年编码的指标不在其中。每个编码基于若干国家专家的评分（V-Dem 编码者级数据中位数为每个编码六人，Coppedge 等 2024b），并用 V-Dem 测量模型计算，该模型校正专家在量表使用上的差异（Pemstein 等，2024）。在 9,063 个国家—指标对中，9,041 对有 V-Dem 编码并纳入评分。

In a single task, Weidmann et al. (2026) code all 53 indicators with GPT-4o and Llama-3.1 70B, and then analyse how the models deviate systematically from V-Dem by country and as a function of disagreement between coders. We run the full task. For JEV, we use the V-Dem question as the instruction and “What is the score for [country] in 2023?” as the input. The ordered response scale gives the levels of a Score, and we remove the instruction to reply with a number. For GPT-6 Luna and Qwen we use the published prompt.

Weidmann 等（2026）在单项任务中用 GPT-4o 与 Llama-3.1 70B 编码全部 53 项指标，再分析模型相对 V-Dem 在国家层面以及随编码者分歧如何系统偏离。我们跑完整任务。对 JEV，我们以 V-Dem 问题为指令，以「What is the score for [country] in 2023?」为输入。有序回答量表给出 Score 的各等级，我们去掉「用数字作答」的指令。对 GPT-6 Luna 与 Qwen 使用已发表提示。

In both recall tasks the models answer from what they learned in training, not from a text they read. The expert placements and V-Dem codes were public before these models were trained, so high agreement may partly reflect recall of the benchmarks themselves. This matters most for V-Dem: Weidmann et al. (2026) used version 14 because it was released after GPT-4o and Llama-3.1 were trained, but JEV, GPT-6 Luna and Qwen3.8-27B were trained after its release, so the comparison with the published models may favour the newer models.

在两项回忆任务中，模型依据训练中学到的内容作答，而非依据正在阅读的文本。专家定位与 V-Dem 编码在这些模型训练之前就已公开，因此高度一致可能部分反映对基准本身的回忆。这对 V-Dem 尤为重要：Weidmann 等（2026）使用第 14 版是因为它在 GPT-4o 与 Llama-3.1 训练之后发布；但 JEV、GPT-6 Luna 与 Qwen3.8-27B 是在该版发布之后训练的，因此与已发表模型的比较可能偏向更新的模型。

In the V-Dem task, JEV has a small but clear lead over every model. Its mean correlation with V-Dem is higher than GPT-6 Luna’s by 0.05 [0.04, 0.07], GPT-4o’s by 0.05 [0.03, 0.06], Qwen3.8-27B’s by 0.09 [0.07, 0.11] and Llama-3.1 70B’s by 0.18 [0.16, 0.20].

在 V-Dem 任务上，JEV 相对每个模型都有小而清晰的领先。其与 V-Dem 的平均相关高于 GPT-6 Luna 0.05 [0.04，0.07]，高于 GPT-4o 0.05 [0.03，0.06]，高于 Qwen3.8-27B 0.09 [0.07，0.11]，高于 Llama-3.1 70B 0.18 [0.16，0.20]。

JEV’s advantage?

JEV 的优势？

JEV annotates and scales text in these political science contexts about as well as a mid-tier commercial LLM and, in many instances, as well as open-weight models that can be run locally. Further, JEV does not show much of a cost advantage over what is commercially available in autumn 2026. JEV does have a clear speed advantage. However, for most scholarly uses, speed is not a deciding factor. On these metrics, there is little reason to adopt JEV in place of traditional, token-based large language models.

在这些政治学场景中，JEV 标注与缩放文本的水平大致相当于中档商业 LLM，并在许多情形下相当于可本地运行的开源权重模型。此外，相对 2026 年秋季市场上可获得的选项，JEV 也没有多少成本优势。JEV 确有清晰的速度优势。但对多数学术用途而言，速度并非决定性因素。就这些指标而论，几乎没有理由用 JEV 取代传统的、基于 token 的大语言模型。

Beyond these metrics, JEV’s supposed calibration advantage might be another reason to prefer it over traditional LLMs. Probabilities help scholars in two ways. First, they rank texts by how likely each answer is, so researchers can screen a corpus for a rare category, or send the answers the model is least sure of to a human or a more capable (and expensive) LLM. For this the probabilities need to rank answers well, which AUROC measures. Second, researchers can weight each answer by its probability to build a continuous scale (Licht et al., 2025), or read the probabilities as rates. For this they also need to be calibrated, which ECE measures. Both uses rely on the probabilities being informative and available.

在这些指标之外，JEV 所谓的校准优势或许是另一项相对传统 LLM 偏好它的理由。概率从两方面帮助学者。第一，它们按各答案的可能性给文本排序，使研究者能在语料中筛出稀有类别，或把模型最不确定的答案交给人类或更强（也更贵）的 LLM。为此概率需要排得好，AUROC 衡量这一点。第二，研究者可用概率给每个答案加权以构建连续量表（Licht 等，2025），或把概率读作比率。为此它们还需要校准良好，ECE 衡量这一点。两种用途都依赖概率有信息量且可得。

As we noted in the introduction, some providers now report few token probabilities or none, and those GPT-6 Luna reports are mostly close to 0 or 1. Figure 1 shows this for the party comparisons in our replication of Di Leo et al. (2025): 74% of Luna’s probabilities are below 0.01 or above 0.99, against 18% for JEV and 16% for Qwen. JEV returns more mid-range probabilities, as we would expect when two parties are close ideologically. Qwen does the same, so near-certainty is not a property of LLMs as such.

如引言所述，部分提供商现在只报告很少甚至不报告 token 概率，而 GPT-6 Luna 报告的那些大多接近 0 或 1。图 1 展示我们复现 Di Leo 等（2025）时的政党比较：Luna 有 74% 的概率低于 0.01 或高于 0.99，JEV 为 18%，Qwen 为 16%。JEV 返回更多中段概率，这在两个政党意识形态接近时是预期表现。Qwen 亦然，因此近乎确定性并非 LLM 本身的属性。

N

N

The difference in probabilities is indicative, but not conclusive, evidence of poor calibration in GPT-6 Luna. Figure 2 presents the expected calibration error (ECE, left panel; Naeini et al. 2015; Guo et al. 2017) and the area under the receiver operating characteristic curve (AUROC, right panel; Hanley and McNeil 1982). The former groups each model’s answers by its stated probability, compares that probability with the share of answers that agree with the human coders in each group, and averages the gaps, weighting by group size. An ECE of zero, for example, means that answers given with 80% probability agree with the human coders 80% of the time. An ECE of one would mean the model is 100% confident and got every answer wrong.

概率差异提示但并不能确证 GPT-6 Luna 校准不佳。图 2 给出期望校准误差（ECE，左图；Naeini 等 2015；Guo 等 2017）与受试者工作特征曲线下面积（AUROC，右图；Hanley 与 McNeil 1982）。前者按模型声称的概率分组答案，将该概率与每组中与人工编码者一致的份额比较，再按组大小加权平均差距。例如 ECE 为零意味着以 80% 概率给出的答案有 80% 与人工编码者一致。ECE 为一意味着模型 100% 自信且每题皆错。

When each question is asked once, GPT-6 Luna has higher calibration error than JEV on all eight reading tasks with a categorical human label. This suggests that TypeSafe’s claims of better calibration might be warranted against Luna. The gap disappears on the open-ended survey pairs when we average over the two presentation orders, and it reverses on the attack-type task, where Luna writes its probabilities in its reply. Qwen3.8-27B’s calibration differs from JEV’s by type of task. Its calibration error is lower on the four pairwise tasks, higher on three of the four labelling tasks, and higher on both attack-type arms and both recall tasks. JEV’s advantage therefore holds relative to Luna’s token probabilities but not consistently relative to Qwen’s. Yet, Where JEV does show better calibration than Qwen, the differences are not dramatic.

当每个问题只问一次时，在全部八项带类别人工标签的阅读任务上，GPT-6 Luna 的校准误差都高于 JEV。这表明 TypeSafe 关于更好校准的主张相对 Luna 可能成立。在开放式调查成对上，当我们对两种呈现顺序取平均时，差距消失；在袭击类型任务上差距反转——该任务中 Luna 在回复里写出其概率。Qwen3.8-27B 的校准相对 JEV 因任务类型而异：四项成对任务上误差更低，四项标注任务中有三项更高，两项袭击类型分支与两项回忆任务上也更高。因此 JEV 的优势相对 Luna 的 token 概率成立，但相对 Qwen 并不一贯。即便如此，在 JEV 校准确优于 Qwen 之处，差异也并不剧烈。

The AUROC presents a similar picture. It is the chance that a randomly chosen right answer gets a higher probability than a randomly chosen wrong one. Luna’s AUROC is lower than JEV’s on seven of the eight tasks and level on the other, and lower than Qwen’s on all eight. On attack type, where it writes its probabilities, Luna’s AUROC is above Qwen’s in both arms and above JEV’s only when given the codebook. JEV and Qwen are not consistently different.

AUROC 呈现相似图景。它是随机抽到的正确答案比随机抽到的错误答案获得更高概率的机会。Luna 的 AUROC 在八项中有七项低于 JEV、一项持平，并在全部八项上低于 Qwen。在袭击类型上——它写出概率——Luna 的 AUROC 在两个分支上都高于 Qwen，仅在给定编码手册时高于 JEV。JEV 与 Qwen 没有一贯差异。

Figure 2: Calibration of JEV, GPT-6 Luna and Qwen3.8-27B against human coders, by task, with 95% bootstrap intervals. Left: calibration error (ECE), the gap between the probability a model gives its answer and the share of such answers that agree with the human label, over ten probability bins; lower is better. Right: AUROC, the chance that a correct answer gets a higher probability than an incorrect one; 0.5 is chance. Top rows: each question asked once. Middle rows: probabilities averaged over the two presentation orders. Bottom rows: on attack type GPT-6 Luna and Qwen write their probabilities in the reply; party positions and democracy indicators are recall tasks. In the attack-type and party-position rows, which have tens or hundreds of thousands of items, the intervals are narrower than the markers. Appendix H describes the Licht and MARPOR tasks.

图 2：JEV、GPT-6 Luna 与 Qwen3.8-27B 相对人工编码者的校准，按任务，含 95% 自助区间。左：校准误差（ECE），即模型赋给其答案的概率与此类答案中与人工标签一致份额之间的差距，按十个概率分箱；越低越好。右：AUROC，正确答案获得高于错误答案之概率的机会；0.5 为随机。上行：每个问题只问一次。中行：对两种呈现顺序的概率取平均。下行：在袭击类型上 GPT-6 Luna 与 Qwen 在回复中写出概率；政党位置与民主指标为回忆任务。袭击类型与政党位置行有数万至数十万条目，区间窄于标记。附录 H 描述 Licht 与 MARPOR 任务。

Conclusion

结论

Our analysis indicates that JEV is a capable model for text scaling and annotation in political science contexts. However, there are no major gains in cost or capabilities that make it a clear choice for most applications. The main advantages lie in its speed and ease of parsing the underlying probabilities. When each question is asked once, JEV’s probabilities are also better calibrated than GPT-6 Luna’s token probabilities, though not consistently better than those of Qwen3.8-27B. As such, the choice to use JEV over other LLMs is task dependent. In some cases, JEV might offer more accurate calibration while still performing on par with LLMs for social science tasks. Our analysis suggest that this still requires that researchers validate JEV as they would other LLMs.

我们的分析表明，在政治学场景中，JEV 是文本缩放与标注的可用模型。但在成本或能力上并无重大收益，足以使其成为多数应用的明确选择。主要优势在于速度，以及底层概率易于解析。当每个问题只问一次时，JEV 的概率也比 GPT-6 Luna 的 token 概率校准更好，尽管并不一贯优于 Qwen3.8-27B。因此，是否用 JEV 而非其他 LLM，取决于任务。在某些情形下，JEV 或许提供更准确的校准，同时在社会科学任务上仍与 LLM 表现相当。我们的分析表明，这仍要求研究者像验证其他 LLM 一样验证 JEV。

References

参考文献

DiGiuseppe et al. (2025) DiGiuseppe, M., A. C. Garriga, and A. Kern (2025). Information, party politics, and public support for central bank independence. SSRN Working Paper 5128037. DiGiuseppe and Shea (2025) DiGiuseppe, M. and P. E. Shea (2025). Information, uncertainty, and public support for brinkmanship during the 2023 debt limit negotiations. British Journal of Political Science 55 , e14. Efron (1979) Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics 7 (1), 1-26. Fernholz (2026) Fernholz, T. (2026, September). A new kind of AI model from a ChatGPT inventor is thrilling developers. TechCrunch. Gilardi et al. (2023) Gilardi, F., M. Alizadeh, and M. Kubli (2023). ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences 120 (30), e2305016120. Greene and Cunningham (2006) Greene, D. and P. Cunningham (2006). Practical solutions to the problem of diagonal dominance in kernel document clustering. In Proceedings of the 23rd International Conference on Machine Learning , pp. 377-384. Association for Computing Machinery. Guo et al. (2017) Guo, C., G. Pleiss, Y. Sun, and K. Q. Weinberger (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning , Volume 70 of Proceedings of Machine Learning Research , pp. 1321-1330. Hanley and McNeil (1982) Hanley, J. A. and B. J. McNeil (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143 (1), 29-36. Jolly et al. (2022) Jolly, S., R. Bakker, L. Hooghe, G. Marks, J. Polk, J. Rovny, M. Steenbergen, and M. A. Vachudova (2022). Chapel Hill Expert Survey trend file, 1999-2019. Electoral Studies 75 , 102420. Kadavath et al. (2022) Kadavath, S., T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, et al. (2022). Language models (mostly) know what they know. arXiv:2207.05221.

Tian et al. (2023) Tian, K., E. Mitchell, A. Zhou, A. Sharma, R. Rafailov, H. Yao, C. Finn, and C. D. Manning (2023). Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pp. 5433-5442. TypeSafe (2026) TypeSafe (2026). Jev documentation. https://typesafe.ai and https://docs.typesafe.ai . Accessed 23 September 2026. Weidmann et al. (2026) Weidmann, N. B., M. Faulborn, and D. García (2026). Large language models are democracy coders with attitudes. PS: Political Science & Politics 59 (1), 17-23.

Use of AI tools

AI 工具使用说明

We used Claude Opus 5 and Opus 5.5 (Anthropic) to generate analysis code and Fable 5.1 (Anthropic) to check the code against reported figures. GPT-6 Astra (OpenAI) drafted the initial manuscript, which the authors heavily edited and added to. Before posting, we used Claude Opus 5.5 to audit the paper’s claims and citations against their sources and our results, and for copy editing. Claude Sonnet 5.5 proofread the text. We reviewed every resulting change. All design choices, analysis and conclusions are ours, and we take full responsibility for the content.

我们使用 Claude Opus 5 与 Opus 5.5（Anthropic）生成分析代码，并用 Fable 5.1（Anthropic）对照报告数字检查代码。GPT-6 Astra（OpenAI）起草初稿，作者做了大量编辑与增补。发布前，我们用 Claude Opus 5.5 对照来源与结果审计论文主张与引用，并做文字润色。Claude Sonnet 5.5 校对正文。我们对每一处改动都做了审阅。全部设计选择、分析与结论均为作者作出，并对内容负全部责任。

Appendix A: prompts as sent

附录 A：实际发送的提示

The survey application’s prompts are given first, followed by those of the other applications, each copied from the first request of its kind in the files that were sent.

先给出调查应用的提示，再给出其他应用的提示；每一条都从所发送文件中该类请求的第一条复制而来。

Study 1: interest-rate knowledge

研究 1：利率知识

Open-ended item. In a few sentences and without looking it up, can you explain how interest rates (i.e. the cost of borrowing money to buy a house or car) go up or down in the US economy?

开放题 。 请用几句话、且不查阅资料，解释在美国经济中利率（即借钱买房或买车的成本）如何上升或下降？

Pairwise Choice with a tie option (Choice). Instructions. You are an expert in US economic policy. Your task is to determine which of two given statements contains a more knowledgeable response to the following question: In a few sentences and without looking it up, can you explain how interest rates (i.e. the cost of borrowing money to buy a house or car) go up or down in the US economy?

带平局选项的成对 Choice（Choice）。 指令。 你是美国经济政策专家。你的任务是判断下列两个陈述中，哪一个对下述问题给出了更有知识含量的回答：请用几句话、且不查阅资料，解释在美国经济中利率（即借钱买房或买车的成本）如何上升或下降？

Options.

选项。

• first: The first statement contains more knowledge. • second: The second statement contains more knowledge. • equal: They are equal or incomparable.

• first：第一个陈述知识含量更高。 • second：第二个陈述知识含量更高。 • equal：两者相当或不可比。

Pairwise Choice without a tie option (Choice). Instructions. You are an expert in US economic policy. Your task is to determine which of two given statements contains a more knowledgeable response to the following question: In a few sentences and without looking it up, can you explain how interest rates (i.e. the cost of borrowing money to buy a house or car) go up or down in the US economy? If both are equally knowledgeable choose one with a slight preference.

无平局选项的成对 Choice（Choice）。 指令。 你是美国经济政策专家。你的任务是判断下列两个陈述中，哪一个对下述问题给出了更有知识含量的回答：请用几句话、且不查阅资料，解释在美国经济中利率（即借钱买房或买车的成本）如何上升或下降？若两者知识含量相当，请带轻微偏好选择其中一个。

Options.

选项。

• first: The first statement contains more knowledge. • second: The second statement contains more knowledge.

• first：第一个陈述知识含量更高。 • second：第二个陈述知识含量更高。

Pairwise Noul (Noul). Instructions. You are an expert in US economic policy. Does the first statement contain a more knowledgeable response than the second statement, in response to the following question: In a few sentences and without looking it up, can you explain how interest rates (i.e. the cost of borrowing money to buy a house or car) go up or down in the US economy?

成对 Noul（Noul）。 指令。 你是美国经济政策专家。针对下述问题，第一个陈述是否比第二个陈述给出了更有知识含量的回答：请用几句话、且不查阅资料，解释在美国经济中利率（即借钱买房或买车的成本）如何上升或下降？

Options.

选项。

• true: The first statement contains more knowledge. • false: The second statement contains more knowledge.

• true：第一个陈述知识含量更高。 • false：第二个陈述知识含量更高。

Score, ten levels (Score). Instructions. You are an expert in US economic policy. Your task is to rate the given statement based on how knowledgeable it is in response to the following question: In a few sentences and without looking it up, can you explain how interest rates (i.e. the cost of borrowing money to buy a house or car) go up or down in the US economy?

Score，十级（Score）。 指令。 你是美国经济政策专家。你的任务是根据对下述问题的回答有多少知识含量，给给定陈述打分：请用几句话、且不查阅资料，解释在美国经济中利率（即借钱买房或买车的成本）如何上升或下降？

Levels, low to high.

等级，由低到高。

Completely incorrect or irrelevant. Almost entirely wrong, with at most a stray relevant word. Mostly wrong, with a trace of relevant knowledge. Largely uninformed, but gestures at something relevant. Partly correct, with significant errors or omissions. Roughly correct in outline, but vague or incomplete. Correct on the main point, with minor gaps. Accurate and reasonably complete. Accurate, complete, and clearly explained. Highly knowledgeable and accurate.

完全错误或不相关。 几乎全错，至多有个别相关词。 大体错误，仅有一丝相关知识。 基本不知情，但隐约触及相关内容。 部分正确，但有明显错误或遗漏。 轮廓大致正确，但含糊或不完整。 主点正确，有小缺口。 准确且相当完整。 准确、完整且解释清楚。 知识含量高且准确。

Score, five levels (Score). Instructions. You are an expert in US economic policy. Your task is to rate the given statement based on how knowledgeable it is in response to the following question: In a few sentences and without looking it up, can you explain how interest rates (i.e. the cost of borrowing money to buy a house or car) go up or down in the US economy?

Score，五级（Score）。 指令。 你是美国经济政策专家。你的任务是根据对下述问题的回答有多少知识含量，给给定陈述打分：请用几句话、且不查阅资料，解释在美国经济中利率（即借钱买房或买车的成本）如何上升或下降？

Levels, low to high.

等级，由低到高。

Completely incorrect or irrelevant. Mostly wrong, with a trace of relevant knowledge. Partly correct, with significant errors or omissions. Correct on the main point, with minor gaps. Highly knowledgeable and accurate.

完全错误或不相关。 大体错误，仅有一丝相关知识。 部分正确，但有明显错误或遗漏。 主点正确，有小缺口。 知识含量高且准确。

Noul for a single answer (Noul). Instructions. You are an expert in US economic policy. Is the statement a knowledgeable and accurate response to the following question: In a few sentences and without looking it up, can you explain how interest rates (i.e. the cost of borrowing money to buy a house or car) go up or down in the US economy?

单条答案的 Noul（Noul）。 指令。 你是美国经济政策专家。该陈述是否是对下述问题有知识含量且准确的回答：请用几句话、且不查阅资料，解释在美国经济中利率（即借钱买房或买车的成本）如何上升或下降？

Options.

选项。

• true: Highly knowledgeable and accurate. • false: Completely incorrect or irrelevant.

• true：知识含量高且准确。 • false：完全错误或不相关。

Study 2: debt-ceiling uncertainty

研究 2：债务上限不确定性

Open-ended item. In one or two sentences, what do you think will happen if the government DOES NOT increase the debt ceiling?

开放题 。 用一两句话说明：若政府不提高债务上限，你认为会发生什么？

Pairwise Choice with a tie option (Choice). Instructions. Right before the 2023 Debt Ceiling Crisis in the United States we asked US citizens the following question: In one or two sentences, what do you think will happen if the government DOES NOT increase the debt ceiling? Your job is to compare each statement and determine which one exhibits the greatest uncertainty about what might happen

带平局选项的成对 Choice（Choice）。 指令。 在美国 2023 年债务上限危机前夕，我们向美国公民问了如下问题：用一两句话说明，若政府不提高债务上限，你认为会发生什么？你的工作是比较各陈述，判定哪一个对可能发生之事表现出更大的不确定性

Options.

选项。

• first: The first statement contains more uncertain. • second: The second statement is more uncertain. • equal: They have the same level of certianty or incomparable.

• first：第一个陈述不确定性更高。 • second：第二个陈述不确定性更高。 • equal：确定性水平相同或不可比。

Pairwise Noul (Noul). Instructions. Right before the 2023 Debt Ceiling Crisis in the United States we asked US citizens the following question: In one or two sentences, what do you think will happen if the government DOES NOT increase the debt ceiling? Does the first statement exhibit greater uncertainty about what might happen than the second statement?

成对 Noul（Noul）。 指令。 在美国 2023 年债务上限危机前夕，我们向美国公民问了如下问题：用一两句话说明，若政府不提高债务上限，你认为会发生什么？第一个陈述是否比第二个陈述对可能发生之事表现出更大的不确定性？

Options.

选项。

• true: The first statement is more uncertain. • false: The second statement is more uncertain.

• true：第一个陈述更不确定。 • false：第二个陈述更不确定。

Score, ten levels (Score). Instructions. Right before the 2023 Debt Ceiling Crisis in the United States we asked US citizens the following question: In one or two sentences, what do you think will happen if the government DOES NOT increase the debt ceiling? Your job is to read one of the answers and rank the statement based on the degree of uncertainty about what might happen

Score，十级（Score）。 指令。 在美国 2023 年债务上限危机前夕，我们向美国公民问了如下问题：用一两句话说明，若政府不提高债务上限，你认为会发生什么？你的工作是阅读其中一个答案，并按对可能发生之事的不确定程度给该陈述排序

Levels, low to high.

等级，由低到高。

Completely uncertain about what might happen. Almost entirely uncertain, naming nothing concrete. Mostly uncertain, with one vague possibility raised. Largely hedged, though some direction is suggested. Mixed, with uncertainty outweighing any stated expectation. Mixed, with a stated expectation outweighing the hedging. Fairly certain, with some hedging remaining. Certain about the main consequence, with minor hedging. Confident and specific about what will happen. Highly certain about what will happen.

对可能发生之事完全不确定。 几乎完全不确定，未点出任何具体内容。 大体不确定，只提出一个含糊可能。 大体在回避，虽暗示某种方向。 混合，不确定性压过任何已陈述预期。 混合，已陈述预期压过回避。 相当确定，仍有一些回避。 对主要后果确定，仅有轻微回避。 对将发生之事自信且具体。 对将发生之事高度确定。

Score, five levels (Score). Instructions. Right before the 2023 Debt Ceiling Crisis in the United States we asked US citizens the following question: In one or two sentences, what do you think will happen if the government DOES NOT increase the debt ceiling? Your job is to read one of the answers and rank the statement based on the degree of uncertainty about what might happen

Score，五级（Score）。 指令。 在美国 2023 年债务上限危机前夕，我们向美国公民问了如下问题：用一两句话说明，若政府不提高债务上限，你认为会发生什么？你的工作是阅读其中一个答案，并按对可能发生之事的不确定程度给该陈述排序

Levels, low to high.

等级，由低到高。

Completely uncertain about what might happen. Mostly uncertain, with one vague possibility raised. Mixed between uncertainty and a stated expectation. Fairly certain, with some hedging remaining. Highly certain about what will happen.

对可能发生之事完全不确定。 大体不确定，只提出一个含糊可能。 在不确定性与已陈述预期之间混合。 相当确定，仍有一些回避。 对将发生之事高度确定。

Noul for a single answer (Noul). Instructions. Right before the 2023 Debt Ceiling Crisis in the United States we asked US citizens the following question: In one or two sentences, what do you think will happen if the government DOES NOT increase the debt ceiling? Is the statement certain about what will happen?

单条答案的 Noul（Noul）。 指令。 在美国 2023 年债务上限危机前夕，我们向美国公民问了如下问题：用一两句话说明，若政府不提高债务上限，你认为会发生什么？该陈述是否对将发生之事确定？

Options.

选项。

• true: Highly certain about what will happen. • false: Completely uncertain about what might happen.

• true：对将发生之事高度确定。 • false：对可能发生之事完全不确定。

Gilardi et al.: tweet relevance and framing

Gilardi 等：推文相关性与框架

The state is the tweet text. The three questions were bundled in one request.

状态为推文文本。三个问题捆在一次请求里。

相关性（Choice）。 指令。 本工作将向你展示从社交媒体平台 Twitter 采集的推文样本。你的任务是判断这些推文是否与「内容审核」有关。「内容审核」指根据具体规则与指南，筛查并监控用户在社交媒体上发布的内容，以决定是否应予发布。每当有人在 Facebook 或 Twitter 一类平台发帖，该内容会经过审核流程（「内容审核」），以确保其不违法、不仇恨、不不当，并符合站点规则。若不符合，该内容可被删除、标记、贴标签或标为「有争议」。决定社交媒体上应允许多少内容并不总是容易。例如许多站点禁止儿童色情与恐怖主义内容，因其违法。但涉及疫苗安全性或政治等内容时则更不清晰。即便人们同意应屏蔽某些内容，也不总是就最佳做法、有效性以及应由谁来做（政府还是私营公司，人工审核员还是人工智能）达成一致。对样本中每条推文：仔细阅读推文文本，注意细节。将推文标为不相关（0）或相关（1）。当推文直接涉及内容审核时应标为相关。这包括讨论社交媒体平台内容审核规则与实践的推文，以及讨论政府规制在线内容审核的推文。也包括讨论轻度内容审核形式（如标记推文）的推文，以及间接涉及内容审核的推文。若不涉及内容审核，或其本身就是被审核内容的例子，则应标为不相关。例如，一条被 Twitter 标为「有争议」的 Donald Trump 推文、一条声称某事为假的推文，或含敏感内容的推文。

Options.

选项。

• relevant: The Tweet directly relates to content moderation, including platforms’ content moderation rules and practices, governments’ regulation of online content moderation, and mild forms such as flagging. • irrelevant: The Tweet does not refer to content moderation, or is itself an example of moderated content.

• relevant：推文直接涉及内容审核，包括平台的内容审核规则与实践、政府对在线内容审核的规制，以及标记等轻度形式。 • irrelevant：推文不涉及内容审核，或其本身是被审核内容的例子。

Problem frame (Choice). Instructions. In this job, you will be shown a sample of Tweets collected from the social media platform Twitter. “Content moderation” refers to the practice of screening and monitoring content posted by users on social media sites to determine if the content should be published or not, based on specific rules and guidelines. Your task is to determine whether the Tweet frames content moderation as a PROBLEM. Tweets should be coded as framing content moderation as a problem when they emphasise negative effects of content moderation, for instance on free speech, or when they describe content moderation as biased, censorious, ineffective or excessive.

问题框架（Choice）。 指令。 本工作将向你展示从社交媒体平台 Twitter 采集的推文样本。「内容审核」指根据具体规则与指南，筛查并监控用户在社交媒体上发布的内容，以决定是否应予发布。你的任务是判断该推文是否把内容审核框定为「问题」。当推文强调内容审核的负面效应（例如对言论自由），或把内容审核描述为有偏见、审查式、无效或过度时，应编码为把内容审核框定为问题。

Options.

选项。

• yes: The Tweet frames content moderation as a problem. • no: The Tweet does not frame content moderation as a problem.

• yes：该推文把内容审核框定为问题。 • no：该推文未把内容审核框定为问题。

Solution frame (Choice). Instructions. In this job, you will be shown a sample of Tweets collected from the social media platform Twitter. “Content moderation” refers to the practice of screening and monitoring content posted by users on social media sites to determine if the content should be published or not, based on specific rules and guidelines. Your task is to determine whether the Tweet frames content moderation as a SOLUTION. Tweets should be coded as framing content moderation as a solution when they emphasise positive effects of content moderation, for instance in reducing harmful, hateful or misleading content, or when they call for more or better moderation.

解决方案框架（Choice）。 指令。 本工作将向你展示从社交媒体平台 Twitter 采集的推文样本。「内容审核」指根据具体规则与指南，筛查并监控用户在社交媒体上发布的内容，以决定是否应予发布。你的任务是判断该推文是否把内容审核框定为「解决方案」。当推文强调内容审核的正面效应（例如减少有害、仇恨或误导内容），或呼吁更多或更好的审核时，应编码为把内容审核框定为解决方案。

Options.

选项。

• yes: The Tweet frames content moderation as a solution. • no: The Tweet does not frame content moderation as a solution.

• yes：该推文把内容审核框定为解决方案。 • no：该推文未把内容审核框定为解决方案。

Ornstein et al.: tweet sentiment

Ornstein 等：推文情感

The state is the tweet text. The few-shot form appends their six examples per case to the zero-shot instruction; both are shown for each case.

状态为推文文本。少样本形式在零样本指令后附加其每个案例的六个示例；两种形式均按案例展示。

Sentiment, zero-shot (Choice). Masterpiece Cakeshop

情感，零样本（Choice）。 Masterpiece Cakeshop

Instructions. Read these tweets posted the day after the US Supreme Court ruled in favor of a baker who refused to bake a wedding cake for a same-sex couple. For each tweet, decide whether its sentiment is Positive, Neutral, or Negative.

指令。 阅读这些在美国最高法院裁决支持一位拒绝为同性伴侣烤婚礼蛋糕的烘焙师次日发布的推文。对每条推文，判断其情感是正向、中性还是负向。

Options.

选项。

• Positive: The tweet expresses positive sentiment about the Court’s decision. • Neutral: The tweet expresses neither positive nor negative sentiment about the Court’s decision. • Negative: The tweet expresses negative sentiment about the Court’s decision.

• Positive：推文对法院裁决表达正向情感。 • Neutral：推文对法院裁决既不表达正向也不表达负向情感。 • Negative：推文对法院裁决表达负向情感。

Sentiment, few-shot (Choice). Masterpiece Cakeshop

情感，少样本（Choice）。 Masterpiece Cakeshop

Instructions. Read these tweets posted the day after the US Supreme Court ruled in favor of a baker who refused to bake a wedding cake for a same-sex couple. For each tweet, decide whether its sentiment is Positive, Neutral, or Negative.

指令。 阅读这些在美国最高法院裁决支持一位拒绝为同性伴侣烤婚礼蛋糕的烘焙师次日发布的推文。对每条推文，判断其情感是正向、中性还是负向。

Here are six labelled examples.

以下是六个已标注示例。

Tweet: Thank you Supreme Court I take pride in your decision!!!![emoji] #SCOTUS Sentiment: Positive

推文：Thank you Supreme Court I take pride in your decision!!!![emoji] #SCOTUS 情感：Positive

Tweet: Supreme Court rules in favor of Colorado baker! This day is getting better by the minute! Sentiment: Positive

推文：Supreme Court rules in favor of Colorado baker! This day is getting better by the minute! 情感：Positive

Tweet: Can’t escape the awful irony of someone allowed to use religion to discriminate against people in love. Not my Jesus. #opentoall #SCOTUS #Hypocrisy #MasterpieceCakeshop Sentiment: Negative

推文：Can’t escape the awful irony of someone allowed to use religion to discriminate against people in love. Not my Jesus. #opentoall #SCOTUS #Hypocrisy #MasterpieceCakeshop 情感：Negative

Tweet: I can’t believe this cake case went all the way to #SCOTUS . Can someone let me know what cake was ultimately served at the wedding? Are they married and living happily ever after? Sentiment: Neutral

推文：I can’t believe this cake case went all the way to #SCOTUS . Can someone let me know what cake was ultimately served at the wedding? Are they married and living happily ever after? 情感：Neutral

Tweet: Supreme Court rules in favor of baker who would not make wedding cake for gay couple Sentiment: Neutral

推文：Supreme Court rules in favor of baker who would not make wedding cake for gay couple 情感：Neutral

Tweet: #SCOTUS set a dangerous precedent today. Although the Court limited the scope to which a business owner could deny services to patrons, the legal argument has been legitimized that one’s subjective religious convictions trump (no pun intended) #humanrights. #LGBTQRights Sentiment: Negative

推文：#SCOTUS set a dangerous precedent today. Although the Court limited the scope to which a business owner could deny services to patrons, the legal argument has been legitimized that one’s subjective religious convictions trump (no pun intended) #humanrights. #LGBTQRights 情感：Negative

Options.

选项。

• Positive: The tweet expresses positive sentiment about the Court’s decision. • Neutral: The tweet expresses neither positive nor negative sentiment about the Court’s decision. • Negative: The tweet expresses negative sentiment about the Court’s decision.

• Positive：推文对法院裁决表达正向情感。 • Neutral：推文对法院裁决既不表达正向也不表达负向情感。 • Negative：推文对法院裁决表达负向情感。

Sentiment, zero-shot (Choice). Mazars

情感，零样本（Choice）。 Mazars

Instructions. Read these tweets posted the day after the US Supreme Court ruled that sitting presidents are not immune to state criminal subpoenas, and that President Trump was obliged to disclose his tax returns to the Manhattan District Attorney. For each tweet, decide whether its sentiment is Positive, Neutral, or Negative.

指令。 阅读这些在美国最高法院裁决现任总统不免于州刑事传票、且特朗普总统须向曼哈顿地区检察官披露其报税表次日发布的推文。对每条推文，判断其情感是正向、中性还是负向。

Options.

选项。

• Positive: The tweet expresses positive sentiment about the Court’s decision. • Neutral: The tweet expresses neither positive nor negative sentiment about the Court’s decision. • Negative: The tweet expresses negative sentiment about the Court’s decision.

• Positive：推文对法院裁决表达正向情感。 • Neutral：推文对法院裁决既不表达正向也不表达负向情感。 • Negative：推文对法院裁决表达负向情感。

Sentiment, few-shot (Choice). Mazars

情感，少样本（Choice）。 Mazars

Instructions. Read these tweets posted the day after the US Supreme Court ruled that sitting presidents are not immune to state criminal subpoenas, and that President Trump was obliged to disclose his tax returns to the Manhattan District Attorney. For each tweet, decide whether its sentiment is Positive, Neutral, or Negative.

指令。 阅读这些在美国最高法院裁决现任总统不免于州刑事传票、且特朗普总统须向曼哈顿地区检察官披露其报税表次日发布的推文。对每条推文，判断其情感是正向、中性还是负向。

Here are six labelled examples.

以下是六个已标注示例。

Tweet: Justice #ClarenceThomas is waste of space on the #scotus Sentiment: Negative

推文：Justice #ClarenceThomas is waste of space on the #scotus 情感：Negative

Tweet: BREAKING: Supreme Court Justice Ruth Bader Ginsburg has been hospitalized for a possible infection, per a SCOTUS spokesperson. @Scotus @ruthbadergins Sentiment: Neutral

推文：BREAKING: Supreme Court Justice Ruth Bader Ginsburg has been hospitalized for a possible infection, per a SCOTUS spokesperson. @Scotus @ruthbadergins 情感：Neutral

Tweet: The Supreme Court is going to disappoint us tomorrow. And trump will feel even more untouchable. He’ll brag about it at his Klan rallies. Sweaty orange spray tan pooling above his lip, smug faced as he gloats and brags. It makes me sick. Sentiment: Negative

推文：The Supreme Court is going to disappoint us tomorrow. And trump will feel even more untouchable. He’ll brag about it at his Klan rallies. Sweaty orange spray tan pooling above his lip, smug faced as he gloats and brags. It makes me sick. 情感：Negative

Tweet: SCOTUS just ruled Manhattan DA CAN get trumps financials and tax returns. This is a great day for the ruke of law and America. Sentiment: Positive

推文：SCOTUS just ruled Manhattan DA CAN get trumps financials and tax returns. This is a great day for the ruke of law and America. 情感：Positive

Tweet: Today the Supreme Court let @realDonaldTrump know that he is not above the law! Sentiment: Positive

推文：Today the Supreme Court let @realDonaldTrump know that he is not above the law! 情感：Positive

Tweet: Both SCOTUS rulings in Trump financial records sent back to lower courts. Practically speaking that means no turnover of records immediately in either case. #7News Sentiment: Neutral

推文：Both SCOTUS rulings in Trump financial records sent back to lower courts. Practically speaking that means no turnover of records immediately in either case. #7News 情感：Neutral

Options.

选项。

• Positive: The tweet expresses positive sentiment about the Court’s decision. • Neutral: The tweet expresses neither positive nor negative sentiment about the Court’s decision. • Negative: The tweet expresses negative sentiment about the Court’s decision.

• Positive：推文对法院裁决表达正向情感。 • Neutral：推文对法院裁决既不表达正向也不表达负向情感。 • Negative：推文对法院裁决表达负向情感。

Di Leo et al.: party positions

Di Leo 等：政党位置

The state lists the two parties by name in their original language with their countries, in the order of the original duplet. The year in the instruction is the reference year; the 1979 request is shown. Their system prompt is reproduced verbatim.

状态按原始二元组顺序，以原语言列出两个政党名称及其国家。指令中的年份为参照年；展示 1979 年请求。其系统提示按原文照录。

State, first request. Party 1: Freiheitliche Partei Österreichs Party 1 country: Austria Party 2: Sozialdemokratische Partei Österreichs Party 2 country: Austria

状态，第一条请求。 政党 1：Freiheitliche Partei Österreichs 政党 1 国家：Austria 政党 2：Sozialdemokratische Partei Österreichs 政党 2 国家：Austria

More right-wing (Choice). Instructions. In political matters, people talk of ’the Left’ and ’the Right’. You will be given the names of two parties in their original language, and the country of origin of each party. Based on their overall ideological stance, which one of these two parties was more right-wing in year 1979?

更右翼（Choice）。 指令。 在政治事务中，人们谈论「左」与「右」。你将得到两个政党以其原语言给出的名称，以及各党的来源国。基于其总体意识形态立场，这两个政党中哪一个在 1979 年更右翼？

Options.

选项。

• Party 1: Party 1 was more right-wing. • Party 2: Party 2 was more right-wing.

• Party 1：政党 1 更右翼。 • Party 2：政党 2 更右翼。

Brandt et al.: attack type

Brandt 等：袭击类型

The state is the incident text as their code builds it (summary, then motive). Only the first incident’s text is shown; every design used the same text.

状态为按其代码构建的事件文本（先摘要，后动机）。只展示第一起事件的文本；每种设计使用同一文本。

State, first incident. 01/02/2017: Assailants abducted eleven construction laborers building a school in Takhta Pul, Kandahar, Afghanistan. The hostages were released on January 20, 2017. No group claimed responsibility for the incident; however, sources attributed the kidnapping to the Taliban.

状态，第一起事件。 01/02/2017：袭击者在阿富汗坎大哈 Takhta Pul 绑架了十一名正在建学校的建筑工人。人质于 2017 年 1 月 20 日获释。无人声称对此事负责；但消息源将绑架归咎于塔利班。

Nine-way Choice, category names only (Choice). Instructions. Classify the following event into these categories: Assassination, Armed Assault, Bombing/Explosion, Hijacking, Hostage Taking (Barricade Incident), Hostage Taking (Kidnapping), Facility/Infrastructure Attack, Unarmed Assault, Unknown

九选一 Choice，仅类别名称（Choice）。 指令。 将下列事件归入这些类别：Assassination, Armed Assault, Bombing/Explosion, Hijacking, Hostage Taking (Barricade Incident), Hostage Taking (Kidnapping), Facility/Infrastructure Attack, Unarmed Assault, Unknown

Options.

选项。

• Assassination: The attack type is Assassination. • Armed Assault: The attack type is Armed Assault. • Bombing/Explosion: The attack type is Bombing/Explosion. • Hijacking: The attack type is Hijacking. • Hostage Taking (Barricade Incident): The attack type is Hostage Taking (Barricade Incident). • Hostage Taking (Kidnapping): The attack type is Hostage Taking (Kidnapping). • Facility/Infrastructure Attack: The attack type is Facility/Infrastructure Attack. • Unarmed Assault: The attack type is Unarmed Assault. • Unknown: The attack type is Unknown.

• Assassination：袭击类型为 Assassination。 • Armed Assault：袭击类型为 Armed Assault。 • Bombing/Explosion：袭击类型为 Bombing/Explosion。 • Hijacking：袭击类型为 Hijacking。 • Hostage Taking (Barricade Incident)：袭击类型为 Hostage Taking (Barricade Incident)。 • Hostage Taking (Kidnapping)：袭击类型为 Hostage Taking (Kidnapping)。 • Facility/Infrastructure Attack：袭击类型为 Facility/Infrastructure Attack。 • Unarmed Assault：袭击类型为 Unarmed Assault。 • Unknown：袭击类型为 Unknown。

Nine-way Choice with the codebook (Choice). The option descriptions are the GTD codebook’s definitions, verbatim.

带编码手册的九选一 Choice（Choice）。 选项描述为 GTD 编码手册定义，原文照录。

Instructions. Classify the following event into these categories: Assassination, Armed Assault, Bombing/Explosion, Hijacking, Hostage Taking (Barricade Incident), Hostage Taking (Kidnapping), Facility/Infrastructure Attack, Unarmed Assault, Unknown

指令。 将下列事件归入这些类别：Assassination, Armed Assault, Bombing/Explosion, Hijacking, Hostage Taking (Barricade Incident), Hostage Taking (Kidnapping), Facility/Infrastructure Attack, Unarmed Assault, Unknown

Use the following definitions from the Global Terrorism Database Codebook.

使用全球恐怖主义数据库编码手册中的下列定义。

This field captures the general method of attack and often reflects the broad class of tactics used. It consists of nine categories, which are defined below. Up to three attack types can be recorded for each incident. Typically, only one attack type is recorded for each incident unless the attack is comprised of a sequence of events.

该字段捕捉一般袭击方式，并常反映所用战术的宽类别。它由下文定义的九个类别组成。每起事件最多可记录三种袭击类型。通常每起事件只记录一种袭击类型，除非攻击由一系列事件构成。

When multiple attack types may apply, the most appropriate value is determined based on the hierarchy below. For example, if an assassination is carried out through the use of an explosive, the Attack Type is coded as Assassination, not Bombing/Explosion. If an attack involves a sequence of events, then the first, the second, and the third attack types are coded in the order of the hierarchy below rather than the order in which they occurred.

当多种袭击类型可能适用时，按下列层级确定最恰当取值。例如，若刺杀通过使用爆炸物实施，袭击类型编码为 Assassination，而非 Bombing/Explosion。若攻击涉及一系列事件，则第一、第二、第三袭击类型按下列层级顺序编码，而非按其发生顺序。

Attack Type Hierarchy: Assassination Hijacking Kidnapping Barricade Incident Bombing/Explosion Armed Assault Unarmed Assault Facility/Infrastructure Attack Unknown

袭击类型层级：Assassination Hijacking Kidnapping Barricade Incident Bombing/Explosion Armed Assault Unarmed Assault Facility/Infrastructure Attack Unknown

Options.

选项。

One question per type (Noul). Nine requests per incident, one per type; the Assassination request is shown and the others substitute the type’s name.

每类型一问（Noul）。 每起事件九次请求，每类型一次；展示 Assassination 请求，其余以类型名称替换。

Instructions. Classify the following event. Does it belong to the category "Assassination"?

指令。 对下列事件分类。它是否属于类别「Assassination」？

Options.

选项。

• true: The event belongs to the category "Assassination". • false: The event does not belong to the category "Assassination".

• true：该事件属于类别「Assassination」。 • false：该事件不属于类别「Assassination」。

Qwen3.5-9B, label names only (their prompt, verbatim). Sent to qwen/qwen3.5-9b as a single user message, with JSON output, seed 66502, at most 300 tokens, reasoning effort none, and the provider pinned to deepinfra at bf16 with no fallback.

Qwen3.5-9B，仅标签名称（其提示，原文照录）。 作为单条用户消息发送至 qwen/qwen3.5-9b，要求 JSON 输出，种子 66502，最多 300 token，推理力度 none，提供商固定为 deepinfra 的 bf16 且无回退。

Message. Classify the following event into up to three of these categories, providing probabilities for each: Assassination, Armed Assault, Bombing/Explosion, Hijacking, Hostage Taking (Barricade Incident), Hostage Taking (Kidnapping), Facility/Infrastructure Attack, Unarmed Assault, Unknown

消息。 将下列事件归入至多三个类别，并为每个类别给出概率：Assassination, Armed Assault, Bombing/Explosion, Hijacking, Hostage Taking (Barricade Incident), Hostage Taking (Kidnapping), Facility/Infrastructure Attack, Unarmed Assault, Unknown

For the event, return only a single JSON object with category names as keys and probabilities as values. Example format: {"Armed Assault": 0.7, "Bombing/Explosion": 0.2, "Unknown": 0.1}

对该事件，只返回一个以类别名为键、概率为值的 JSON 对象。示例格式：{"Armed Assault": 0.7, "Bombing/Explosion": 0.2, "Unknown": 0.1}

Event: ""

事件：""

Qwen3.5-9B, codebook. Sent to qwen/qwen3.5-9b as a single user message, with JSON output, seed 66502, at most 300 tokens, reasoning effort none, and the provider pinned to deepinfra at bf16 with no fallback.

Qwen3.5-9B，编码手册。 作为单条用户消息发送至 qwen/qwen3.5-9b，要求 JSON 输出，种子 66502，最多 300 token，推理力度 none，提供商固定为 deepinfra 的 bf16 且无回退。

Message. Classify the following event into up to three of these categories, providing probabilities for each: Assassination, Armed Assault, Bombing/Explosion, Hijacking, Hostage Taking (Barricade Incident), Hostage Taking (Kidnapping), Facility/Infrastructure Attack, Unarmed Assault, Unknown

消息。 将下列事件归入至多三个类别，并为每个类别给出概率：Assassination, Armed Assault, Bombing/Explosion, Hijacking, Hostage Taking (Barricade Incident), Hostage Taking (Kidnapping), Facility/Infrastructure Attack, Unarmed Assault, Unknown

Use the following definitions from the Global Terrorism Database Codebook.

使用全球恐怖主义数据库编码手册中的下列定义。

This field captures the general method of attack and often reflects the broad class of tactics used. It consists of nine categories, which are defined below. Up to three attack types can be recorded for each incident. Typically, only one attack type is recorded for each incident unless the attack is comprised of a sequence of events.

该字段捕捉一般袭击方式，并常反映所用战术的宽类别。它由下文定义的九个类别组成。每起事件最多可记录三种袭击类型。通常每起事件只记录一种袭击类型，除非攻击由一系列事件构成。

When multiple attack types may apply, the most appropriate value is determined based on the hierarchy below. For example, if an assassination is carried out through the use of an explosive, the Attack Type is coded as Assassination, not Bombing/Explosion. If an attack involves a sequence of events, then the first, the second, and the third attack types are coded in the order of the hierarchy below rather than the order in which they occurred.

当多种袭击类型可能适用时，按下列层级确定最恰当取值。例如，若刺杀通过使用爆炸物实施，袭击类型编码为 Assassination，而非 Bombing/Explosion。若攻击涉及一系列事件，则第一、第二、第三袭击类型按下列层级顺序编码，而非按其发生顺序。

Attack Type Hierarchy: Assassination Hijacking Kidnapping Barricade Incident Bombing/Explosion Armed Assault Unarmed Assault Facility/Infrastructure Attack Unknown

袭击类型层级：Assassination Hijacking Kidnapping Barricade Incident Bombing/Explosion Armed Assault Unarmed Assault Facility/Infrastructure Attack Unknown

For the event, return only a single JSON object with category names as keys and probabilities as values. Example format: {"Armed Assault": 0.7, "Bombing/Explosion": 0.2, "Unknown": 0.1}

对该事件，只返回一个以类别名为键、概率为值的 JSON 对象。示例格式：{"Armed Assault": 0.7, "Bombing/Explosion": 0.2, "Unknown": 0.1}

Event: ""

事件：""

Le Mens and Gallego: manifesto and speech sentences

Le Mens 与 Gallego：党纲与演讲句子

The state is the sentence. Each request bundles a Score and a Noul per dimension: two of each for a manifesto sentence, one of each for a speech sentence. The Score levels are listed in order and rescaled to 0-100.

状态为句子。每次请求按维度捆绑一个 Score 与一个 Noul：党纲句子各两个，演讲句子各一个。Score 等级按顺序列出并缩放到 0-100。

Manifestos, economic. Score. “You will be provided with a sentence from a party manifesto. Where does this sentence stand on the ‘left’ to ‘right’ wing scale, in terms of economic policy?” Levels: Extremely left, Very left, Left, Somewhat left, Centre, Somewhat right, Right, Very right, Extremely right. Noul. The same opening sentence of instructions, then “Does this sentence refer to economic policy?”; true: “The sentence refers to economic policy.”; false: “The sentence does not refer to economic policy.”

党纲，经济。 Score。 「你将得到一句来自政党党纲的句子。就经济政策而言，该句在『左』到『右』翼量表上处于何处？」等级：Extremely left, Very left, Left, Somewhat left, Centre, Somewhat right, Right, Very right, Extremely right。 Noul。 相同的开场指令句，然后「该句是否涉及经济政策？」；true：「该句涉及经济政策。」；false：「该句不涉及经济政策。」

Manifestos, social. As above with “Where does this sentence stand on the ‘liberal’ to ‘conservative’ scale, in terms of social policy?” and levels from Extremely liberal to Extremely conservative, with “Neither liberal nor conservative” at the centre; the Noul asks about social policy.

党纲，社会。 同上，改为「就社会政策而言，该句在『自由』到『保守』量表上处于何处？」，等级从 Extremely liberal 到 Extremely conservative，中心为「Neither liberal nor conservative」；Noul 询问是否涉及社会政策。

Speeches. The instructions open with their background on the coal subsidy debate and the coding job verbatim (their config_EU_debate), then ask “Where does this sentence stand on the ‘anti-subsidy’ to ‘pro-subsidy’ scale?”, with levels from Extremely anti-subsidy to Extremely pro-subsidy and “Neutral” at the centre. The Noul asks “Does this sentence refer to subsidy policy?”. The full text of every question is in data/batch_inputs/lemens_prompts_used.json.

演讲。 指令以其关于煤炭补贴辩论与编码工作的背景原文开场（其 config_EU_debate），然后问「该句在『反补贴』到『亲补贴』量表上处于何处？」，等级从 Extremely anti-subsidy 到 Extremely pro-subsidy，中心为「Neutral」。Noul 问「该句是否涉及补贴政策？」。每个问题的全文见 data/batch_inputs/lemens_prompts_used.json。

Weidmann et al.: democracy indicators from country names

Weidmann 等：从国名推断民主指标

The published prompt (their online Appendix B) with V-Dem v14 codebook wording, one request per country and indicator. The first request sent, for v2clacfree. Instructions:

已发表提示（其在线附录 B）采用 V-Dem v14 编码手册措辞，每个国家与指标一次请求。发送的第一条请求针对 v2clacfree。指令：

You are an assistant who evaluates political systems in different countries and years. You will be asked to produce numeric scores derived from your knowledge of this country. The question is as follows: Is there academic freedom and freedom of cultural expression related to political issues?

你是一名评估不同国家与年份政治制度的助手。你将被要求根据对该国的知识给出数值分数。问题如下：是否存在与政治议题相关的学术自由与文化表达自由？

State: What is the score for United States of America in 2023? Score levels, 0 upward:

状态： What is the score for United States of America in 2023? Score 等级，自 0 起：

Not respected by public authorities. Censorship and intimidation are frequent. Academic activities and cultural expressions are severely restricted or controlled by the government. Weakly respected by public authorities. Academic freedom and freedom of cultural expression are practiced occasionally, but direct criticism of the government is mostly met with repression. Somewhat respected by public authorities. Academic freedom and freedom of cultural expression are practiced routinely, but strong criticism of the government is sometimes met with repression. Mostly respected by public authorities. There are few limitations on academic freedom and freedom of cultural expression, and resulting sanctions tend to be infrequent and soft. Fully respected by public authorities. There are no restrictions on academic freedom or cultural expression.

公共当局不予尊重。审查与恐吓频繁。学术活动与文化表达受到政府严重限制或控制。 公共当局勉强尊重。学术自由与文化表达自由偶有实践，但对政府的直接批评大多遭遇压制。 公共当局有所尊重。学术自由与文化表达自由例行实践，但对政府的强烈批评有时遭遇压制。 公共当局大体尊重。对学术自由与文化表达自由限制很少，相应制裁往往少见且温和。 公共当局完全尊重。对学术自由或文化表达没有任何限制。

The published closing line, “Please only return the score as a number, without any explanation.”, is dropped for JEV and kept for GPT-6 Luna, whose single user message was:

已发表的收尾句「Please only return the score as a number, without any explanation.」对 JEV 去掉，对 GPT-6 Luna 保留；后者的单条用户消息为：

You are an assistant who evaluates political systems in different countries and years. You will be asked to produce numeric scores derived from your knowledge of this country. The question is as follows: Is there academic freedom and freedom of cultural expression related to political issues? The numeric scores are as follows: 0: Not respected by public authorities. Censorship and intimidation are frequent. Academic activities and cultural expressions are severely restricted or controlled by the government. 1: Weakly respected by public authorities. Academic freedom and freedom of cultural expression are practiced occasionally, but direct criticism of the government is mostly met with repression. 2: Somewhat respected by public authorities. Academic freedom and freedom of cultural expression are practiced routinely, but strong criticism of the government is sometimes met with repression. 3: Mostly respected by public authorities. There are few limitations on academic freedom and freedom of cultural expression, and resulting sanctions tend to be infrequent and soft. 4: Fully respected by public authorities. There are no restrictions on academic freedom or cultural expression. What is the score for United States of America in 2023? Please only return the score as a number, without any explanation.

你是一名评估不同国家与年份政治制度的助手。你将被要求根据对该国的知识给出数值分数。问题如下：是否存在与政治议题相关的学术自由与文化表达自由？数值分数如下：0：公共当局不予尊重。审查与恐吓频繁。学术活动与文化表达受到政府严重限制或控制。1：公共当局勉强尊重。学术自由与文化表达自由偶有实践，但对政府的直接批评大多遭遇压制。2：公共当局有所尊重。学术自由与文化表达自由例行实践，但对政府的强烈批评有时遭遇压制。3：公共当局大体尊重。对学术自由与文化表达自由限制很少，相应制裁往往少见且温和。4：公共当局完全尊重。对学术自由或文化表达没有任何限制。What is the score for United States of America in 2023? Please only return the score as a number, without any explanation.

Appendix B: position bias, transitivity and test-retest

附录 B：位置偏差、传递性与重测

Only some of these diagnostics need the probability. Position bias, the choice-based transitivity measures and test-retest agreement on the winner can be measured from choices alone, and the published article reports the sparse-triplet measure for its six models (below). Calibration, the stochastic forms of transitivity and any conditioning on reported confidence require the probability, which a generative model that exposes token log-probabilities can also supply, so a typed output does not make any of these diagnostics possible for the first time. What it changes is that the probability arrives with every judgment, defined over the named options and at no extra cost, and that no answer falls outside the set. Nothing has to be extracted from token probabilities. Whether the probability carries information is then a question about this model, and the flip rates in Table 3 bear on it. We have no token probabilities for the published generative scorers. We do for GPT-6 Luna. Averaged over the two orders, its calibration error against the raters is level with JEV’s; in single calls it is higher (0.145 against 0.071). Its interface omits low-probability alternatives, so many of its probabilities are exactly 0 or 1. Qwen3.8-27B, whose token probabilities we also have, is better calibrated than JEV on these pairs (0.011 against 0.071 in single calls).

这些诊断中只有一部分需要概率。位置偏差、基于选择的传递性度量以及对胜者的重测一致，仅凭选择即可测量；已发表文章报告了其六个模型的稀疏三元组度量（见下）。校准、传递性的随机形式以及任何以报告置信度为条件的分析都需要概率；暴露 token 对数概率的生成模型也能提供，因此类型化输出并非第一次使这些诊断成为可能。它改变的是：概率随每次判断而来，定义在具名选项上且无额外成本，且没有答案落在集合之外。无需从 token 概率中抽取。概率是否携带信息，随之成为关于该模型的问题，表 3 的翻转率与此相关。我们对已发表的生成式评分者没有 token 概率。对 GPT-6 Luna 有。对两种顺序取平均后，其相对评分者的校准误差与 JEV 持平；在单次调用中则更高（0.145 对 0.071）。其接口省略低概率备选项，因此许多概率恰好为 0 或 1。我们同样拥有其 token 概率的 Qwen3.8-27B，在这些对上校准优于 JEV（单次调用 0.011 对 0.071）。

Test-retest

重测

r=0.998{}

r=0.998{}

Position bias

位置偏差

-

-

Transitivity

传递性

On the sparse-triplet measure used in the published article, the model reaches 98.18%, against a published range of 95.95 to 99.06%. A complete round robin over 80 responses, in which all 82,160 triads are observable, yields 639 circular triads and a Kendall coefficient of consistence of 0.9700. Over 11,841 ordered triples, weak stochastic transitivity holds in 98.37% of cases and the strong form in 80.21%, indicating that the model is more reliable about the direction of a difference than about its magnitude.

在已发表文章使用的稀疏三元组度量上，模型达到 98.18%，而已发表区间为 95.95 到 99.06%。对 80 条回答做完整循环赛——全部 82,160 个三元组可观测——得到 639 个循环三元组，Kendall 一致性系数为 0.9700。在 11,841 个有序三元组上，弱随机传递性成立于 98.37% 的情形，强形式成立于 80.21%，表明模型对差异方向的可靠性高于对其量级的可靠性。

Flip rates

翻转率

-

Probability of choice	Pairs	Flip rate	Position bias
<.50	5,537	0.3655	$-$ 0.1104
.50–.70	2,984	0.1146	$-$ 0.0721
.70–.85	2,881	0.0139	$-$ 0.0263
.85–.95	3,862	0.0003	$-$ 0.0079
>.95	12,103	0.0000	$-$ 0.0005

-

所选选项的概率	对数	翻转率	位置偏差
<.50	5,537	0.3655	$-$ 0.1104
.50–.70	2,984	0.1146	$-$ 0.0721
.70–.85	2,881	0.0139	$-$ 0.0263
.85–.95	3,862	0.0003	$-$ 0.0079
>.95	12,103	0.0000	$-$ 0.0005

Flip rates decline monotonically across the five probability bins of Table 3, which cover 27,367 decisive pairs.

翻转率在表 3 的五个概率分箱上单调下降；这些分箱覆盖 27,367 对有决断的对。

Appendix C: the annotation benchmarks in full

附录 C：标注基准全文

The Gilardi et al. relevance benchmark

Gilardi 等的相关性基准

The replication archive of Gilardi et al. (2023) supplies the tweets, the instruction they gave ChatGPT, gold labels from two trained research assistants, and published accuracy figures for ChatGPT and for Amazon Mechanical Turk.

Gilardi 等（2023）的复现档案提供推文、其给 ChatGPT 的指令、两位受训研究助理的金标准标签，以及 ChatGPT 与 Amazon Mechanical Turk 的已发表准确率数字。

We use their relevance task, in which a tweet is coded as relating to content moderation or not, with their instruction verbatim apart from the closing clause specifying reply format. A Choice question also needs a description of each answer option, and the two we wrote condense the instruction’s own definitions of relevant and irrelevant tweets, which the instruction states in full. Their gold standard is the label the two assistants agree on, with disagreements dropped; applying their construction to the archive reproduces their reported intercoder agreement of 93.9% exactly, which confirms our gold labels match theirs. This yields 2,403 labelled tweets in the 2021 sample.

我们使用其相关性任务：将推文编码为是否涉及内容审核；指令除指定回复格式的收尾句外原文照用。Choice 问题还需要各答案选项的描述，我们写的两条浓缩了指令本身对相关与不相关推文的定义——指令中已完整陈述。其金标准是两位助理达成一致的标签，分歧则丢弃；对其档案应用其构造恰好复现其报告的 93.9% 编码者间一致，确认我们的金标准标签与其一致。由此得到 2021 样本中 2,403 条已标注推文。

They report two metrics: accuracy against that gold standard, and intercoder agreement, meaning agreement between two runs at the same temperature, which they report at two temperatures (four runs in all). We therefore ran JEV four times, as they did, 12,156 requests in total for $0.61.

他们报告两项指标：相对该金标准的准确率，以及编码者间一致——指相同温度下两次运行之间的一致，并在两个温度上报告（共四次运行）。因此我们像他们一样跑了四次 JEV，相关性任务合计 12,156 次请求，费用 $0.61。

Annotator Accuracy Intercoder agreement GPT-6 Luna, single run 91.1 — JEV 1.13, single run 89.0 — Qwen3.8-27B, single run 88.2 — JEV 1.13, four runs, unanimity required 88.2 98.7 ChatGPT (temp 1) 72.8 92.0 Mechanical Turk 71.2 79.2 ChatGPT (temp 0.2) 70.2 95.0 Trained annotators — 93.9 Table 4: Tweet relevance classification, 2021 sample (2,403 tweets with agreed gold labels). Rows for ChatGPT, Mechanical Turk and trained annotators are published figures from Gilardi et al. (2023); the others are ours. JEV’s intercoder agreement is agreement among its four runs. GPT-6 Luna and the open-weight Qwen3.8-27B were each run once with the published instruction (Appendix H).

Annotator	Accuracy	Intercoder agreement
GPT-6 Luna, single run	91.1	—
JEV 1.13, single run	89.0	—
Qwen3.8-27B, single run	88.2	—
JEV 1.13, four runs, unanimity required	88.2	98.7
ChatGPT (temp 1)	72.8	92.0
Mechanical Turk	71.2	79.2
ChatGPT (temp 0.2)	70.2	95.0
Trained annotators	—	93.9

标注者 准确率 编码者间一致 GPT-6 Luna，单次运行 91.1 — JEV 1.13，单次运行 89.0 — Qwen3.8-27B，单次运行 88.2 — JEV 1.13，四次运行，要求全体一致 88.2 98.7 ChatGPT (temp 1) 72.8 92.0 Mechanical Turk 71.2 79.2 ChatGPT (temp 0.2) 70.2 95.0 受训标注员 — 93.9 表 4：推文相关性分类，2021 样本（2,403 条达成一致金标准标签的推文）。ChatGPT、Mechanical Turk 与受训标注员行为 Gilardi 等（2023）已发表数字；其余为我们所得。JEV 的编码者间一致指其四次运行之间的一致。GPT-6 Luna 与开源权重 Qwen3.8-27B 各按已发表指令运行一次（附录 H）。

标注者	准确率	编码者间一致
GPT-6 Luna，单次运行	91.1	—
JEV 1.13，单次运行	89.0	—
Qwen3.8-27B，单次运行	88.2	—
JEV 1.13，四次运行，要求全体一致	88.2	98.7
ChatGPT (temp 1)	72.8	92.0
Mechanical Turk	71.2	79.2
ChatGPT (temp 0.2)	70.2	95.0
受训标注员	—	93.9

Accuracy is 89.0% on a single run (Table 4). Their scoring of crowd workers marks a tweet wrong whenever the two crowd workers who coded it disagree, and applying that rule to our four runs costs only 0.8 points, taking accuracy to 88.2%, because the four runs are unanimous on 97.9% of tweets. Self-consistency is 98.7%, above the 93.9% agreement of the two trained annotators, which is the same pattern Gilardi et al. (2023) report for ChatGPT at temperature 0.2. On the smaller 2023 sample (411 tweets) accuracy is 76.2% on a single run and 74.2% under unanimity, above every published accuracy on that sample.

单次运行准确率为 89.0%（表 4）。其对众包工人的计分规则是：只要编码该推文的两名众包工人不同意就标错；将该规则用于我们的四次运行仅损失 0.8 个百分点，准确率到 88.2%，因为四次运行在 97.9% 的推文上全体一致。自一致为 98.7%，高于两位受训标注员的 93.9% 一致，这与 Gilardi 等（2023）对温度 0.2 的 ChatGPT 报告的模式相同。在较小的 2023 样本（411 条推文）上，单次运行准确率 76.2%，要求全体一致时 74.2%，高于该样本上所有已发表准确率。

Two qualifications matter more than the margin. The task is easy: a binary judgment on which two trained humans agree 93.9% of the time. A result of this kind shows only that the model clears a floor of basic competence. And the published comparator is a 2023 model: part of the 15.5-point gap between JEV under unanimity and ChatGPT at temperature 1 reflects three years of model development rather than anything about JEV’s constrained output. What the comparison does establish is that this model, which returns only a constrained decision, handles a standard annotation task at least as well as the 2023 ChatGPT did, at a fraction of the cost.

两项限定比差距本身更重要。任务容易：两位受训人类在其上 93.9% 一致的二元判断。这类结果只表明模型跨过了基本能力的下限。而已发表对照是 2023 年的模型：JEV 在全体一致要求下与温度 1 的 ChatGPT 之间 15.5 个百分点的差距，部分反映三年的模型发展，而非 JEV 受约束输出本身。该比较确实确立的是：这个只返回受约束决策的模型，在标准标注任务上至少不亚于 2023 年的 ChatGPT，且成本只是其一小部分。

Accuracy alone flatters a classifier when the classes are unbalanced. On relevance, where 46.2% of the agreed 2021 labels are positive and always answering the larger class would score 53.8%, the model’s F1 on the relevant class is 0.874 (precision 0.931, recall 0.824).

当类别不平衡时，准确率单独会抬高分类器。在相关性上，2021 年达成一致标签中 46.2% 为正，总答较大类会得 53.8%；模型在相关类上的 F1 为 0.874（精确率 0.931，召回 0.824）。

The archive does not contain the instructions for the two framing tasks, so we reconstructed them from the published task definitions. Accuracy there is 92.4% for problem framing (2,361 tweets) and 84.7% for solution framing (2,398), against 83.3% and 77.5% for always answering no, since only 16.7% and 22.5% of those tweets frame content moderation as a problem or a solution. F1 on the positive class is 0.806 for problem framing and 0.546 for solution framing, where the model finds only 0.408 of the tweets the coders marked. On the 2023 sample, where 3.0% of 166 tweets frame content moderation as a problem, the model’s problem-framing accuracy is 61.4%, below the 97.0% of always answering no (precision 0.072, recall 1.000). Because the instructions are ours rather than theirs, none of these figures is comparable to their published ChatGPT numbers. They show a model that finds most problem framing and misses most solution framing in the 2021 sample, and that marks far more problem framing than the coders did in the 2023 sample.

档案不含两项框架任务的指令，因此我们按已发表任务定义重建。问题框架准确率 92.4%（2,361 条推文），解决方案框架 84.7%（2,398），而总答「否」分别为 83.3% 与 77.5%，因为仅 16.7% 与 22.5% 的推文把内容审核框定为问题或解决方案。正类 F1 在问题框架为 0.806，在解决方案框架为 0.546——模型只找出编码者所标推文的 0.408。在 2023 样本上，166 条推文中 3.0% 把内容审核框定为问题，模型的问题框架准确率 61.4%，低于总答「否」的 97.0%（精确率 0.072，召回 1.000）。因指令是我们的而非他们的，这些数字皆不可与其已发表 ChatGPT 数字比较。它们显示：在 2021 样本上模型找到多数问题框架、错过多数解决方案框架；在 2023 样本上标出的问题框架远多于编码者。

-3.4

-3.4

The Ornstein et al. sentiment benchmark

Ornstein 等的情感基准

For the replication of Ornstein et al. (2025), the Masterpiece Cakeshop preamble is verbatim from their Table 1 and the six few-shot examples per case are theirs, taken from the promptr package. The classification form of their Mazars preamble sits in their Supplementary Materials, which we could not retrieve; we took its first sentence verbatim from the version in their Appendix B of the same instruction and appended the classification sentence from their Table 1. Correlations use the 907 of 945 tweets on which every measure is available; pairwise deletion leaves the ordering unchanged. The intervals below come from 2,000 resamples of those tweets, with each gap taken as a paired difference within a resample.

复现 Ornstein 等（2025）时， Masterpiece Cakeshop 前导语原文取自其表 1，每个案例的六个少样本示例亦为其所有，取自 promptr 包。其 Mazars 前导语的分类形式在我们未能取得的补充材料中；我们从其附录 B 同一指令版本原文取首句，并附上其表 1 的分类句。相关使用每项度量皆可得的 945 条中的 907 条；成对删除不改变排序。下列区间来自对这些推文的 2,000 次重抽样，每个差距取为一次重抽样内的成对差。

-

-

With the few-shot examples, and with the GPT-4 rule applied to its twenty most probable first tokens, Qwen3.8-27B scores 0.841 [0.811, 0.868], which is 0.052 [0.025, 0.078] higher than GPT-4 and 0.182 [0.141, 0.226] higher than the RoBERTa model. Without examples, Qwen3.8-27B produced a prose answer (rather than a label) for 941 out of 945 tweets. Its zero-shot score of 0.668 is therefore based on the first token of a prose answer, and is not comparable with the other scores (Table 5).

有少样本示例、并对二十个最可能首 token 应用 GPT-4 规则时，Qwen3.8-27B 得 0.841 [0.811，0.868]，高于 GPT-4 0.052 [0.025，0.078]，高于 RoBERTa 模型 0.182 [0.141，0.226]。没有示例时，Qwen3.8-27B 对 945 条中 941 条给出散文回答（而非标签）。因此其零样本分数 0.668 基于散文回答的首 token，不可与其他分数比较（表 5）。

\rho

Measure	Kind	Scoring rule	$\rho$ with experts
Qwen3.8-27B (few-shot)	generative	positive $-$ negative	0.841
GPT-6 Luna (few-shot)	generative	positive $-$ negative	0.816
GPT-3 (few-shot)	generative	principal component	0.807
GPT-6 Luna (zero-shot)	generative	positive $-$ negative	0.791
GPT-4 (few-shot)	generative	positive $-$ negative	0.789
JEV (few-shot)	constrained output	positive $-$ negative	0.740
JEV (few-shot)	constrained output	principal component	0.714
JEV (zero-shot)	constrained output	positive $-$ negative	0.701
Qwen3.8-27B (zero-shot)a	generative	positive $-$ negative	0.668
TweetNLP RoBERTa	fine-tuned encoder	principal component	0.659
JEV (zero-shot)	constrained output	principal component	0.604
Naive Bayes	supervised	P(positive)	0.291
a 941 of 945 replies were prose, not a label.

\rho

度量	类型	计分规则	与专家的 $\rho$
Qwen3.8-27B（少样本）	生成式	正向 $-$ 负向	0.841
GPT-6 Luna（少样本）	生成式	正向 $-$ 负向	0.816
GPT-3（少样本）	生成式	主成分	0.807
GPT-6 Luna（零样本）	生成式	正向 $-$ 负向	0.791
GPT-4（少样本）	生成式	正向 $-$ 负向	0.789
JEV（少样本）	受约束输出	正向 $-$ 负向	0.740
JEV（少样本）	受约束输出	主成分	0.714
JEV（零样本）	受约束输出	正向 $-$ 负向	0.701
Qwen3.8-27B（零样本）a	生成式	正向 $-$ 负向	0.668
TweetNLP RoBERTa	微调编码器	主成分	0.659
JEV（零样本）	受约束输出	主成分	0.604
朴素贝叶斯	有监督	P(positive)	0.291
a 945 条回复中 941 条为散文而非标签。

Appendix D: A/B correctness

附录 D：A/B 正确性

Correlation between two latent scales can hide disagreement about the individual comparisons that produced them. Table 6 therefore scores each scorer on the decision it actually makes. Column 3 is accuracy on the expert-judged pairs, the share of the 460 judgments matched; the main text reports F1 on the same pairs. Column 4 is agreement with the majority of the other scorers (the six published models and JEV) across all 28,040 published pairs, which is a far larger sample but measures agreement with the other models rather than correctness.

两个潜在量表之间的相关可能掩盖产生它们的个体比较上的分歧。因此表 6 按评分者实际做出的决策计分。第 3 列是专家判断对上的准确率，即 460 次判断中匹配的份额；正文在同一批对上报告 F1。第 4 列是在全部 28,040 对已发表对上与其他评分者（六个已发表模型与 JEV）多数的一致，样本大得多，但衡量的是与其他模型的一致而非正确性。

Nine further models from an unpublished extension by the same authors drew their own pairs, sharing only 782 of the 28,040, so they are not in the table; their scores are in the replication archive. GPT-6 Luna and Qwen3.8-27B did not judge the full published grid either, so they cannot enter column 4; their expert-pair accuracy (0.780 and 0.787) and F1 are in Table 8.

同一作者未发表扩展中的另九个模型抽取了自己的对，仅与 28,040 对中的 782 对重叠，故不在表中；其分数见复现档案。GPT-6 Luna 与 Qwen3.8-27B 也未判断完整已发表网格，故不能进入第 4 列；其专家对准确率（0.780 与 0.787）与 F1 见表 8。

Scorer Mode Expert pairs Other models’ majority Llama 3.1 405B pairwise 0.815 0.949 Gemma 3 27B pairwise 0.813 0.928 JEV pairwise 0.796 0.910 GPT-4o pairwise 0.817 0.906 GPT-4o mini pairwise 0.824 0.899 Gemma 3 4B pairwise 0.739 0.875 Llama 3.1 8B pairwise 0.737 0.870 Table 6: A/B correctness rather than correlation. Column 3 is accuracy on the expert-judged pairs; column 4 is agreement with the majority of the other six scorers across all 28,040 published pairs. JEV’s accuracy in column 3 is averaged over both presentation orders (0.800 in the published order alone; Table 8). Nine further models from an unpublished extension by the same authors drew their own pairs and share only 782 of those pairs, so they are not shown.

Scorer	Mode	Expert pairs	Other models’ majority
Llama 3.1 405B	pairwise	0.815	0.949
Gemma 3 27B	pairwise	0.813	0.928
JEV	pairwise	0.796	0.910
GPT-4o	pairwise	0.817	0.906
GPT-4o mini	pairwise	0.824	0.899
Gemma 3 4B	pairwise	0.739	0.875
Llama 3.1 8B	pairwise	0.737	0.870

评分者 模式 专家对 其他模型多数 Llama 3.1 405B 成对 0.815 0.949 Gemma 3 27B 成对 0.813 0.928 JEV 成对 0.796 0.910 GPT-4o 成对 0.817 0.906 GPT-4o mini 成对 0.824 0.899 Gemma 3 4B 成对 0.739 0.875 Llama 3.1 8B 成对 0.737 0.870 表 6：A/B 正确性而非相关。第 3 列是专家判断对上的准确率；第 4 列是在全部 28,040 对已发表对上与其他六个评分者多数的一致。JEV 第 3 列准确率为两种呈现顺序的平均（仅已发表顺序为 0.800；表 8）。同一作者未发表扩展中的另九个模型抽取了自己的对，仅重叠 782 对，故未展示。

评分者	模式	专家对	其他模型多数
Llama 3.1 405B	成对	0.815	0.949
Gemma 3 27B	成对	0.813	0.928
JEV	成对	0.796	0.910
GPT-4o	成对	0.817	0.906
GPT-4o mini	成对	0.824	0.899
Gemma 3 4B	成对	0.739	0.875
Llama 3.1 8B	成对	0.737	0.870

Appendix E: what is known about the model

附录 E：关于该模型已知什么

We requested JEV 1.13 as typesafe/jev-1.13 through OpenRouter on 19, 22 and 23 September 2026; every response reported typesafe/jev-1.13-20260917, which we take to be TypeSafe’s jev-1.13.0. The commercial term is “System One”. The vendor documents price, throughput and nine failure modes but publishes no public-benchmark results or calibration metrics (TypeSafe, 2026). This appendix sets out what the vendor states, what the outputs reveal and what remains unknown.

我们于 2026 年 9 月 19、22 与 23 日经 OpenRouter 以 typesafe/jev-1.13 请求 JEV 1.13；每次响应报告 typesafe/jev-1.13-20260917，我们视其为 TypeSafe 的 jev-1.13.0。商业用语是「System One」。厂商文档给出价格、吞吐与九种失败模式，但不公布公开基准结果或校准指标（TypeSafe，2026）。本附录陈述厂商说法、输出揭示了什么，以及仍未知什么。

What the vendor states

厂商陈述

TypeSafe says it built JEV with “a new architecture, a new sampler, and a new training algorithm” (TypeSafe, 2026). It describes none of the three technically. It calls the sampler parallel, producing every output in one query instead of token by token, and says JEV writes no text and is not a chat or code-completion LLM. It describes its training method, Reinforcement Learning for Calibrated Decisions (RLCD), only by its aim, calibrated probabilities, and gives no formal objective for it. TypeSafe says it makes its own training data, and its chief executive told TechCrunch that the data are exclusively synthetic while saying little about the architecture (Fernholz, 2026). That account would not rule out that the model has seen our benchmarks, since text another model writes carries what that model learned; a benchmark built after the version date would settle it. TypeSafe’s primer presents RLCD as a third way, after reinforcement learning from human feedback and from verifiable rewards, to post-train pretrained language models, which we read as implying an unnamed pretrained base. The same weights serve every customer.

TypeSafe 称其以「新架构、新采样器与新训练算法」构建 JEV（TypeSafe，2026）。三者皆无技术描述。它称采样器为并行，在一次查询中产生全部输出而非逐 token，并称 JEV 不写文本，不是聊天或代码补全 LLM。它对其训练方法「校准决策强化学习」（RLCD）仅以其目标——校准概率——描述，未给出形式化目标。TypeSafe 称自行制作训练数据；其首席执行官告诉 TechCrunch 数据全为合成，但对架构所言甚少（Fernholz，2026）。该说法不能排除模型见过我们的基准，因为另一模型写出的文本携带该模型所学；在版本日期之后构建的基准才能裁定。TypeSafe 的入门材料把 RLCD 呈现为人类反馈强化学习与可验证奖励强化学习之后的第三种后训练预训练语言模型的路径，我们读作暗示存在未具名的预训练基座。同一套权重服务每位客户。

The documentation says the state is read once and every question is evaluated against it in parallel and in isolation, with each Score level judged against the state on its own wording, without its number or neighbours. It offers no determinism guarantee. The API takes no temperature or seed.

文档称状态读一次，每个问题并行且孤立地对照它评估，每个 Score 等级仅按其自身措辞对照状态判断，不看其编号或邻居。它不提供确定性保证。API 不接受温度或种子。

TypeSafe lists nine known failure modes for version 1.13 (page last reviewed 2 October 2026). Among them, it says, the model reads questions literally, handles double negatives and multi-hop questions less reliably, counts poorly, loses accuracy as irrelevant text fills the state, and leans toward the first option of a Choice. It also warns that the expected Score should not be read as an exact number between two levels.

TypeSafe 列出版本 1.13 的九种已知失败模式（页面最近审阅于 2026 年 10 月 2 日）。其中称：模型按字面阅读问题，处理双重否定与多跳问题较不可靠，计数差，当无关文本填满状态时准确率下降，并偏向 Choice 的第一选项。它还警告：期望 Score 不应被读作两等级之间的精确数字。

-

-

What the outputs reveal

输出揭示了什么

Everything below comes from the 495,943 successful responses of the first four applications, collected on 19 September 2026 and 22 September 2026, and their run logs. We flag each inference about how JEV is built. Although no response carries generated text, each reports an output-token count, constant within each of our 6 question schemas and running from 38 for party comparisons to 97 for three bundled Gilardi questions. We infer that the schema fixes the count and the answer never moves it. That fits the vendor’s statement that JEV writes no text. Only input tokens are billed, so the count may be nominal.

以下全部来自前四项应用的 495,943 次成功响应（采集于 2026 年 9 月 19 日与 22 日）及其运行日志。我们对每条关于 JEV 如何构建的推断都做标记。尽管响应不带生成文本，每次都报告输出 token 计数，在我们的 6 种问题模式内恒定，从政党比较的 38 到捆绑三个 Gilardi 问题的 97。我们推断模式固定该计数，答案从不移动它。这符合厂商「JEV 不写文本」的说法。只对输入 token 计费，因此该计数可能只是名义上的。

Every probability is a multiple of 0.01, and 14.8% of Choice and Score probabilities are exactly zero, so we infer rounding to the nearest hundredth: a zero stands for a value below 0.005. Probability vectors sum to 1.00 in 99.86% of cases and to 0.99 in the other 721, never above one, which we read as a sum-to-one adjustment after rounding. The returned choice is the most probable option in 99.987% of 517,039 Choice answers, as documented. The Score is the documented expected level counted from zero. Allowing for rounding, that expectation reproduces 99.8% of 6,432 returned Scores, and an expectation counted from one never matches exactly. The rescaling of JEV’s Scores to the 0-10 range in the survey application relies on this.

每个概率都是 0.01 的倍数，14.8% 的 Choice 与 Score 概率恰好为零，因此我们推断四舍五入到最近的百分之一：零代表低于 0.005 的值。概率向量在 99.86% 的情形求和为 1.00，其余 721 次为 0.99，从不高于一，我们读作四舍五入后的和一调整。返回的选择在 517,039 次 Choice 答案的 99.987% 中是最可能选项，与文档一致。Score 是从零计起的文档化期望等级。计入四舍五入后，该期望复现 6,432 个返回 Score 的 99.8%，而从一计起的期望从不精确匹配。调查应用中把 JEV 的 Score 缩放到 0-10 范围依赖这一点。

n

n

Answers vary slightly on replay. Across 1,000 byte-identical pairwise replays from both studies, 98.0% returned the same choice (Appendix B covers Study 1 alone), but the probability vector was identical in only 40.7%, and each option’s probability moved by 0.0120 on average. Asking the same questions together or one at a time, seconds apart (150 pairs), moved Choice probabilities by 0.0085 on average, between the shifts for duplicates within one run (0.0061) and for replays a median 47 minutes later (0.0120). We read this as bounding any bundling effect at about the size of replay variation, as the vendor’s account predicts.

重放时答案略有变化。在两项研究共 1,000 次字节完全相同的成对重放中，98.0% 返回相同选择（附录 B 仅覆盖研究 1），但概率向量仅在 40.7% 中相同，各选项概率平均移动 0.0120。将相同问题一起或逐个、相隔数秒询问（150 对），Choice 概率平均移动 0.0085，介于同一次运行内重复的偏移（0.0061）与中位 47 分钟后重放的偏移（0.0120）之间。我们读作：任何捆绑效应的上界大约相当于重放变异的大小，与厂商说法一致。

What remains unknown

仍未知什么

We do not know the architecture (encoder, decoder, diffusion or otherwise), whether a request is one forward pass, the parameter count, or which base or teacher model, if any, JEV derives from. Nor do we know how options and levels are scored, what an output token counts, or whether replay variation arises in the model or the serving stack. TypeSafe publishes no model changelog. We therefore cannot tell whether the weights changed between our collection dates.

我们不知道架构（编码器、解码器、扩散或其他）、一次请求是否为一次前向传播、参数量，或 JEV 源自何种基座或教师模型（若有）。也不知道选项与等级如何计分、输出 token 计什么，或重放变异来自模型还是服务栈。TypeSafe 不公布模型变更日志。因此无法判断权重是否在我们的采集日之间发生变化。

Nothing in the outputs separates a new architecture from a pretrained or distilled language model whose scores over the answer set are read off, rounded and renormalised. Rounding to hundredths, the sum-to-one adjustment, a confidence computed from the top probability, schema-fixed output-token counts and small replay variation are properties of an interface, and a language model behind a constrained output head could produce every one of them. A technical report, open weights or a named base model would settle the question. Short of that, two checks on the archive would bear on it without settling it: whether the input-token counts the endpoint reports match one tokenizer family exactly, which would point to a base model from that family if the vendor counts with the model’s own tokenizer, and whether the model’s disagreements with other scorers coincide with one generative model’s more often than shared difficulty explains, which would point to a teacher model. We have run neither.

输出中没有任何东西能把新架构与预训练或蒸馏语言模型区分开——后者在答案集上的分数被读出、四舍五入并重新归一化。四舍五入到百分之一、和一调整、由顶部概率计算的置信度、模式固定的输出 token 计数以及小幅重放变异，都是接口属性；受约束输出头背后的语言模型可以产生其中每一项。技术报告、开放权重或具名基座模型才能裁定该问题。在此之外，对档案的两项检查与之相关但无法裁定：端点报告的输入 token 计数是否精确匹配某一分词器家族——若厂商用模型自有分词器计数，则指向该家族的基座模型；以及模型与其他评分者的分歧是否比共享难度所能解释的更常与某一生成模型重合——这会指向教师模型。两项我们都未做。

Appendix F: the conflict benchmark in full

附录 F：冲突基准全文

Test set

测试集

Brandt et al. (2026) evaluate on GTD incidents from 2017 to 2020. Their posted code, applied to the 0522 GTD release, yields 38,157 incidents, while their archived predictions cover 37,709. The difference is one filter that neither the paper nor the code states: dropping the 448 incidents whose summary does not open with a complete calendar date, 443 of them dated to a month only. With that filter our rebuilt test set matches their event identifiers exactly. The text each model sees is the summary followed by the motive, as their code builds it.

Brandt 等（2026）在 2017 至 2020 年的 GTD 事件上评估。其发布代码应用于 0522 版 GTD 得到 38,157 起事件，而其存档预测覆盖 37,709 起。差额是一项论文与代码皆未说明的过滤：丢掉摘要不以完整日历日期开头的 448 起事件，其中 443 起仅标到月份。加上该过滤后，我们重建的测试集与其事件标识符精确匹配。每个模型看到的文本是摘要后接动机，按其代码构建。

Reproducing the published figures

复现已发表数字

Scoring their per-incident predictions with our reimplementation of their parser and metrics reproduces all 71 cells of their Tables 2, 5 and 6 within rounding. Their macro averages are unweighted means over the nine types, with a type never predicted scoring zero. A footnote describes them as support-weighted, but only the unweighted mean reproduces the printed values. Every model in Table 11 is scored by that rule on the same 37,709 incidents, with any answer outside the nine types counted wrong. Qwen3.5-9B is parsed exactly as their generative models were, taking the highest-probability key of the returned JSON object.

用我们对其解析器与指标的重实现对其逐事件预测计分，在舍入范围内复现其表 2、5、6 的全部 71 格。其宏平均是九类上的未加权均值，从未被预测的类型记零。脚注描述为按支持加权，但只有未加权均值复现印刷值。表 11 中每个模型都按该规则在同一批 37,709 起事件上计分，任何落在九类之外的答案计错。Qwen3.5-9B 的解析方式与其对生成模型完全相同：取返回 JSON 对象中概率最高的键。

The runs and the prompted baselines

运行与提示式基线

Qwen3.5-9B ran through OpenRouter, pinned to a single provider’s bf16 deployment with no fallback, reasoning disabled, JSON output and a fixed seed (Appendix A). All 37,709 responses in each arm came from that deployment. The original authors ran mid-2024 versions of their generative models locally at 4-bit precision (Brandt et al., 2026).

Qwen3.5-9B 经 OpenRouter 运行，固定到单一提供商的 bf16 部署且无回退，关闭推理，JSON 输出，固定种子（附录 A）。每条分支的全部 37,709 次响应都来自该部署。原作者以 4 比特精度在本地运行其中期 2024 版生成模型（Brandt 等，2026）。

This design cannot say how much of Qwen3.5-9B’s lead over the published prompted models reflects their age and how much their precision or serving. The same holds for JEV’s lead over the published prompted models. Its lead over Qwen3.5-9B, run at the same time through the same router, is the cleaner comparison, though the two arms still differ in prompt and interface. Running at 4-bit is costly on this task even for a fine-tuned model: ConflLlama reaches macro F1 0.571 at 4-bit against 0.647 at 8-bit. Qwen3.5-9B is one model of 9 billion parameters, and JEV’s size is unknown (Appendix E).

该设计无法说明 Qwen3.5-9B 相对已发表提示式模型的领先有多少来自后者的年代，有多少来自精度或服务。JEV 相对已发表提示式模型的领先同样如此。其相对同一时间经同一路由运行的 Qwen3.5-9B 的领先是更干净的比较，尽管两分支在提示与接口上仍有差异。即便对微调模型，在本任务上以 4 比特运行也代价不菲：ConflLlama 在 4 比特宏 F1 为 0.571，8 比特为 0.647。Qwen3.5-9B 是 90 亿参数的单一模型，JEV 体量未知（附录 E）。

-0.003

-0.003

Ties and decoding

平局与解码

JEV reports probabilities to the hundredth, so ties occur. For the nine-way question the prediction is the option JEV returns, whose probability is tied at the top in 44 incidents. With nine separate questions the highest probability is tied in 380 incidents; we break those ties at random with a fixed seed, and breaking them by the codebook hierarchy instead moves accuracy by less than 0.004. Under hierarchy decoding we take the highest-ranked type whose probability reaches 0.5, and the maximum when none does.

JEV 将概率报告到百分位，因此会出现平局。对九选一问题，预测取 JEV 返回的选项，其概率在 44 起事件中并列最高。用九个独立问题时，最高概率在 380 起事件中并列；我们以固定种子随机打破这些平局，若改按编码手册层级打破，准确率变动小于 0.004。在层级解码下，我们取概率达到 0.5 的最高层级类型；若皆未达到则取最大值。

Where only one of the nine-way question and the nine separate questions is right, it is the single question for 3,534 incidents and the separate questions for 728.

在九选一问题与九个独立问题仅一侧正确的情形中，单问正确的有 3,534 起，独立问题正确的有 728 起。

Recall by type

按类型的召回

Attack type Support Confli- BERT ConflLlama 8-bit JEV names JEV codebook Qwen 3.5 names Qwen 3.5 codebook GPT-6 Luna names GPT-6 Luna codebook Qwen 27B names Qwen 27B codebook Assassination 2,990 0.74 0.59 0.29 0.54 0.02 0.38 0.48 0.66 0.28 0.75 Armed Assault 9,079 0.81 0.75 0.90 0.94 0.94 0.89 0.79 0.89 0.88 0.91 Bombing/Explosion 14,508 0.97 0.92 0.91 0.81 0.84 0.65 0.92 0.86 0.85 0.73 Hijacking 154 0.64 0.54 0.60 0.70 0.44 0.58 0.38 0.50 0.55 0.56 Hostage taking, barricade 230 0.38 0.27 0.33 0.46 0.26 0.37 0.23 0.46 0.29 0.41 Hostage taking, kidnapping 3,495 0.88 0.86 0.75 0.82 0.77 0.81 0.71 0.81 0.69 0.71 Facility/infrastructure 2,624 0.85 0.74 0.72 0.79 0.33 0.72 0.79 0.87 0.69 0.78 Unarmed Assault 301 0.62 0.50 0.25 0.44 0.18 0.31 0.35 0.40 0.18 0.31 Unknown 4,328 0.55 0.39 0.00 0.01 0.03 0.07 0.00 0.03 0.00 0.01 Table 7: Recall by attack type on the 37,709 test incidents. Support is the number of incidents of that type. ConfliBERT and ConflLlama are recomputed from Brandt et al.’s per-incident predictions. GPT-6 Luna is a current generative model (Appendix H). Qwen 3.5 is Qwen3.5-9B and Qwen 27B is the open-weight Qwen3.8-27B. “Names” columns give the model only the category names; “codebook” columns add the GTD codebook definitions.

Attack type	Support	Confli- BERT	ConflLlama 8-bit	JEV names	JEV codebook	Qwen 3.5 names	Qwen 3.5 codebook	GPT-6 Luna names	GPT-6 Luna codebook	Qwen 27B names	Qwen 27B codebook
Assassination	2,990	0.74	0.59	0.29	0.54	0.02	0.38	0.48	0.66	0.28	0.75
Armed Assault	9,079	0.81	0.75	0.90	0.94	0.94	0.89	0.79	0.89	0.88	0.91
Bombing/Explosion	14,508	0.97	0.92	0.91	0.81	0.84	0.65	0.92	0.86	0.85	0.73
Hijacking	154	0.64	0.54	0.60	0.70	0.44	0.58	0.38	0.50	0.55	0.56
Hostage taking, barricade	230	0.38	0.27	0.33	0.46	0.26	0.37	0.23	0.46	0.29	0.41
Hostage taking, kidnapping	3,495	0.88	0.86	0.75	0.82	0.77	0.81	0.71	0.81	0.69	0.71
Facility/infrastructure	2,624	0.85	0.74	0.72	0.79	0.33	0.72	0.79	0.87	0.69	0.78
Unarmed Assault	301	0.62	0.50	0.25	0.44	0.18	0.31	0.35	0.40	0.18	0.31
Unknown	4,328	0.55	0.39	0.00	0.01	0.03	0.07	0.00	0.03	0.00	0.01

袭击类型 支持数 Confli- BERT ConflLlama 8-bit JEV 名称 JEV 编码手册 Qwen 3.5 名称 Qwen 3.5 编码手册 GPT-6 Luna 名称 GPT-6 Luna 编码手册 Qwen 27B 名称 Qwen 27B 编码手册 Assassination 2,990 0.74 0.59 0.29 0.54 0.02 0.38 0.48 0.66 0.28 0.75 Armed Assault 9,079 0.81 0.75 0.90 0.94 0.94 0.89 0.79 0.89 0.88 0.91 Bombing/Explosion 14,508 0.97 0.92 0.91 0.81 0.84 0.65 0.92 0.86 0.85 0.73 Hijacking 154 0.64 0.54 0.60 0.70 0.44 0.58 0.38 0.50 0.55 0.56 Hostage taking, barricade 230 0.38 0.27 0.33 0.46 0.26 0.37 0.23 0.46 0.29 0.41 Hostage taking, kidnapping 3,495 0.88 0.86 0.75 0.82 0.77 0.81 0.71 0.81 0.69 0.71 Facility/infrastructure 2,624 0.85 0.74 0.72 0.79 0.33 0.72 0.79 0.87 0.69 0.78 Unarmed Assault 301 0.62 0.50 0.25 0.44 0.18 0.31 0.35 0.40 0.18 0.31 Unknown 4,328 0.55 0.39 0.00 0.01 0.03 0.07 0.00 0.03 0.00 0.01 表 7：37,709 起测试事件上按袭击类型的召回。支持数是该类型事件数。ConfliBERT 与 ConflLlama 由 Brandt 等的逐事件预测重算。GPT-6 Luna 是当前生成模型（附录 H）。Qwen 3.5 为 Qwen3.5-9B，Qwen 27B 为开源权重 Qwen3.8-27B。「名称」列只给模型类别名称；「编码手册」列加入 GTD 编码手册定义。

袭击类型	支持数	Confli- BERT	ConflLlama 8-bit	JEV 名称	JEV 编码手册	Qwen 3.5 名称	Qwen 3.5 编码手册	GPT-6 Luna 名称	GPT-6 Luna 编码手册	Qwen 27B 名称	Qwen 27B 编码手册
Assassination	2,990	0.74	0.59	0.29	0.54	0.02	0.38	0.48	0.66	0.28	0.75
Armed Assault	9,079	0.81	0.75	0.90	0.94	0.94	0.89	0.79	0.89	0.88	0.91
Bombing/Explosion	14,508	0.97	0.92	0.91	0.81	0.84	0.65	0.92	0.86	0.85	0.73
Hijacking	154	0.64	0.54	0.60	0.70	0.44	0.58	0.38	0.50	0.55	0.56
Hostage taking, barricade	230	0.38	0.27	0.33	0.46	0.26	0.37	0.23	0.46	0.29	0.41
Hostage taking, kidnapping	3,495	0.88	0.86	0.75	0.82	0.77	0.81	0.71	0.81	0.69	0.71
Facility/infrastructure	2,624	0.85	0.74	0.72	0.79	0.33	0.72	0.79	0.87	0.69	0.78
Unarmed Assault	301	0.62	0.50	0.25	0.44	0.18	0.31	0.35	0.40	0.18	0.31
Unknown	4,328	0.55	0.39	0.00	0.01	0.03	0.07	0.00	0.03	0.00	0.01

Table 7 shows where the codebook helps. For JEV it raises recall on eight of the nine types, Unknown only trivially, and lowers it on Bombing/Explosion. That fall is consistent with the coding hierarchy, which ranks Bombing/Explosion below four other types.

表 7 显示编码手册在何处有帮助。对 JEV，它提高九类中八类的召回，Unknown 仅微不足道，并降低 Bombing/Explosion 的召回。该下降与编码层级一致——该层级把 Bombing/Explosion 排在另外四类之下。

GTD reserves Assassination for attacks on specific, prominent individuals, and JEV’s recall on it rises from 0.285 to 0.542 (Table 7). The codebook’s hierarchy also fixes the gold label. Of the test incidents, 7.2% carry more than one recorded type, and in all 2,709 of them the first, the label every model is scored on, is the type the hierarchy ranks highest. A model given only the names is not told that rule, so on those incidents an answer naming another recorded type, such as Bombing/Explosion for an assassination by bomb, counts as wrong.

GTD 将 Assassination 保留给针对特定、显要个人的攻击，JEV 在其上的召回从 0.285 升至 0.542（表 7）。编码手册的层级也固定金标准标签。测试事件中 7.2% 带有一种以上记录类型，且在全部 2,709 起中，每个模型据以计分的第一标签都是层级排名最高的类型。只给名称的模型未被告知该规则，因此在这些事件上，若答案点名另一记录类型——例如对炸弹刺杀答 Bombing/Explosion——计为错误。

Macro F1 weights the nine types equally and accuracy weights incidents, and for both JEV and Qwen3.5-9B the codebook raised recall on rare and narrowly defined types while lowering it on Bombing/Explosion, the most common (Table 7). Qwen3.5-9B also started lower, recalling almost no Assassinations from the names alone (0.015), so it had more to gain in macro F1, which scores a type a model seldom predicts near zero. For Qwen3.8-27B too, the codebook increases recall for Assassination (0.28 to 0.75), which is the biggest increase in Table 7. It decreases recall for Bombing/Explosion (0.85 to 0.73).

宏 F1 对九类等权，准确率对事件加权；对 JEV 与 Qwen3.5-9B，编码手册都提高了稀有且定义狭窄类型的召回，同时降低了最常见的 Bombing/Explosion（表 7）。Qwen3.5-9B 起点也更低，仅凭名称几乎召回不了 Assassination（0.015），因此在宏 F1 上更有可提升空间——该指标对模型很少预测的类型接近打零。对 Qwen3.8-27B，编码手册同样提高 Assassination 召回（0.28 到 0.75），这是表 7 中最大增幅；并降低 Bombing/Explosion 召回（0.85 到 0.73）。

Unknown

JEV almost never answers Unknown: it gives that label to 0.17% of incidents from the names alone and to 0.95% when the codebook defines it. A model reproducing memorised codes would give Unknown as readily as any other, so this is also weak evidence against JEV having memorised the test set’s codes, public in GTD releases before its version date, though not against partial exposure. Comparisons restricted to incidents of known type are also not neutral: the fine-tuned models’ Unknown answers on the incidents kept still count as errors. Qwen3.5-9B, which writes its answer, also seldom answers Unknown even with the codebook (recall 0.072, against 0.009 for JEV), so most of the shortfall is shared by a generative model given the same definition. Qwen3.8-27B also has a recall for Unknown of 0.000 from the names and 0.006 with the codebook.

JEV 几乎从不答 Unknown：仅名称时给该标签的事件占 0.17%，编码手册定义时占 0.95%。若模型在复现记忆中的编码，会像对待其他标签一样轻易给出 Unknown，因此这也是反对 JEV 已记忆测试集编码的弱证据——这些编码在其版本日期前的 GTD 发布中已公开——尽管不能反对部分暴露。仅限已知类型事件的比较也不中立：被保留事件上微调模型的 Unknown 答案仍计为错误。写出答案的 Qwen3.5-9B 即便有编码手册也很少答 Unknown（召回 0.072，JEV 为 0.009），因此短缺的大部分由给定同一定义的生成模型所共享。Qwen3.8-27B 对 Unknown 的召回在仅名称时为 0.000，有编码手册时为 0.006。

Whatever part of the fine-tuned models’ lead the codebook leaves is credited to examples only by elimination. It includes base rates, conventions the codebook does not spell out and ConfliBERT’s pretraining on conflict text. The Unknown result also suggests that what a code means in practice is partly learned from examples, so the design cannot divide the fine-tuned models’ lead between examples and meaning.

编码手册留下的微调模型领先部分，只是通过排除法归功于示例。它包括基率、编码手册未写明的惯例，以及 ConfliBERT 在冲突文本上的预训练。Unknown 结果也表明，编码在实践中的含义部分从示例中学得，因此该设计无法把微调模型的领先在示例与含义之间切开。

Multi-label classification

多标签分类

The GTD records up to three attack types per incident. Scored as a multi-label task by their rule, which keeps every type with a probability of at least 0.5, the nine-way question reaches a subset accuracy of 66.6% (category names only; 67.5% with the codebook), against 79.4% for ConfliBERT and 72.4% for the 8-bit ConflLlama. Its probabilities sum to one, so that rule gives it at most one type per incident, and since most incidents carry a single type, this costs little. The nine separate questions reach 24.9%, because they say yes too readily: on average 1.76 of the eight known types clear 0.5 per incident, where coders record 0.96. With its written probabilities, Qwen3.8-27B reaches a subset accuracy of 63.7% from the category names and 65.2% with the codebook, lower than JEV. GPT-6 Luna’s written probabilities give 66.6% and 70.1%. Neither model wrote two types at 0.5 or above for any incident, so both are effectively single-label.

GTD 每起事件最多记录三种袭击类型。按其规则——保留概率至少 0.5 的每一类型——作为多标签任务计分时，九选一问题子集准确率为 66.6%（仅类别名称；有编码手册为 67.5%），而 ConfliBERT 为 79.4%，8 比特 ConflLlama 为 72.4%。其概率和为一，因此该规则每起事件至多给一种类型；由于多数事件只带一种类型，代价不大。九个独立问题达到 24.9%，因为它们过轻易说是：平均每起事件有 1.76 个已知八类中超过 0.5，而编码者记录 0.96。用其写出的概率，Qwen3.8-27B 子集准确率在仅类别名称时为 63.7%，有编码手册时为 65.2%，低于 JEV。GPT-6 Luna 写出的概率给出 66.6% 与 70.1%。两个模型都未对任何事件写出两个达到 0.5 及以上的类型，因此实质上均为单标签。

The BBC task

BBC 任务

-0.182

-0.182

Appendix G: further results and checks

附录 G：进一步结果与检查

Party positions

政党位置

Di Leo et al. (2025) run each comparison seven times and keep the modal answer, reporting 93.23% congruence across repeats; we run a single pass. Modal voting reduces run-to-run noise, but on the survey comparisons JEV returns the same winner on 98.6% of replays (Appendix B), so we expect seven passes would change few of its decisions here either, though we have no replays of the party comparisons.

Di Leo 等（2025）每对比较跑七次并保留众数答案，报告跨重复一致率 93.23%；我们跑单次。众数投票降低运行间噪声，但在调查比较上 JEV 在 98.6% 的重放中返回同一胜者（附录 B），因此我们预期七次通过在此也改变不了多少决策，尽管我们没有政党比较的重放。

Their scores are not in the archive and their figures are transcribed to two decimals, so only our correlations carry intervals (Table 13). The voter comparison rests on 17 to 93 parties a year. Our CHES correlation is stable, running 0.856 to 0.891 across the 8 elections with a CHES wave, which span four decades of European party systems. Di Leo et al. (2025) attribute the lower agreement with manifesto scores, which we reproduce, to the sensitivity of manifesto scaling. In panel b of Table 13 we add Qwen3.8-27B and a fourth benchmark, voters’ placements in the CSES, which are available for 1999 to 2014 (between 95 and 142 parties a year). The correlations with this benchmark are between 0.76 and 0.84 for JEV’s scale, between 0.70 and 0.78 for GPT-6 Luna’s and between 0.58 and 0.67 for Qwen’s, so the ordering is the same as with the experts. These are single passes, without paired intervals.

其分数不在档案中，数字转录到两位小数，因此只有我们的相关带区间（表 13）。选民比较每年基于 17 到 93 个政党。我们的 CHES 相关稳定，在有 CHES 波次的 8 次选举上为 0.856 到 0.891，跨越四十年欧洲政党体系。Di Leo 等（2025）将与宣言分数较低的一致——我们予以复现——归因于宣言缩放的敏感性。在表 13 的 b 面板我们加入 Qwen3.8-27B 与第四个基准：CSES 中选民定位，可得于 1999 至 2014（每年 95 到 142 个政党）。与该基准的相关对 JEV 量表在 0.76 到 0.84 之间，对 GPT-6 Luna 在 0.70 到 0.78，对 Qwen 在 0.58 到 0.67，因此排序与专家一致。这些是单次通过，无成对区间。

Because every pair here is presented in one order, as in their design, the first-listed party is chosen 46.6% of the time. A single-order study cannot distinguish that departure from genuine imbalance between slots, and here the slots are not exchangeable: the published duplet files fix each pair’s order by country and then by party name, so a preference for one slot, of the kind Study 1 shows (Appendix B), would move parties listed early in that order relative to those listed late rather than average out.

因这里每对按单一顺序呈现——与其设计相同——首选前列政党的比例为 46.6%。单序研究无法将该偏离与槽位间真实不平衡区分开；此处槽位不可交换：已发表二元组文件按国家再按党名固定每对顺序，因此对某一槽的偏好——研究 1 所示类型（附录 B）——会移动该顺序中靠前相对靠后列出的政党，而非平均掉。

Nor can this design exclude contamination. The model sees nothing but each party’s name and country, so it answers from what it has stored about the parties, and the expert, manifesto and voter placements it is validated against were public before the model’s version date, and are listed beside each party’s name in [9]’s own files. A model that had learned those placements would also agree most closely with the experts, so the ordering across benchmarks cannot separate recall of the benchmark from knowledge of the parties. Being newer, JEV has also had more chance to see the placements than GPT-3.5 had, and part of its margin over the published figures may reflect that exposure.

该设计也不能排除污染。模型除各党名称与国家外什么都看不到，因此从其所存储的关于政党的信息作答；而据以验证的专家、宣言与选民定位在模型版本日期前已公开，并在 [9] 自有文件中列于各党名称旁。学过这些定位的模型也会与专家最一致，因此跨基准的排序无法把对基准的回忆与对政党的知识分开。作为更新的模型，JEV 也比 GPT-3.5 更有机会见过这些定位，其相对已发表数字的部分优势可能反映该暴露。

Splitting the parties by coverage bears on this without settling it. A model recalling the expert placements should place the parties the expert survey covers better than those it leaves out, and since only covered parties have a CHES placement, we compare the two groups on the manifesto scores, which most parties in both carry. Over the 8 years with a CHES wave, JEV’s scale correlates with them at a mean of 0.618 [0.525, 0.699] among parties CHES places that year and 0.469 [0.393, 0.542] among the rest, a gap of 0.149 [0.036, 0.255] in a bootstrap over parties that holds JEV’s scores fixed, positive in 6 of the 8 years. That is the pattern recall predicts. But 774 of the 966 party-years without a placement belong to countries CHES did not survey that year, and a model that knew less about those countries’ parties would show the same gap; the manifesto scores spread about as widely in both groups (standard deviations averaging 20.1 and 19.8 points), so a narrower range does not explain it. The split cannot tell recall from salience, and a model could also have recalled the manifesto scores themselves.

按覆盖度拆分政党与此相关但无法裁定。回忆专家定位的模型应把专家调查覆盖的政党放得比未覆盖的更好；由于只有被覆盖政党有 CHES 定位，我们在两组都大多带有的宣言分数上比较。在有 CHES 波次的 8 年里，JEV 量表与它们的平均相关：当年 CHES 定位的政党为 0.618 [0.525，0.699]，其余为 0.469 [0.393，0.542]，差距 0.149 [0.036，0.255]——在固定 JEV 分数、对政党做自助法时——8 年中有 6 年为正。这正是回忆所预测的模式。但 966 个无定位的政党—年中有 774 个属于当年 CHES 未调查的国家，对那些国家政党了解更少的模型也会显示同样差距；两组中宣言分数散布大致同样宽（标准差平均 20.1 与 19.8 分），因此更窄的范围不能解释。该拆分无法区分回忆与显著性，且模型也可能直接回忆了宣言分数本身。

The survey application

调查应用

Measure N Precision Recall F1 95% CI Accuracy GPT-4o mini (paper) 460 0.835 0.845 0.840 [0.797, 0.876] 0.824 Gemma 3 27B (paper) 460 0.821 0.841 0.831 [0.787, 0.869] 0.813 GPT-4o (paper) 460 0.852 0.805 0.828 [0.780, 0.869] 0.817 JEV (Noul) 460 0.812 0.829 0.821 [0.772, 0.858] 0.802 Llama 3.1 405B (paper) 459 0.877 0.769 0.820 [0.770, 0.859] 0.815 JEV (rev) 460 0.799 0.841 0.819 [0.773, 0.858] 0.798 GPT-6 Luna (fwd) 460 0.748 0.900 0.817 [0.772, 0.856] 0.780 JEV (order-averaged) 460 0.803 0.829 0.816 [0.769, 0.856] 0.796 JEV (fwd) 460 0.827 0.801 0.814 [0.764, 0.852] 0.800 Qwen3.8-27B (fwd) 460 0.884 0.701 0.782 [0.721, 0.831] 0.787 Gemma 3 4B (paper) 460 0.796 0.701 0.746 [0.685, 0.799] 0.739 Llama 3.1 8B (paper) 460 0.849 0.629 0.723 [0.656, 0.781] 0.737 Table 8: Agreement with expert raters: 460 judgments of 243 pairs. F1, defined as in the published article, treats “human picked statement 1” as the positive class, so it depends on order; accuracy is also shown. “fwd” is the published presentation order, “rev” the reverse and “(paper)” marks the published scorers. Intervals: 2,000 bootstrap resamples of pairs; rater clustering is not modelled. GPT-6 Luna and the open-weight Qwen3.8-27B are ours, in the published order (Appendix H).

Measure	N	Precision	Recall	F1	95% CI	Accuracy
GPT-4o mini (paper)	460	0.835	0.845	0.840	[0.797, 0.876]	0.824
Gemma 3 27B (paper)	460	0.821	0.841	0.831	[0.787, 0.869]	0.813
GPT-4o (paper)	460	0.852	0.805	0.828	[0.780, 0.869]	0.817
JEV (Noul)	460	0.812	0.829	0.821	[0.772, 0.858]	0.802
Llama 3.1 405B (paper)	459	0.877	0.769	0.820	[0.770, 0.859]	0.815
JEV (rev)	460	0.799	0.841	0.819	[0.773, 0.858]	0.798
GPT-6 Luna (fwd)	460	0.748	0.900	0.817	[0.772, 0.856]	0.780
JEV (order-averaged)	460	0.803	0.829	0.816	[0.769, 0.856]	0.796
JEV (fwd)	460	0.827	0.801	0.814	[0.764, 0.852]	0.800
Qwen3.8-27B (fwd)	460	0.884	0.701	0.782	[0.721, 0.831]	0.787
Gemma 3 4B (paper)	460	0.796	0.701	0.746	[0.685, 0.799]	0.739
Llama 3.1 8B (paper)	460	0.849	0.629	0.723	[0.656, 0.781]	0.737

度量 N 精确率 召回 F1 95% CI 准确率 GPT-4o mini（论文） 460 0.835 0.845 0.840 [0.797, 0.876] 0.824 Gemma 3 27B（论文） 460 0.821 0.841 0.831 [0.787, 0.869] 0.813 GPT-4o（论文） 460 0.852 0.805 0.828 [0.780, 0.869] 0.817 JEV (Noul) 460 0.812 0.829 0.821 [0.772, 0.858] 0.802 Llama 3.1 405B（论文） 459 0.877 0.769 0.820 [0.770, 0.859] 0.815 JEV (rev) 460 0.799 0.841 0.819 [0.773, 0.858] 0.798 GPT-6 Luna (fwd) 460 0.748 0.900 0.817 [0.772, 0.856] 0.780 JEV（顺序平均） 460 0.803 0.829 0.816 [0.769, 0.856] 0.796 JEV (fwd) 460 0.827 0.801 0.814 [0.764, 0.852] 0.800 Qwen3.8-27B (fwd) 460 0.884 0.701 0.782 [0.721, 0.831] 0.787 Gemma 3 4B（论文） 460 0.796 0.701 0.746 [0.685, 0.799] 0.739 Llama 3.1 8B（论文） 460 0.849 0.629 0.723 [0.656, 0.781] 0.737 表 8：与专家评分者的一致：对 243 对的 460 次判断。F1 定义同已发表文章，将「人类选陈述 1」视为正类，因此依赖顺序；同时给出准确率。「fwd」为已发表呈现顺序，「rev」为反向，「（论文）」标记已发表评分者。区间：对对的 2,000 次自助重抽样；未对评分者聚类建模。GPT-6 Luna 与开源权重 Qwen3.8-27B 为我们所得，按已发表顺序（附录 H）。

度量	N	精确率	召回	F1	95% CI	准确率
GPT-4o mini（论文）	460	0.835	0.845	0.840	[0.797, 0.876]	0.824
Gemma 3 27B（论文）	460	0.821	0.841	0.831	[0.787, 0.869]	0.813
GPT-4o（论文）	460	0.852	0.805	0.828	[0.780, 0.869]	0.817
JEV (Noul)	460	0.812	0.829	0.821	[0.772, 0.858]	0.802
Llama 3.1 405B（论文）	459	0.877	0.769	0.820	[0.770, 0.859]	0.815
JEV (rev)	460	0.799	0.841	0.819	[0.773, 0.858]	0.798
GPT-6 Luna (fwd)	460	0.748	0.900	0.817	[0.772, 0.856]	0.780
JEV（顺序平均）	460	0.803	0.829	0.816	[0.769, 0.856]	0.796
JEV (fwd)	460	0.827	0.801	0.814	[0.764, 0.852]	0.800
Qwen3.8-27B (fwd)	460	0.884	0.701	0.782	[0.721, 0.831]	0.787
Gemma 3 4B（论文）	460	0.796	0.701	0.746	[0.685, 0.799]	0.739
Llama 3.1 8B（论文）	460	0.849	0.629	0.723	[0.656, 0.781]	0.737

-

-

Two features of the benchmark bound what it can show. JEV was asked about these pairs with the published human-comparison prompt, which has no “equal” option because the raters had none, so the specification scored here differs from the grid’s by that option. And the raters disagree with one another. On the 140 pairs judged more than once (357 judgments), two judgments of the same pair pick the same statement 75.2% of the time. No scorer could agree with more than 87.1% of those judgments, and JEV agrees with 80.1%. A scorer that follows the raters’ majority agrees with any one of them more often than two of them agree with each other, so differences of a few points between the rows of Table 8 are small against the raters’ own disagreement.

该基准的两项特征限定了它能显示什么。JEV 用已发表的人类比较提示被问及这些对——因评分者没有「相等」选项，提示也没有——因此此处计分的规格因该选项而不同于网格。且评分者彼此不同意。在被判断超过一次的 140 对（357 次判断）上，同一对的两次判断选出同一陈述的比例为 75.2%。没有评分者能与超过 87.1% 的这些判断一致，JEV 与 80.1% 一致。跟随评分者多数的评分者与其中任一位的一致，多于两位评分者彼此之间的一致，因此表 8 各行间几个百分点的差异相对评分者自身分歧很小。

-

-

Appendix H: a current generative comparator

附录 H：当前生成式对照

The published prompted models are the ones the original authors used, from 2023 to mid-2024. To set JEV beside a current one, we ran OpenAI’s GPT-6 Luna on every test through its Batch API on 25 September 2026 (on 28 September for the manifesto and speech sentences and, as described below, the democracy indicators). Each paper’s published prompt was sent verbatim, reply-format lines included, with temperature 0, a fixed seed and reasoning effort set to none. The one change the interface forced is in the sentiment task, where [33]’s request for a single token with twenty alternatives became a short reply with the five alternatives the interface allows; their score reads the first token either way. The conflict prompts are, byte for byte, those Qwen3.5-9B received. Tests with more than 3,000 items were piloted first (by respondent in the survey, within 2009 for party positions, stratified by attack type for the GTD) before the full run.

已发表的提示式模型是原作者使用的那些，从 2023 年到 2024 年中。为把 JEV 与当前模型并列，我们于 2026 年 9 月 25 日经 Batch API 在每项测试上运行 OpenAI 的 GPT-6 Luna（党纲与演讲句子以及下文所述民主指标于 9 月 28 日）。每篇论文的已发表提示原文发送，含回复格式行，温度 0，固定种子，推理力度设为 none。接口迫使的唯一改动在情感任务：[33] 要求带二十个备选的单 token，变为接口允许的五个备选的短回复；其计分无论哪种方式都读首 token。冲突提示与 Qwen3.5-9B 收到的逐字节相同。超过 3,000 条目的测试先试点（调查按受访者，政党位置限 2009 年内，GTD 按袭击类型分层）再全量运行。

We compare GPT-6 Luna with JEV on eight checks. Each is scored by the original paper’s rule, through code that first reproduced JEV’s reported figures exactly, and the difference from JEV is read from a paired bootstrap. Table 9 gives the differences from JEV and the reading of each. The scores themselves stand beside JEV’s in Tables 11, 12, 13 and 14 and in the appendix tables of the same tests (Tables 4, 5, 7 and 8). Those tables also hold cells outside the eight checks, such as GPT-6 Luna’s correlations with the voter and manifesto benchmarks. They are descriptive, and no claim rests on them. On the two reconstructed framing tasks, which Table 9 omits, GPT-6 Luna is level with JEV in all four task-by-sample cells. With the codebook, it reaches accuracy 0.743 and macro F1 0.634 on the GTD incidents, above JEV’s codebook arm in accuracy and level in macro F1. The 510,681 requests cost $7.17. Across 10 matched tasks, GPT-6 Luna’s price per decision was 0.45 to 1.03 times JEV’s, and at most 1.27 times if every uncached input token is billed at the posted cache-write rate.

我们在八项检查上比较 GPT-6 Luna 与 JEV。每项按原论文规则计分，所用代码先精确复现了 JEV 的报告数字；与 JEV 的差距由成对自助法读出。表 9 给出相对 JEV 的差距及各项解读。分数本身与 JEV 并列于表 11、12、13、14 及相同测试的附录表（表 4、5、7、8）。那些表还含八项检查之外的格子，例如 GPT-6 Luna 与选民及宣言基准的相关。它们是描述性的，无主张依赖它们。在表 9 省略的两项重建框架任务上，GPT-6 Luna 在全部四个任务\times样本格上与 JEV 持平。有编码手册时，它在 GTD 事件上达到准确率 0.743、宏 F1 0.634，准确率高于 JEV 的编码手册分支，宏 F1 持平。510,681 次请求费用 $7.17。在 10 项匹配任务上，GPT-6 Luna 每个决策的价格为 JEV 的 0.45 到 1.03 倍；若每个未缓存输入 token 都按公布的缓存写入费率计费，至多为 1.27 倍。

\rho

Test	Statistic	GPT-6 Luna	JEV	Difference	Reading
Tweet relevance, 2021	accuracy (%)	91.1	89.0	2.1 [0.7, 3.5]	above
Tweet relevance, 2023	accuracy (%)	84.2	76.2	8.0 [4.1, 11.9]	above
Tweet sentiment, few-shot	$\rho$ with experts	0.816	0.740	0.076 [0.039, 0.112]	above
GTD attack type, names	accuracy	0.709	0.715	$-$ 0.006 [ $-$ 0.009, $-$ 0.003]	below
GTD attack type, names	macro F1	0.544	0.555	$-$ 0.011 [ $-$ 0.025, 0.002]	level
BBC conflict	conflict F1	0.139	0.321	$-$ 0.182 [ $-$ 0.308, $-$ 0.050]	below
Party positions	CHES $r$ , mean of 8 years	0.844	0.877	$-$ 0.051 to $-$ 0.017	level in 8 of 8
Survey, expert pairs	F1	0.817	0.814	0.004 [ $-$ 0.031, 0.043]	level
Survey, Study 1 scale	$r$ with GPT-4o’s scale	0.935	0.937	$-$ 0.002 [ $-$ 0.008, 0.004]	level
Survey, calibration	error vs raters	0.114	0.109	0.005 [ $-$ 0.031, 0.040]	level
Democracy indicators	mean country $r$ with V-Dem	0.628	0.680	$-$ 0.052 [ $-$ 0.068, $-$ 0.037]	below
Democracy indicators	exact agreement	0.516	0.543	$-$ 0.027 [ $-$ 0.038, $-$ 0.016]	below
Democracy indicators	calibration error	0.390	0.057	0.333 [0.318, 0.347]	above

\rho

检验	统计量	GPT-6 Luna	JEV	差距	解读
推文相关性，2021	准确率 (%)	91.1	89.0	2.1 [0.7, 3.5]	更高
推文相关性，2023	准确率 (%)	84.2	76.2	8.0 [4.1, 11.9]	更高
推文情感，少样本	与专家的 $\rho$	0.816	0.740	0.076 [0.039, 0.112]	更高
GTD 袭击类型，名称	准确率	0.709	0.715	$-$ 0.006 [ $-$ 0.009, $-$ 0.003]	更低
GTD 袭击类型，名称	宏 F1	0.544	0.555	$-$ 0.011 [ $-$ 0.025, 0.002]	持平
BBC 冲突	冲突 F1	0.139	0.321	$-$ 0.182 [ $-$ 0.308, $-$ 0.050]	更低
政党位置	CHES $r$ ，8 年平均	0.844	0.877	$-$ 0.051 到 $-$ 0.017	8/8 持平
调查，专家对	F1	0.817	0.814	0.004 [ $-$ 0.031, 0.043]	持平
调查，研究 1 量表	与 GPT-4o 量表的 $r$	0.935	0.937	$-$ 0.002 [ $-$ 0.008, 0.004]	持平
调查，校准	相对评分者的误差	0.114	0.109	0.005 [ $-$ 0.031, 0.040]	持平
民主指标	与 V-Dem 的国家平均 $r$	0.628	0.680	$-$ 0.052 [ $-$ 0.068, $-$ 0.037]	更低
民主指标	精确一致	0.516	0.543	$-$ 0.027 [ $-$ 0.038, $-$ 0.016]	更低
民主指标	校准误差	0.390	0.057	0.333 [0.318, 0.347]	更高

Three further observations stand out. First, GPT-6 Luna is not deterministic. Requests sent twice with identical bodies, once in a pilot and again in the full run, returned the same answer 95.3% to 97.1% of the time, against 98.6% for JEV’s replays (Appendix B). No interval in Table 9 includes that variation. Second, it has position effects of its own. On the expert-judged survey pairs it chose the statement shown first more often than the raters did, and on the party duplets it chose the party listed second more often than the expert placements imply. Third, its interface omits low-probability alternatives, so many of its token probabilities are exactly 0, 0.5 or 1, and a score built from them is coarser than GPT-4’s. The scripts, request files and every response are in the replication archive.

还有三点值得注意。第一，GPT-6 Luna 不确定。主体相同的请求发送两次——一次在试点、一次在全量——时，相同答案比例为 95.3% 到 97.1%，而 JEV 重放为 98.6%（附录 B）。表 9 中没有任何区间纳入该变异。第二，它自有位置效应。在专家判断的调查对上，它比评分者更常选先展示的陈述；在政党二元组上，它比专家定位所暗示的更常选后列政党。第三，其接口省略低概率备选，因此许多 token 概率恰好为 0、0.5 或 1，由它们构建的分数比 GPT-4 更粗。脚本、请求文件与每条响应见复现档案。

The democracy indicators were added later. GPT-6 Luna ran on them on 28 September 2026 under the same settings. The last rows of Table 9 give the paired differences and Table 12 its scores. Against JEV, with a paired country bootstrap of 2,000 draws, it trails by 0.05 [0.04, 0.07] in mean per-country correlation and by 2.7 [1.6, 3.8] points in exact agreement. Its probabilities, read from the five alternatives the interface returns for the answer token, put more than 0.99 on the chosen level for 52% of pairs, with a mean modal probability of 0.91 against a hit rate of 52%. Its calibration error is 0.39, higher than JEV’s by 0.33 [0.32, 0.35]. The five alternatives hold almost all of the probability, so the truncation does not explain the gap. JEV’s probabilities also separate right from wrong codes better than GPT-6 Luna’s and Qwen3.8-27B’s (AUROC 0.692 against 0.612 and 0.673; JEV minus Luna 0.080 [0.063, 0.095], JEV minus Qwen 0.019 [0.005, 0.032]). On the survey pairs the two are level in calibration when probabilities are averaged over both presentation orders (Appendix B); in single calls JEV’s calibration error is lower there and on the seven other labelled tasks of Figure 2.

民主指标为后加。GPT-6 Luna 于 2026 年 9 月 28 日在相同设定下运行。表 9 末行给出成对差距，表 12 给出其分数。相对 JEV，以 2,000 次成对国家自助法，其国家平均相关落后 0.05 [0.04，0.07]，精确一致落后 2.7 [1.6，3.8] 个百分点。其概率——从接口为答案 token 返回的五个备选取——在 52% 的对上把超过 0.99 放在所选等级，平均众数概率 0.91，而命中率 52%。校准误差 0.39，高于 JEV 0.33 [0.32，0.35]。五个备选几乎持有全部概率，因此截断不能解释差距。JEV 的概率也比 GPT-6 Luna 与 Qwen3.8-27B 更好地区分对错编码（AUROC 0.692 对 0.612 与 0.673；JEV 减 Luna 0.080 [0.063，0.095]，JEV 减 Qwen 0.019 [0.005，0.032]）。在调查对上，当概率对两种呈现顺序取平均时二者校准持平（附录 B）；在单次调用中，JEV 在那里以及图 2 的另外七项有标签任务上校准误差更低。

An open-weight comparator: Qwen3.8-27B

开源权重对照：Qwen3.8-27B

We also performed all of the tests on an open-weight model, Qwen3.8-27B, on 30 September 2026. We ran the model through OpenRouter, pinned to a single host (Parasail, which serves the model at 8-bit precision, FP8), with reasoning off, a temperature of 0, and the twenty most probable tokens returned for each position in its reply. We sent it the same messages, byte for byte, as in the matching GPT-6 Luna request. All replies came from that host, and the only replies lost, to a host error, were those for one party duplet and one GTD incident. We tabulate the differences from JEV in Table 10, using the same conventions as in Table 9. Qwen is level with or above JEV for four of the seven applications (tweets, sentiment, survey, and manifestos and speeches) and below JEV for three (attack type, party positions and democracy indicators). It scores above JEV for sentiment and, in the 2023 sample, for tweet relevance and problem framing, but below it for solution framing in the 2021 sample. For the seven primary tasks, it cost between 2.35 and 5.41 times as much per decision as JEV, in part because this route does not have a batch rate.

我们还于 2026 年 9 月 30 日在开源权重模型 Qwen3.8-27B 上做了全部检验。经 OpenRouter 运行，固定到单一主机（Parasail，以 8 比特精度 FP8 提供模型），关闭推理，温度 0，并对回复每一位置返回二十个最可能 token。发送与匹配的 GPT-6 Luna 请求逐字节相同的消息。全部回复来自该主机；仅因主机错误丢失的回复是一个政党二元组与一起 GTD 事件。我们在表 10 中按与表 9 相同约定列出相对 JEV 的差距。Qwen 在七项应用中有四项与 JEV 持平或更高（推文、情感、调查，以及党纲与演讲），三项更低（袭击类型、政党位置与民主指标）。情感以及 2023 样本的推文相关性与问题框架上高于 JEV，但 2021 样本的解决方案框架上更低。对七项主任务，每个决策成本为 JEV 的 2.35 到 5.41 倍，部分因为该路由没有批处理费率。

-

Test	Statistic	Qwen	JEV	Difference	Reading
Tweet relevance, 2021	accuracy (%)	88.2	89.0	$-$ 0.8 [ $-$ 2.1, 0.4]	level
Tweet relevance, 2023	accuracy (%)	79.6	76.2	3.4 [0.2, 6.8]	abovea
Problem frame, 2021	accuracy (%)	92.4	92.4	0.0 [ $-$ 1.2, 1.2]	level
Problem frame, 2023	accuracy (%)	69.3	61.4	7.8 [1.2, 14.5]	above
Solution frame, 2021	accuracy (%)	81.4	84.7	$-$ 3.4 [ $-$ 4.4, $-$ 2.3]	below
Solution frame, 2023	accuracy (%)	96.5	98.2	$-$ 1.8 [ $-$ 4.1, 0.0]	level
Tweet sentiment, few-shot	$\rho$ with experts	0.841	0.740	0.102 [0.071, 0.131]	above
GTD attack type, names	accuracy	0.680	0.715	$-$ 0.034 [ $-$ 0.037, $-$ 0.032]	below
GTD attack type, names	macro F1	0.518	0.555	$-$ 0.038 [ $-$ 0.049, $-$ 0.026]	below
GTD attack type, codebook	accuracy	0.689	0.721	$-$ 0.032 [ $-$ 0.035, $-$ 0.029]	below
GTD attack type, codebook	macro F1	0.593	0.638	$-$ 0.045 [ $-$ 0.057, $-$ 0.034]	below
BBC conflict	conflict F1	0.176	0.321	$-$ 0.145 [ $-$ 0.278, $-$ 0.017]	below
Party positions	CHES $r$ , mean of 8 years	0.755	0.877	$-$ 0.154 to $-$ 0.091	below in 8 of 8
Survey, expert pairs	F1	0.782	0.814	$-$ 0.032 [ $-$ 0.070, 0.003]	level
Survey, Study 1 scale	$r$ with GPT-4o	0.942	0.937	0.004 [ $-$ 0.000, 0.009]	level
Survey, calibration	error vs raters	0.049	0.109	$-$ 0.060 [ $-$ 0.091, $-$ 0.009]	below
Manifestos, economic	$r$ with experts	0.974	0.969	0.005 [ $-$ 0.004, 0.022]	level
Manifestos, social	$r$ with experts	0.940	0.918	0.022 [ $-$ 0.005, 0.063]	level
Speeches, coal subsidies	$r$ with crowd	0.905	0.814	0.092 [ $-$ 0.001, 0.186]	level
Democracy indicators	mean country $r$ with V-Dem	0.590	0.680	$-$ 0.090 [ $-$ 0.107, $-$ 0.074]	below
Democracy indicators	exact agreement	0.511	0.543	$-$ 0.032 [ $-$ 0.044, $-$ 0.020]	below
Democracy indicators	calibration error	0.152	0.057	0.095 [0.082, 0.107]	above
a Depends on the bootstrap seed: level under half of 200 other seeds.

-

检验	统计量	Qwen	JEV	差距	解读
推文相关性，2021	准确率 (%)	88.2	89.0	$-$ 0.8 [ $-$ 2.1, 0.4]	持平
推文相关性，2023	准确率 (%)	79.6	76.2	3.4 [0.2, 6.8]	更高a
问题框架，2021	准确率 (%)	92.4	92.4	0.0 [ $-$ 1.2, 1.2]	持平
问题框架，2023	准确率 (%)	69.3	61.4	7.8 [1.2, 14.5]	更高
解决方案框架，2021	准确率 (%)	81.4	84.7	$-$ 3.4 [ $-$ 4.4, $-$ 2.3]	更低
解决方案框架，2023	准确率 (%)	96.5	98.2	$-$ 1.8 [ $-$ 4.1, 0.0]	持平
推文情感，少样本	与专家的 $\rho$	0.841	0.740	0.102 [0.071, 0.131]	更高
GTD 袭击类型，名称	准确率	0.680	0.715	$-$ 0.034 [ $-$ 0.037, $-$ 0.032]	更低
GTD 袭击类型，名称	宏 F1	0.518	0.555	$-$ 0.038 [ $-$ 0.049, $-$ 0.026]	更低
GTD 袭击类型，编码手册	准确率	0.689	0.721	$-$ 0.032 [ $-$ 0.035, $-$ 0.029]	更低
GTD 袭击类型，编码手册	宏 F1	0.593	0.638	$-$ 0.045 [ $-$ 0.057, $-$ 0.034]	更低
BBC 冲突	冲突 F1	0.176	0.321	$-$ 0.145 [ $-$ 0.278, $-$ 0.017]	更低
政党位置	CHES $r$ ，8 年平均	0.755	0.877	$-$ 0.154 到 $-$ 0.091	8/8 更低
调查，专家对	F1	0.782	0.814	$-$ 0.032 [ $-$ 0.070, 0.003]	持平
调查，研究 1 量表	与 GPT-4o 的 $r$	0.942	0.937	0.004 [ $-$ 0.000, 0.009]	持平
调查，校准	相对评分者的误差	0.049	0.109	$-$ 0.060 [ $-$ 0.091, $-$ 0.009]	更低
党纲，经济	与专家的 $r$	0.974	0.969	0.005 [ $-$ 0.004, 0.022]	持平
党纲，社会	与专家的 $r$	0.940	0.918	0.022 [ $-$ 0.005, 0.063]	持平
演讲，煤炭补贴	与众包的 $r$	0.905	0.814	0.092 [ $-$ 0.001, 0.186]	持平
民主指标	与 V-Dem 的国家平均 $r$	0.590	0.680	$-$ 0.090 [ $-$ 0.107, $-$ 0.074]	更低
民主指标	精确一致	0.511	0.543	$-$ 0.032 [ $-$ 0.044, $-$ 0.020]	更低
民主指标	校准误差	0.152	0.057	0.095 [0.082, 0.107]	更高
a 依赖自助法种子：在另外 200 个种子的半数下为持平。

Additional calibration tasks

额外校准任务

Four tasks in Figure 2 (seven of its rows) come from outside the seven replications. We added them because each has a human label and all three models give probabilities. Three use pairwise comparison data that Licht et al. (2025) reuse. In WiscAds (Carlson and Montgomery, 2017), the model reads the text of two television advertisements from the 2008 US Senate elections and picks the more negative one (1,944 pairs). In Immigration (Carlson and Montgomery, 2017), it picks the statement that expresses more fear or worry about the effects of immigration (3,265 pairs). In Park (Park, 2021), it picks the paragraph from a House committee hearing that is more grandstanding and less information-seeking (2,228 pairs). The answer counted as correct is the text with the higher score on the Bradley-Terry scale estimated from the human coders’ comparisons. Each model sees every pair in all four combinations of presentation order and label order, using the prompts of Licht et al. (2025).

图 2 中四项任务（其七行）来自七项复现之外。我们加入它们是因为每项都有人工标签，且三个模型都给出概率。三项使用 Licht 等（2025）复用的成对比较数据。在 WiscAds（Carlson 与 Montgomery，2017）中，模型阅读 2008 年美国参议院选举两则电视广告文本并选出更负向的一则（1,944 对）。在 Immigration（Carlson 与 Montgomery，2017）中，选出对移民效应表达更多恐惧或担忧的陈述（3,265 对）。在 Park（Park，2021）中，选出众议院委员会听证会中更爱作秀、更少求信息的段落（2,228 对）。计为正确的答案是在由人工编码者比较估计的 Bradley-Terry 量表上分数更高的文本。每个模型以呈现顺序与标签顺序的全部四种组合看到每对，使用 Licht 等（2025）的提示。

The fourth task screens manifesto sentences for the European Union. On a random 10% of the Manifesto Project corpus (Lehmann et al., 2025), 121,877 human-coded sentences, each model gives the probability that a sentence concerns the EU, and we score it against the human codes for the two EU categories (108 and 110). EU sentences are rare, so the calibration error is small for every model.

第四项任务筛查涉及欧盟的党纲句子。在 Manifesto Project 语料（Lehmann 等，2025）随机 10%、共 121,877 句经人工编码的句子上，每个模型给出句子涉及欧盟的概率，我们对照两个欧盟类别（108 与 110）的人工编码计分。欧盟句子稀少，因此每个模型的校准误差都很小。

Appendix I: additional tables

附录 I：附加表

Macro F1 Unknown Model Accuracy Nine types Known types recall Fine-tuned on GTD labels (published) ConfliBERT 0.840 0.744 0.775 0.550 ConflLlama, 8-bit 0.765 0.647 0.687 0.387 ConflLlama, 4-bit 0.729 0.571 0.608 0.323 Prompted with the codebook JEV 1.13 0.721 0.638 0.734 0.009 GPT-6 Luna 0.743 0.634 0.725 0.029 Qwen3.5-9B 0.636 0.559 0.634 0.072 Qwen3.8-27B 0.689 0.593 0.683 0.006 Prompted with label names only JEV 1.13 0.715 0.555 0.644 0.002 JEV 1.13, one question per type 0.640 0.413 0.478 0.001 GPT-6 Luna 0.709 0.544 0.631 0.001 Qwen3.5-9B 0.654 0.435 0.503 0.028 Qwen3.8-27B 0.680 0.518 0.599 0.000 Qwen 2.5 14B (published) 0.541 0.397 0.457 0.060 Llama 3.1 8B (published) 0.346 0.208 0.236 0.072 Gemma 2 9B (published) 0.325 0.214 0.230 0.280 Table 11: Attack type of 37,709 GTD incidents, replicating Brandt et al. (2026). Known types: macro F1 over the eight types, on the 33,381 incidents not coded Unknown. Unknown recall: share of the 4,328 Unknown incidents labelled Unknown. GPT-6 Luna was sent Qwen3.5-9B’s prompts, and the open-weight Qwen3.8-27B received GPT-6 Luna’s requests (Appendix H).

		Macro F1	Unknown
Model	Accuracy	Nine types	Known types	recall
Fine-tuned on GTD labels (published)
ConfliBERT	0.840	0.744	0.775	0.550
ConflLlama, 8-bit	0.765	0.647	0.687	0.387
ConflLlama, 4-bit	0.729	0.571	0.608	0.323
Prompted with the codebook
JEV 1.13	0.721	0.638	0.734	0.009
GPT-6 Luna	0.743	0.634	0.725	0.029
Qwen3.5-9B	0.636	0.559	0.634	0.072
Qwen3.8-27B	0.689	0.593	0.683	0.006
Prompted with label names only
JEV 1.13	0.715	0.555	0.644	0.002
JEV 1.13, one question per type	0.640	0.413	0.478	0.001
GPT-6 Luna	0.709	0.544	0.631	0.001
Qwen3.5-9B	0.654	0.435	0.503	0.028
Qwen3.8-27B	0.680	0.518	0.599	0.000
Qwen 2.5 14B (published)	0.541	0.397	0.457	0.060
Llama 3.1 8B (published)	0.346	0.208	0.236	0.072
Gemma 2 9B (published)	0.325	0.214	0.230	0.280

宏 F1 Unknown 模型 准确率 九类 已知类型 召回 在 GTD 标签上微调（已发表） ConfliBERT 0.840 0.744 0.775 0.550 ConflLlama，8-bit 0.765 0.647 0.687 0.387 ConflLlama，4-bit 0.729 0.571 0.608 0.323 用编码手册提示 JEV 1.13 0.721 0.638 0.734 0.009 GPT-6 Luna 0.743 0.634 0.725 0.029 Qwen3.5-9B 0.636 0.559 0.634 0.072 Qwen3.8-27B 0.689 0.593 0.683 0.006 仅用标签名称提示 JEV 1.13 0.715 0.555 0.644 0.002 JEV 1.13，每类型一问 0.640 0.413 0.478 0.001 GPT-6 Luna 0.709 0.544 0.631 0.001 Qwen3.5-9B 0.654 0.435 0.503 0.028 Qwen3.8-27B 0.680 0.518 0.599 0.000 Qwen 2.5 14B（已发表） 0.541 0.397 0.457 0.060 Llama 3.1 8B（已发表） 0.346 0.208 0.236 0.072 Gemma 2 9B（已发表） 0.325 0.214 0.230 0.280 表 11：37,709 起 GTD 事件的袭击类型，复现 Brandt 等（2026）。已知类型：在未编码为 Unknown 的 33,381 起事件上八类的宏 F1。Unknown 召回：4,328 起 Unknown 事件中被标为 Unknown 的份额。向 GPT-6 Luna 发送 Qwen3.5-9B 的提示，开源权重 Qwen3.8-27B 收到 GPT-6 Luna 的请求（附录 H）。

		宏 F1	Unknown
模型	准确率	九类	已知类型	召回
在 GTD 标签上微调（已发表）
ConfliBERT	0.840	0.744	0.775	0.550
ConflLlama，8-bit	0.765	0.647	0.687	0.387
ConflLlama，4-bit	0.729	0.571	0.608	0.323
用编码手册提示
JEV 1.13	0.721	0.638	0.734	0.009
GPT-6 Luna	0.743	0.634	0.725	0.029
Qwen3.5-9B	0.636	0.559	0.634	0.072
Qwen3.8-27B	0.689	0.593	0.683	0.006
仅用标签名称提示
JEV 1.13	0.715	0.555	0.644	0.002
JEV 1.13，每类型一问	0.640	0.413	0.478	0.001
GPT-6 Luna	0.709	0.544	0.631	0.001
Qwen3.5-9B	0.654	0.435	0.503	0.028
Qwen3.8-27B	0.680	0.518	0.599	0.000
Qwen 2.5 14B（已发表）	0.541	0.397	0.457	0.060
Llama 3.1 8B（已发表）	0.346	0.208	0.236	0.072
Gemma 2 9B（已发表）	0.325	0.214	0.230	0.280

-

Coder	Correlation	Difference	Exact	Changed	Within one	Missing
JEV, most probable level	0.68 [0.66, 0.70]	$-$ 0.16	54	36	92	0
JEV, expected level	0.75 [0.73, 0.76]	$-$ 0.20				0
GPT-6 Luna	0.63 [0.61, 0.64]	$-$ 0.09	52	38	91	0
Qwen3.8-27B	0.59 [0.57, 0.61]	$-$ 0.17	51	35	89	0
Qwen3.8-27B, expected level	0.67 [0.66, 0.69]	$-$ 0.18				0
GPT-4o	0.63 [0.62, 0.65]	$-$ 0.28	49	36	90	5
Llama-3.1 70B	0.50 [0.49, 0.52]	$+$ 0.50	46	29	82	129
GPT-4o and Llama, averaged	0.64 [0.62, 0.65]	$+$ 0.11				134

-

编码者	相关	差距	精确	已变	相差不超过一	缺失
JEV，最可能等级	0.68 [0.66, 0.70]	$-$ 0.16	54	36	92	0
JEV，期望等级	0.75 [0.73, 0.76]	$-$ 0.20				0
GPT-6 Luna	0.63 [0.61, 0.64]	$-$ 0.09	52	38	91	0
Qwen3.8-27B	0.59 [0.57, 0.61]	$-$ 0.17	51	35	89	0
Qwen3.8-27B，期望等级	0.67 [0.66, 0.69]	$-$ 0.18				0
GPT-4o	0.63 [0.62, 0.65]	$-$ 0.28	49	36	90	5
Llama-3.1 70B	0.50 [0.49, 0.52]	$+$ 0.50	46	29	82	129
GPT-4o 与 Llama，平均	0.64 [0.62, 0.65]	$+$ 0.11				134

CHES experts TEV voters CMP manifestos Year GPT-3.5 JEV GPT-6 Luna GPT-3.5 JEV GPT-6 Luna GPT-3.5 JEV GPT-6 Luna 1979 — — — 0.75 0.92 [0.79, 0.97] 0.93 0.49 0.57 [0.44, 0.69] 0.51 1984 0.77 0.88 [0.81, 0.92] 0.83 0.78 0.92 [0.84, 0.96] 0.89 0.53 0.59 [0.46, 0.69] 0.56 1989 0.77 0.88 [0.81, 0.92] 0.84 0.74 0.82 [0.65, 0.91] 0.79 0.60 0.59 [0.46, 0.69] 0.59 1994 0.81 0.89 [0.83, 0.93] 0.84 0.84 0.89 [0.79, 0.94] 0.83 0.44 0.52 [0.43, 0.60] 0.50 1999 0.75 0.88 [0.83, 0.91] 0.84 0.77 0.88 [0.81, 0.93] 0.85 0.44 0.49 [0.39, 0.57] 0.48 2004 0.76 0.86 [0.81, 0.89] 0.81 0.68 0.81 [0.72, 0.87] 0.76 0.53 0.55 [0.46, 0.64] 0.59 2009 0.78 0.88 [0.84, 0.91] 0.86 0.58 0.64 [0.48, 0.76] 0.60 0.44 0.45 [0.34, 0.55] 0.52 2014 0.76 0.87 [0.84, 0.90] 0.85 0.59 0.84 [0.74, 0.90] 0.80 0.55 0.55 [0.46, 0.63] 0.58 2019 0.78 0.89 [0.86, 0.92] 0.87 — — — 0.59 0.62 [0.54, 0.69] 0.65 Qwen3.8-27B CSES voters Year CHES TEV CMP JEV GPT-6 Luna Qwen 27B 1979 — 0.86 0.46 — — — 1984 0.79 0.81 0.51 — — — 1989 0.76 0.78 0.50 — — — 1994 0.79 0.82 0.37 — — — 1999 0.78 0.84 0.38 0.84 0.78 0.66 2004 0.70 0.72 0.48 0.80 0.73 0.67 2009 0.74 0.57 0.42 0.82 0.77 0.67 2014 0.73 0.65 0.54 0.76 0.70 0.58 2019 0.75 — 0.57 — — — Table 13: European party positions, replicating Di Leo et al. (2025): Pearson correlation of Bradley-Terry scores with each benchmark by year. CHES: Chapel Hill Expert Survey (before 1999, the Ray-Marks-Steenbergen values); TEV: True European Voter surveys; CMP: Manifesto Project scores. Their GPT-3.5 figures, the modal answer over seven runs, are transcribed from their Figure 1; dashes mark panels they leave empty. JEV’s are a single pass with 95% Fisher intervals; GPT-6 Luna and, in panel b, Qwen3.8-27B are single passes without intervals (Appendix H). Panel b adds voters’ placements in the Comparative Study of Electoral Systems (CSES), available in their files for 1999 to 2014.

	CHES experts	TEV voters	CMP manifestos
Year	GPT-3.5	JEV	GPT-6 Luna	GPT-3.5	JEV	GPT-6 Luna	GPT-3.5	JEV	GPT-6 Luna
1979	—	—	—	0.75	0.92 [0.79, 0.97]	0.93	0.49	0.57 [0.44, 0.69]	0.51
1984	0.77	0.88 [0.81, 0.92]	0.83	0.78	0.92 [0.84, 0.96]	0.89	0.53	0.59 [0.46, 0.69]	0.56
1989	0.77	0.88 [0.81, 0.92]	0.84	0.74	0.82 [0.65, 0.91]	0.79	0.60	0.59 [0.46, 0.69]	0.59
1994	0.81	0.89 [0.83, 0.93]	0.84	0.84	0.89 [0.79, 0.94]	0.83	0.44	0.52 [0.43, 0.60]	0.50
1999	0.75	0.88 [0.83, 0.91]	0.84	0.77	0.88 [0.81, 0.93]	0.85	0.44	0.49 [0.39, 0.57]	0.48
2004	0.76	0.86 [0.81, 0.89]	0.81	0.68	0.81 [0.72, 0.87]	0.76	0.53	0.55 [0.46, 0.64]	0.59
2009	0.78	0.88 [0.84, 0.91]	0.86	0.58	0.64 [0.48, 0.76]	0.60	0.44	0.45 [0.34, 0.55]	0.52
2014	0.76	0.87 [0.84, 0.90]	0.85	0.59	0.84 [0.74, 0.90]	0.80	0.55	0.55 [0.46, 0.63]	0.58
2019	0.78	0.89 [0.86, 0.92]	0.87	—	—	—	0.59	0.62 [0.54, 0.69]	0.65

	Qwen3.8-27B	CSES voters
Year	CHES	TEV	CMP	JEV	GPT-6 Luna	Qwen 27B
1979	—	0.86	0.46	—	—	—
1984	0.79	0.81	0.51	—	—	—
1989	0.76	0.78	0.50	—	—	—
1994	0.79	0.82	0.37	—	—	—
1999	0.78	0.84	0.38	0.84	0.78	0.66
2004	0.70	0.72	0.48	0.80	0.73	0.67
2009	0.74	0.57	0.42	0.82	0.77	0.67
2014	0.73	0.65	0.54	0.76	0.70	0.58
2019	0.75	—	0.57	—	—	—

CHES 专家 TEV 选民 CMP 宣言 年 GPT-3.5 JEV GPT-6 Luna GPT-3.5 JEV GPT-6 Luna GPT-3.5 JEV GPT-6 Luna 1979 — — — 0.75 0.92 [0.79, 0.97] 0.93 0.49 0.57 [0.44, 0.69] 0.51 1984 0.77 0.88 [0.81, 0.92] 0.83 0.78 0.92 [0.84, 0.96] 0.89 0.53 0.59 [0.46, 0.69] 0.56 1989 0.77 0.88 [0.81, 0.92] 0.84 0.74 0.82 [0.65, 0.91] 0.79 0.60 0.59 [0.46, 0.69] 0.59 1994 0.81 0.89 [0.83, 0.93] 0.84 0.84 0.89 [0.79, 0.94] 0.83 0.44 0.52 [0.43, 0.60] 0.50 1999 0.75 0.88 [0.83, 0.91] 0.84 0.77 0.88 [0.81, 0.93] 0.85 0.44 0.49 [0.39, 0.57] 0.48 2004 0.76 0.86 [0.81, 0.89] 0.81 0.68 0.81 [0.72, 0.87] 0.76 0.53 0.55 [0.46, 0.64] 0.59 2009 0.78 0.88 [0.84, 0.91] 0.86 0.58 0.64 [0.48, 0.76] 0.60 0.44 0.45 [0.34, 0.55] 0.52 2014 0.76 0.87 [0.84, 0.90] 0.85 0.59 0.84 [0.74, 0.90] 0.80 0.55 0.55 [0.46, 0.63] 0.58 2019 0.78 0.89 [0.86, 0.92] 0.87 — — — 0.59 0.62 [0.54, 0.69] 0.65 Qwen3.8-27B CSES 选民 年 CHES TEV CMP JEV GPT-6 Luna Qwen 27B 1979 — 0.86 0.46 — — — 1984 0.79 0.81 0.51 — — — 1989 0.76 0.78 0.50 — — — 1994 0.79 0.82 0.37 — — — 1999 0.78 0.84 0.38 0.84 0.78 0.66 2004 0.70 0.72 0.48 0.80 0.73 0.67 2009 0.74 0.57 0.42 0.82 0.77 0.67 2014 0.73 0.65 0.54 0.76 0.70 0.58 2019 0.75 — 0.57 — — — 表 13：欧洲政党位置，复现 Di Leo 等（2025）：Bradley-Terry 分数与各基准按年的 Pearson 相关。CHES：Chapel Hill 专家调查（1999 年前为 Ray-Marks-Steenbergen 值）；TEV：True European Voter 调查；CMP：Manifesto Project 分数。其 GPT-3.5 数字——七次运行的众数答案——转录自其图 1；破折号标记其留空的面板。JEV 为单次通过，含 95% Fisher 区间；GPT-6 Luna 以及 b 面板中的 Qwen3.8-27B 为无区间的单次通过（附录 H）。b 面板加入比较选举制度研究（CSES）中选民定位，其文件中可得于 1999 至 2014。

	CHES 专家	TEV 选民	CMP 宣言
年	GPT-3.5	JEV	GPT-6 Luna	GPT-3.5	JEV	GPT-6 Luna	GPT-3.5	JEV	GPT-6 Luna
1979	—	—	—	0.75	0.92 [0.79, 0.97]	0.93	0.49	0.57 [0.44, 0.69]	0.51
1984	0.77	0.88 [0.81, 0.92]	0.83	0.78	0.92 [0.84, 0.96]	0.89	0.53	0.59 [0.46, 0.69]	0.56
1989	0.77	0.88 [0.81, 0.92]	0.84	0.74	0.82 [0.65, 0.91]	0.79	0.60	0.59 [0.46, 0.69]	0.59
1994	0.81	0.89 [0.83, 0.93]	0.84	0.84	0.89 [0.79, 0.94]	0.83	0.44	0.52 [0.43, 0.60]	0.50
1999	0.75	0.88 [0.83, 0.91]	0.84	0.77	0.88 [0.81, 0.93]	0.85	0.44	0.49 [0.39, 0.57]	0.48
2004	0.76	0.86 [0.81, 0.89]	0.81	0.68	0.81 [0.72, 0.87]	0.76	0.53	0.55 [0.46, 0.64]	0.59
2009	0.78	0.88 [0.84, 0.91]	0.86	0.58	0.64 [0.48, 0.76]	0.60	0.44	0.45 [0.34, 0.55]	0.52
2014	0.76	0.87 [0.84, 0.90]	0.85	0.59	0.84 [0.74, 0.90]	0.80	0.55	0.55 [0.46, 0.63]	0.58
2019	0.78	0.89 [0.86, 0.92]	0.87	—	—	—	0.59	0.62 [0.54, 0.69]	0.65

	Qwen3.8-27B	CSES 选民
年	CHES	TEV	CMP	JEV	GPT-6 Luna	Qwen 27B
1979	—	0.86	0.46	—	—	—
1984	0.79	0.81	0.51	—	—	—
1989	0.76	0.78	0.50	—	—	—
1994	0.79	0.82	0.37	—	—	—
1999	0.78	0.84	0.38	0.84	0.78	0.66
2004	0.70	0.72	0.48	0.80	0.73	0.67
2009	0.74	0.57	0.42	0.82	0.77	0.67
2014	0.73	0.65	0.54	0.76	0.70	0.58
2019	0.75	—	0.57	—	—	—

-

	UK manifestos (experts)	EU speeches (crowd)
	Economic	Social	Subsidies
JEV	0.97	0.92	0.81
JEV, relevance-weighted	0.97	0.92	0.86
GPT-4o (published)	0.98	0.94	0.93
GPT-6 Luna	0.98	0.95	0.95
Qwen3.8-27B	0.97	0.94	0.90
Other published models	0.95 to 0.98	0.88 to 0.95	$-$ 0.52 to 0.86
JEV $-$ GPT-4o	$-$ 0.01 [ $-$ 0.03, 0.00]	$-$ 0.02 [ $-$ 0.05, 0.00]	$-$ 0.12 [ $-$ 0.20, $-$ 0.04]
JEV, weighted $-$ GPT-4o	$-$ 0.01 [ $-$ 0.02, 0.00]	$-$ 0.02 [ $-$ 0.05, 0.00]	$-$ 0.07 [ $-$ 0.15, +0.02]
JEV $-$ GPT-6 Luna	$-$ 0.01 [ $-$ 0.02, $-$ 0.00]	$-$ 0.03 [ $-$ 0.08, $-$ 0.01]	$-$ 0.14 [ $-$ 0.21, $-$ 0.07]
JEV $-$ Qwen3.8-27B	$-$ 0.01 [ $-$ 0.02, 0.00]	$-$ 0.02 [ $-$ 0.06, +0.01]	$-$ 0.09 [ $-$ 0.19, 0.00]
Documents	18	18	36
Sentences	18,263	18,263	429

-

	英国党纲（专家）	欧盟演讲（众包）
	经济	社会	补贴
JEV	0.97	0.92	0.81
JEV，相关性加权	0.97	0.92	0.86
GPT-4o（已发表）	0.98	0.94	0.93
GPT-6 Luna	0.98	0.95	0.95
Qwen3.8-27B	0.97	0.94	0.90
其他已发表模型	0.95 到 0.98	0.88 到 0.95	$-$ 0.52 到 0.86
JEV $-$ GPT-4o	$-$ 0.01 [ $-$ 0.03, 0.00]	$-$ 0.02 [ $-$ 0.05, 0.00]	$-$ 0.12 [ $-$ 0.20, $-$ 0.04]
JEV，加权 $-$ GPT-4o	$-$ 0.01 [ $-$ 0.02, 0.00]	$-$ 0.02 [ $-$ 0.05, 0.00]	$-$ 0.07 [ $-$ 0.15, +0.02]
JEV $-$ GPT-6 Luna	$-$ 0.01 [ $-$ 0.02, $-$ 0.00]	$-$ 0.03 [ $-$ 0.08, $-$ 0.01]	$-$ 0.14 [ $-$ 0.21, $-$ 0.07]
JEV $-$ Qwen3.8-27B	$-$ 0.01 [ $-$ 0.02, 0.00]	$-$ 0.02 [ $-$ 0.06, +0.01]	$-$ 0.09 [ $-$ 0.19, 0.00]
文档数	18	18	36
句子数	18,263	18,263	429

JEV versus LLMs: Accuracy, Cost and Calibration on Seven Political Science Replications

JEV 对比大语言模型：七项政治学复现中的准确率、成本与校准

Abstract

摘要

What JEV returns

JEV 返回什么

How does JEV compare to LLMs and humans?

JEV 与 LLM、人类相比如何？

Tweet relevance

推文相关性

Tweet sentiment

推文情感

Terrorist attack type

恐袭类型

Open-ended survey responses

开放式调查回答

Manifestos and speeches

党纲与演讲

Party positions from party names

从党名推断政党位置

Democracy indicators from country names

从国名推断民主指标

JEV’s advantage?

JEV 的优势？

Conclusion

结论

References

参考文献

Use of AI tools

AI 工具使用说明

Appendix A: prompts as sent

附录 A：实际发送的提示

Study 1: interest-rate knowledge

研究 1：利率知识

Pairwise Choice with a tie option (Choice).

带平局选项的成对 Choice（Choice）。

Pairwise Choice without a tie option (Choice).

无平局选项的成对 Choice（Choice）。

Pairwise Noul (Noul).

成对 Noul（Noul）。

Score, ten levels (Score).

Score，十级（Score）。

Score, five levels (Score).

Score，五级（Score）。

Noul for a single answer (Noul).

单条答案的 Noul（Noul）。

Study 2: debt-ceiling uncertainty

研究 2：债务上限不确定性

Pairwise Choice with a tie option (Choice).

带平局选项的成对 Choice（Choice）。

Pairwise Noul (Noul).

成对 Noul（Noul）。

Score, ten levels (Score).

Score，十级（Score）。

Score, five levels (Score).

Score，五级（Score）。

Noul for a single answer (Noul).

单条答案的 Noul（Noul）。

Gilardi et al.: tweet relevance and framing

Gilardi 等：推文相关性与框架

Relevance (Choice).

相关性（Choice）。

Problem frame (Choice).

问题框架（Choice）。

Solution frame (Choice).

解决方案框架（Choice）。

Ornstein et al.: tweet sentiment

Ornstein 等：推文情感

Sentiment, zero-shot (Choice).

情感，零样本（Choice）。

Sentiment, few-shot (Choice).

情感，少样本（Choice）。

Sentiment, zero-shot (Choice).

情感，零样本（Choice）。

Sentiment, few-shot (Choice).

情感，少样本（Choice）。

Di Leo et al.: party positions

Di Leo 等：政党位置

More right-wing (Choice).

更右翼（Choice）。