Mission: Impossible Language Models

Julie Kallini Email:  kallini@stanford.edu    Isabel Papadimitriou    Richard Futrell    Kyle Mahowald, Christopher Potts Affiliation:  Stanford University; University of California, Irvine; University of Texas, Austin

Abstract

摘要

Chomsky and others have very directly claimed that large language models (LLMs) are equally capable of learning languages that are possible and impossible for humans to learn. However, there is very little published experimental evidence to support such a claim. Here, we develop a set of synthetic impossible languages of differing complexity, each designed by systematically altering English data with unnatural word orders and grammar rules. These languages lie on an impossibility continuum: at one end are languages that are inherently impossible, such as random and irreversible shuffles of English words, and on the other, languages that may not be intuitively impossible but are often considered so in linguistics, particularly those with rules based on counting word positions. We report on a wide range of evaluations to assess the capacity of GPT-2 small models to learn these uncontroversially impossible languages, and crucially, we perform these assessments at various stages throughout training to compare the learning process for each language. Our core finding is that GPT-2 struggles to learn impossible languages when compared to English as a control, challenging the core claim. More importantly, we hope our approach opens up a productive line of inquiry in which different LLM architectures are tested on a variety of impossible languages in an effort to learn more about how LLMs can be used as tools for these cognitive and typological investigations.

Chomsky 等人曾非常直接地主张：大语言模型（LLM）学习人类可能学会的语言与人类不可能学会的语言，能力是一样的。然而，支持这一主张的已发表实验证据极少。本文构造了一组复杂度不同的合成不可能语言：系统性地改写英语数据，引入不自然的词序与语法规则。这些语言落在一条「不可能性连续统」上：一端是固有不可能的语言，例如对英语词做随机且不可逆的打乱；另一端是直觉上未必「不可能」、但在语言学里常被当作不可能的语言，尤其是依赖清点词位置的规则。我们报告一系列评估，衡量 GPT-2 small 学习这些无可争议的不可能语言的能力；更关键的是，我们在训练过程的多个阶段做评估，以比较各语言的学习轨迹。核心发现是：相对英语对照，GPT-2 学习不可能语言明显更吃力，从而挑战了上述核心主张。更重要的是，我们希望这一做法能打开一条可推进的研究路线：在多种不可能语言上检验不同 LLM 架构，借此更好理解如何把 LLM 用作认知与类型学调查的工具。

1 Introduction

1 引言

Figure 1: Partial impossibility continuum of languages based on complexity. We assess the learnability of languages at different points in the continuum and push the (currently unclear) boundary between possible and impossible.

图 1：按复杂度排列的部分不可能性连续统。我们评估连续统不同位置上语言的可学性，并推动（目前仍不清晰的）可能与不可能之间的边界。

Chomsky (2023), Chomsky et al. (2023), Moro et al. (2023), and Bolhuis et al. (2024) make very broad claims to the effect that large language models (LLMs) are equally capable of learning possible and impossible human languages. For these authors, it follows from this claim that LLMs cannot teach us anything about language, and so the claim (if true) would have significant consequences for linguistic methodology and potentially also for the viability of LLMs as the basis for robust language capabilities.

Chomsky (2023)、Chomsky et al. (2023)、Moro et al. (2023) 与 Bolhuis et al. (2024) 提出了相当宽泛的主张：大语言模型（LLM）学习人类意义上「可能」与「不可能」语言的能力是等同的。在这些作者看来，由此可推出 LLM 无法告诉我们任何关于语言的东西；若主张成立，将对语言学方法论、乃至 LLM 作为稳健语言能力基础的可行性产生重大影响。

These authors state this claim in absolute terms. For example, Chomsky et al. (2023) flatly assert that LLMs “are incapable of distinguishing the possible from the impossible,” Chomsky (2023) says this property “can’t be modified,” and Moro et al. (2023) write that “the distinction between possible versus impossible languages cannot be formulated by definition for LLM.” Bolhuis et al. (2024) go so far as to claim that “LLMs can produce ‘impossible’ languages […] just as well as (if not better than) natural language output.” One might expect such strong claims to be supported by extensive formal analysis and/or experimental evidence. However, as far as we are aware, this is not the case. The sole experimental paper cited by the above authors is Mitchell and Bowers 2020—an important and inspiring paper but not one that can resolve these questions on its own. In addition, linguists themselves do not even have an agreed upon notion of what defines the possible or the impossible languages, to say nothing of having formal results with respect to LLMs.

这些作者以绝对口吻陈述该主张。例如，Chomsky et al. (2023) 断然声称 LLM「无力区分可能与不可能」，Chomsky (2023) 说这一性质「无法被修改」，Moro et al. (2023) 则写到「可能语言与不可能语言的区分，按定义就无法对 LLM 加以表述」。Bolhuis et al. (2024) 甚至主张「LLM 可以生成『不可能』语言……其表现与自然语言输出一样好（即便不更好）。」如此强硬的主张，人们理应期待大量形式分析与／或实验证据支撑。但就我们所知，事实并非如此。上述作者引用的唯一实验论文是 Mitchell and Bowers 2020——这篇工作重要且富有启发，但单靠它并不能裁定这些问题。此外，语言学家自己对「什么界定可能／不可能语言」也尚无共识，更谈不上关于 LLM 的形式结论。

Here we provide extensive new experimental evidence to inform the claim that LLMs are equally capable of learning possible and impossible languages in the human sense. Arguably, the central challenge for such work is the fact that there is no agreed-upon way of distinguishing these two groups. We do not feel positioned ourselves to assert such a definition, so we instead offer some examples of impossible languages on a continuum of intuitive complexity (Figure 1). Some of these examples seem intuitively impossible, such as random sentence-level shuffling of English words. Others operationalize less obvious but common claims in the linguistics literature about rules that are impossible, like those that depend on counting words.

本文提供大量新的实验证据，以检验「LLM 学习人类意义上可能与不可能语言的能力相等」这一主张。可以说，此类工作的核心难点在于：两种语言之间并没有公认的划分方式。我们并不打算自行断言一套定义，而是给出若干落在直观复杂度连续统上的不可能语言例子（图 1）。有些例子直觉上就不可能，例如对英语词做句级随机打乱；另一些则把语言学文献中关于「不可能规则」的常见说法操作化，例如依赖清点词数的规则。

All of our examples are, we take it, uncontroversial instances of impossible languages. Thus, our experiments can inform the core hypotheses as follows: if LLMs learn these languages as well as they learn natural languages, then the claims of Chomsky and others are supported (for the specific class of LLMs tested). Conversely, if LLMs do not learn these languages as well as the possible ones, it would call into question those assertions. In that case, proponents of those claims ought to provide examples of impossible languages that they find more informative, which we can then evaluate using our approach to further advance the discussion.

我们认为，所有例子都是无可争议的不可能语言实例。因此，实验可按如下方式支撑核心假设：若 LLM 学习这些语言的表现与自然语言相当，则 Chomsky 等人的主张得到支持（至少对所测这一类 LLM 而言）。反之，若 LLM 学习这些语言不如学习可能语言，则那些断言就值得怀疑。那时，主张的支持者应提供他们认为更有信息量的不可能语言例子；我们再用同一套方法评估，以继续推进讨论。

Our experiments use GPT-2 small models Radford et al. (2018); Radford et al. (2019), and our base training corpus is the BabyLM dataset Warstadt et al. (2023), which we modify in various ways to implement our impossible languages. What we find is that these models indeed struggle to learn impossible languages, shown through three core experiments:

实验使用 GPT-2 small 模型（Radford et al., 2018；Radford et al., 2019），基础训练语料为 BabyLM 数据集（Warstadt et al., 2023），我们以多种方式改写该数据以实现不可能语言。我们发现：这些模型学习不可能语言确实更吃力，体现在三项核心实验中：

• • In Experiment 1, we train GPT-2 models on our set of defined possible and impossible languages, measuring their learning efficiency through test set perplexities. We find that models trained on possible languages learn more efficiently, evident from lower perplexities achieved in fewer training steps.

• • 在实验 1中，我们在所定义的可能与不可能语言上训练 GPT-2，用测试集困惑度衡量学习效率。结果表明：在可能语言上训练的模型学得更高效，表现为用更少训练步数达到更低困惑度。

• • In Experiment 2, we more closely examine a set of languages that exhibit count-based verb marking rules, using surprisal comparisons to target the relevant patterns. We find that GPT-2s trained on possible languages are more surprised by ungrammatical constructions, indicating that models disprefer agreement rules involving counting.

• • 在实验 2中，我们更细致地考察一组含基于计数的动词标记规则的语言，用惊讶度对比锁定相关模式。结果表明：在可能语言上训练的 GPT-2 对不合语法的结构更「惊讶」，说明模型不偏好涉及计数的一致关系规则。

• • In Experiment 3, we dive deeper into the internal mechanisms that models may develop to learn such count-based grammar rules using causal abstraction analysis. We find that models develop natural, modular solutions to unnatural grammatical patterns.

• • 在实验 3中，我们用因果抽象分析深入考察模型可能形成的内部机制，以学习这类基于计数的语法规则。结果表明：模型会为不自然的语法模式发展出自然的、模块化的解法。

Overall, our experimental results strongly challenge the claims of Chomsky and others given above, and we believe they pave the way for even deeper discussions of LLMs as models of language learning. At the same time, we recognize that models and humans exhibit fundamental differences, but the extent to which models favor or disfavor natural languages can be influenced by specific architectural decisions (as demonstrated by our findings on tokenization and positional encodings). We hope this paper initiates a new line of work that explores how different model architectures can distinguish between the possible and impossible languages.[^1]

总体而言，实验结果有力挑战了上文所引 Chomsky 等人的主张，并我们认为它们为把 LLM 当作语言学习模型展开更深讨论铺平了道路。同时我们也承认，模型与人存在根本差异；但模型偏爱或排斥自然语言的程度，会受具体架构选择影响（如我们对分词与位置编码的发现所示）。希望本文开启一条新工作线：探索不同模型架构如何区分可能与不可能语言。[^1]

2 Background and Related Work

2 背景与相关工作

2.1 Impossible Human Languages and Language Universals

2.1 不可能的人类语言与语言共性

The notion of an impossible human language is elusive and difficult to define, in part due to a lack of consensus on which properties are universal in human language and which properties are “impossible” (Comrie, 1989; Evans and Levinson, 2009; Nefdt, 2024). For instance, recursion, or the principle that all languages produce hierarchical syntactic structures via recursive procedures, has been claimed to be a universal property of human language (Chomsky, 1957; Chomsky, 1965; Chomsky, 2002; Hauser et al., 2002). However, the motivations for recursion have been questioned, with empirical limits on the maximum depth of nested phrases Karlsson (2007); Jin et al. (2018) and counterevidence from at least one natural language that seems to lack embedded structures Everett (2012). Still, if we grant that possible languages are defined by hierarchical, recursive rules, what defines the impossible languages? Moro et al. (2023) claim that the class of impossible languages would use the “opposite” type of rules: those based on the linear order of words. Musso et al. (2003) provide a few concrete examples that involve counting word positions to mark features like negation and agreement, and we include languages with similar rules in our set of tested impossible languages.

「不可能的人类语言」这一概念难以捉摸、也难定义，部分原因是：哪些性质在人类语言中是普遍的、哪些性质是「不可能」的，学界并无共识（Comrie, 1989；Evans and Levinson, 2009；Nefdt, 2024）。例如，递归——即所有语言都经由递归程序生成层次句法结构——曾被主张为人类语言的普遍性质（Chomsky, 1957；Chomsky, 1965；Chomsky, 2002；Hauser et al., 2002）。然而，递归的动机受到质疑：嵌套短语最大深度存在经验上限（Karlsson, 2007；Jin et al., 2018），并且至少有一种自然语言似乎缺乏嵌入结构（Everett, 2012）。即便我们承认可能语言由层次化、递归的规则界定，什么又界定不可能语言？Moro et al. (2023) 主张，不可能语言会使用「相反」类型的规则：基于词的线性顺序。Musso et al. (2003) 给出了若干具体例子，涉及清点词位置以标记否定与一致等特征；我们在所测不可能语言中纳入了带有类似规则的语言。

It is important to also distinguish what is impossible from what is merely typologically marked, such as the word order patterns listed in Greenberg’s (1963) language universals. Previous work has shown that such word order universals can arise through a language’s optimization of communication efficiency, achieved by balancing complexity and ambiguity Hahn et al. (2020); Futrell and Hahn (2022). While our current exploration does not encompass attested languages, various impossible languages can similarly differ in their information-theoretic complexity, informing the patterns that lie at the boundary between possible and impossible.

还必须把「不可能」与「仅仅类型学上标记」区分开，后者如 Greenberg (1963) 语言共性中列出的词序模式。已有工作表明，这类词序共性可经由语言对交际效率的优化产生，即在复杂性与歧义之间取得平衡（Hahn et al., 2020；Futrell and Hahn, 2022）。虽然当前探索并不涵盖已证实存在的语言，但各类不可能语言同样可在信息论复杂度上有所不同，从而提示落在可能与不可能边界上的那些模式。

2.2 Training Language Models with Unnatural Word Orders

2.2 用不自然词序训练语言模型

The only work cited by Chomsky that investigates neural language models’ ability to learn impossible languages is Mitchell and Bowers 2020, which finds that recurrent neural networks (RNNs; Elman, 1990) trained on various unnatural language constructs, such as reversed sentences and randomized vocabularies, achieve high accuracy on a subject–verb number agreement task. Other work turns to more recent Transformer-based language models Vaswani et al. (2017), observing their sensitivity to word order and phrase structure Alleman et al. (2021); Galke et al. (2023) as well as their surprising ability to learn from syntactic information alone Huang et al. (2023). Studies by Sinha et al. (2021) and Abdou et al. (2022) debate the impact of tokenization, pretraining adjustments, and positional encodings in recovering word order information from shuffled languages. Further investigations into BERT’s Devlin et al. (2019) reliance on word order for grammatical role classification suggest that lexical cues alone may not always be sufficient for good performance (Papadimitriou et al., 2022; see also Hessel and Schofield, 2021; Pham et al., 2021).

Chomsky 方面引用的、专门考察神经语言模型学习不可能语言能力的工作，只有 Mitchell and Bowers 2020：它发现，在反向句子、随机化词表等多种不自然语言构造上训练的循环神经网络（RNN；Elman, 1990），在主谓数一致任务上仍能取得高准确率。另一些工作转向更近的基于 Transformer 的语言模型（Vaswani et al., 2017），观察它们对词序与短语结构的敏感性（Alleman et al., 2021；Galke et al., 2023），以及仅从句法信息中学习的惊人能力（Huang et al., 2023）。Sinha et al. (2021) 与 Abdou et al. (2022) 讨论了分词、预训练调整与位置编码在从打乱语言中恢复词序信息时的作用。关于 BERT（Devlin et al., 2019）在语法角色分类上对词序依赖的进一步研究则提示：仅靠词汇线索并不总能保证好表现（Papadimitriou et al., 2022；另见 Hessel and Schofield, 2021；Pham et al., 2021）。

2.3 Language Models and Formal Languages

2.3 语言模型与形式语言

A related line of research examines the abilities of neural language models to express formal languages, as defined by the Chomsky hierarchy Chomsky (1956); Chomsky (1959). Human language is considered to be slightly more expressive than context-free languages due to certain syntactic phenomena that interleave constituents Shieber (1985); Joshi (1985). Previous work has shown that RNNs or related models can represent variants of counter and Dyck languages, which are context-free Weiss et al. (2018); Merrill (2019); Merrill et al. (2020); Hewitt et al. (2020).[^2] Similar work on Transformer architectures has shown that, while they are theoretically Turing-complete provided arbitrary precision and decoder steps Pérez et al. (2021), they cannot empirically model many regular and non-regular languages Hahn (2020); Ebrahimi et al. (2020); Deletang et al. (2023).

相关研究方向考察神经语言模型表达形式语言的能力，形式语言由 Chomsky 谱系界定（Chomsky, 1956；Chomsky, 1959）。由于某些把成分交错穿插的句法现象，人类语言被认为比上下文无关语言略强一些（Shieber, 1985；Joshi, 1985）。已有工作表明，RNN 或相关模型可以表示计数语言与 Dyck 语言的变体，这些属于上下文无关语言（Weiss et al., 2018；Merrill, 2019；Merrill et al., 2020；Hewitt et al., 2020）。[^2]对 Transformer 架构的类似工作则表明：尽管在任意精度与足够解码步数下它们理论上是图灵完备的（Pérez et al., 2021），经验上却难以建模许多正则与非正则语言（Hahn, 2020；Ebrahimi et al., 2020；Deletang et al., 2023）。

The inability of Transformer-based language models to learn more complex languages in the Chomsky hierarchy seems surprising, given their impressive performance on natural language. This could be interpreted as evidence that theoretically weak computational models are sufficient for expressing human language. Alternatively, Transformer-based models can be augmented to have inductive biases for nested, hierarchical structures through architecture changes, like the addition of a stack component Hao et al. (2018); Murty et al. (2023), or data-centered approaches, like structural pretraining Papadimitriou and Jurafsky (2023).

基于 Transformer 的语言模型难以学习 Chomsky 谱系中更复杂的语言，这与它们在自然语言上的出色表现形成反差，显得出人意料。一种解读是：理论能力偏弱的计算模型已足以表达人类语言。另一种做法是：通过架构改动（例如加入栈组件，Hao et al., 2018；Murty et al., 2023）或数据侧方法（如结构预训练，Papadimitriou and Jurafsky, 2023），为 Transformer 模型注入对嵌套层次结构的归纳偏置。

3 Impossible Languages

3 不可能语言

Class	Language	Example 1	Example 2
*Shuffle	NoShuffle	He  cleans  his  very  messy  books he lf .	They  clean  his  very  messy  books he lf .
NondeterministicShuffle	messy  books  his he  very . lf He  cleans	his .  very he They  messy lf  books  clean
DeterministicShuffle$(s=21)$	cleans He  messy  books he lf  very .  his	clean They  messy  books he lf  very .  his
DeterministicShuffle$(s=57)$	cleans  his He  messy . he  very lf  books	clean  his They  messy . he  very lf  books
DeterministicShuffle$(s=84)$	He  messy . lf  his  very  books  cleans he	They  messy . lf  his  very  books  clean he
LocalShuffle$(w=3)$	his He  cleans  books  very  messy . he lf	his They  clean  books  very  messy . he lf
	LocalShuffle$(w=5)$	his  messy  very He  cleans lf  books he .	his  messy  very They  clean lf  books he .
	LocalShuffle$(w=10)$	messy  books  his he  very . lf He  cleans	messy  books  his he  very . lf They  clean
	EvenOddShuffle	He  his  messy he .  cleans  very  books lf	They  his  messy he .  clean  very  books lf
*Reverse	NoReverse	He  cleans  his  very  messy  books R he lf .	They  clean  his R  very  messy  books he lf .
PartialReverse	He  cleans  his  very  messy  books R . lf he	They  clean  his R . lf he  books  messy  very
	FullReverse	. lf he R  books  messy  very  his  cleans He	. lf he  books  messy  very R  his  clean They
*Hop	NoHop	He  clean S  his  very  messy  books he lf .	They  clean P  his  very  messy  books he lf .
TokenHop	He  clean  his  very  messy  books S he lf .	They  clean  his  very  messy  books P he lf .
	WordHop	He  clean  his  very  messy  books he lf S .	They  clean  his  very  messy  books he lf P .

Table 1: List of impossible languages with examples. Control (‘No*’) languages have patterns that resemble English. Differently colored blocks represent different GPT-2 tokens.

类别	语言	示例 1	示例 2
*Shuffle	NoShuffle	He  cleans  his  very  messy  books he lf .	They  clean  his  very  messy  books he lf .
NondeterministicShuffle	messy  books  his he  very . lf He  cleans	his .  very he They  messy lf  books  clean
DeterministicShuffle$(s=21)$	cleans He  messy  books he lf  very .  his	clean They  messy  books he lf  very .  his
DeterministicShuffle$(s=57)$	cleans  his He  messy . he  very lf  books	clean  his They  messy . he  very lf  books
DeterministicShuffle$(s=84)$	He  messy . lf  his  very  books  cleans he	They  messy . lf  his  very  books  clean he
LocalShuffle$(w=3)$	his He  cleans  books  very  messy . he lf	his They  clean  books  very  messy . he lf
	LocalShuffle$(w=5)$	his  messy  very He  cleans lf  books he .	his  messy  very They  clean lf  books he .
	LocalShuffle$(w=10)$	messy  books  his he  very . lf He  cleans	messy  books  his he  very . lf They  clean
	EvenOddShuffle	He  his  messy he .  cleans  very  books lf	They  his  messy he .  clean  very  books lf
*Reverse	NoReverse	He  cleans  his  very  messy  books R he lf .	They  clean  his R  very  messy  books he lf .
PartialReverse	He  cleans  his  very  messy  books R . lf he	They  clean  his R . lf he  books  messy  very
	FullReverse	. lf he R  books  messy  very  his  cleans He	. lf he  books  messy  very R  his  clean They
*Hop	NoHop	He  clean S  his  very  messy  books he lf .	They  clean P  his  very  messy  books he lf .
TokenHop	He  clean  his  very  messy  books S he lf .	They  clean  his  very  messy  books P he lf .
	WordHop	He  clean  his  very  messy  books he lf S .	They  clean  his  very  messy  books he lf P .

表 1：不可能语言一览及示例。对照（「No*」）语言的模式接近英语。不同颜色块表示不同的 GPT-2 token。

Core to our experiments are the set of impossible languages we synthesize. In constructing these artificial counterfactual languages, we consider their information-theoretic attributes relevant to machine learning, such as entropy rate, as well as their formal linguistic characteristics, such as adherence to hierarchical grammatical structures. We believe that our choice of languages broadly spans the impossibility continuum hypothesized in Figure 1.

实验的核心是我们合成的一组 不可能语言。构造这些人工反事实语言时，我们既考虑与机器学习相关的信息论属性（如熵率），也考虑形式语言学特征（如是否遵守层次语法结构）。我们认为，所选语言大体覆盖了图 1 所假设的不可能性连续统。

Concretely, we specify impossible languages by defining perturbation functions of English sentences. These perturbation functions map English input sentences to sequences of tokens. We categorize our languages into three classes: *Shuffle, *Reverse, and *Hop, defined in the next subsections. Each class has one control language that represents unaltered English, or a pattern that is very similar to English. Table 1 provides examples of perturbed sentences in each language.

具体而言，我们通过对英语句子定义 扰动函数 来规定不可能语言。这些扰动函数把英语输入句映射为 token 序列。语言分为三类：*Shuffle、*Reverse 与 *Hop，见以下各小节。每一类都有一种对照语言，代表未改动的英语，或与英语非常接近的模式。表 1 给出了各语言扰动句子的示例。

3.1 *Shuffle Languages.

3.1 *Shuffle 语言。

The first set of impossible languages, which we call the *Shuffle languages, involve different shuffles of tokenized English sentences.

第一组不可能语言，我们称为 *Shuffle 语言，对分词后的英语句子做不同形式的打乱。

1. NoShuffle: The input sentence is tokenized, and the token sequence is unaltered. This language is simply English, used for comparison with other *Shuffle languages.

1. NoShuffle：输入句经分词后，token 序列保持不变。该语言就是英语本身，用于与其他 *Shuffle 语言对照。

2. NondeterministicShuffle: The tokenized input sentence is randomly shuffled. A different random shuffle is used for each input sentence, with no consistency across inputs.

2. NondeterministicShuffle：对分词后的输入句做随机打乱。每个输入句使用不同的随机打乱，跨输入之间没有一致性。

3. DeterministicShuffle($s$): The tokenized input sentence is deterministically shuffled based on the length of the token sequence. For example, all token sequences of length 5 are shuffled in the same order. We create several languages by varying the random seed $s$ that produces the shuffle.

3. DeterministicShuffle($s$)：对分词后的输入句按 token 序列长度做确定性打乱。例如，所有长度为 5 的 token 序列以同一顺序打乱。我们通过改变产生打乱的随机种子 $s$，构造若干种此类语言。

4. LocalShuffle($w$): The tokenized input sentence is deterministically shuffled in local windows of a fixed size $w$. We create several languages by varying $w$.

4. LocalShuffle($w$)：在固定大小为 $w$ 的局部窗口内，对分词后的输入句做确定性打乱。我们通过改变 $w$ 构造若干种此类语言。

5. EvenOddShuffle: The tokenized input sentence is reordered such that all even-indexed tokens appear first, followed by all odd-indexed tokens.

5. EvenOddShuffle：重排分词后的输入句，使全部偶数下标 token 先出现，随后是全部奇数下标 token。

The random shuffling function that generates the NondeterministicShuffle language is irreversible, resulting in sentences that are purely bags of words—any structural information in the original linguistic signal is irretrievable. While the DeterministicShuffle languages are created using a reversible perturbation function, this function operates in an entirely non-linguistic manner; words are ordered based solely on the random seed and sentence length, without considerations for linguistic features or information locality—the property that, when parts of text predict each other, they are often close together (Futrell, 2019; Mansfield and Kemp, 2023). This method is arguably even less humanly feasible than NondeterministicShuffle, as it relies on an arbitrarily complex yet consistent rule to determine word order.[^3] The question of ranking these two families of languages in the impossibility continuum probes at the definition of impossibility and whether reversibility to an attested language like English is a relevant quantity.

生成 NondeterministicShuffle 语言的随机打乱函数不可逆，句子退化为纯粹的词袋——原语言信号中的任何结构信息都无法找回。DeterministicShuffle 语言虽由可逆扰动函数生成，但该函数完全非语言学：词序仅由随机种子与句子长度决定，不考虑语言特征或 信息局部性——即当文本片段彼此可预测时，它们往往彼此靠近这一性质（Futrell, 2019；Mansfield and Kemp, 2023）。这种方法可以说比 NondeterministicShuffle 更不适合人类：它依赖一条任意复杂却前后一致的规则来决定词序。[^3]如何在不可能性连续统上对这两族语言排序，触及「不可能性」的定义本身，以及「能否可逆地回到像英语这样的已证实语言」是否是一个相关量。

The LocalShuffle languages offer a finer-grained testbed for the importance of information locality, since we can observe the effects of different window sizes. Finally, EvenOddShuffle also manipulates locality, but interestingly preserves part of the linear word order of English while introducing new long-distance dependencies.

LocalShuffle 语言为信息局部性的重要性提供了更细粒度的试验场，因为我们可以观察不同窗口大小的效应。最后，EvenOddShuffle 同样操纵局部性，但有趣的是：它保留了英语线性词序的一部分，同时引入了新的长距离依赖。

3.2 *Reverse Languages.

3.2 *Reverse 语言。

The *Reverse impossible languages involve reversals of all or part of input sentences.

*Reverse 不可能语言对输入句的全部或部分做反转。

1. NoReverse: The input sentence is tokenized, and a special marker token R is inserted at a random position in the token list. Like NoShuffle, this language is most similar to English. We use it for comparison with other *Reverse languages.

1. NoReverse：输入句经分词后，在 token 列表的随机位置插入特殊标记 token R。与 NoShuffle 一样，该语言最接近英语。我们用它与其他 *Reverse 语言对照。

2. PartialReverse: The input sentence is tokenized, a special marker token R is inserted at a random position in the list of tokens, and the following tokens are reversed.

2. PartialReverse：输入句经分词后，在 token 列表的随机位置插入特殊标记 token R，并反转其后的 token。

3. FullReverse: The input sentence is tokenized, a special marker token R is inserted at a random position in the token list, and all tokens are reversed.

3. FullReverse：输入句经分词后，在 token 列表的随机位置插入特殊标记 token R，并反转全部 token。

The PartialReverse language is inspired by the experiments of Mitchell and Bowers (2020) on partially reversed English data, though our experiments are not a direct replication, since we use a different model architecture and dataset. FullReverse may seem like a plausible language syntactically, but higher-level linguistic concepts like anaphora would be highly disrupted. The R tokens are placed at the same positions across the data in all *Reverse languages to control for the entropy introduced by their random placement.

PartialReverse 语言受 Mitchell and Bowers (2020) 在部分反转英语数据上实验的启发；不过我们的实验并非直接复现，因为所用模型架构与数据集不同。FullReverse 在句法上看似可能说得通，但回指等更高层语言概念会被严重打乱。所有 *Reverse 语言中，R token 的位置在数据间保持一致，以控制其随机放置所引入的熵。

3.3 *Hop Languages.

3.3 *Hop 语言。

The *Hop languages perturb verb inflection with counting rules.

*Hop 语言用计数规则扰动动词屈折。

1. NoHop: All 3rd-person present tense verbs in the input sentence are lemmatized, and the sentence is tokenized. For each 3rd-person present tense verb, a special marker representing the verb’s number and tense is placed right after the lemmatized verb. Singular verbs are marked with a special token S, and plural verbs are marked with P. Like the other control languages, NoHop has a pattern that is most similar to English.

1. NoHop：将输入句中所有第三人称现在时动词还原为词元，再分词。对每个第三人称现在时动词，在词元化动词正后方放置表示该动词数与时态的特殊标记。单数动词标为特殊 token S，复数标为 P。与其他对照语言一样，NoHop 的模式最接近英语。

2. TokenHop: Identical transformation to NoHop, but the special number/tense markers are placed 4 tokens after the verb.

2. TokenHop：变换与 NoHop 相同，但特殊的数／时态标记放在动词之后第 4 个 token 处。

3. WordHop: Identical transformation to NoHop and TokenHop, but the special number/tense markers are placed 4 words after the verb, skipping punctuation.

3. WordHop：变换与 NoHop、TokenHop 相同，但特殊的数／时态标记放在动词之后第 4 个词处，并跳过标点。

These languages specifically investigate GPT-2’s ability to learn grammar rules that involve counting the positions of words or tokens.

这些语言专门考察 GPT-2 学习涉及清点词或 token 位置的语法规则的能力。

4 Experiments

4 实验

Figure 2: Perplexities on a sample of 10K test sentences for each impossible language model over training steps. Error bars indicate 95% confidence intervals across 5 training runs initialized with different random seeds and evaluated on different test samples.

图 2：各不可能语言模型在 1 万条测试句样本上、随训练步数变化的困惑度。误差条表示 5 次以不同随机种子初始化、并在不同测试样本上评估的训练运行的 95% 置信区间。

We run several experiments to assess GPT-2’s learning of our impossible languages. Our first experiment (Section 4.2) uses perplexities as a general evaluation to compare how well each impossible language model has learned its own perturbed language and see whether this reflects the hypothesized impossibility continuum. In our second and third experiments, we conduct a closer examination of the *Hop languages. Given that their count-based verb marking rules appear to be the least clearly implausible among our proposed languages, we focus on examining these rules specifically through targeted assessments using surprisal theory (Section 4.3). Finally, we dive deeper into the mechanisms each *Hop model uses to predict their respective verb marking rules using causal abstraction analysis (Section 4.4). For all evaluations, we run tests on several model checkpoints to observe the learning process over intervals of training steps.[^4]

我们运行多项实验，评估 GPT-2 对我们不可能语言的学习。第一项实验（第 4.2 节）用困惑度做总体评估，比较各不可能语言模型对自己扰动语言的掌握程度，并看其是否反映所假设的不可能性连续统。第二、三项实验更细致地考察 *Hop 语言。鉴于其基于计数的动词标记规则在我们提出的语言中「不可能性」最不鲜明，我们通过惊讶度理论做针对性评估（第 4.3 节）。最后，我们用因果抽象分析深入考察各 *Hop 模型预测各自动词标记规则时所使用的机制（第 4.4 节）。所有评估都在多个模型检查点上运行，以观察训练步数间隔上的学习过程。[^4]

4.1 Implementation Details

4.1 实现细节

For each impossible language, we apply its perturbation function to each sentence of the BabyLM dataset Warstadt et al. (2023) to create a transformed dataset. Appendix A provides details on preprocessing and formatting, and describes the language-specific filtering needed to achieve the criteria that define each language.

对每种不可能语言，我们将其扰动函数作用于 BabyLM 数据集（Warstadt et al., 2023）的每一句，以生成变换后的数据集。附录 A 给出预处理与格式细节，并描述为实现各语言定义标准所需的语言特定过滤。

We train standard GPT-2 small models Radford et al. (2018); Radford et al. (2019) on each impossible language. To produce confidence intervals for our experiments, we train 5 sets of models for each language using different random seeds, which affect the model parameter initialization and dataset shuffling during training. Training and model hyperparameter choices are detailed in Appendix B. The primary set of GPT-2 models we train have absolute positional encodings. We also train a set of GPT-2 small models with an architecture in which the positional encodings are removed, so that the models’ only notion of word order is derived from GPT-2’s causal language modeling learning objective Kazemnejad et al. (2023). Results for these additional experiments supported our main findings on the unaltered GPT-2 architecture. These results are provided in Appendix C.

我们在每种不可能语言上训练标准 GPT-2 small 模型（Radford et al., 2018；Radford et al., 2019）。为给出实验置信区间，我们对每种语言用不同随机种子训练 5 组模型；随机种子影响模型参数初始化与训练时的数据集打乱。训练与模型超参见附录 B。主实验所用 GPT-2 模型带绝对位置编码。我们还训练了一组去掉位置编码的 GPT-2 small：模型对词序的唯一感知来自 GPT-2 的因果语言建模目标（Kazemnejad et al., 2023）。这些附加实验的结果支持我们对未改动 GPT-2 架构的主要发现，见附录 C。

4.2 Experiment 1: Language Models Reflect the Impossibility Continuum

4.2 实验 1：语言模型反映不可能性连续统

We train GPT-2 models on all of the languages described in Table 1, and evaluate each model’s perplexities on a test set over the course of training. Test perplexities provide a general metric for the extent to which a model has learned a language.

我们在表 1 所述全部语言上训练 GPT-2，并在训练过程中评估各模型在测试集上的困惑度。测试困惑度提供了模型掌握一门语言程度的总体指标。

Setup.

We sample 10K sentences from the BabyLM test set and perturb this sample for each impossible language. For a given impossible language model, we report the geometric mean of the individual sentence perplexities in the corresponding test sample.

设置。

我们从 BabyLM 测试集抽样 1 万句，并对每种不可能语言做相应扰动。对给定的不可能语言模型，我们报告对应测试样本中各句困惑度的几何平均。

Hypothesis.

Models trained on possible languages will achieve lower average perplexities more quickly (as measured in training steps) than those trained on impossible languages.

假设。

在可能语言上训练的模型，将比在不可能语言上训练的模型更快（以训练步数计）达到更低的平均困惑度。

Results.

Our results are in Figure 2. There are clear distinctions between model perplexities after about 500 training steps. First considering the *Shuffle models, the NondeterministicShuffle model has the highest perplexities, followed by the three DeterministicShuffle models, indicating that GPT-2 is better at learning shuffling patterns when they are deterministic, invertible functions.[^5] The prevalence of certain sentence lengths in the corpus could also limit the variety of sentence shuffles in the DeterministicShuffle languages, potentially resulting in similarly functioning words frequently occupying the same token positions, thus increasing their predictability.

结果。

结果见图 2。约 500 个训练步之后，各模型困惑度出现清晰分野。先看 *Shuffle 模型：NondeterministicShuffle 困惑度最高，随后是三种 DeterministicShuffle，说明当打乱模式是确定性、可逆函数时，GPT-2 学得更好。[^5]语料中某些句子长度占优，也可能限制 DeterministicShuffle 语言中句级打乱的多样性，使功能相近的词经常占据相同 token 位置，从而提高可预测性。

Following the sentence-level shuffles, the next models in the order of decreasing perplexity are the three LocalShuffle models, with smaller window sizes having lower perplexities. LocalShuffle$(w=3)$ and EvenOddShuffle have perplexities closest to the NoShuffle model (which represents unaltered English), but NoShuffle consistently has the lowest perplexities throughout the training process.

在句级打乱之后，按困惑度递减，接下来是三种 LocalShuffle 模型，窗口越小困惑度越低。LocalShuffle$(w=3)$ 与 EvenOddShuffle 的困惑度最接近 NoShuffle 模型（代表未改动英语），但整个训练过程中 NoShuffle 始终困惑度最低。

Compared to the *Shuffle models, the experimental *Reverse models have perplexities that are much closer to the NoReverse model, and PartialReverse is slightly better than FullReverse. For the *Hop languages, their respective control model again has the lowest perplexities, although differences among the models are quite minimal. This warrants our deep-dive into the particular verb marking patterns for this set of models.

相对 *Shuffle 模型，实验性 *Reverse 模型的困惑度更接近 NoReverse，且 PartialReverse 略优于 FullReverse。对 *Hop 语言，各自的对照模型同样困惑度最低，但模型间差异相当小。这促使我们深入考察这组模型具体的动词标记模式。

4.3 Experiment 2: Language Models Disprefer Counting Rules

4.3 实验 2：语言模型不偏好计数规则

(a) Test 1: mean surprisals of the verb marker token (S or P) for each *Hop model.

(a) 测试 1：各 *Hop 模型上动词标记 token（S 或 P）的平均惊讶度。

In Experiment 1, we show that impossible languages are harder for GPT-2 to learn. However, perplexity is a coarse-grained metric of language learning, and the question remains: do language models learn natural grammatical structures better than impossible grammars?

实验 1 表明不可能语言对 GPT-2 更难学。但困惑度是粗粒度指标；问题仍在：语言模型是否比学习不可能语法更好地学会自然语法结构？

The structure of the *Hop languages invites a finer-grained evaluation of their verb marking rules. We use surprisals to measure how well each *Hop model can predict the placement of its verb marker tokens, S and P. The surprisal $S(w_{i})$ of a word $w_{i}$ is the negative log probability of $w_{i}$ given the context words $w_{1},\ldots,w_{i-1}$ that precede it: $S(w_{i})=-\log_{2}p(w_{i}|w_{1},\ldots,w_{i-1})$. Surprisals have been used as acceptability judgments from neural language models to probe for their processing of syntactic information Wilcox et al. (2018); Futrell et al. (2019); Hu et al. (2020); Wilcox et al. (2023) and have been shown to correlate with human sentence processing difficulty Hale (2001); Levy (2008).

Hop 语言的结构邀请对其动词标记规则做更细粒度评估。我们用惊讶度*衡量各 *Hop 模型预测其动词标记 token S 与 P 放置位置的好坏。词 $w_{i}$ 的惊讶度 $S(w_{i})$ 是给定其前导上下文词 $w_{1},\ldots,w_{i-1}$ 时 $w_{i}$ 的负对数概率：$S(w_{i})=-\log_{2}p(w_{i}|w_{1},\ldots,w_{i-1})$。惊讶度已被用作神经语言模型的可接受性判断，以探测其对句法信息的加工（Wilcox et al., 2018；Futrell et al., 2019；Hu et al., 2020；Wilcox et al., 2023），并被证明与人类句子加工难度相关（Hale, 2001；Levy, 2008）。

Setup.

To test the *Hop models’ sensitivity to marker placement, we conduct two tests on a sample of 10K sentences extracted from the BabyLM dataset containing the verb marker tokens (S or P). As an example, consider the following pair of sentences for the NoHop language shown in 4.3.

设置。

为检验 *Hop 模型对标记放置的敏感性，我们在 BabyLM 数据集抽取的、含动词标记 token（S 或 P）的 1 万句样本上做两项测试。作为例子，考虑 4.3 中所示 NoHop 语言的下列句子对。

. ˙[1]He  clean S  his  very  messy  books he lf . .̱ *He  clean__ his  very  messy  books he lf .

Sentence 4.3 is an example in the NoHop language, and 4.3 is an ungrammatical counterfactual in which the marker token does not appear.

句子 4.3 是 NoHop 语言中的例子，而 4.3 是标记 token 未出现的不合语法反事实。

In the first test, we compare the average surprisals of the marker tokens across the three *Hop languages, using grammatical examples like 4.3. In the case of 4.3, the marker is singular, and its surprisal $S(\texttt{\,\lx@texthl@color{\framebox{{S}}}\,})$ is defined as:

第一项测试比较三种 *Hop 语言上标记 token 的平均惊讶度，使用如 4.3 这类合语法例子。在 4.3 中，标记为单数，其惊讶度 $S(\texttt{\,\lx@texthl@color{\framebox{{S}}}\,})$ 定义为：

$$S(\texttt{\,\lx@texthl@color{\framebox{{S}}}\,})=-\log_{2}p(\texttt{\,\lx@texthl@color{\framebox{{S}}}\,$|$\,\lx@texthl@color{He}\,\lx@texthl@color{ clean}\,})$$

We average this surprisal value for instances of S or P in the test sample.

我们对测试样本中 S 或 P 的实例平均该惊讶度。

In the second test, we construct minimal pairs from the example sentences in which the marker token appears and does not appear, and then compare the surprisal of the marker token to the surprisal of the token that follows it, both conditioned on the same context. In example 4.3, the surprisal of the following token $S(\texttt{\,\lx@texthl@color{ his}\,})$ is defined as:

第二项测试由示例句构造最小对：标记 token 出现与不出现，然后比较标记 token 的惊讶度与紧随其后 token 的惊讶度，二者条件于同一上下文。在例子 4.3 中，后续 token 的惊讶度 $S(\texttt{\,\lx@texthl@color{ his}\,})$ 定义为：

$$S(\texttt{\,\lx@texthl@color{ his}\,})=-\log_{2}p(\texttt{\,\lx@texthl@color{ his}\,$|$\,\lx@texthl@color{He}\,\lx@texthl@color{ clean}\,})$$

We expect $S(\texttt{\,\lx@texthl@color{ his}\,})-S(\texttt{\,\lx@texthl@color{\framebox{{S}}}\,})$ to be a large positive value. We average such surprisal differences over instances of the marker tokens in the test sample and similarly define marker surprisals and minimal pair configurations for the other *Hop languages.

我们期望 $S(\texttt{\,\lx@texthl@color{ his}\,})-S(\texttt{\,\lx@texthl@color{\framebox{{S}}}\,})$ 是较大的正值。我们对测试样本中标记 token 的实例平均此类惊讶度差，并对其他 *Hop 语言类似地定义标记惊讶度与最小对配置。

Hypothesis.

For the first surprisal test, our hypothesis is that the mean surprisal of the marker tokens across test examples will be smaller for the control language than for the impossible languages. For the second test, our hypothesis is that the mean surprisal difference across all test pairs will be larger for possible languages than for impossible ones.

假设。

对第一项惊讶度测试，我们假设：跨测试例子，对照语言上标记 token 的平均惊讶度小于不可能语言。对第二项测试，我们假设：跨全部测试对，可能语言的平均惊讶度差大于不可能语言。

Results.

Our results are presented in Figure 3. The NoHop model, which has the verb marking pattern most similar to English, consistently has the lowest mean marker surprisal across training steps in test 1 (Figure 3(a)). The NoHop model also has the highest mean surprisal difference across training steps in test 2 (Figure 3(b)). Both of these results indicate that GPT-2 has learned to expect the marker tokens when they follow a more natural grammatical pattern and was very surprised when they did not appear at the correct positions.

结果。

结果见图 3。动词标记模式最接近英语的 NoHop 模型，在测试 1 中跨训练步数始终具有最低的平均标记惊讶度（图 3(a)）。NoHop 模型在测试 2 中跨训练步数也具有最高的平均惊讶度差（图 3(b)）。这两项结果都表明：当标记 token 遵循更自然的语法模式时，GPT-2 已学会期待它们；当它们未出现在正确位置时，模型非常「惊讶」。

GPT-2 learns to expect marker tokens at the right locations in the other *Hop models, just not as well as the control. TokenHop tends to have a lower marker surprisal and a higher mean surprisal difference compared to WordHop across training steps, indicating that GPT-2 is better at learning the verb marking rule when the units being counted are tokens instead of words.

在其他 *Hop 模型中，GPT-2 也会学会在正确位置期待标记 token，只是不如对照好。跨训练步数，TokenHop 往往比 WordHop 有更低的标记惊讶度与更高的平均惊讶度差，说明当被计数的单位是 token 而非词时，GPT-2 更容易学会动词标记规则。

4.4 Experiment 3: Language Models Develop Natural Solutions to Unnatural Patterns

4.4 实验 3：语言模型为不自然模式发展出自然解法

Experiment 2 demonstrates that, while GPT-2 favors natural grammar rules, it is also capable of acquiring count-based grammar rules like those seen in the verb marking patterns of our *Hop languages. But what sorts of internal mechanisms does it implement to learn such grammar rules, and how do these mechanisms compare to the more natural control? To address this, we conduct a final experiment using causal abstraction analysis, which offers an interpretability framework for identifying and examining causal mechanisms within neural models Geiger et al. (2020); Geiger et al. (2021); Wu et al. (2022); Wu et al. (2023a); Wu et al. (2023b); Geiger et al. (2023). We employ the interchange intervention technique on our *Hop models. To perform a basic interchange intervention on a neural model $M$, we create two instances of $M$ that are provided two different inputs, the base input $b$ and the source input $s$. Then, we interchange representations created while processing $b$ with representations created while processing $s$ and observe the effect on the output of $M$. Such interventions allow us to piece together a causal understanding of how the model processes inputs.

实验 2 表明：虽然 GPT-2 偏爱自然语法规则，它也能习得如 Hop 语言动词标记模式那样的基于计数的语法规则。但为学会这类语法规则，它实现了怎样的内部机制，又如何与更自然的对照比较？为此，我们做最后一项实验，使用因果抽象分析*——这是一套识别并检验神经模型内部因果机制的可解释性框架（Geiger et al., 2020；Geiger et al., 2021；Wu et al., 2022；Wu et al., 2023a；Wu et al., 2023b；Geiger et al., 2023）。我们在 Hop 模型上采用互换干预*技术。对神经模型 $M$ 做基本互换干预时，创建 $M$ 的两个实例，分别给定两个不同输入：基输入 $b$ 与源输入 $s$。然后，用处理 $s$ 时产生的表征去互换处理 $b$ 时产生的表征，并观察对 $M$ 输出的影响。此类干预让我们拼合出模型如何处理输入的因果图景。

Setup.

Figure 4: An interchange intervention on the NoHop model with base input $b=\texttt{The man be}$ and source input $s=\texttt{The men be}$. The intervention is performed at the second layer and second token position, causing a change in prediction from S to P.

设置。

图 4：对 NoHop 模型的一次互换干预，基输入 $b=\texttt{The man be}$，源输入 $s=\texttt{The men be}$。干预在第二层、第二个 token 位置执行，使预测由 S 变为 P。

Figure 5: Subject–verb agreement interchange intervention accuracies (IIA) for each *Hop model over training steps. Vertical axes denote the GPT-2 layer of the intervention, and horizontal axes denote the token position of the intervention. $t_{d}$, $t_{s}$, and $t_{v}$ represent the tokens for the determiner, subject, and verb, respectively. $t_{1}\dots t_{4}$ represent the four tokens/words between the verb and its marker for TokenHop and WordHop. IIA values are averaged over results from 5 models initialized on different random seeds. See Appendix F for confidence intervals.

图 5：各 *Hop 模型随训练步数变化的主谓一致互换干预准确率（IIA）。纵轴表示干预所在的 GPT-2 层，横轴表示干预所在的 token 位置。$t_{d}$、$t_{s}$、$t_{v}$ 分别表示限定词、主语与动词的 token。$t_{1}\dots t_{4}$ 表示 TokenHop 与 WordHop 中动词与其标记之间的四个 token／词。IIA 值对以不同随机种子初始化的 5 个模型结果取平均。置信区间见附录 F。

We use interchange interventions to identify representations in our *Hop models that have causal effects on their output behaviors on a subject–verb agreement task. In our experimental setup, $b$ is a sentence prefix with a singular subject and $s$ is an identical prefix with the plural form of the subject. These prefixes include all tokens up to but not including the markers (S and P). We interchange the GPT-2 block outputs from processing $b$ with GPT-2 block outputs from processing $s$ and observe whether the probability of plural marker P is higher than the probability of singular marker S after the intervention. This is shown more concretely in Figure 4.

我们用互换干预识别 *Hop 模型中对主谓一致任务输出行为有因果效应的表征。实验设置中，$b$ 是带单数主语的句子前缀，$s$ 是将主语改为复数形式后的相同前缀。这些前缀包含截至标记（S 与 P）之前的全部 token。我们互换处理 $b$ 与处理 $s$ 时得到的 GPT-2 块输出，并观察干预后复数标记 P 的概率是否高于单数标记 S。图 4 给出更具体的示意。

We run such interventions at each GPT-2 layer and token position to see which parts of the model cause a change in the marker prediction. We run all of these interventions over several test examples and report the interchange intervention accuracy (IIA), a metric that represents the subject–verb agreement accuracy if the counterfactual (i.e. plural) were the ground truth. The test examples for each *Hop model are extracted from their respective versions of the BabyLM test set, and minimally-different counterfactual examples are created by changing the singular subjects to plural subjects. To ensure that interventions on different examples are analogous, we use regular expressions to locate examples that follow the same structure (i.e. subjects and verbs at the same positions).

我们在每一 GPT-2 层与每一 token 位置运行此类干预，以观察模型的哪些部分导致标记预测发生变化。我们在若干测试例子上运行全部这些干预，并报告互换干预准确率（IIA）：该指标表示若反事实（即复数）为真值时的主谓一致准确率。各 *Hop 模型的测试例子取自各自版本的 BabyLM 测试集，并通过把单数主语改成复数主语构造最小差异反事实。为确保不同例子上的干预可类比，我们用正则表达式定位遵循同一结构的例子（即主语与动词处于相同位置）。

Results.

Our results are presented in Figure 5. The IIA graphs demonstrate how information about the marker tokens flows through the models. We can see that, in all three *Hop models, IIA is high at the token position of the subject up until about layer 3; then there is a transition to the position of the last token in the prefix, preceding the location where the marker should be predicted. All models develop the same modular solution to the task by tracking agreement through the representations at the relevant positions, but the NoHop model obtains nearly 100% IIA earlier during training, at about 1,500 training steps, supporting the previous surprisal results.

结果。

结果见图 5。IIA 图展示了关于标记 token 的信息如何在模型中流动。可以看到，在全部三种 *Hop 模型中，IIA 在主语 token 位置上直到约第 3 层都很高；随后过渡到前缀最后一个 token 的位置，也就是应预测标记之处的前一位置。所有模型都通过对相关位置上的表征跟踪一致关系，为该任务发展出相同的模块化解法；但 NoHop 模型在更早的训练阶段——约 1500 个训练步——就达到近 100% 的 IIA，支持此前的惊讶度结果。

5 Discussion and Conclusion

5 讨论与结论

Contra claims by Chomsky and others that LLMs cannot possibly inform our understanding of human language, we argue there is great value in treating LLMs as a comparative system for human language and in understanding what systems like LLMs can and cannot learn. Prior explorations of neural language models have already been fruitful for understanding the generalization of syntactic principles from data Wilcox et al. (2018); Marvin and Linzen (2018); Futrell et al. (2019); Prasad et al. (2019); Hu et al. (2020). Our paper complements this line of work. We have shown that GPT-2 models do not master our set of synthetic impossible languages as well as natural ones, challenging the unfounded assertions stated previously.

与 Chomsky 等人「LLM 不可能增进我们对人类语言理解」的主张相反，我们认为：把 LLM 当作人类语言的对照系统、并弄清 LLM 能学什么不能学什么，具有很大价值。对神经语言模型的既有探索，已有助于理解从数据中概括句法原则（Wilcox et al., 2018；Marvin and Linzen, 2018；Futrell et al., 2019；Prasad et al., 2019；Hu et al., 2020）。本文补充这一工作线。我们表明：相对自然语言，GPT-2 模型对我们这组合成不可能语言的掌握更差，从而挑战了此前缺乏根据的断言。

Even in the absence of a clear definition of what constitutes a possible or impossible language, we believe that our investigations advance this debate regarding LLMs. The lack of a definition does not hinder inquiry into this topic; in fact, it beckons further explorations of the boundary between the possible and impossible languages, as shown in our hypothesized continuum in Figure 1. We believe that the *Hop languages we propose closely approach this boundary.

即便对何为可能／不可能语言尚无清晰定义，我们仍认为本研究推进了关于 LLM 的这场辩论。缺少定义并不妨碍探究；事实上，它呼唤对可能与不可能边界的进一步探索，正如图 1 所假设的连续统所示。我们认为，所提出的 *Hop 语言相当接近这条边界。

At the same time, conclusions about LLMs’ linguistic competence and preferences for natural languages should be informed by an understanding of the ways that models fundamentally differ from humans. For instance, we saw that models can perform operations that involve counting tokens because LLMs rely on tokens as basic units. While humans are sensitive to morpheme boundaries and word boundaries, it is unlikely humans rely on atomic tokens in the way that LLMs do. This does not mean that LLMs can fundamentally tell us nothing about human language. Rather, as we did here, it is valuable to consider and control for this difference before making generalizations.

同时，关于 LLM 语言能力及其对自然语言偏好的结论，应建立在理解模型与人根本差异的基础上。例如，我们看到模型能执行涉及清点 token 的操作，因为 LLM 以 token 为基本单位。人虽对语素边界与词边界敏感，却不太可能像 LLM 那样依赖原子 token。这并不意味着 LLM 从根本上无法告诉我们关于人类语言的任何事。相反，如本文所做，在做出概括之前考虑并控制这一差异是有价值的。

Since at least the 1950s, a major line of linguistic inquiry has focused on what aspects of syntactic structure can be learned just from data, without domain-specific innate priors (e.g. a Universal Grammar). LLMs lack strong in-built linguistic priors, yet they can learn complex syntactic structures. While many LLMs are trained with vastly more data than children see, there is increasing evidence that even systems trained on smaller amounts of data can learn interesting linguistic information (Warstadt et al., 2023). The current paper raises further questions along similar lines. Since we do find that real languages are more learnable by GPT-2, this leads us to wonder what inductive bias of GPT language models matches natural language. We believe that this inductive bias is related to information locality, the tendency for statistical correlations in text to be short range. Information locality arises in GPTs due to their autoregressive training objective and has been argued to arise in humans due to the incremental nature of real-time language processing (Futrell, 2019; Hahn et al., 2021).

至少自 1950 年代以来，语言学的一条主线追问：句法结构的哪些方面可以仅从数据中学会，而无需领域特定的先天先验（例如普遍语法）。LLM 缺乏强内置语言先验，却能学会复杂句法结构。许多 LLM 的训练数据远超儿童所见，但越来越多证据表明，即便在较小数据量上训练的系统也能学到有意思的语言信息（Warstadt et al., 2023）。本文沿着相似线索提出进一步问题。既然我们发现真实语言对 GPT-2 更易学，便想知道 GPT 语言模型的何种归纳偏置与自然语言相匹配。我们认为这一归纳偏置与信息局部性相关：文本中的统计相关倾向为短程。信息局部性在 GPT 中因其自回归训练目标而产生，也有人主张它在人类中因实时语言加工的增量性而产生（Futrell, 2019；Hahn et al., 2021）。

Since LLMs have been shown to learn the complex structures of human language and have a preference for learning such structures over unnatural counterfactuals, it follows that they are clearly relevant to investigations and claims about the necessary innate priors for language learning. Arguments that they are “by design, unlimited in what they can ‘learn”’ and “incapable of distinguishing the possible from the impossible” (Chomsky et al., 2023) do not offer convincing evidence otherwise.

既然已表明 LLM 能学会人类语言的复杂结构，并相对不自然的反事实更偏好学习这些结构，那么它们显然与「语言学习需要哪些先天先验」的调查与主张相关。声称它们「在设计上对所能『学习』的内容没有限制」、并「无力区分可能与不可能」（Chomsky et al., 2023）的论点，并不能提供令人信服的反证。

6 Acknowledgments

6 致谢

The authors would like to thank Aryaman Arora, Christiane Fellbaum, Roger Levy, Tristan Thrush, and Diyi Yang for helpful comments on the project. We would also like to thank the members of the Stanford NLP Group, the MIT Computational Psycholinguistics Lab, and the anonymous reviewers for useful discussions. Julie Kallini is supported by a National Science Foundation Graduate Research Fellowship under grant number DGE-2146755.

作者感谢 Aryaman Arora、Christiane Fellbaum、Roger Levy、Tristan Thrush 与 Diyi Yang 对本项目的有益意见。也感谢 Stanford NLP Group、MIT Computational Psycholinguistics Lab 的成员以及匿名审稿人的有益讨论。Julie Kallini 受美国国家科学基金会研究生研究奖学金资助，编号 DGE-2146755。

7 Limitations

7 局限

Due to resource constraints, we exclusively use the GPT-2 architecture to train models on our various synthetic impossible languages. Each of our experiments involves training a GPT-2 model from scratch on a different language dataset, and for every such language, we train multiple GPT-2 models to establish confidence intervals for our evaluation metrics. Applying this approach to several different model architectures would be quite resource-intensive, so we opted to choose a single architecture in this paper. Future work could apply our methodology to models trained with different architectures or training objectives.

受资源所限，我们仅使用 GPT-2 架构在各类合成不可能语言上训练模型。每项实验都需要在不同语言数据集上从零训练一个 GPT-2，且对每种语言训练多个 GPT-2 以建立评估指标的置信区间。把这一做法推广到多种不同模型架构将非常耗费资源，因此本文选择单一架构。未来工作可将我们的方法应用于以不同架构或训练目标训练的模型。

Our impossible languages are derived by manipulating an English dataset. While we do not conduct experiments that use other natural languages as a starting point, our experimental choices (i.e. the synthetic languages we design) are informed by linguistic diversity and typology, distinguishing our impossible languages from those that are rare but attested. However, future work might involve deriving impossible languages from base languages other than English and include more morphological manipulations.

我们的不可能语言由操纵英语数据集得到。虽然我们未以其他自然语言为起点做实验，但实验选择（即所设计的合成语言）受到语言多样性与类型学的启发，从而把不可能语言与「罕见但已证实」的语言区分开。不过，未来工作可从英语以外的基语言派生不可能语言，并纳入更多形态层面的操纵。

8 Ethics Statement

8 伦理声明

While this work makes the case for language models as useful tools for cognitive science and linguistics research, these models learn and generate language through processes that are fundamentally different from those employed by humans. Making direct claims about human language learning based on the results of this paper could pose potential risks and harms. This research merely aims to explore the learnability of different languages (specifically, those languages that cannot be acquired by humans and are not representative of any known human language) through the lens of neural models.

虽然本文主张语言模型可作为认知科学与语言学研究的有用工具，这些模型学习与生成语言的过程与人类根本不同。若直接依据本文结果对人类语言学习做出断言，可能带来潜在风险与伤害。本研究仅旨在从神经模型的视角，探索不同语言的可学性（具体而言，是那些无法被人类习得、也不代表任何已知人类语言的语言）。

References

参考文献

Abdou et al. (2022) Mostafa Abdou, Vinit Ravishankar, Artur Kulmizev, and Anders Søgaard. 2022. Word order does matter and shuffled language models know it . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 6907–6919, Dublin, Ireland. Association for Computational Linguistics.

Alleman et al. (2021) Matteo Alleman, Jonathan Mamou, Miguel A Del Rio, Hanlin Tang, Yoon Kim, and SueYeon Chung. 2021. Syntactic perturbations reveal representational correlates of hierarchical phrase structure in pretrained language models . In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021) , pages 263–276, Online. Association for Computational Linguistics.

Bolhuis et al. (2024) Johan J. Bolhuis, Stephen Crain, Sandiway Fong, and Andrea Moro. 2024. Three reasons why AI doesn’t model human language . Nature , 627(8004):489–489.

Chomsky (1956) Noam Chomsky. 1956. Three models for the description of language . IRE Transactions on Information Theory , 2(3):113–124.

Chomsky (1957) Noam Chomsky. 1957. Syntactic Structures . De Gruyter Mouton, Berlin, Boston.

Chomsky (1959) Noam Chomsky. 1959. On certain formal properties of grammars . Information and Control , 2(2):137–167.

Chomsky (1965) Noam Chomsky. 1965. Aspects of the Theory of Syntax . The MIT Press.

Chomsky (2002) Noam Chomsky. 2002. On Nature and Language . Cambridge University Press.

Chomsky (2023) Noam Chomsky. 2023. Conversations with Tyler: Noam Chomsky . Conversations with Tyler Podcast.

Chomsky et al. (2023) Noam Chomsky, Ian Roberts, and Jeffrey Watumull. 2023. Noam Chomsky: The false promise of ChatGPT . The New York Times .

Comrie (1989) Bernard Comrie. 1989. Language universals and linguistic typology: Syntax and morphology . University of Chicago press.

Deletang et al. (2023) Gregoire Deletang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, and Pedro A Ortega. 2023. Neural networks and the Chomsky hierarchy . In The Eleventh International Conference on Learning Representations .

Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Ebrahimi et al. (2020) Javid Ebrahimi, Dhruv Gelda, and Wei Zhang. 2020. How can self-attention networks recognize Dyck-n languages? In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 4301–4306, Online. Association for Computational Linguistics.

Elman (1990) Jeffrey L. Elman. 1990. Finding structure in time . Cognitive Science , 14(2):179–211.

Evans and Levinson (2009) Nicholas Evans and Stephen C Levinson. 2009. The myth of language universals: Language diversity and its importance for cognitive science. Behavioral and brain sciences , 32(5):429–448.

Everett (2012) Daniel L. Everett. 2012. What does Pirahã grammar have to teach us about human language and the mind? WIREs Cognitive Science , 3(6):555–563.

Futrell (2019) Richard Futrell. 2019. Information-theoretic locality properties of natural language . In Proceedings of the First Workshop on Quantitative Syntax (Quasy, SyntaxFest 2019) , pages 2–15, Paris, France. Association for Computational Linguistics.

Futrell and Hahn (2022) Richard Futrell and Michael Hahn. 2022. Information theory as a bridge between language function and language form . Frontiers in Communication , 7.

Futrell et al. (2019) Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state . In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 32–42, Minneapolis, Minnesota. Association for Computational Linguistics.

Galke et al. (2023) Lukas Galke, Yoav Ram, and Limor Raviv. 2023. What makes a language easy to deep-learn?

Geiger et al. (2021) Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. 2021. Causal abstractions of neural networks . In Advances in Neural Information Processing Systems , volume 34, pages 9574–9586. Curran Associates, Inc.

Geiger et al. (2020) Atticus Geiger, Kyle Richardson, and Christopher Potts. 2020. Neural natural language inference models partially embed theories of lexical entailment and negation . In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP , pages 163–173, Online. Association for Computational Linguistics.

Geiger et al. (2023) Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D. Goodman. 2023. Finding alignments between interpretable causal variables and distributed neural representations. In Proceedings of Causal Learning and Reasoning 2024.

Greenberg (1963) Joseph Greenberg. 1963. Some universals of grammar with particular reference to the order of meaningful elements . Universals of Language , pages 73–113.

Hahn (2020) Michael Hahn. 2020. Theoretical limitations of self-attention in neural sequence models . Transactions of the Association for Computational Linguistics , 8:156–171.

Hahn et al. (2021) Michael Hahn, Judith Degen, and Richard Futrell. 2021. Modeling word and morpheme order in natural language as an efficient trade-off of memory and surprisal. Psychological Review , 128(4):726–756.

Hahn et al. (2020) Michael Hahn, Dan Jurafsky, and Richard Futrell. 2020. Universals of word order reflect optimization of grammars for efficient communication . Proceedings of the National Academy of Sciences , 117(5):2347–2353.

Hale (2001) John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model . In Second Meeting of the North American Chapter of the Association for Computational Linguistics .

Hao et al. (2018) Yiding Hao, William Merrill, Dana Angluin, Robert Frank, Noah Amsel, Andrew Benz, and Simon Mendelsohn. 2018. Context-free transductions with neural stacks . In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pages 306–315, Brussels, Belgium. Association for Computational Linguistics.

Hauser et al. (2002) Marc D. Hauser, Noam Chomsky, and W. Tecumseh Fitch. 2002. The faculty of language: What is it, who has it, and how did it evolve? Science , 298(5598):1569–1579.

Hessel and Schofield (2021) Jack Hessel and Alexandra Schofield. 2021. How effective is BERT without word ordering? implications for language understanding and data privacy . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) , pages 204–211, Online. Association for Computational Linguistics.

Hewitt et al. (2020) John Hewitt, Michael Hahn, Surya Ganguli, Percy Liang, and Christopher D. Manning. 2020. RNNs can generate bounded hierarchical languages with optimal memory . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1978–2010, Online. Association for Computational Linguistics.

Hu et al. (2020) Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 1725–1744, Online. Association for Computational Linguistics.

Huang et al. (2023) Qian Huang, Eric Zelikman, Sarah Li Chen, Yuhuai Wu, Gregory Valiant, and Percy Liang. 2023. Lexinvariant language models .

Jin et al. (2018) Lifeng Jin, Finale Doshi-Velez, Timothy Miller, William Schuler, and Lane Schwartz. 2018. Depth-bounding is effective: Improvements and evaluation of unsupervised PCFG induction . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 2721–2731, Brussels, Belgium. Association for Computational Linguistics.

Joshi (1985) Aravind K. Joshi. 1985. Tree adjoining grammars: How much context-sensitivity is required to provide reasonable structural descriptions? , Studies in Natural Language Processing, page 206–250. Cambridge University Press.

Karamcheti et al. (2021) Siddharth* Karamcheti, Laurel* Orr, Jason Bolton, Tianyi Zhang, Karan Goel, Avanika Narayan, Rishi Bommasani, Deepak Narayanan, Tatsunori Hashimoto, Dan Jurafsky, Christopher D. Manning, Christopher Potts, Christopher Ré, and Percy Liang. 2021. Mistral - a journey towards reproducible language model training .

Karlsson (2007) Fred Karlsson. 2007. Constraints on multiple center-embedding of clauses . Journal of Linguistics , 43(2):365–392.

Kazemnejad et al. (2023) Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. 2023. The impact of positional encoding on length generalization in transformers. arXiv preprint arXiv:2305.19466 .

Levy (2008) Roger Levy. 2008. Expectation-based syntactic comprehension . Cognition , 106(3):1126–1177.

Mansfield and Kemp (2023) John Mansfield and Charles Kemp. 2023. The emergence of grammatical structure from inter-predictability.

Marvin and Linzen (2018) Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics.

Merrill (2019) William Merrill. 2019. Sequential neural networks as automata . In Proceedings of the Workshop on Deep Learning and Formal Languages: Building Bridges , pages 1–13, Florence. Association for Computational Linguistics.

Merrill et al. (2020) William Merrill, Gail Weiss, Yoav Goldberg, Roy Schwartz, Noah A. Smith, and Eran Yahav. 2020. A formal hierarchy of RNN architectures . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 443–459, Online. Association for Computational Linguistics.

Mitchell and Bowers (2020) Jeff Mitchell and Jeffrey Bowers. 2020. Priorless recurrent networks learn curiously . In Proceedings of the 28th International Conference on Computational Linguistics , pages 5147–5158, Barcelona, Spain (Online). International Committee on Computational Linguistics.

Moro et al. (2023) Andrea Moro, Matteo Greco, and Stefano F. Cappa. 2023. Large languages, impossible languages and human brains . Cortex , 167:82–85.

Murty et al. (2023) Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D. Manning. 2023. Pushdown layers: Encoding recursive structure in transformer language models .

Musso et al. (2003) Mariacristina Musso, Andrea Moro, Volkmar Glauche, Michel Rijntjes, Jürgen Reichenbach, Christian Büchel, and Cornelius Weiller. 2003. Broca’s area and the language instinct. Nature Neuroscience , 6(7):774–781.

Nefdt (2024) Ryan M. Nefdt. 2024. The Philosophy of Theoretical Linguistics: A Contemporary Outlook . Cambridge University Press.

Papadimitriou et al. (2022) Isabel Papadimitriou, Richard Futrell, and Kyle Mahowald. 2022. When classifying grammatical role, BERT doesn’t care about word order… except when it matters . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 636–643, Dublin, Ireland. Association for Computational Linguistics.

Papadimitriou and Jurafsky (2023) Isabel Papadimitriou and Dan Jurafsky. 2023. Injecting structural hints: Using language models to study inductive biases in language learning .

Pham et al. (2021) Thang Pham, Trung Bui, Long Mai, and Anh Nguyen. 2021. Out of order: How important is the sequential order of words in a sentence in natural language understanding tasks? In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages 1145–1160, Online. Association for Computational Linguistics.

Prasad et al. (2019) Grusha Prasad, Marten van Schijndel, and Tal Linzen. 2019. Using priming to uncover the organization of syntactic representations in neural language models . In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) , pages 66–76, Hong Kong, China. Association for Computational Linguistics.

Pérez et al. (2021) Jorge Pérez, Pablo Barceló, and Javier Marinkovic. 2021. Attention is Turing-complete . Journal of Machine Learning Research , 22(75):1–35.

Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations .

Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training . Ms, OpenAI.

Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Ms, OpenAI.

Shieber (1985) Stuart M. Shieber. 1985. Evidence against the context-freeness of natural language. Linguistics and Philosophy , 8(3):333–343.

Sinha et al. (2021) Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 2888–2913, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Tenney et al. (2019) Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? Probing for sentence structure in contextualized word representations . In International Conference on Learning Representations .

Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need . In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc.

Warstadt et al. (2023) Alex Warstadt, Leshem Choshen, Aaron Mueller, Adina Williams, Ethan Wilcox, and Chengxu Zhuang. 2023. Call for papers – the BabyLM challenge: Sample-efficient pretraining on a developmentally plausible corpus .

Weiss et al. (2018) Gail Weiss, Yoav Goldberg, and Eran Yahav. 2018. On the practical computational power of finite precision RNNs for language recognition . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 740–745, Melbourne, Australia. Association for Computational Linguistics.

Wilcox et al. (2018) Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018. What do RNN language models learn about filler–gap dependencies? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pages 211–221, Brussels, Belgium. Association for Computational Linguistics.

Wilcox et al. (2023) Ethan Gotlieb Wilcox, Richard Futrell, and Roger Levy. 2023. Using computational models to test syntactic learnability . Linguistic Inquiry , pages 1–44.

Wu et al. (2023a) Zhengxuan Wu, Karel D’Oosterlinck, Atticus Geiger, Amir Zur, and Christopher Potts. 2023a. Causal proxy models for concept-based model explanations . In Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 37313–37334. PMLR.

Wu et al. (2023b) Zhengxuan Wu, Atticus Geiger, Thomas Icard, Christopher Potts, and Noah Goodman. 2023b. Interpretability at scale: Identifying causal mechanisms in Alpaca . In Advances in Neural Information Processing Systems , volume 36, pages 78205–78226. Curran Associates, Inc.

Wu et al. (2022) Zhengxuan Wu, Atticus Geiger, Joshua Rozner, Elisa Kreiss, Hanson Lu, Thomas Icard, Christopher Potts, and Noah Goodman. 2022. Causal distillation for language models . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 4288–4295, Seattle, United States. Association for Computational Linguistics.

Supplementary Materials

补充材料

Appendix A Dataset Filters

附录 A 数据集过滤

The BabyLM dataset Warstadt et al. (2023) is an English-language dataset of about 100 million words intended to approximate the amount of linguistic data available to an English-speaking child. To create a dataset for an impossible language, we first pre-process the BabyLM dataset using Stanza Qi et al. (2020). We perform sentence segmentation on each dataset file and then extract part-of-speech (POS) and morphological feature tags for all the sentences, which are required for the *Hop transformations. We transform each tagged sentence in the original BabyLM dataset using the impossible language’s rule-based perturbation function, as described in Section 3. Depending on the class of the impossible language and the specific features of the input sentence, perturbed sentences may be included or excluded from the final dataset used for model training (see below for details on this filtering). Since we apply these filters, the language classes have datasets of slightly different sizes. The *Shuffle and *Reverse languages have training sets of about 9.69 million sentences, and the *Hop languages have training sets of about 8.43 million sentences.

BabyLM 数据集（Warstadt et al., 2023）是约 1 亿词的英语数据集，意在近似英语儿童可获得的语言数据量。为构造不可能语言的数据集，我们先用 Stanza（Qi et al., 2020）预处理 BabyLM：对每个数据文件做句子切分，再为所有句子抽取词性（POS）与形态特征标签，这些是 *Hop 变换所必需的。然后用该不可能语言基于规则的扰动函数（见第 3 节）变换原 BabyLM 中每个已标注句子。取决于不可能语言类别与输入句的具体特征，扰动后的句子可能被纳入或排除出最终用于模型训练的数据集（过滤细节见下）。由于应用了这些过滤，各类语言的数据集大小略有不同。*Shuffle 与 *Reverse 语言的训练集约 969 万句，*Hop 语言的训练集约 843 万句。

*Shuffle Filters

For the *Shuffle languages, we filter sentences from the BabyLM dataset such that the set of token sequence lengths seen in the validation and test sets are also seen in the training set. This ensures that any shuffles for the DeterministicShuffle perturbation (which are determined by the token sequence length) in the test set have also occurred at least once in the training set. We apply these filters for all *Shuffle languages such that their datasets are comprised of the same subset of original sentences.

*Shuffle 过滤

对 *Shuffle 语言，我们过滤 BabyLM 句子，使得验证集与测试集中出现过的 token 序列长度集合，在训练集中也出现过。这保证 DeterministicShuffle 扰动（由 token 序列长度决定）在测试集中的任何打乱，至少也在训练集中出现过一次。我们对全部 *Shuffle 语言应用这些过滤，使它们的数据集由同一批原始句子子集构成。

*Reverse Filters

For the *Reverse languages, we do not apply any sentence filtering, so their models are trained on the entire BabyLM dataset.

*Reverse 过滤

对 *Reverse 语言，我们不做任何句子过滤，因此其模型在整个 BabyLM 数据集上训练。

*Hop Filters

For the *Hop languages, we filter out sentences from the BabyLM dataset that would not allow the special markers to fully complete 4 hops in the TokenHop or WordHop perturbations, i.e. sentences in which a 3rd-person present tense verb is too close to the end of the sentence. We again filter out these sentences from all perturbations, so TokenHop, WordHop, and NoHop are comprised of the same subset of original sentences from the BabyLM dataset.

*Hop 过滤

对 *Hop 语言，我们过滤掉 BabyLM 中无法让特殊标记在 TokenHop 或 WordHop 扰动中完整完成 4 次跳跃的句子，即第三人称现在时动词过于靠近句末的句子。我们再次从全部扰动中滤掉这些句子，因此 TokenHop、WordHop 与 NoHop 由 BabyLM 的同一批原始句子子集构成。

Appendix B GPT-2 Training Details and Hyperparameters

附录 B GPT-2 训练细节与超参

We train GPT-2 small models with a standard training regime Radford et al. (2018); Radford et al. (2019) using the library of Karamcheti et al. (2021). We mostly use the default GPT-2 small hyperparameters to train our models (context length of 1024, batch size of 512, etc.). We only change the total number of training steps and the number of warm-up steps. We train with a learning rate that linearly warms up from 0 to 6e-4 over 300 steps. While 10% of steps for warm-up is typical for LLM training, we acknowledge that the best warm-up may be different when using a small pretraining dataset, so we also tried 1,000 warm-up steps and 4,000 warm-up steps. (4,000 steps is the GPT-2 default. Since we only train for 3,000 steps, this effectively means we have a learning-rate that linearly warms up from 0 to 4.5e-4.) Using a different warm-up did not change the ranking of impossible language model perplexities.

我们用标准训练流程（Radford et al., 2018；Radford et al., 2019）与 Karamcheti et al. (2021) 的库训练 GPT-2 small 模型。大体采用默认 GPT-2 small 超参（上下文长度 1024、批大小 512 等）。我们只改动总训练步数与预热步数。学习率在 300 步内从 0 线性预热到 6e-4。LLM 训练通常用 10% 步数做预热；我们承认在小预训练数据上最佳预热可能不同，因此也尝试了 1000 与 4000 预热步。（4000 步是 GPT-2 默认。由于我们只训练 3000 步，这实际上意味着学习率从 0 线性预热到 4.5e-4。）改用不同预热并未改变不可能语言模型困惑度的排序。

We train the models for 3,000 training steps, which equates to about 11.03 epochs for the *Shuffle languages, 10.05 epochs for the *Reverse languages, and 12.04 epochs for the *Hop languages. The vocabulary set also varies based on the language. The *Shuffle languages use the standard GPT-2 vocabulary containing 50,257 tokens; the *Reverse languages add one special token R, for a vocabulary size of 50,258; and the *Hop languages add two special tokens S and P for verb inflection, for a vocabulary size of 50,259. We train on NVIDIA RTX 3090 (24GB) GPUs and NVIDIA RTX A6000 (48GB) GPUs. The runtime for each pretraining experiment was $\sim$24 hours (for one language and one random seed), for a total experiment runtime of $\sim$1800 hours.

我们训练 3000 步，约合 *Shuffle 语言 11.03 个 epoch、*Reverse 语言 10.05 个 epoch、*Hop 语言 12.04 个 epoch。词表也随语言而变。*Shuffle 语言使用含 50257 个 token 的标准 GPT-2 词表；*Reverse 语言增加一个特殊 token R，词表大小 50258；*Hop 语言为动词屈折增加两个特殊 token S 与 P，词表大小 50259。我们在 NVIDIA RTX 3090（24GB）与 NVIDIA RTX A6000（48GB）GPU 上训练。每次预训练实验运行时间约 24 小时（一种语言、一个随机种子），总实验运行时间约 1800 小时。

Appendix C Results for Models without Positional Encodings

附录 C 无位置编码模型的结果

Here, we present results for each of our experiments using GPT-2 models we trained without positional encodings. All other aspects of the experiments are the same, including the impossible language datasets and training hyperparameters. We again train 5 sets of models initialized using different random seeds. Figure 8 presents the perplexity results; Figure 8 presents the surprisal results; and Figure 8 presents the causal intervention results.

此处给出在去掉位置编码的 GPT-2 模型上各实验的结果。实验其余方面相同，包括不可能语言数据集与训练超参。我们同样用不同随机种子初始化训练 5 组模型。图 6 给出困惑度结果；图 7 给出惊讶度结果；图 8 给出因果干预结果。

Figure 6: Perplexities on a sample of 10K test sentences for each impossible language model trained without positional encodings. Error bars indicate 95% confidence intervals across 5 training runs initialized with different random seeds and evaluated on different test samples.

图 6：在无位置编码条件下训练的各不可能语言模型，在 1 万条测试句样本上的困惑度。误差条表示 5 次以不同随机种子初始化、并在不同测试样本上评估的训练运行的 95% 置信区间。

Appendix D Constituency Probing Evaluation

附录 D 成分探测评估

We also test how perturbations might influence latent linguistic properties in sentences that are seemingly unaffected by the perturbations. For this, we develop a constituency probing experiment to examine whether the contextual representations generated by different models are effective in classifying a sequence of tokens with an appropriate constituent label, similar to the edge probing experiments of Tenney et al. 2019. For example, if the input sentence is “I enjoy strawberry ice cream” and the span of tokens in question represents the constituent “strawberry ice cream,” the span should be labeled as a noun phrase (NP).

我们还检验扰动如何影响那些表面上未受扰动影响的句子中的潜在语言属性。为此，我们设计成分探测实验，考察不同模型生成的上下文表征，能否有效地把一段 token 序列分类到合适的成分标签，类似 Tenney et al. 2019 的边界探测实验。例如，若输入句是 “I enjoy strawberry ice cream”，所关注的 token 跨度表示成分 “strawberry ice cream”，则该跨度应标为名词短语（NP）。

Setup.

We conduct these experiments for *Reverse and *Hop languages, since these languages have constituents in contiguous token sequences. For NoReverse and PartialReverse, we take a sample of unaltered BabyLM test sentences and omit the reversal token R. For FullReverse, we use the same sample sentences, but reverse the tokens. For the *Hop languages, we use a sample of BabyLM test sentences that are unaffected by the perturbation, which are sentences that do not contain 3rd-person present tense verbs. To extract constituents for testing, we parse the sample sentences using Stanza’s BERT-based consituency parser. We include noun phrases (NP), verb phrases (VP), adjective phrases (ADJP), adverb phrases (ADVP), and prepositional phrases (PP), and we stratify the samples so that there are equal numbers of example constituents for each phrasal category. We obtain a total of 10K examples for probe training and testing for each language class, where an example is comprised of a tokenized sentence, indices of the constituent span, and the constituent label.

设置。

我们对 *Reverse 与 *Hop 语言做这些实验，因为这些语言中的成分位于连续 token 序列中。对 NoReverse 与 PartialReverse，我们取未改动的 BabyLM 测试句样本并省略反转 token R。对 FullReverse，使用相同样本句，但反转 token。对 *Hop 语言，我们使用未受扰动影响的 BabyLM 测试句样本，即不含第三人称现在时动词的句子。为抽取测试用成分，我们用 Stanza 基于 BERT 的成分句法分析器解析样本句。纳入名词短语（NP）、动词短语（VP）、形容词短语（ADJP）、副词短语（ADVP）与介词短语（PP），并对样本分层，使各短语类别的成分例子数量相等。每一语言类共得到 1 万个用于探测训练与测试的例子；每个例子由分词句子、成分跨度索引与成分标签组成。

Our probes are L2-regularized logistic regression classifiers trained on the span representations of the tokens corresponding to constituents in the examples. To obtain span representations for training the probes, we mean-pool the representations of the tokens within the span. We try extracting representations from GPT-2 by averaging the last four hidden layers of the model or using different layers individually. We train each probe for a maximum of 10 iterations and hold out 20% of constituent examples for testing.

我们的探测器是 L2 正则化的逻辑回归分类器，在对应成分 token 的跨度表征上训练。为获得训练用跨度表征，我们对跨度内 token 的表征做均值池化。我们尝试从 GPT-2 提取表征：对模型最后四层隐状态取平均，或分别使用不同层。每个探测器最多训练 10 轮迭代，并留出 20% 成分例子作测试。

Hypothesis.

Constituency probes will achieve higher accuracy for possible languages than impossible ones, in virtue of the fact that the impossible languages are defined by some rules that do not respect constituency boundaries.

假设。

成分探测器在可能语言上的准确率将高于不可能语言，因为不可能语言由某些不尊重成分边界的规则所定义。

Results.

The results of the probing experiment using the average of the last four GPT-2 layers are presented in Figure 9. Across *Reverse and *Hop models trained with positional encodings, there are not any clear trends indicating that certain models have better representations of constituents than others, as differences among probe accuracies are minimal and unstable across training steps. However, looking closely at the *Reverse models without positional encodings, we can see that PartialReverse has significantly lower probe accuracy than the other models up until 2K training steps. We found similar results when using different layers for span representations, as shown in Figure 10. These results might indicate that the *Hop perturbations were too weak to fundamentally affect the models’ representations of latent linguistic structure, but quite unnatural reversal rule of the PartialReverse language disturbed consituency boundaries in a way that could not be recovered by GPT-2 models without positional encodings.

结果。

使用 GPT-2 最后四层平均的探测实验结果见图 9。在带位置编码训练的 Reverse 与 Hop 模型之间，并无清晰趋势表明某些模型的成分表征更好：探测准确率差异很小，且跨训练步数不稳定。但仔细看无位置编码的 *Reverse 模型，可见 PartialReverse 在直至 2K 训练步之前，探测准确率显著低于其他模型。使用不同层作跨度表征时结果类似，见图 10。这些结果可能表明：*Hop 扰动过弱，不足以从根本上影响模型对潜在语言结构的表征；而 PartialReverse 语言相当不自然的反转规则扰乱了成分边界，以至于无位置编码的 GPT-2 模型无法恢复。

(a) Probe accuracy for *Reverse and *Hop models.

(a) *Reverse 与 *Hop 模型的探测准确率。

(a) Probe accuracy for *Reverse models.

(a) *Reverse 模型的探测准确率。

Appendix E Additional DeterministicShuffle Results

附录 E 额外的 DeterministicShuffle 结果

In addition to perplexities of each impossible language model on its own test data, we also obtain perplexities for each DeterministicShuffle model on the NondeterministicShuffle test sample and all other DeterministicShuffle test samples. This measures whether these models have learned to distinguish their own shuffles from other shuffles. We found that this was indeed the case, as shown in the results in Figure 11.

除各不可能语言模型在自身测试数据上的困惑度外，我们还取得每个 DeterministicShuffle 模型在 NondeterministicShuffle 测试样本以及所有其他 DeterministicShuffle 测试样本上的困惑度。这衡量模型是否学会把自己的打乱与其他打乱区分开。结果确实如此，见图 11。

(a) Test perplexities for models with positional encodings.

(a) 带位置编码模型的测试困惑度。

Appendix F Confidence Intervals for Interchange Intervention Accuracies

附录 F 互换干预准确率的置信区间

We present the same results of our causal abstraction experiments from Section 4.4, but include confidence intervals for results across models initialized on different random seeds. Figure 12 presents the results for NoHop; Figure 13 presents the results for TokenHop; and Figure 14 presents the results for WordHop. Figures 15, 16, and 17 show the same plots for each *Hop model trained without positional encodings, respectively.

我们呈现第 4.4 节因果抽象实验的相同结果，但加入不同随机种子初始化模型结果的置信区间。图 12 为 NoHop；图 13 为 TokenHop；图 14 为 WordHop。图 15、16、17 分别给出在无位置编码条件下训练的各 *Hop 模型的相同图。

(a) 300 Training Steps.

(a) 300 训练步。

(a) 300 Training Steps.

(a) 300 训练步。

(a) 300 Training Steps.

(a) 300 训练步。

(a) 300 Training Steps.

(a) 300 训练步。

(a) 300 Training Steps.

(a) 300 训练步。

(a) 300 Training Steps.

(a) 300 训练步。