SciBERT: A Pretrained Language Model for Scientific Text
SciBERT: A Pretrained Language Model for Scientific Text
Abstract
摘要
Obtaining large-scale annotated data for NLP tasks in the scientific domain is challenging and expensive. We release SciBERT, a pretrained language model based on BERT (Devlin et al., 2019) to address the lack of high-quality, large-scale labeled scientific data. SciBERT leverages unsupervised pretraining on a large multi-domain corpus of scientific publications to improve performance on downstream scientific NLP tasks. We evaluate on a suite of tasks including sequence tagging, sentence classification and dependency parsing, with datasets from a variety of scientific domains. We demonstrate statistically significant improvements over BERT and achieve new state-of-the-art results on several of these tasks. The code and pretrained models are available at https://github.com/allenai/scibert/.
在科学领域为自然语言处理任务获取大规模标注数据既困难又昂贵。我们发布 SciBERT,这是一个基于 BERT(Devlin et al., 2019)的预训练语言模型,用来弥补高质量、大规模科学标注数据的不足。SciBERT 在大规模、跨领域的科学出版物语料上做无监督预训练,以提升下游科学 NLP 任务的表现。我们在一组任务上做了评估,包括序列标注、句子分类和依存句法分析,所用数据集来自多个科学领域。结果表明,SciBERT 相对 BERT 具有统计显著的提升,并在其中若干任务上达到了新的当时最优。代码和预训练模型见 https://github.com/allenai/scibert/。
1 Introduction
1 引言
The exponential increase in the volume of scientific publications in the past decades has made NLP an essential tool for large-scale knowledge extraction and machine reading of these documents. Recent progress in NLP has been driven by the adoption of deep neural models, but training such models often requires large amounts of labeled data. In general domains, large-scale training data is often possible to obtain through crowdsourcing, but in scientific domains, annotated data is difficult and expensive to collect due to the expertise required for quality annotation.
过去几十年里,科学出版物的数量呈指数增长,自然语言处理因此成为从这些文献中做大规模知识抽取和机器阅读的必要工具。近年来 NLP 的进展很大程度上来自深度神经模型的普及,但训练这类模型往往需要大量标注数据。在通用领域,大规模训练数据常常可以靠众包获得;在科学领域,高质量标注却需要专业知识,收集既困难又昂贵。
As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks. These models return contextualized embeddings for each token which can be passed into minimal task-specific neural architectures. Leveraging the success of unsupervised pretraining has become especially important especially when task-specific annotations are difficult to obtain, like in scientific NLP. Yet while both BERT and ELMo have released pretrained models, they are still trained on general domain corpora such as news articles and Wikipedia.
正如 ELMo(Peters et al., 2018)、GPT(Radford et al., 2018)和 BERT(Devlin et al., 2019)所表明的,在大规模语料上对语言模型做无监督预训练,能显著提升许多 NLP 任务的表现。这些模型为每个 token 返回上下文相关的嵌入,随后只需接入很轻量的任务专用神经网络。当任务标注很难获得时——科学 NLP 正是如此——借助无监督预训练就显得格外重要。不过,BERT 和 ELMo 虽都发布了预训练模型,它们仍是在新闻报道、维基百科这类通用领域语料上训练的。
In this work, we make the following contributions:
在本工作中,我们做出如下贡献:
(i) We release SciBERT, a new resource demonstrated to improve performance on a range of NLP tasks in the scientific domain. SciBERT is a pretrained language model based on BERT but trained on a large corpus of scientific text.
(i)我们发布 SciBERT,这一新资源被证明能够提升科学领域一系列 NLP 任务的表现。SciBERT 是基于 BERT 的预训练语言模型,但训练语料是大规模科学文本。
(ii) We perform extensive experimentation to investigate the performance of finetuning versus task-specific architectures atop frozen embeddings, and the effect of having an in-domain vocabulary.
(ii)我们进行了广泛实验,比较微调与在冻住的嵌入之上搭建任务专用架构这两种用法,并考察使用领域内词表的效果。
(iii) We evaluate SciBERT on a suite of tasks in the scientific domain, and achieve new state-of-the-art (SOTA) results on many of these tasks.
(iii)我们在一组科学领域任务上评估 SciBERT,并在其中许多任务上取得了新的当时最优(SOTA)结果。
2 Methods
2 方法
Background The BERT model architecture (Devlin et al., 2019) is based on a multilayer bidirectional Transformer (Vaswani et al., 2017). Instead of the traditional left-to-right language modeling objective, BERT is trained on two tasks: predicting randomly masked tokens and predicting whether two sentences follow each other. SciBERT follows the same architecture as BERT but is instead pretrained on scientific text.
Background BERT 的模型架构(Devlin et al., 2019)基于多层双向 Transformer(Vaswani et al., 2017)。它没有采用传统的从左到右语言建模目标,而是在两项任务上训练:预测被随机掩码的 token,以及判断两个句子是否前后相接。SciBERT 沿用与 BERT 相同的架构,但改为在科学文本上预训练。
Vocabulary BERT uses WordPiece (Wu et al., 2016) for unsupervised tokenization of the input text. The vocabulary is built such that it contains the most frequently used words or subword units. We refer to the original vocabulary released with BERT as BASEVOCAB.
Vocabulary BERT 使用 WordPiece(Wu et al., 2016)对输入文本做无监督切分。词表的构建方式是收纳最常用的词或子词单元。我们将随 BERT 发布的原词表称为 BASEVOCAB。
We construct SCIVOCAB, a new WordPiece vocabulary on our scientific corpus using the SentencePiece library. We produce both cased and uncased vocabularies and set the vocabulary size to 30K to match the size of BASEVOCAB. The resulting token overlap between BASEVOCAB and SCIVOCAB is 42%, illustrating a substantial difference in frequently used words between scientific and general domain texts.
我们用 SentencePiece 库在科学语料上构造了一套新的 WordPiece 词表 SCIVOCAB。我们同时制作区分大小写和不区分大小写的版本,并将词表规模设为 3 万,以便与 BASEVOCAB 的规模对齐。BASEVOCAB 与 SCIVOCAB 的 token 重叠仅为 42%,说明科学文本和通用领域文本在常用词上存在明显差别。
Corpus We train SciBERT on a random sample of 1.14M papers from Semantic Scholar (Ammar et al., 2018). This corpus consists of 18% papers from the computer science domain and 82% from the broad biomedical domain. We use the full text of the papers, not just the abstracts. The average paper length is 154 sentences (2,769 tokens) resulting in a corpus size of 3.17B tokens, similar to the 3.3B tokens on which BERT was trained. We split sentences using ScispaCy (Neumann et al., 2019), which is optimized for scientific text.
Corpus 我们在从 Semantic Scholar(Ammar et al., 2018)随机抽样的 114 万篇论文上训练 SciBERT。该语料中 18% 的论文来自计算机科学,82% 来自更宽的生物医学领域。我们使用的是论文全文,而不仅仅是摘要。论文平均长度为 154 句(2,769 个 token),语料总规模为 31.7 亿 token,与训练 BERT 所用的 33 亿 token 相近。分句使用针对科学文本优化的 ScispaCy(Neumann et al., 2019)。
3 Experimental Setup
3 实验设置
3.1 Tasks
3.1 任务
We experiment on the following core NLP tasks:
我们在下列核心 NLP 任务上做实验:
- Named Entity Recognition (NER)
- 命名实体识别(NER)
- PICO Extraction (PICO)
- PICO 抽取(PICO)
- Text Classification (CLS)
- 文本分类(CLS)
- Relation Classification (REL)
- 关系分类(REL)
- Dependency Parsing (DEP)
- 依存句法分析(DEP)
PICO, like NER, is a sequence labeling task where the model extracts spans describing the Participants, Interventions, Comparisons, and Outcomes in a clinical trial paper (Kim et al., 2011). REL is a special case of text classification where the model predicts the type of relation expressed between two entities, which are encapsulated in the sentence by inserted special tokens.
PICO 与 NER 一样属于序列标注任务:模型要抽出临床试验论文中描述参与者、干预、对照和结局的片段(Kim et al., 2011)。REL 是文本分类的一种特例,模型预测两个实体之间表达的关系类型;这两个实体通过插入的特殊 token 封装在句子里。
3.2 Datasets
3.2 数据集
For brevity, we only describe the newer datasets here, and refer the reader to the references in Table 1 for the older datasets. EBM-NLP (Nye et al., 2018) annotates PICO spans in clinical trial abstracts. SciERC (Luan et al., 2018) annotates entities and relations from computer science abstracts.
为节省篇幅,这里只介绍较新的数据集,更早的数据集请参见表 1 中的文献。EBM-NLP(Nye et al., 2018)在临床试验摘要中标注 PICO 片段。SciERC(Luan et al., 2018)在计算机科学摘要中标注实体和关系。
ACL-ARC (Jurgens et al., 2018) and SciCite (Cohan et al., 2019) assign intent labels (e.g. Comparison, Extension, etc.) to sentences from scientific papers that cite other papers. The Paper Field dataset is built from the Microsoft Academic Graph (Sinha et al., 2015) and maps paper titles to one of 7 fields of study. Each field of study (i.e. geography, politics, economics, business, sociology, medicine, and psychology) has approximately 12K training examples.
ACL-ARC(Jurgens et al., 2018)和 SciCite(Cohan et al., 2019)为科学论文中引用其他论文的句子标注意图标签(例如 Comparison、Extension 等)。Paper Field 数据集由 Microsoft Academic Graph(Sinha et al., 2015)构建,把论文标题映射到 7 个学科之一。每个学科(即地理学、政治学、经济学、商学、社会学、医学和心理学)大约有 1.2 万条训练样本。
3.3 Pretrained BERT Variants
3.3 预训练 BERT 变体
BERT-Base We use the pretrained weights for BERT-Base (Devlin et al., 2019) released with the original BERT code. The vocabulary is BASEVOCAB. We evaluate both cased and uncased versions of this model.
BERT-Base 我们使用随原始 BERT 代码发布的 BERT-Base(Devlin et al., 2019)预训练权重。词表为 BASEVOCAB。我们同时评估该模型区分大小写和不区分大小写的版本。
SciBERT We use the original BERT code to train SciBERT on our corpus with the same configuration and size as BERT-Base. We train 4 different versions of SciBERT: (i) cased or uncased and (ii) BASEVOCAB or SCIVOCAB. The two models that use BASEVOCAB are finetuned from the corresponding BERT-Base models. The other two models that use the new SCIVOCAB are trained from scratch.
SciBERT 我们使用原始 BERT 代码,在自己的语料上按与 BERT-Base 相同的配置和规模训练 SciBERT。我们训练了 4 个不同版本的 SciBERT:(i)区分或不区分大小写;(ii)使用 BASEVOCAB 或 SCIVOCAB。使用 BASEVOCAB 的两个模型从对应的 BERT-Base 模型微调得到。使用新的 SCIVOCAB 的另外两个模型则从头训练。
Pretraining BERT for long sentences can be slow. Following the original BERT code, we set a maximum sentence length of 128 tokens, and train the model until the training loss stops decreasing. We then continue training the model allowing sentence lengths up to 512 tokens.
对长句预训练 BERT 可能会很慢。我们沿用原始 BERT 代码的做法,先把最大句长设为 128 个 token,训练到训练损失不再下降,再继续训练并允许句长达到 512 个 token。
We use a single TPU v3 with 8 cores. Training the SCIVOCAB models from scratch on our corpus takes 1 week (5 days with max length 128, then 2 days with max length 512). The BASEVOCAB models take 2 fewer days of training because they aren’t trained from scratch.
我们使用一枚 8 核的 TPU v3。在语料上从头训练 SCIVOCAB 模型需要 1 周(最大长度 128 时 5 天,随后最大长度 512 时再 2 天)。BASEVOCAB 模型少花 2 天,因为它们不是从头训练的。
5 BERT’s largest model was trained on 16 Cloud TPUs for 4 days. Expected 40-70 days (Dettmers, 2019) on an 8-GPU machine.
5 BERT 最大的模型是在 16 块 Cloud TPU 上训练了 4 天。若改在 8 GPU 的机器上,预计需要 40–70 天(Dettmers, 2019)。
All pretrained BERT models are converted to be compatible with PyTorch using the pytorch-transformers library. All our models (Sections 3.4 and 3.5) are implemented in PyTorch using AllenNLP (Gardner et al., 2017).
所有预训练 BERT 模型都用 pytorch-transformers 库转换成与 PyTorch 兼容的格式。我们的全部模型(第 3.4 和 3.5 节)都在 PyTorch 中基于 AllenNLP(Gardner et al., 2017)实现。
Casing We follow Devlin et al. (2019) in using the cased models for NER and the uncased models for all other tasks. We also use the cased models for parsing. Some light experimentation showed that the uncased models perform slightly better (even sometimes on NER) than cased models.
Casing 我们沿用 Devlin et al. (2019) 的做法:NER 使用区分大小写的模型,其余任务使用不区分大小写的模型。句法分析也使用区分大小写的模型。一些初步实验显示,不区分大小写的模型表现略好(有时甚至在 NER 上也是如此)。
3.4 Finetuning BERT
3.4 微调 BERT
We mostly follow the same architecture, optimization, and hyperparameter choices used in Devlin et al. (2019). For text classification (i.e. CLS and REL), we feed the final BERT vector for the [CLS] token into a linear classification layer. For sequence labeling (i.e. NER and PICO), we feed the final BERT vector for each token into a linear classification layer with softmax output. We differ slightly in using an additional conditional random field, which made evaluation easier by guaranteeing well-formed entities. For DEP, we use the model from Dozat and Manning (2017) with dependency tag and arc embeddings of size 100 and biaffine matrix attention over BERT vectors instead of stacked BiLSTMs.
我们大体沿用 Devlin et al. (2019) 的架构、优化和超参数选择。对于文本分类(即 CLS 和 REL),我们将 [CLS] token 的最终 BERT 向量送入线性分类层。对于序列标注(即 NER 和 PICO),我们将每个 token 的最终 BERT 向量送入带 softmax 输出的线性分类层。我们略有不同之处是额外使用了条件随机场,这样可以保证实体形式合法,使评估更方便。对于 DEP,我们采用 Dozat and Manning (2017) 的模型,依存标签和弧嵌入维度为 100,并在 BERT 向量上使用双仿射矩阵注意力,而不再堆叠 BiLSTM。
In all settings, we apply a dropout of 0.1 and optimize cross entropy loss using Adam (Kingma and Ba, 2015). We finetune for 2 to 5 epochs using a batch size of 32 and a learning rate of 5e-6, 1e-5, 2e-5, or 5e-5 with a slanted triangular schedule (Howard and Ruder, 2018) which is equivalent to the linear warmup followed by linear decay (Devlin et al., 2019). For each dataset and BERT variant, we pick the best learning rate and number of epochs on the development set and report the corresponding test results.
在所有设置中,我们使用 0.1 的 dropout,并用 Adam(Kingma and Ba, 2015)优化交叉熵损失。微调进行 2 到 5 个 epoch,批次大小为 32,学习率为 5e-6、1e-5、2e-5 或 5e-5,并采用斜三角学习率日程(Howard and Ruder, 2018);这等价于线性预热后再线性衰减(Devlin et al., 2019)。对每个数据集和每种 BERT 变体,我们在开发集上选取最佳学习率和 epoch 数,并报告对应的测试结果。
We found the setting that works best across most datasets and models is 2 or 4 epochs and a learning rate of 2e-5. While task-dependent, optimal hyperparameters for each task are often the same across BERT variants.
我们发现,在大多数数据集和模型上效果最好的设置是 2 或 4 个 epoch,学习率为 2e-5。尽管会因任务而异,但各任务的最优超参数在不同 BERT 变体之间往往相同。
3.5 Frozen BERT Embeddings
3.5 冻住的 BERT 嵌入
We also explore the usage of BERT as pretrained contextualized word embeddings, like ELMo (Peters et al., 2018), by training simple task-specific models atop frozen BERT embeddings.
我们还探索把 BERT 当作类似 ELMo(Peters et al., 2018)的预训练上下文词嵌入来用:在冻住的 BERT 嵌入之上训练简单的任务专用模型。
For text classification, we feed each sentence of BERT vectors into a 2-layer BiLSTM of size 200 and apply a multilayer perceptron (with hidden size 200) on the concatenated first and last BiLSTM vectors. For sequence labeling, we use the same BiLSTM layers and use a conditional random field to guarantee well-formed predictions.
对于文本分类,我们将一句中的 BERT 向量送入大小为 200 的 2 层 BiLSTM,再在首尾 BiLSTM 向量的拼接上接一个隐层大小为 200 的多层感知机。对于序列标注,我们使用同样的 BiLSTM 层,并用条件随机场来保证预测形式合法。
For DEP, we use the full model from Dozat and Manning (2017) with dependency tag and arc embeddings of size 100 and the same BiLSTM setup as other tasks. We did not find changing the depth or size of the BiLSTMs to significantly impact results (Reimers and Gurevych, 2017).
对于 DEP,我们使用 Dozat and Manning (2017) 的完整模型,依存标签和弧嵌入维度为 100,BiLSTM 设置与其他任务相同。我们没有发现改变 BiLSTM 的深度或宽度会显著影响结果(Reimers and Gurevych, 2017)。
We optimize cross entropy loss using Adam, but holding BERT weights frozen and applying a dropout of 0.5. We train with early stopping on the development set (patience of 10) using a batch size of 32 and a learning rate of 0.001.
我们仍用 Adam 优化交叉熵损失,但冻住 BERT 权重,并将 dropout 设为 0.5。训练时在开发集上做早停(耐心值为 10),批次大小为 32,学习率为 0.001。
We did not perform extensive hyperparameter search, but while optimal hyperparameters are going to be task-dependent, some light experimentation showed these settings work fairly well across most tasks and BERT variants.
我们没有做大规模超参数搜索;最优超参数固然会因任务而异,但一些初步实验表明,这些设置在大多数任务和 BERT 变体上效果都相当不错。
4 Results
4 结果
Table 1 summarizes the experimental results. We observe that SciBERT outperforms BERT-Base on scientific tasks (+2.11 F1 with finetuning and +2.43 F1 without). We also achieve new SOTA results on many of these tasks using SciBERT.
表 1 汇总了实验结果。可以看到,SciBERT 在科学任务上超过了 BERT-Base(微调时高 2.11 个 F1,不微调时高 2.43 个 F1)。使用 SciBERT,我们还在其中许多任务上取得了新的当时最优。
8 For rest of this paper, all results reported in this manner are averaged over datasets excluding UAS for DEP since we already include LAS.
8 在本文其余部分,所有按此方式报告的结果都是在各数据集上取平均,并排除 DEP 的 UAS,因为我们已经计入了 LAS。
4.1 Biomedical Domain
4.1 生物医学领域
We observe that SciBERT outperforms BERT-Base on biomedical tasks (+1.92 F1 with finetuning and +3.59 F1 without). In addition, SciBERT achieves new SOTA results on BC5CDR and ChemProt (Lee et al., 2019), and EBM-NLP (Nye et al., 2018).
可以看到,SciBERT 在生物医学任务上超过了 BERT-Base(微调时高 1.92 个 F1,不微调时高 3.59 个 F1)。此外,SciBERT 在 BC5CDR 和 ChemProt(Lee et al., 2019)以及 EBM-NLP(Nye et al., 2018)上取得了新的当时最优。
SciBERT performs slightly worse than SOTA on 3 datasets. The SOTA model for JNLPBA is a BiLSTM-CRF ensemble trained on multiple NER datasets not just JNLPBA (Yoon et al., 2018). The SOTA model for NCBI-disease is BioBERT (Lee et al., 2019), which is BERT-Base finetuned on 18B tokens from biomedical papers. The SOTA result for GENIA is in Nguyen and Verspoor (2019) which uses the model from Dozat and Manning (2017) with part-of-speech (POS) features, which we do not use.
在 3 个数据集上,SciBERT 略低于当时最优。JNLPBA 的最优模型是在多个 NER 数据集(而不仅仅是 JNLPBA)上训练的 BiLSTM-CRF 集成(Yoon et al., 2018)。NCBI-disease 的最优模型是 BioBERT(Lee et al., 2019),即在 180 亿生物医学论文 token 上微调的 BERT-Base。GENIA 的当时最优来自 Nguyen and Verspoor (2019),他们使用 Dozat and Manning (2017) 的模型并加入了词性(POS)特征,而我们没有使用这些特征。
In Table 2, we compare SciBERT results with reported BioBERT results on the subset of datasets included in (Lee et al., 2019). Interesting, SciBERT outperforms BioBERT results on BC5CDR and ChemProt, and performs similarly on JNLPBA despite being trained on a substantially smaller biomedical corpus.
在表 2 中,我们把 SciBERT 的结果与已报道的 BioBERT 结果,放在 Lee et al. (2019) 所包含的那部分数据集上进行比较。有意思的是,SciBERT 在 BC5CDR 和 ChemProt 上超过了 BioBERT 的结果,在 JNLPBA 上表现相近,尽管它所训练的生物医学语料要小得多。
Table 1: Test performances of all BERT variants on all tasks and datasets. Bold indicates the SOTA result (multiple results bolded if difference within 95% bootstrap confidence interval). Keeping with past work, we report macro F1 scores for NER (span-level), macro F1 scores for REL and CLS (sentence-level), and macro F1 for PICO (token-level), and micro F1 for ChemProt specifically. For DEP, we report labeled (LAS) and unlabeled (UAS) attachment scores (excluding punctuation) for the same model with hyperparameters tuned for LAS. All results are the average of multiple runs with different random seeds.
| Field | Task | Dataset | SOTA | BERT-Base Frozen | BERT-Base Finetune | SciBERT Frozen | SciBERT Finetune |
|---|---|---|---|---|---|---|---|
| Bio | NER | BC5CDR (Li et al., 2016) | 88.85⁷ | 85.08 | 86.72 | 88.73 | 90.01 |
| Bio | NER | JNLPBA (Collier and Kim, 2004) | 78.58 | 74.05 | 76.09 | 75.77 | 77.28 |
| Bio | NER | NCBI-disease (Dogan et al., 2014) | 89.36 | 84.06 | 86.88 | 86.39 | 88.57 |
| Bio | PICO | EBM-NLP (Nye et al., 2018) | 66.30 | 61.44 | 71.53 | 68.30 | 72.28 |
| Bio | DEP | GENIA (Kim et al., 2003) - LAS | 91.92 | 90.22 | 90.33 | 90.36 | 90.43 |
| Bio | DEP | GENIA (Kim et al., 2003) - UAS | 92.84 | 91.84 | 91.89 | 92.00 | 91.99 |
| Bio | REL | ChemProt (Kringelum et al., 2016) | 76.68 | 68.21 | 79.14 | 75.03 | 83.64 |
| CS | NER | SciERC (Luan et al., 2018) | 64.20 | 63.58 | 65.24 | 65.77 | 67.57 |
| CS | REL | SciERC (Luan et al., 2018) | n/a | 72.74 | 78.71 | 75.25 | 79.97 |
| CS | CLS | ACL-ARC (Jurgens et al., 2018) | 67.9 | 62.04 | 63.91 | 60.74 | 70.98 |
| Multi | CLS | Paper Field | n/a | 63.64 | 65.37 | 64.38 | 65.71 |
| Multi | CLS | SciCite (Cohan et al., 2019) | 84.0 | 84.31 | 84.85 | 85.42 | 85.49 |
| Average | 73.58 | 77.16 | 76.01 | 79.27 |
表 1:所有 BERT 变体在全部任务和数据集上的测试表现。加粗表示当时最优结果(若差异落在 95% 自助法置信区间内,则多项结果同时加粗)。与以往工作保持一致,NER 报告跨度级宏平均 F1,REL 和 CLS 报告句子级宏平均 F1,PICO 报告 token 级宏平均 F1,ChemProt 则特别报告微平均 F1。对于 DEP,我们报告同一模型在按 LAS 调好超参数后的带标签(LAS)和无标签(UAS)附着分数(不含标点)。所有结果均为使用不同随机种子多次运行的平均值。
| 领域 | 任务 | 数据集 | SOTA | BERT-Base 冻结 | BERT-Base 微调 | SciBERT 冻结 | SciBERT 微调 |
|---|---|---|---|---|---|---|---|
| Bio | NER | BC5CDR (Li et al., 2016) | 88.85⁷ | 85.08 | 86.72 | 88.73 | 90.01 |
| Bio | NER | JNLPBA (Collier and Kim, 2004) | 78.58 | 74.05 | 76.09 | 75.77 | 77.28 |
| Bio | NER | NCBI-disease (Dogan et al., 2014) | 89.36 | 84.06 | 86.88 | 86.39 | 88.57 |
| Bio | PICO | EBM-NLP (Nye et al., 2018) | 66.30 | 61.44 | 71.53 | 68.30 | 72.28 |
| Bio | DEP | GENIA (Kim et al., 2003) - LAS | 91.92 | 90.22 | 90.33 | 90.36 | 90.43 |
| Bio | DEP | GENIA (Kim et al., 2003) - UAS | 92.84 | 91.84 | 91.89 | 92.00 | 91.99 |
| Bio | REL | ChemProt (Kringelum et al., 2016) | 76.68 | 68.21 | 79.14 | 75.03 | 83.64 |
| CS | NER | SciERC (Luan et al., 2018) | 64.20 | 63.58 | 65.24 | 65.77 | 67.57 |
| CS | REL | SciERC (Luan et al., 2018) | n/a | 72.74 | 78.71 | 75.25 | 79.97 |
| CS | CLS | ACL-ARC (Jurgens et al., 2018) | 67.9 | 62.04 | 63.91 | 60.74 | 70.98 |
| Multi | CLS | Paper Field | n/a | 63.64 | 65.37 | 64.38 | 65.71 |
| Multi | CLS | SciCite (Cohan et al., 2019) | 84.0 | 84.31 | 84.85 | 85.42 | 85.49 |
| 平均 | 73.58 | 77.16 | 76.01 | 79.27 |
7 The SOTA paper did not report a single score. We compute the average of the reported results for each class weighted by number of examples in each class.
7 该当时最优论文并未报告单一分数。我们对各类别已报告的结果按各类别样本数加权,计算平均值。
Table 2: Comparing SciBERT with the reported BioBERT results on biomedical datasets.
| Task | Dataset | BioBERT | SciBERT |
|---|---|---|---|
| NER | BC5CDR | 88.85 | 90.01 |
| NER | JNLPBA | 77.59 | 77.28 |
| NER | NCBI-disease | 89.36 | 88.57 |
| REL | ChemProt | 76.68 | 83.64 |
表 2:在生物医学数据集上,将 SciBERT 与已报道的 BioBERT 结果进行比较。
| 任务 | 数据集 | BioBERT | SciBERT |
|---|---|---|---|
| NER | BC5CDR | 88.85 | 90.01 |
| NER | JNLPBA | 77.59 | 77.28 |
| NER | NCBI-disease | 89.36 | 88.57 |
| REL | ChemProt | 76.68 | 83.64 |
4.2 Computer Science Domain
4.2 计算机科学领域
We observe that SciBERT outperforms BERT-Base on computer science tasks (+3.55 F1 with finetuning and +1.13 F1 without). In addition, SciBERT achieves new SOTA results on ACL-ARC (Cohan et al., 2019), and the NER part of SciERC (Luan et al., 2018). For relations in SciERC, our results are not comparable with those in Luan et al. (2018) because we are performing relation classification given gold entities, while they perform joint entity and relation extraction.
可以看到,SciBERT 在计算机科学任务上超过了 BERT-Base(微调时高 3.55 个 F1,不微调时高 1.13 个 F1)。此外,SciBERT 在 ACL-ARC(Cohan et al., 2019)以及 SciERC 的 NER 部分(Luan et al., 2018)上取得了新的当时最优。对于 SciERC 中的关系,我们的结果无法与 Luan et al. (2018) 直接比较,因为我们是在给定黄金实体的条件下做关系分类,而他们做的是实体与关系的联合抽取。
4.3 Multiple Domains
4.3 多领域
We observe that SciBERT outperforms BERT-Base on the multidomain tasks (+0.49 F1 with finetuning and +0.93 F1 without). In addition, SciBERT outperforms the SOTA on SciCite (Cohan et al., 2019). No prior published SOTA results exist for the Paper Field dataset.
可以看到,SciBERT 在多领域任务上超过了 BERT-Base(微调时高 0.49 个 F1,不微调时高 0.93 个 F1)。此外,SciBERT 在 SciCite(Cohan et al., 2019)上超过了当时最优。Paper Field 数据集此前没有已发表的最优结果。
5 Discussion
5 讨论
5.1 Effect of Finetuning
5.1 微调的效果
We observe improved results via BERT finetuning rather than task-specific architectures atop frozen embeddings (+3.25 F1 with SciBERT and +3.58 with BERT-Base, on average). For each scientific domain, we observe the largest effects of finetuning on the computer science (+5.59 F1 with SciBERT and +3.17 F1 with BERT-Base) and biomedical tasks (+2.94 F1 with SciBERT and +4.61 F1 with BERT-Base), and the smallest effect on multidomain tasks (+0.7 F1 with SciBERT and +1.14 F1 with BERT-Base).
可以看到,对 BERT 做微调,比在冻住的嵌入之上搭建任务专用架构效果更好(平均而言,SciBERT 高 3.25 个 F1,BERT-Base 高 3.58 个 F1)。分科学领域来看,微调的效果在计算机科学(SciBERT 高 5.59 个 F1,BERT-Base 高 3.17 个 F1)和生物医学任务(SciBERT 高 2.94 个 F1,BERT-Base 高 4.61 个 F1)上最大,在多领域任务上最小(SciBERT 高 0.7 个 F1,BERT-Base 高 1.14 个 F1)。
On every dataset except BC5CDR and SciCite, BERT-Base with finetuning outperforms (or performs similarly to) a model using frozen SciBERT embeddings.
除 BC5CDR 和 SciCite 外,在每一个数据集上,经过微调的 BERT-Base 都超过(或接近)使用冻住 SciBERT 嵌入的模型。
5.2 Effect of SCIVOCAB
5.2 SCIVOCAB 的效果
We assess the importance of an in-domain scientific vocabulary by repeating the finetuning experiments for SciBERT with BASEVOCAB. We find the optimal hyperparameters for SciBERT-BASEVOCAB often coincide with those of SciBERT-SCIVOCAB.
为评估领域内科学词表的重要性,我们用 BASEVOCAB 重复了 SciBERT 的微调实验。我们发现,SciBERT-BASEVOCAB 的最优超参数常常与 SciBERT-SCIVOCAB 一致。
Averaged across datasets, we observe +0.60 F1 when using SCIVOCAB. For each scientific domain, we observe +0.76 F1 for biomedical tasks, +0.61 F1 for computer science tasks, and +0.11 F1 for multidomain tasks.
在各数据集上平均,使用 SCIVOCAB 可带来 0.60 个 F1 的提升。分科学领域来看,生物医学任务高 0.76 个 F1,计算机科学任务高 0.61 个 F1,多领域任务高 0.11 个 F1。
Given the disjoint vocabularies (Section 2) and the magnitude of improvement over BERT-Base (Section 4), we suspect that while an in-domain vocabulary is helpful, SciBERT benefits most from the scientific corpus pretraining.
考虑到两套词表几乎不相交(第 2 节),以及相对 BERT-Base 的提升幅度(第 4 节),我们推测:领域内词表固然有帮助,但 SciBERT 的主要收益仍来自在科学语料上的预训练。
6 Related Work
6 相关工作
Recent work on domain adaptation of BERT includes BioBERT (Lee et al., 2019) and ClinicalBERT (Alsentzer et al., 2019; Huang et al., 2019). BioBERT is trained on PubMed abstracts and PMC full text articles, and ClinicalBERT is trained on clinical text from the MIMIC-III database (Johnson et al., 2016). In contrast, SciBERT is trained on the full text of 1.14M biomedical and computer science papers from the Semantic Scholar corpus (Ammar et al., 2018). Furthermore, SciBERT uses an in-domain vocabulary (SCIVOCAB) while the other above-mentioned models use the original BERT vocabulary (BASEVOCAB).
近期对 BERT 做领域适配的工作包括 BioBERT(Lee et al., 2019)和 ClinicalBERT(Alsentzer et al., 2019; Huang et al., 2019)。BioBERT 在 PubMed 摘要和 PMC 全文上训练,ClinicalBERT 则在 MIMIC-III 数据库(Johnson et al., 2016)的临床文本上训练。与此不同,SciBERT 是在 Semantic Scholar 语料(Ammar et al., 2018)中 114 万篇生物医学和计算机科学论文的全文上训练的。此外,SciBERT 使用领域内词表(SCIVOCAB),而上述其他模型使用的是 BERT 原词表(BASEVOCAB)。
7 Conclusion and Future Work
7 结论与未来工作
We released SciBERT, a pretrained language model for scientific text based on BERT. We evaluated SciBERT on a suite of tasks and datasets from scientific domains. SciBERT significantly outperformed BERT-Base and achieves new SOTA results on several of these tasks, even compared to some reported BioBERT (Lee et al., 2019) results on biomedical tasks.
我们发布了 SciBERT,这是一个基于 BERT、面向科学文本的预训练语言模型。我们在来自科学领域的一组任务和数据集上评估了 SciBERT。SciBERT 显著超过了 BERT-Base,并在其中若干任务上取得了新的当时最优,即便与部分已报道的 BioBERT(Lee et al., 2019)生物医学结果相比也是如此。
For future work, we will release a version of SciBERT analogous to BERT-Large, as well as experiment with different proportions of papers from each domain. Because these language models are costly to train, we aim to build a single resource that’s useful across multiple domains.
在后续工作中,我们将发布一个类似于 BERT-Large 的 SciBERT 版本,并试验来自各领域论文的不同比例。由于训练这类语言模型成本很高,我们希望做出一份能在多个领域都有用的单一资源。
Acknowledgment
致谢
We thank the anonymous reviewers for their comments and suggestions. We also thank Waleed Ammar, Noah Smith, Yoav Goldberg, Daniel King, Doug Downey, and Dan Weld for their helpful discussions and feedback. All experiments were performed on beaker.org and supported in part by credits from Google Cloud.
感谢匿名审稿人的意见和建议。我们也感谢 Waleed Ammar、Noah Smith、Yoav Goldberg、Daniel King、Doug Downey 和 Dan Weld 的讨论与反馈。全部实验在 beaker.org 上完成,并部分得到了 Google Cloud 额度的支持。
References
参考文献
Emily Alsentzer, John R. Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew B. A. McDermott. 2019. Publicly available clinical bert embeddings. In ClinicalNLP workshop at NAACL.
Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, Rodney Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler Murray, Hsu-Han Ooi, Matthew Peters, Joanna Power, Sam Skjonsberg, Lucy Lu Wang, Chris Wilhelm, Zheng Yuan, Madeleine van Zuylen, and Oren Etzioni. 2018. Construction of the literature graph in semantic scholar. In NAACL.
Arman Cohan, Waleed Ammar, Madeleine van Zuylen, and Field Cady. 2019. Structural scaffolds for citation intent classification in scientific publications. In NAACL-HLT, pages 3586–3596, Minneapolis, Minnesota. Association for Computational Linguistics.
Nigel Collier and Jin-Dong Kim. 2004. Introduction to the bio-entity recognition task at jnlpba. In NLPBA/BioNLP.
Tim Dettmers. 2019. TPUs vs GPUs for Transformers (BERT). http://timdettmers.com/2018/10/17/tpus-vs-gpus-for-transformers-bert/. Accessed: 2019-02-22.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT.
Rezarta Islamaj Dogan, Robert Leaman, and Zhiyong Lu. 2014. NCBI disease corpus: A resource for disease name recognition and concept normalization. Journal of biomedical informatics, 47:1–10.
Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. ICLR.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017. Allennlp: A deep semantic natural language processing platform. In arXiv:1803.07640.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL.
Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv:1904.05342.
Alistair E. W. Johnson, Tom J. Pollard aand Lu Shen, Liwei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, , and Roger G. Mark. 2016. Mimic-iii, a freely accessible critical care database. In Scientific Data, 3:160035.
David Jurgens, Srijan Kumar, Raine Hoover, Daniel A. McFarland, and Daniel Jurafsky. 2018. Measuring the evolution of a scientific field through citation frames. TACL, 06:391–406.
Jin-Dong Kim, Tomoko Ohta, Yuka Tateisi, and Jun’ichi Tsujii. 2003. GENIA corpus - a semantically annotated corpus for bio-textmining. Bioinformatics, 19:i180i182.
Su Kim, David Martínez, Lawrence Cavedon, and Lars Yencken. 2011. Automatic classification of sentences to support evidence based medicine. In BMC Bioinformatics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. ICLR.
Jens Kringelum, Sonny Kim Kjærulff, Søren Brunak, Ole Lund, Tudor I. Oprea, and Olivier Taboureau. 2016. ChemProt-3.0: a global chemical biology diseases mapping. In Database.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. In arXiv:1901.08746.
Jiao Li, Yueping Sun, Robin J. Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Thomas C. Wiegers, and Zhiyong Lu. 2016. BioCreative V CDR task corpus: a resource for chemical disease relation extraction. Database : the journal of biological databases and curation.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In EMNLP.
Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019. ScispaCy: Fast and robust models for biomedical natural language processing. In arXiv:1902.07669.
Dat Quoc Nguyen and Karin M. Verspoor. 2019. From pos tagging to dependency parsing for biomedical event extraction. BMC Bioinformatics, 20:1–13.
Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain James Marshall, Ani Nenkova, and Byron C. Wallace. 2018. A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. In ACL.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke S. Zettlemoyer. 2018. Deep contextualized word representations. In NAACL-HLT.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Nils Reimers and Iryna Gurevych. 2017. Optimal hyperparameters for deep lstm-networks for sequence labeling tasks. In EMNLP.
Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Paul Hsu, and Kuansan Wang. 2015. An overview of microsoft academic service (MAS) and applications. In WWW.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. abs/1609.08144.
Wonjin Yoon, Chan Ho So, Jinhyuk Lee, and Jaewoo Kang. 2018. CollaboNet: collaboration of deep neural networks for biomedical named entity recognition. In DTMBio workshop at CIKM.