Decision Transformer: Reinforcement Learning via Sequence Modeling

Decision Transformer: Reinforcement Learning via Sequence Modeling

Decision Transformer：用序列建模做强化学习

Lili Chen Kevin Lu Aravind Rajeswaran Kimin Lee Aditya Grover Michael Laskin Pieter Abbeel Aravind Srinivas Affiliation: equal contribution equal advising Igor Mordatch Affiliation: equal contribution equal advising Affiliation: UC Berkeley Facebook AI Research Google Brain {lilichen, kzl}@berkeley.edu

Abstract

摘要

We introduce a framework that abstracts Reinforcement Learning (RL) as a sequence modeling problem. This allows us to draw upon the simplicity and scalability of the Transformer architecture, and associated advances in language modeling such as GPT-x and BERT. In particular, we present Decision Transformer, an architecture that casts the problem of RL as conditional sequence modeling. Unlike prior approaches to RL that fit value functions or compute policy gradients, Decision Transformer simply outputs the optimal actions by leveraging a causally masked Transformer. By conditioning an autoregressive model on the desired return (reward), past states, and actions, our Decision Transformer model can generate future actions that achieve the desired return. Despite its simplicity, Decision Transformer matches or exceeds the performance of state-of-the-art model-free offline RL baselines on Atari, OpenAI Gym, and Key-to-Door tasks.

我们提出一个把强化学习（RL）抽象为序列建模问题的框架，从而借用 Transformer 架构的简洁与可扩展性，以及 GPT-x、BERT 等语言模型进展。具体地，我们提出 Decision Transformer：把 RL 问题写成条件序列建模。与拟合价值函数或计算策略梯度的先前 RL 路线不同，Decision Transformer 仅借助因果掩码 Transformer 输出最优动作。通过对期望回报（奖励）、过去状态与动作条件化一个自回归模型，Decision Transformer 可生成达成期望回报的未来动作。尽管形式简单，它在 Atari、OpenAI Gym 与 Key-to-Door 任务上匹配或超过当时最强的无模型离线 RL 基线。

Figure 1: Decision Transformer architecture[^1]. States, actions, and returns are fed into modality-specific linear embeddings and a positional episodic timestep encoding is added. Tokens are fed into a GPT architecture which predicts actions autoregressively using a causal self-attention mask.

图 1：Decision Transformer 结构[^1]。状态、动作与回报送入模态专用线性嵌入，并加上按回合时间步的位置编码。token 送入 GPT 架构，在因果自注意力掩码下自回归预测动作。

1 Introduction

1 引言

Contents Recent work has shown transformers [1] can model high-dimensional distributions of semantic concepts at scale, including effective zero-shot generalization in language [2] and out-of-distribution image generation [3]. Given the diversity of successful applications of such models, we seek to examine their application to sequential decision making problems formalized as reinforcement learning (RL). In contrast to prior work using transformers as an architectural choice for components within traditional RL algorithms [4, 5], we seek to study if generative trajectory modeling - i.e. modeling the joint distribution of the sequence of states, actions, and rewards - can serve as a replacement for conventional RL algorithms.

目录 近期工作表明，Transformer [1] 能大规模建模语义概念的高维分布，包括语言中的有效零样本泛化 [2] 与分布外图像生成 [3]。鉴于此类模型应用多样，我们考察其在形式化为强化学习（RL）的序列决策问题上的应用。与把 Transformer 仅当作传统 RL 算法内部部件的先前工作 [4, 5] 不同，我们想研究：生成式轨迹建模——即建模状态、动作与奖励序列的联合分布——能否 替代 常规 RL 算法。

We consider the following shift in paradigm: instead of training a policy through conventional RL algorithms like temporal difference (TD) learning [6], we will train transformer models on collected experience using a sequence modeling objective. This will allow us to bypass the need for bootstrapping for long term credit assignment - thereby avoiding one of the “deadly triad” [6] known to destabilize RL. It also avoids the need for discounting future rewards, as typically done in TD learning, which can induce undesirable short-sighted behaviors. Additionally, we can make use of existing transformer frameworks widely used in language and vision that are easy to scale, utilizing a large body of work studying stable training of transformer models.

我们考虑如下范式转变：不再用时序差分（TD）学习 [6] 等常规 RL 算法训策略，而是用序列建模目标、在已收集经验上训练 Transformer。由此可绕开为长期信用分配做 bootstrap，从而避开已知会使 RL 不稳的「致命三角」之一 [6]；也不必像 TD 学习那样对未来奖励打折，后者常诱发短视行为。此外，可直接使用语言与视觉中广泛、易扩展的现成 Transformer 框架，以及大量关于稳定训练 Transformer 的研究。

In addition to their demonstrated ability to model long sequences, transformers also have other advantages. Transformers can perform credit assignment directly via self-attention, in contrast to Bellman backups which slowly propagate rewards and are prone to “distractor” signals [7]. This can enable transformers to still work effectively in the presence of sparse or distracting rewards. Finally, empirical evidence suggest that a transformer modeling approach can model a wide distribution of behaviors, enabling better generalization and transfer [3].

除擅长建模长序列外，Transformer 还有其他优点。它可通过自注意力直接做信用分配，相对缓慢传播奖励、易受「干扰」信号影响的 Bellman 备份 [7]。这使得在稀疏或干扰奖励下 Transformer 仍可有效工作。经验证据也表明，Transformer 建模方法能刻画广泛行为分布，利于更好泛化与迁移 [3]。

We explore our hypothesis by considering offline RL, where we will task agents with learning policies from suboptimal data - producing maximally effective behavior from fixed, limited experience. This task is traditionally challenging due to error propagation and value overestimation [8]. However, it is a natural task when training with a sequence modeling objective. By training an autoregressive model on sequences of states, actions, and returns, we reduce policy sampling to autoregressive generative modeling. We can specify the expertise of the policy - which “skill” to query - by selecting the desired return tokens, acting as a prompt for generation.

我们通过 离线 RL 检验假设：智能体需从次优数据学习策略——在固定、有限经验上产出尽可能有效的行为。该设定因误差传播与价值高估而一向困难 [8]，但对序列建模目标却很自然。在状态、动作与回报序列上训练自回归模型后，策略采样就变成自回归生成。通过选择期望回报 token 作为生成提示，可指定策略专长——查询何种「技能」。

Figure 2: Illustrative example of finding shortest path for a fixed graph (left) posed as reinforcement learning. Training dataset consists of random walk trajectories and their per-node returns-to-go (middle). Conditioned on a starting state and generating largest possible return at each node, Decision Transformer sequences optimal paths.

图 2：把固定图上找最短路（左）写成强化学习的示意。训练数据由随机游走轨迹及其每节点回报-to-go 组成（中）。条件于起始状态并在每个节点生成尽可能大的回报，Decision Transformer 可串出最优路径。

0

0

Motivated by this observation, we propose Decision Transformer, where we use the GPT architecture to autoregressively model trajectories (shown in Figure 1). We study whether sequence modeling can perform policy optimization by evaluating Decision Transformer on offline RL benchmarks in Atari [10], OpenAI Gym [11], and Key-to-Door [12] environments. We show that - without using dynamic programming - Decision Transformer matches or exceeds the performance of state-of-the-art model-free offline RL algorithms [13, 14]. Furthermore, in tasks where long-term credit assignment is required, Decision Transformer capably outperforms the RL baselines. With this work, we aim to bridge sequence modeling and transformers with RL, and hope that sequence modeling serves as a strong algorithmic paradigm for RL.

受此启发，我们提出 Decision Transformer：用 GPT 架构自回归建模轨迹（见图 1）。我们在 Atari [10]、OpenAI Gym [11] 与 Key-to-Door [12] 的离线 RL 基准上评估，检验序列建模能否做策略优化。结果表明—— 不使用动态规划 ——Decision Transformer 匹配或超过当时最强无模型离线 RL 算法 [13, 14]。在需要长期信用分配的任务上，它明显优于 RL 基线。本文旨在把序列建模与 Transformer 和 RL 桥接起来，并希望序列建模成为 RL 的有力算法范式。

2 Preliminaries

2 预备知识

2.1 Offline reinforcement learning

2.1 离线强化学习

\mathcal{S}

(\mathcal{S}

2.2 Transformers

2.2 Transformer

n

n

z_{i}=\sum_{j=1}^{n}\texttt{softmax}(\{\langle q_{i},k_{j^{\prime}}\rangle\}_{j^{\prime}=1}^{n})_{j}\cdot v_{j}. \tag{1}

n

n

3 Method

3 方法

In this section, we present Decision Transformer, which models trajectories autoregressively with minimal modification to the transformer architecture, as summarized in Figure 1 and Algorithm 1.

本节介绍 Decision Transformer：以对 Transformer 结构的最小改动自回归建模轨迹，概要见图 1 与算法 1。

\widehat{R}_{t}=\sum_{t^{\prime}=t}^{T}r_{t^{\prime}}

\widehat{R}_{t}=\sum_{t^{\prime}=t}^{T}r_{t^{\prime}}

\tau=\left(\widehat{R}_{1},s_{1},a_{1},\widehat{R}_{2},s_{2},a_{2},\dots,\widehat{R}_{T},s_{T},a_{T}\right). \tag{2}

At test time, we can specify the desired performance (e.g. 1 for success or 0 for failure), as well as the environment starting state, as the conditioning information to initiate generation. After executing the generated action for the current state, we decrement the target return by the achieved reward and repeat until episode termination.

测试时可指定期望表现（如成功为 1、失败为 0）以及环境起始状态，作为启动生成的条件信息。对当前状态执行生成动作后，用已获奖励递减目标回报，重复直至回合结束。

K

K

K

K

# R, s, a, t: returns-to-go, states, actions, or timesteps # transformer: transformer with causal masking (GPT) # embed_s, embed_a, embed_R: linear embedding layers # embed_t: learned episode positional embedding # pred_a: linear action prediction layer # main model def** DecisionTransformer (R, s, a, t):** # compute embeddings for tokens pos_embedding = embed_t (t)****# per-timestep (note: not per-token) s_embedding **= embed_s (s) + pos_embedding a_embedding = embed_a (a) + pos_embedding R_embedding = embed_R (R)****+** pos_embedding # interleave tokens as (R_1, s_1, a_1, ..., R_K, s_K) input_embeds = stack (R_embedding, s_embedding, a_embedding) # use transformer to get hidden states hidden_states = transformer (input_embeds = input_embeds) # select hidden states for action prediction tokens a_hidden **= unstack (hidden_states).** actions # predict action return** pred_a (a_hidden)** # training loop for (R, s, a, t) in dataloader****:****# dims: (batch_size, K, dim) a_preds = DecisionTransformer (R, s, a, t) loss = mean ((a_preds - a) 2)****# L2 loss for continuous actions optimizer . zero_grad (); loss . backward (); optimizer . step ()** # evaluation loop target_return = 1 # for instance, expert-level return R, s, a, t, done = [target_return], [env . reset ()], [], [1], False while not done :****# autoregressive generation/sampling # sample next action action = DecisionTransformer (R, s, a, t)[-1]****# for cts actions new_s, r, done,_= env . step (action) # append new tokens to sequence R = R + [R [-1] - r]****# decrement returns-to-go with reward s, a, t = s + [new_s], a + [action], t +****[len (R)] R, s, a, t = R [- K :], ... # only keep context length of K Algorithm 1 Decision Transformer Pseudocode (for continuous actions)

# R, s, a, t: returns-to-go, states, actions, or timesteps # transformer: transformer with causal masking (GPT) # embed_s, embed_a, embed_R: linear embedding layers # embed_t: learned episode positional embedding # pred_a: linear action prediction layer # main model def** DecisionTransformer (R, s, a, t) : # compute embeddings for tokens pos_embedding = embed_t (t) # per-timestep (note: not per-token) s_embedding = embed_s (s) + pos_embedding a_embedding = embed_a (a) + pos_embedding R_embedding = embed_R (R)****+** pos_embedding # interleave tokens as (R_1, s_1, a_1, ..., R_K, s_K) input_embeds = stack (R_embedding, s_embedding, a_embedding) # use transformer to get hidden states hidden_states = transformer (input_embeds = input_embeds) # select hidden states for action prediction tokens a_hidden = unstack (hidden_states) . actions # predict action return** pred_a (a_hidden)** # training loop for (R, s, a, t) in dataloader**:**# dims: (batch_size, K, dim) a_preds = DecisionTransformer (R, s, a, t) loss = mean ((a_preds - a) 2)****# L2 loss for continuous actions optimizer . zero_grad (); loss . backward (); optimizer . step ()** # evaluation loop target_return = 1 # for instance, expert-level return R, s, a, t, done =****[target_return], [env . reset ()], [], [1], False while not done : # autoregressive generation/sampling # sample next action action = DecisionTransformer (R, s, a, t)[-1]****# for cts actions new_s, r, done,_****= env . step (action) # append new tokens to sequence R = R + [R [-1] - r]****# decrement returns-to-go with reward s, a, t = s +****[new_s], a +****[action], t +****[len (R)] R, s, a, t = R [- K :],... # only keep context length of K 算法 1 Decision Transformer 伪代码（连续动作）

4 Evaluations on Offline RL Benchmarks

4 离线 RL 基准上的评估

In this section, we investigate the performance of Decision Transformer relative to dedicated offline RL and imitation learning algorithms. In particular, our primary points of comparison are model-free offline RL algorithms based on TD-learning, since our Decision Transformer architecture is fundamentally model-free in nature as well. Furthermore, TD-learning is the dominant paradigm in RL for sample efficiency, and also features prominently as a sub-routine in many model-based RL algorithms [16, 17]. We also compare with behavior cloning and variants, since it also involves a likelihood based policy learning formulation similar to ours. The exact algorithms depend on the environment but our motivations are as follows:

本节考察 Decision Transformer 相对专用离线 RL 与模仿学习算法的表现。主要对照是基于 TD 学习的无模型离线 RL，因为 Decision Transformer 本质上也是无模型的。TD 学习仍是样本效率上的主导范式，也是许多基于模型 RL 算法的子程序 [16, 17]。我们也与行为克隆及其变体比较，因其同样是基于似然的策略学习。具体算法随环境而变，动机如下：

• TD learning : most of these methods use an action-space constraint or value pessimism, and will be the most faithful comparison to Decision Transformer, representing standard RL methods. A state-of-the-art model-free method is Conservative Q-Learning (CQL) [14] which serves as our primary comparison. In addition, we also compare against other prior model-free RL algorithms like BEAR [18] and BRAC [19].

• TD 学习 ：这类方法大多使用动作空间约束或价值悲观，是对 Decision Transformer 最忠实的对照，代表标准 RL。当时最强无模型方法是 Conservative Q-Learning（CQL）[14]，作为主要对照。另与 BEAR [18]、BRAC [19] 等先前无模型 RL 比较。

• Imitation learning : this regime similarly uses supervised losses for training, rather than Bellman backups. We use behavior cloning here, and include a more detailed discussion in Section 5.1.

• 模仿学习 ：该设定同样用监督损失训练，而非 Bellman 备份。我们使用行为克隆，并在第 5.1 节更细讨论。

We evaluate on both discrete (Atari [10]) and continuous (OpenAI Gym [11]) control tasks. The former involves high-dimensional observation spaces and requires long-term credit assignment, while the latter requires fine-grained continuous control, representing a diverse set of tasks. Our main results are summarized in Figure 3, where we show averaged normalized performance for each domain.

我们在离散（Atari [10]）与连续（OpenAI Gym [11]）控制任务上评估。前者观察空间高维、需要长期信用分配；后者需要细粒度连续控制，任务多样。主要结果汇总于图 3，给出各领域平均归一化表现。

Figure 3: Results comparing Decision Transformer (ours) to TD learning (CQL) and behavior cloning across Atari, OpenAI Gym, and Minigrid. On a diverse set of tasks, Decision Transformer performs comparably or better than traditional approaches. Performance is measured by normalized episode return (see text for details).

图 3：在 Atari、OpenAI Gym 与 Minigrid 上，比较 Decision Transformer（本文）、TD 学习（CQL）与行为克隆。任务多样时，Decision Transformer 与传统方法相当或更好。表现以归一化回合回报衡量（细节见正文）。

4.1 Atari

The Atari benchmark [10] is challenging due to its high-dimensional visual inputs and difficulty of credit assignment arising from the delay between actions and resulting rewards. We evaluate our method on 1% of all samples in the DQN-replay dataset as per Agarwal et al. 2020, representing 500 thousand of the 50 million transitions observed by an online DQN agent [20] during training; we report the mean and standard deviation of 3 seeds. We normalize scores based on a professional gamer, following the protocol of Hafner et al. 2020, where 100 represents the professional gamer score and 0 represents a random policy.

Atari 基准 [10] 因高维视觉输入、以及动作与奖励延迟带来的信用分配困难而具挑战。我们按 Agarwal et al. 2020，在 DQN-replay 数据集全部样本的 1% 上评估，即在线 DQN 智能体 [20] 训练期间观察到的 5000 万转移中的 50 万；报告 3 个种子的均值与标准差。分数按职业玩家归一化，遵循 Hafner et al. 2020：100 为职业玩家，0 为随机策略。

\bf{267.5}\pm 97.5

Game	DT (Ours)	CQL	QR-DQN	REM	BC
Breakout	$\bf{267.5}\pm 97.5$	$211.1$	$17.1$	$8.9$	$138.9\pm 61.7$
Qbert	$15.4\pm 11.4$	$\bf{104.2}$	$0.0$	$0.0$	$17.3\pm 14.7$
Pong	$106.1\pm 8.1$	$\bf{111.9}$	$18.0$	$0.5$	$85.2\pm 20.0$
Seaquest	$\bf{2.5}\pm 0.4$	$1.7$	$0.4$	$0.7$	$2.1\pm 0.3$

\bf{267.5}\pm 97.5

游戏	DT（本文）	CQL	QR-DQN	REM	BC
Breakout	$\bf{267.5}\pm 97.5$	$211.1$	$17.1$	$8.9$	$138.9\pm 61.7$
Qbert	$15.4\pm 11.4$	$\bf{104.2}$	$0.0$	$0.0$	$17.3\pm 14.7$
Pong	$106.1\pm 8.1$	$\bf{111.9}$	$18.0$	$0.5$	$85.2\pm 20.0$
Seaquest	$\bf{2.5}\pm 0.4$	$1.7$	$0.4$	$0.7$	$2.1\pm 0.3$

K=30

K=30

4.2 OpenAI Gym

In this section, we consider the continuous control tasks from the D4RL benchmark [23]. We also consider a 2D reacher environment that is not part of the benchmark, and generate the datasets using a similar methodology to the D4RL benchmark. Reacher is a goal-conditioned task and has sparse rewards, so it represents a different setting than the standard locomotion environments (HalfCheetah, Hopper, and Walker). The different dataset settings are described below.

本节考虑 D4RL 基准 [23] 的连续控制任务。另考虑不在基准中的 2D reacher，并用类似 D4RL 的方法生成数据。Reacher 是目标条件任务且奖励稀疏，与标准运动环境（HalfCheetah、Hopper、Walker）设定不同。不同数据设定如下。

Medium: 1 million timesteps generated by a “medium” policy that achieves approximately one-third the score of an expert policy.

Medium：由达到约专家三分之一分数的「中等」策略生成的 100 万时间步。

Medium-Replay: the replay buffer of an agent trained to the performance of a medium policy (approximately 25k-400k timesteps in our environments).

Medium-Replay：训练到中等策略水平的智能体回放缓冲（我们环境中约 2.5 万-40 万时间步）。

Medium-Expert: 1 million timesteps generated by the medium policy concatenated with 1 million timesteps generated by an expert policy.

Medium-Expert：中等策略 100 万时间步与专家策略 100 万时间步拼接。

We compare to CQL [14], BEAR [18], BRAC [19], and AWR [24]. CQL represents the state-of-the-art in model-free offline RL, an instantiation of TD learning with value pessimism. Score are normalized so that 100 represents an expert policy, as per Fu et al. 2020. CQL numbers are reported from the original paper; BC numbers are run by us; and the other methods are reported from the D4RL paper. Our results are shown in Table 3. Decision Transformer achieves the highest scores in a majority of the tasks and is competitive with the state of the art in the remaining tasks.

我们与 CQL [14]、BEAR [18]、BRAC [19]、AWR [24] 比较。CQL 代表当时无模型离线 RL 前沿，是带价值悲观的 TD 学习实例。分数按 Fu et al. 2020 归一化，使 100 对应专家策略。CQL 数字取自原论文；BC 由我们运行；其余取自 D4RL 论文。结果见表 3。Decision Transformer 在多数任务取得最高分，其余任务也与当时最优竞争。

\bf{86.8}\pm 1.3

Dataset	Environment	DT (Ours)	CQL	BEAR	BRAC-v	AWR	BC
Medium-Expert	HalfCheetah	$\bf{86.8}\pm 1.3$	$62.4$	$53.4$	$41.9$	$52.7$	$59.9$
Medium-Expert	Hopper	$107.6\pm 1.8$	$\bf{111.0}$	$96.3$	$0.8$	$27.1$	$79.6$
Medium-Expert	Walker	$\bf{108.1}\pm 0.2$	$98.7$	$40.1$	$81.6$	$53.8$	$36.6$
Medium-Expert	Reacher	$\bf{89.1\pm 1.3}$	$30.6$	-	-	-	$73.3$
Medium	HalfCheetah	$42.6\pm 0.1$	$44.4$	$41.7$	$\bf{46.3}$	$37.4$	$43.1$
Medium	Hopper	$\bf{67.6}\pm 1.0$	$58.0$	$52.1$	$31.1$	$35.9$	$63.9$
Medium	Walker	$74.0\pm 1.4$	$79.2$	$59.1$	$\bf{81.1}$	$17.4$	$77.3$
Medium	Reacher	$\bf{51.2\pm 3.4}$	$26.0$	-	-	-	$\bf{48.9}$
Medium-Replay	HalfCheetah	$36.6\pm 0.8$	$46.2$	$38.6$	$\bf{47.7}$	$40.3$	$4.3$
Medium-Replay	Hopper	$\bf{82.7\pm 7.0}$	$48.6$	$33.7$	$0.6$	$28.4$	$27.6$
Medium-Replay	Walker	$\bf{66.6\pm 3.0}$	$26.7$	$19.2$	$0.9$	$15.5$	$36.9$
Medium-Replay	Reacher	$\bf{18.0\pm 2.4}$	$\bf{19.0}$	-	-	-	$5.4$
Average (Without Reacher)	$\bf{74.7}$	$63.9$	$48.2$	$36.9$	$34.3$	$46.4$
Average (All Settings)	$\bf{69.2}$	$54.2$	-	-	-	$47.7$

\bf{86.8}\pm 1.3

数据集	环境	DT（本文）	CQL	BEAR	BRAC-v	AWR	BC
Medium-Expert	HalfCheetah	$\bf{86.8}\pm 1.3$	$62.4$	$53.4$	$41.9$	$52.7$	$59.9$
Medium-Expert	Hopper	$107.6\pm 1.8$	$\bf{111.0}$	$96.3$	$0.8$	$27.1$	$79.6$
Medium-Expert	Walker	$\bf{108.1}\pm 0.2$	$98.7$	$40.1$	$81.6$	$53.8$	$36.6$
Medium-Expert	Reacher	$\bf{89.1\pm 1.3}$	$30.6$	-	-	-	$73.3$
Medium	HalfCheetah	$42.6\pm 0.1$	$44.4$	$41.7$	$\bf{46.3}$	$37.4$	$43.1$
Medium	Hopper	$\bf{67.6}\pm 1.0$	$58.0$	$52.1$	$31.1$	$35.9$	$63.9$
Medium	Walker	$74.0\pm 1.4$	$79.2$	$59.1$	$\bf{81.1}$	$17.4$	$77.3$
Medium	Reacher	$\bf{51.2\pm 3.4}$	$26.0$	-	-	-	$\bf{48.9}$
Medium-Replay	HalfCheetah	$36.6\pm 0.8$	$46.2$	$38.6$	$\bf{47.7}$	$40.3$	$4.3$
Medium-Replay	Hopper	$\bf{82.7\pm 7.0}$	$48.6$	$33.7$	$0.6$	$28.4$	$27.6$
Medium-Replay	Walker	$\bf{66.6\pm 3.0}$	$26.7$	$19.2$	$0.9$	$15.5$	$36.9$
Medium-Replay	Reacher	$\bf{18.0\pm 2.4}$	$\bf{19.0}$	-	-	-	$5.4$
平均（不含 Reacher）	$\bf{74.7}$	$63.9$	$48.2$	$36.9$	$34.3$	$46.4$
平均（全部设定）	$\bf{69.2}$	$54.2$	-	-	-	$47.7$

5 Discussion

5 讨论

5.1 Does Decision Transformer perform behavior cloning on a subset of the data?

5.1 Decision Transformer 是否只是在数据子集上做行为克隆？

X\%

X\%

X\in[10\%,25\%,40\%,100\%]

X\in[10\%,25\%,40\%,100\%]

42.6\pm 0.1

Dataset	Environment	DT (Ours)	10%BC	25%BC	40%BC	100%BC	CQL
Medium	HalfCheetah	$42.6\pm 0.1$	$42.9$	$43.0$	$43.1$	$43.1$	$\bf 44.4$
Medium	Hopper	$\bf 67.6\pm 1.0$	$65.9$	$65.2$	$65.3$	$63.9$	$58.0$
Medium	Walker	$74.0\pm 1.4$	$78.8$	$\bf 80.9$	$78.8$	$77.3$	$79.2$
Medium	Reacher	$51.2\pm 3.4$	$51.0$	$48.9$	$58.2$	$\bf 58.4$	$26.0$
Medium-Replay	HalfCheetah	$36.6\pm 0.8$	$40.8$	$40.9$	$41.1$	$4.3$	$\bf 46.2$
Medium-Replay	Hopper	$\bf 82.7\pm 7.0$	$70.6$	$58.6$	$31.0$	$27.6$	$48.6$
Medium-Replay	Walker	$66.6\pm 3.0$	$\bf 70.4$	$67.8$	$67.2$	$36.9$	$26.7$
Medium-Replay	Reacher	$18.0\pm 2.4$	$\bf 33.1$	$16.2$	$10.7$	$5.4$	$19.0$
Average	$56.1$	$\bf{56.7}$	$52.7$	$49.4$	$39.5$	$43.5$

42.6\pm 0.1

数据集	环境	DT（本文）	10%BC	25%BC	40%BC	100%BC	CQL
Medium	HalfCheetah	$42.6\pm 0.1$	$42.9$	$43.0$	$43.1$	$43.1$	$\bf 44.4$
Medium	Hopper	$\bf 67.6\pm 1.0$	$65.9$	$65.2$	$65.3$	$63.9$	$58.0$
Medium	Walker	$74.0\pm 1.4$	$78.8$	$\bf 80.9$	$78.8$	$77.3$	$79.2$
Medium	Reacher	$51.2\pm 3.4$	$51.0$	$48.9$	$58.2$	$\bf 58.4$	$26.0$
Medium-Replay	HalfCheetah	$36.6\pm 0.8$	$40.8$	$40.9$	$41.1$	$4.3$	$\bf 46.2$
Medium-Replay	Hopper	$\bf 82.7\pm 7.0$	$70.6$	$58.6$	$31.0$	$27.6$	$48.6$
Medium-Replay	Walker	$66.6\pm 3.0$	$\bf 70.4$	$67.8$	$67.2$	$36.9$	$26.7$
Medium-Replay	Reacher	$18.0\pm 2.4$	$\bf 33.1$	$16.2$	$10.7$	$5.4$	$19.0$
平均	$56.1$	$\bf{56.7}$	$52.7$	$49.4$	$39.5$	$43.5$

In contrast, when we study low data regimes - such as Atari, where we use 1% of a replay buffer as the dataset - %BC is weak (shown in Table 4). This suggests that in scenarios with relatively low amounts of data, Decision Transformer can outperform %BC by using all trajectories in the dataset to improve generalization, even if those trajectories are dissimilar from the return conditioning target. Our results indicate that Decision Transformer can be more effective than simply performing imitation learning on a subset of the dataset. On the tasks we considered, Decision Transformer either outperforms or is competitive to %BC, without the confound of having to select the optimal subset.

相较之下，在低数据设定——如只用回放缓冲 1% 的 Atari——%BC 很弱（表 4）。说明数据相对少时，Decision Transformer 可通过使用全部轨迹改善泛化，即使这些轨迹与回报条件目标不相似，从而超过 %BC。结果表明它可比「只在数据子集上模仿」更有效。在所考虑任务上，Decision Transformer 超过或持平 %BC，且无需选择最优子集这一混淆因素。

\bf{267.5}\pm 97.5

Game	DT (Ours)	10%BC	25%BC	40%BC	100%BC
Breakout	$\bf{267.5}\pm 97.5$	$28.5\pm 8.2$	$73.5\pm 6.4$	$108.2\pm 67.5$	$138.9\pm 61.7$
Qbert	$15.4\pm 11.4$	$6.6\pm 1.7$	$16.0\pm 13.8$	$11.8\pm 5.8$	$\bf{17.3}\pm 14.7$
Pong	$\bf{106.1}\pm 8.1$	$2.5\pm 0.2$	$13.3\pm 2.7$	$72.7\pm 13.3$	$85.2\pm 20.0$
Seaquest	$\bf{2.5}\pm 0.4$	$1.1\pm 0.2$	$1.1\pm 0.2$	$1.6\pm 0.4$	$2.1\pm 0.3$

\bf{267.5}\pm 97.5

游戏	DT（本文）	10%BC	25%BC	40%BC	100%BC
Breakout	$\bf{267.5}\pm 97.5$	$28.5\pm 8.2$	$73.5\pm 6.4$	$108.2\pm 67.5$	$138.9\pm 61.7$
Qbert	$15.4\pm 11.4$	$6.6\pm 1.7$	$16.0\pm 13.8$	$11.8\pm 5.8$	$\bf{17.3}\pm 14.7$
Pong	$\bf{106.1}\pm 8.1$	$2.5\pm 0.2$	$13.3\pm 2.7$	$72.7\pm 13.3$	$85.2\pm 20.0$
Seaquest	$\bf{2.5}\pm 0.4$	$1.1\pm 0.2$	$1.1\pm 0.2$	$1.6\pm 0.4$	$2.1\pm 0.3$

5.2 How well does Decision Transformer model the distribution of returns?

5.2 Decision Transformer 对回报分布建模得如何？

We evaluate the ability of Decision Transformer to understand return-to-go tokens by varying the desired target return over a wide range - evaluating the multi-task distribution modeling capability of transformers. Figure 4 shows the average sampled return accumulated by the agent over the course of the evaluation episode for varying values of target return. On every task, the desired target returns and the true observed returns are highly correlated. On some tasks like Pong, HalfCheetah and Walker, Decision Transformer generates trajectories that almost perfectly match the desired returns (as indicated by the overlap with the oracle line). Furthermore, on some Atari tasks like Seaquest, we can prompt the Decision Transformer with higher returns than the maximum episode return available in the dataset, demonstrating that Decision Transformer is sometimes capable of extrapolation.

我们在宽范围内改变期望目标回报，评估 Decision Transformer 对回报-to-go token 的理解——检验 Transformer 的多任务分布建模能力。图 4 给出不同目标回报下，智能体在评估回合中累积的平均采样回报。每项任务上，期望目标回报与真实观测回报高度相关。Pong、HalfCheetah、Walker 等任务上，生成轨迹几乎完美匹配期望回报（与 oracle 线重合）。此外在 Seaquest 等部分 Atari 任务上，可用高于数据集最大回合回报的值提示 Decision Transformer，说明有时具备外推能力。

Figure 4: Sampled (evaluation) returns accumulated by Decision Transformer when conditioned on the specified target (desired) returns. Top: Atari. Bottom: D4RL medium-replay datasets.

图 4：条件于指定目标（期望）回报时，Decision Transformer 累积的采样（评估）回报。 上 ： Atari。 下 ： D4RL medium-replay 数据集。

5.3 What is the benefit of using a longer context length?

5.3 更长上下文有何好处？

K

K

K=1

Game	DT (Ours)	DT with no context ( $K=1$ )
Breakout	$\bf{267.5}\pm 97.5$	$73.9\pm 10$
Qbert	$\bf{15.1}\pm 11.4$	$13.6\pm 11.3$
Pong	$\bf{106.1}\pm 8.1$	$2.5\pm 0.2$
Seaquest	$\bf{2.5}\pm 0.4$	$0.6\pm 0.1$

K=1

游戏	DT（本文）	无上下文 DT（ $K=1$ ）
Breakout	$\bf{267.5}\pm 97.5$	$73.9\pm 10$
Qbert	$\bf{15.1}\pm 11.4$	$13.6\pm 11.3$
Pong	$\bf{106.1}\pm 8.1$	$2.5\pm 0.2$
Seaquest	$\bf{2.5}\pm 0.4$	$0.6\pm 0.1$

5.4 Does Decision Transformer perform effective long-term credit assignment?

5.4 Decision Transformer 能否有效做长期信用分配？

To evaluate long-term credit assignment capabilities of our model, we consider a variant of the Key-to-Door environment proposed in Mesnard et al. 2020. This is a grid-based environment with a sequence of three phases: (1) in the first phase, the agent is placed in a room with a key; (2) then, the agent is placed in an empty room; (3) and finally, the agent is placed in a room with a door. The agent receives a binary reward when reaching the door in the third phase, but only if it picked up the key in the first phase. This problem is difficult for credit assignment because credit must be propagated from the beginning to the end of the episode, skipping over actions taken in the middle.

为评估长期信用分配能力，我们采用 Mesnard et al. 2020 提出的 Key-to-Door 环境变体。这是基于网格的环境，含三阶段序列：（1）智能体被放在有钥匙的房间；（2）再放到空房间；（3）最后放到有门的房间。仅当第一阶段拾起钥匙后，第三阶段到达门才给二元奖励。该问题信用分配困难：信用须从回合开头传到末尾，并跳过中间动作。

We train on datasets of trajectories generated by applying random actions and report success rates in Table 6. Furthermore, for the Key-to-Door environment we use the entire episode length as the context, rather than having a fixed content window as in the other environments. Methods that use highsight return information: our Decision Transformer model and %BC (trained only on successful episodes) are able to learn effective policies - producing near-optimal paths, despite only training on random walks. TD learning (CQL) cannot effectively propagate Q-values over the long horizons involved and gets poor performance.

我们在随机动作生成的轨迹数据集上训练，成功率见表 6。Key-to-Door 上我们用整段回合长度作上下文，而非其他环境的固定窗口。使用后见回报信息的方法——Decision Transformer 与仅在成功回合上训练的 %BC——能学到有效策略，走出近乎最优路径，尽管只在随机游走上训练。TD 学习（CQL）无法在所涉长视野上有效传播 Q 值，表现很差。

\bf{71.8}\%

Dataset	DT (Ours)	CQL	BC	%BC	Random
1K Random Trajectories	$\bf{71.8}\%$	$13.1\%$	$1.4\%$	$69.9\%$	$3.1\%$
10K Random Trajectories	$94.6\%$	$13.3\%$	$1.6\%$	$\bf{95.1}\%$	$3.1\%$

\bf{71.8}\%

数据集	DT（本文）	CQL	BC	%BC	随机
1K 随机轨迹	$\bf{71.8}\%$	$13.1\%$	$1.4\%$	$69.9\%$	$3.1\%$
10K 随机轨迹	$94.6\%$	$13.3\%$	$1.6\%$	$\bf{95.1}\%$	$3.1\%$

5.5 Can transformers be accurate critics in sparse reward settings?

5.5 稀疏奖励设定下 Transformer 能否成为准确评判器？

p(\hat{R}_{1})

p(\hat{R}_{1})

Figure 5: Left: Averages of running return probabilities predicted by the transformer model for three types of episode outcomes. Right: Transformer attention weights from all timesteps superimposed for a particular successful episode. The model attends to steps near pivotal events in the episode, such as picking up the key and reaching the door.

图 5： 左 ： Transformer 对三类回合结果预测的运行回报概率均值。 右 ： 某一成功回合上，所有时间步 Transformer 注意力权重叠加。模型关注接近关键事件的步骤，如拾起钥匙与到达门。

5.6 Does Decision Transformer perform well in sparse reward settings?

5.6 Decision Transformer 在稀疏奖励设定下表现如何？

A known weakness of TD learning algorithms is that they require densely populated rewards in order to perform well, which can be unrealistic and/or expensive. In contrast, Decision Transformer can improve robustness in these settings since it makes minimal assumptions on the density of the reward. To evaluate this, we consider a delayed return version of the D4RL benchmarks where the agent does not receive any rewards along the trajectory, and instead receives the cumulative reward of the trajectory in the final timestep. Our results for delayed returns are shown in Table 7. Delayed returns minimally affect Decision Transformer; and due to the nature of the training process, while imitation learning methods are reward agnostic. While TD learning collapses, Decision Transformer and %BC still perform well, indicating that Decision Transformer can be more robust to delayed rewards.

TD 学习算法的已知弱点是需要稠密奖励才能表现好，这往往不现实或昂贵。相较之下，Decision Transformer 对奖励密度假设很少，可提升这类设定下的稳健性。为此我们考虑 D4RL 基准的延迟回报版本：轨迹途中不给奖励，只在最后一步给整段累积奖励。延迟回报结果见表 7。延迟回报对 Decision Transformer 影响很小；训练过程本身也使模仿学习方法对奖励无关。TD 学习崩溃时，Decision Transformer 与 %BC 仍表现良好，说明前者对延迟奖励更稳健。

\bf{107.3\pm 3.5}

		Delayed (Sparse)	Agnostic	Original (Dense)
Dataset	Environment	DT (Ours)	CQL	BC	%BC	DT (Ours)	CQL
Medium-Expert	Hopper	$\bf{107.3\pm 3.5}$	$9.0$	$59.9$	$102.6$	$107.6$	$111.0$
Medium	Hopper	$60.7\pm 4.5$	$5.2$	$63.9$	$\bf{65.9}$	$67.6$	$58.0$
Medium-Replay	Hopper	$\bf{78.5\pm 3.7}$	$2.0$	$27.6$	$70.6$	$82.7$	$48.6$

\bf{107.3\pm 3.5}

		延迟（稀疏）	无关	原始（稠密）
数据集	环境	DT（本文）	CQL	BC	%BC	DT（本文）	CQL
Medium-Expert	Hopper	$\bf{107.3\pm 3.5}$	$9.0$	$59.9$	$102.6$	$107.6$	$111.0$
Medium	Hopper	$60.7\pm 4.5$	$5.2$	$63.9$	$\bf{65.9}$	$67.6$	$58.0$
Medium-Replay	Hopper	$\bf{78.5\pm 3.7}$	$2.0$	$27.6$	$70.6$	$82.7$	$48.6$

5.7 Why does Decision Transformer avoid the need for value pessimism or behavior regularization?

5.7 为何 Decision Transformer 不需要价值悲观或行为正则？

One key difference between Decision Transformer and prior offline RL algorithms is that we do not require policy regularization or conservatism to achieve good performance. Our conjecture is that TD-learning based algorithms learn an approximate value function and improve the policy by optimizing this value function. This act of optimizing a learned function can exacerbate and exploit any inaccuracies in the value function approximation, causing failures in policy improvement. Since Decision Transformer does not require explicit optimization using learned functions as objectives, it avoids the need for regularization or conservatism.

Decision Transformer 与先前离线 RL 算法的一个关键差别是：我们不需要策略正则或保守性即可取得好表现。我们的猜想是：基于 TD 学习的算法学习近似价值函数，再通过优化该价值函数改进策略。优化所学函数会放大并利用价值近似中的任何误差，导致策略改进失败。Decision Transformer 不以所学函数为显式优化目标，因而回避了正则或保守的需要。

5.8 How can Decision Transformer benefit online RL regimes?

5.8 Decision Transformer 如何惠及在线 RL？

Offline RL and the ability to model behaviors has the potential to enable sample-efficient online RL for downstream tasks. Works studying the transition from offline to online generally find that likelihood-based approaches, like our sequence modeling objective, are more successful [26, 27]. As a result, although we studied offline RL in this work, we believe Decision Transformer can meaningfully improve online RL methods by serving as a strong model for behavior generation. For instance, Decision Transformer can serve as a powerful “memorization engine” and in conjunction with powerful exploration algorithms like Go-Explore [28], has the potential to simultaneously model and generative a diverse set of behaviors.

离线 RL 与行为建模能力有望使下游任务的在线 RL 更样本高效。研究离线到在线过渡的工作普遍发现，基于似然的方法——如我们的序列建模目标——更成功 [26, 27]。因此，尽管本文研究离线 RL，我们相信 Decision Transformer 可作为强行为生成模型，有意义地改进在线 RL。例如，它可充当强大的「记忆引擎」，与 Go-Explore [28] 等强探索算法结合，有望同时建模并生成多样化行为。

6 Related Work

6 相关工作

6.1 Offline reinforcement learning

6.1 离线强化学习

To mitigate the impact of distribution shift in offline RL, prior algorithms either (a) constrain the policy action space [29, 30, 31] or (b) incorporate value pessimism [29, 14], or (c) incorporate pessimism into learned dynamics models [32, 33]. Since we do not use Decision Transformers to explicitly learn the dynamics model, we primarily compare against model-free algorithms in our work; in particular, adding a dynamics model tends to improve the performance of model-free algorithms. Another line of work explores learning wide behavior distribution from an offline dataset by learning a task-agnostic set of skills, either with likelihood-based approaches [34, 35, 36, 37] or by maximizing mutual information [38, 39, 40]. Our work is similar to the likelihood-based approaches, which do not use iterative Bellman updates - although we use a simpler sequence modeling objective instead of a variational method, and use rewards for conditional generation of behaviors.

为减轻离线 RL 中的分布偏移，先前算法或（a）约束策略动作空间 [29, 30, 31]，或（b）引入价值悲观 [29, 14]，或（c）把悲观引入所学动力学模型 [32, 33]。因我们不用 Decision Transformer 显式学动力学，主要与无模型算法比较；加入动力学模型往往会提升无模型方法。另一路线从离线数据学习宽行为分布，用似然方法 [34, 35, 36, 37] 或最大化互信息 [38, 39, 40] 学任务无关技能集。我们的工作接近似然路线——同样不用迭代 Bellman 更新——但用更简单的序列建模目标而非变分方法，并用奖励条件化生成行为。

6.2 Supervised learning in reinforcement learning settings

6.2 强化学习设定中的监督学习

K=1

K=1

Concurrent to our work, Janner et al. 2021 propose Trajectory Transformer, which is similar to Decision Transformer but additionally uses state and return prediction, as well as discretization, which incorporates model-based components. We believe that their experiments, in addition to our results, highlight the potential for sequence modeling to be a generally applicable idea for reinforcement learning.

与我们同期，Janner et al. 2021 提出 Trajectory Transformer，与 Decision Transformer 类似，但额外使用状态与回报预测以及离散化，引入基于模型的成分。我们相信他们的实验加上我们的结果，共同凸显序列建模作为强化学习一般思路的潜力。

6.3 Credit assignment

6.3 信用分配

Many works have studied better credit assignment via state-association, learning an architecture which decomposes the reward function such that certain “important” states comprise most of the credit [50, 51, 12]. They use the learned reward function to change the reward of an actor-critic algorithm to help propagate signal over long horizons. In particular, similar to our long-term setting, some works have specifically shown such state-associative architectures can perform better in delayed reward settings [52, 7, 53, 25]. In contrast, we allow these properties to naturally emerge in a transformer architecture, without having to explicitly learn a reward function or a critic.

许多工作通过状态关联研究更好的信用分配：学习分解奖励函数的结构，使某些「重要」状态占据大部分信用 [50, 51, 12]。它们用所学奖励函数改变 actor-critic 的奖励，以帮助长视野传播信号。与我们的长期设定类似，一些工作专门表明此类状态关联结构在延迟奖励设定下更好 [52, 7, 53, 25]。相较之下，我们让这些性质在 Transformer 结构中自然涌现，而无需显式学习奖励函数或评判器。

6.4 Conditional language generation

6.4 条件语言生成

Various works have studied guided generation for images [54] and language [55, 56]. Several works [57, 58, 59, 60, 61, 62] have explored training or fine-tuning of models for controllable text generation. Class-conditional language models can also be used to learn disciminators to guide generation [63, 55, 64, 65]. However, these approaches mostly assume constant “classes”, while in reinforcement learning the reward signal is time-varying. Furthermore, it is more natural to prompt the model desired target return and continuously decrease it by the observed rewards over time, since the transformer model and environment jointly generate the trajectory.

多篇工作研究图像 [54] 与语言 [55, 56] 的引导生成。若干工作 [57, 58, 59, 60, 61, 62] 探索为可控文本生成训练或微调模型。类条件语言模型也可学习判别器来引导生成 [63, 55, 64, 65]。但这些方法大多假设恒定「类别」，而强化学习中奖励信号随时间变化。此外，更自然的是用期望目标回报提示模型，并随时间用观测奖励不断递减——因为轨迹由 Transformer 与环境共同生成。

6.5 Attention and transformer models

6.5 注意力与 Transformer 模型

Transformers [1] have been applied successfully to many tasks in natural language processing [66, 9] and computer vision [67, 68]. However, transformers are relatively unstudied in RL, mostly due to differing nature of the problem, such as higher variance in training. Zambaldi et al. 2018 showed that augmenting transformers with relational reasoning improve performance in combinatorial environments and Ritter et al. 2020 showed iterative self-attention allowed for RL agents to better utilize episodic memories. Parisotto et al. 2020 discussed design decisions for more stable training of transformers in the high-variance RL setting. Unlike our work, these still use actor-critic algorithms for optimization, focusing on novelty in architecture. Additionally, in imitation learning, some works have studied transformers as a replacement for LSTMs: Dasari and Gupta 2020 study one-shot imitation learning, and Abramson et al. 2020 combine language and image modalities for text-conditioned behavior generation.

Transformer [1] 已成功用于自然语言处理 [66, 9] 与计算机视觉 [67, 68] 的多项任务。但在 RL 中研究相对较少，多因问题性质不同，例如训练方差更高。Zambaldi et al. 2018 表明用关系推理增强 Transformer 可提升组合环境表现；Ritter et al. 2020 表明迭代自注意力让 RL 智能体更好利用情景记忆。Parisotto et al. 2020 讨论了在高方差 RL 设定下更稳定训练 Transformer 的设计选择。与我们不同，这些工作仍用 actor-critic 优化，创新在结构。此外在模仿学习中，有工作把 Transformer 当作 LSTM 替代：Dasari and Gupta 2020 研究一次性模仿学习；Abramson et al. 2020 结合语言与图像模态做文本条件行为生成。

7 Conclusion

7 结论

We proposed Decision Transformer, seeking to unify ideas in language/sequence modeling and reinforcement learning. On standard offline RL benchmarks, we showed Decision Transformer can match or outperform strong algorithms designed explicitly for offline RL with minimal modifications from standard language modeling architectures.

我们提出 Decision Transformer，旨在统一语言/序列建模与强化学习的思路。在标准离线 RL 基准上，它以相对标准语言建模结构的最小改动，即可匹配或超过专为离线 RL 设计的强算法。

We hope this work inspires more investigation into using large transformer models for RL. We used a simple supervised loss that was effective in our experiments, but applications to large-scale datasets could benefit from self-supervised pretraining tasks. In addition, one could consider more sophisticated embeddings for returns, states, and actions - for instance, conditioning on return distributions to model stochastic settings instead of deterministic returns. Transformer models can also be used to model the state evolution of trajectory, potentially serving as an alternative to model-based RL, and we hope to explore this in future work.

希望本文激发更多用大型 Transformer 做 RL 的研究。我们使用在实验中有效的简单监督损失，但面向大规模数据的应用或可受益于自监督预训练任务。此外可考虑更精细的回报、状态与动作嵌入——例如条件于回报分布以建模随机设定，而非确定性回报。Transformer 也可建模轨迹的状态演化，有望作为基于模型 RL 的替代，我们希望在未来工作中探索。

For real-world applications, it is important to understand the types of errors transformers make in MDP settings and possible negative consequences, which are underexplored. It will also be important to consider the datasets we train models on, which can potentially add destructive biases, particularly as we consider studying augmenting RL agents with more data which may come from questionable sources. For instance, reward design by nefarious actors can potentially generate unintended behaviors as our model generates behaviors by conditioning on desired returns.

面向真实世界应用，需要理解 Transformer 在 MDP 设定中会犯何种错误及可能负面后果，这方面仍探索不足。同样重要的是审视训练数据：它们可能引入破坏性偏见，尤其当我们研究用更多可能来源可疑的数据增强 RL 智能体时。例如，恶意行为者设计的奖励可能产生非预期行为，因为我们的模型通过条件于期望回报生成行为。

Acknowledgements

致谢

This research was supported by Berkeley Deep Drive, Open Philanthropy, and the National Science Foundation under NSF:NRI #2024675. Part of this work was completed when Aravind Rajeswaran was a PhD student at the University of Washington, where he was supported by the J.P. Morgan PhD Fellowship in AI (2020-21). We also thank Luke Metz and Daniel Freeman for valuable feedback and discussions, as well as Justin Fu for assistance in setting up D4RL benchmarks, and Aviral Kumar for assistance with the CQL baselines and hyperparameters.

本研究得到 Berkeley Deep Drive、Open Philanthropy，以及国家科学基金会 NSF:NRI #2024675 的支持。部分工作完成于 Aravind Rajeswaran 在华盛顿大学攻读博士期间，当时他获 J.P. Morgan AI 博士奖学金（2020-21）支持。我们也感谢 Luke Metz 与 Daniel Freeman 的宝贵反馈与讨论，Justin Fu 协助搭建 D4RL 基准，以及 Aviral Kumar 协助 CQL 基线与超参。

References

参考文献

Hafner et al. [2020] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193 , 2020. Dabney et al. [2018] Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Conference on Artificial Intelligence , 2018. Fu et al. [2020] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020. Peng et al. [2019] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 , 2019. Raposo et al. [2021] David Raposo, Sam Ritter, Adam Santoro, Greg Wayne, Theophane Weber, Matt Botvinick, Hado van Hasselt, and Francis Song. Synthetic returns for long-term credit assignment. arXiv preprint arXiv:2102.12425 , 2021. Gao et al. [2018] Yang Gao, Huazhe Xu, Ji Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement learning from imperfect demonstrations. arXiv preprint arXiv:1802.05313 , 2018. Nair et al. [2020] Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 , 2020. Ecoffet et al. [2019] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901.10995 , 2019. Fujimoto et al. [2019] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning , 2019. Kumar et al. [2019b] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems , 2019b.

Watkins [1989] Christopher Watkins. Learning from delayed rewards. 01 1989. Mnih et al. [2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013. Srivastava et al. [2019] Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, Wojciech Jaśkowski, and Jürgen Schmidhuber. Training agents using upside-down reinforcement learning. arXiv preprint arXiv:1912.02877 , 2019. Kumar et al. [2019c] Aviral Kumar, Xue Bin Peng, and Sergey Levine. Reward-conditioned policies. arXiv preprint arXiv:1912.13465 , 2019c. ogm [2019] Acting without rewards. 2019. URL https://ogma.ai/2019/08/acting-without-rewards/ . Chen et al. [2020] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning , pages 1691-1703. PMLR, 2020. Ghosh et al. [2019] Dibya Ghosh, Abhishek Gupta, Justin Fu, Ashwin Reddy, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals without reinforcement learning. arXiv preprint arXiv:1912.06088 , 2019. Paster et al. [2020] Keiran Paster, Sheila A McIlraith, and Jimmy Ba. Planning from pixels using inverse dynamics models. arXiv preprint arXiv:2012.02419 , 2020. Janner et al. [2021] Michael Janner, Qiyang Li, and Sergey Levine. Reinforcement learning as one big sequence modeling problem. arXiv preprint arXiv:2106.02039 , 2021. Ferret et al. [2019] Johan Ferret, Raphaël Marinier, Matthieu Geist, and Olivier Pietquin. Self-attentional credit assignment for transfer in reinforcement learning. arXiv preprint arXiv:1907.08027 , 2019.

Abramson et al. [2020] Josh Abramson, Arun Ahuja, Iain Barr, Arthur Brussee, Federico Carnevale, Mary Cassin, Rachita Chhaparia, Stephen Clark, Bogdan Damoc, Andrew Dudzik, et al. Imitating interactive intelligence. arXiv preprint arXiv:2012.05672 , 2020. Wolf et al. [2020] Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. Transformers: State-of-the-art natural language processing. In Conference on Empirical Methods in Natural Language Processing: System Demonstrations , 2020. Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Appendix A Experimental Details

附录 A 实验细节

Code for experiments can be found in the supplementary material.

实验代码见补充材料。

A.1 Atari

We build our Decision Transformer implementation for Atari games off of minGPT (https://github.com/karpathy/minGPT), a publicly available re-implementation of GPT. We use most of the default hyperparameters from their character-level GPT example (https://github.com/karpathy/minGPT/blob/master/play_char.ipynb). We reduce the batch size (except in Pong), block size, number of layers, attention heads, and embedding dimension for faster training. For processing the observations, we use the DQN encoder from Mnih et al. 2015 with an additional linear layer to project to the embedding dimension.

Atari 上的 Decision Transformer 实现基于公开的 GPT 复现 minGPT（ https://github.com/karpathy/minGPT）。多数超参取自其字符级 GPT 示例（ https://github.com/karpathy/minGPT/blob/master/play_char.ipynb）。为加快训练，我们减小 batch size（Pong 除外）、block size、层数、注意力头数与嵌入维。观察处理使用 Mnih et al. 2015 的 DQN 编码器，再加线性层投影到嵌入维。

1\times

1\times

6

Hyperparameter	Value
Number of layers	$6$
Number of attention heads	$8$
Embedding dimension	$128$
Batch size	$512$ Pong
	$128$ Breakout, Qbert, Seaquest
Context length $K$	$50$ Pong
	$30$ Breakout, Qbert, Seaquest
Return-to-go conditioning	$90$ Breakout ( $\approx 1\times$ max in dataset)
	$2500$ Qbert ( $\approx 5\times$ max in dataset)
	$20$ Pong ( $\approx 1\times$ max in dataset)
	$1450$ Seaquest ( $\approx 5\times$ max in dataset)
Nonlinearity	ReLU, encoder
	GeLU, otherwise
Encoder channels	$32,64,64$
Encoder filter sizes	$8\times 8,4\times 4,3\times 3$
Encoder strides	$4,2,1$
Max epochs	$5$
Dropout	$0.1$
Learning rate	$6*10^{-4}$
Adam betas	$(0.9,0.95)$
Grad norm clip	$1.0$
Weight decay	$0.1$
Learning rate decay	Linear warmup and cosine decay (see code for details)
Warmup tokens	$512*20$
Final tokens	$2500000K$

6

超参数	取值
层数	$6$
注意力头数	$8$
嵌入维	$128$
Batch size	$512$ Pong
	$128$ Breakout, Qbert, Seaquest
上下文长度 $K$	$50$ Pong
	$30$ Breakout, Qbert, Seaquest
回报-to-go 条件化	$90$ Breakout（约数据集最大的 $1\times$ ）
	$2500$ Qbert（约数据集最大的 $5\times$ ）
	$20$ Pong（约数据集最大的 $1\times$ ）
	$1450$ Seaquest（约数据集最大的 $5\times$ ）
非线性	ReLU，编码器
	GeLU，其余
编码器通道	$32,64,64$
编码器滤波器尺寸	$8\times 8,4\times 4,3\times 3$
编码器步长	$4,2,1$
最大 epoch	$5$
Dropout	$0.1$
学习率	$6*10^{-4}$
Adam betas	$(0.9,0.95)$
梯度范数裁剪	$1.0$
权重衰减	$0.1$
学习率衰减	线性预热与余弦衰减（细节见代码）
Warmup tokens	$512*20$
Final tokens	$2500000K$

A.2 OpenAI Gym

10^{5}

10^{5}

3

Hyperparameter	Value
Number of layers	$3$
Number of attention heads	$1$
Embedding dimension	$128$
Nonlinearity function	ReLU
Batch size	$64$
Context length $K$	$20$ HalfCheetah, Hopper, Walker
	$5$ Reacher
Return-to-go conditioning	$6000$ HalfCheetah
	$3600$ Hopper
	$5000$ Walker
	$50$ Reacher
Dropout	$0.1$
Learning rate	$10^{-4}$
Grad norm clip	$0.25$
Weight decay	$10^{-4}$
Learning rate decay	Linear warmup for first $10^{5}$ training steps

3

超参数	取值
层数	$3$
注意力头数	$1$
嵌入维	$128$
非线性	ReLU
Batch size	$64$
上下文长度 $K$	$20$ HalfCheetah, Hopper, Walker
	$5$ Reacher
回报-to-go 条件化	$6000$ HalfCheetah
	$3600$ Hopper
	$5000$ Walker
	$50$ Reacher
Dropout	$0.1$
学习率	$10^{-4}$
梯度范数裁剪	$0.25$
权重衰减	$10^{-4}$
学习率衰减	前 $10^{5}$ 训练步线性预热

2.5\times 10^{4}

2.5\times 10^{4}

3

Hyperparameter	Value
Number of layers	$3$
Embedding dimension	$256$
Nonlinearity function	ReLU
Batch size	$64$
Dropout	$0.1$
Learning rate	$10^{-4}$
Weight decay	$10^{-4}$
Learning rate decay	Linear warmup for first $10^{5}$ training steps

3

超参数	取值
层数	$3$
嵌入维	$256$
非线性	ReLU
Batch size	$64$
Dropout	$0.1$
学习率	$10^{-4}$
权重衰减	$10^{-4}$
学习率衰减	前 $10^{5}$ 训练步线性预热

A.3 Graph Shortest Path

A.3 图最短路

0

0

\hat{R}

\hat{R}

1,000

T=10

\infty

\infty

15.8\%

15.8\%

While this is a simple example and uses a prior on generation that we do not use in other experiments for simplicity, it illustrates how hindsight return information can be used with generation priors to avoid the need for explicit dynamic programming.

虽是简单例子，且为简洁在其他实验未使用生成先验，但它说明：后见回报信息配合生成先验，可避免显式动态规划。

Appendix B Atari Task Scores

附录 B Atari 任务分数

Table 11 shows the normalized scores used for normalization used in Hafner et al. 2020. Tables 12 and 13 show the raw scores corresponding to Tables 1 and 4, respectively. For %BC scores, we use the same hyperparameters as Decision Transformer for fair comparison. For REM and QR-DQN, there is a slight discrepancy between Agarwal et al. 2020 and Kumar et al. 2020; we report raw data provided to us by REM authors.

表 11 给出 Hafner et al. 2020 归一化所用分数。表 12 与 13 分别对应表 1 与 4 的原始分数。%BC 分数使用与 Decision Transformer 相同的超参以便公平比较。REM 与 QR-DQN 在 Agarwal et al. 2020 与 Kumar et al. 2020 之间略有出入；我们报告 REM 作者提供的原始数据。

2

Game	Random	Gamer
Breakout	$2$	$30$
Qbert	$164$	$13455$
Pong	$-21$	$15$
Seaquest	$68$	$42055$

2

游戏	随机	玩家
Breakout	$2$	$30$
Qbert	$164$	$13455$
Pong	$-21$	$15$
Seaquest	$68$	$42055$

\bf{76.9}\pm 27.3

Game	DT (Ours)	CQL	QR-DQN	REM	BC
Breakout	$\bf{76.9}\pm 27.3$	$61.1$	$6.8$	$4.5$	$40.9\pm 17.3$
Qbert	$2215.8\pm 1523.7$	$\bf{14012.0}$	$156.0$	$160.1$	$2464.1\pm 1948.2$
Pong	$17.1\pm 2.9$	$\bf{19.3}$	$-14.5$	$-20.8$	$9.7\pm 7.2$
Seaquest	$\bf{1129.3}\pm 189.0$	$779.4$	$250.1$	$370.5$	$968.6\pm 133.8$

\bf{76.9}\pm 27.3

游戏	DT（本文）	CQL	QR-DQN	REM	BC
Breakout	$\bf{76.9}\pm 27.3$	$61.1$	$6.8$	$4.5$	$40.9\pm 17.3$
Qbert	$2215.8\pm 1523.7$	$\bf{14012.0}$	$156.0$	$160.1$	$2464.1\pm 1948.2$
Pong	$17.1\pm 2.9$	$\bf{19.3}$	$-14.5$	$-20.8$	$9.7\pm 7.2$
Seaquest	$\bf{1129.3}\pm 189.0$	$779.4$	$250.1$	$370.5$	$968.6\pm 133.8$

\bf{76.9}\pm 27.3

Game	DT (Ours)	10%BC	25%BC	40%BC	100%BC
Breakout	$\bf{76.9}\pm 27.3$	$10.0\pm 2.3$	$22.6\pm 1.8$	$32.3\pm 18.9$	$40.9\pm 17.3$
Qbert	$2215.8\pm 1523.7$	$1045\pm 232.0$	$2302.5\pm 1844.1$	$1674.1\pm 776.0$	$\bf{2464.1}\pm 1948.2$
Pong	$\bf{17.1}\pm 2.9$	$-20.3\pm 0.1$	$-16.2\pm 1.0$	$5.2\pm 4.8$	$9.7\pm 7.2$
Seaquest	$\bf{1129.3}\pm 189.0$	$521.3\pm 103.0$	$549.3\pm 96.2$	$758\pm 169.1$	$968.6\pm 133.8$

\bf{76.9}\pm 27.3

Decision Transformer: Reinforcement Learning via Sequence Modeling

Decision Transformer：用序列建模做强化学习

Abstract

摘要

1 Introduction

1 引言

Contents

目录

2 Preliminaries

2 预备知识

2.1 Offline reinforcement learning

2.1 离线强化学习

2.2 Transformers

2.2 Transformer

3 Method

3 方法

4 Evaluations on Offline RL Benchmarks

4 离线 RL 基准上的评估

4.1 Atari

4.2 OpenAI Gym

5 Discussion

5 讨论

5.1 Does Decision Transformer perform behavior cloning on a subset of the data?

5.1 Decision Transformer 是否只是在数据子集上做行为克隆？

5.2 How well does Decision Transformer model the distribution of returns?

5.2 Decision Transformer 对回报分布建模得如何？

5.3 What is the benefit of using a longer context length?

5.3 更长上下文有何好处？

5.4 Does Decision Transformer perform effective long-term credit assignment?

5.4 Decision Transformer 能否有效做长期信用分配？

5.5 Can transformers be accurate critics in sparse reward settings?

5.5 稀疏奖励设定下 Transformer 能否成为准确评判器？

5.6 Does Decision Transformer perform well in sparse reward settings?

5.6 Decision Transformer 在稀疏奖励设定下表现如何？

5.7 Why does Decision Transformer avoid the need for value pessimism or behavior regularization?

5.7 为何 Decision Transformer 不需要价值悲观或行为正则？

5.8 How can Decision Transformer benefit online RL regimes?

5.8 Decision Transformer 如何惠及在线 RL？

6 Related Work

6 相关工作

6.1 Offline reinforcement learning

6.1 离线强化学习

6.2 Supervised learning in reinforcement learning settings

6.2 强化学习设定中的监督学习

6.3 Credit assignment

6.3 信用分配

6.4 Conditional language generation

6.4 条件语言生成

6.5 Attention and transformer models

6.5 注意力与 Transformer 模型

7 Conclusion

7 结论

Acknowledgements

致谢

References

参考文献

Appendix A Experimental Details

附录 A 实验细节

A.1 Atari

A.2 OpenAI Gym

A.2.1 Decision Transformer

A.2.1 Decision Transformer

A.2.2 Behavior Cloning

A.2.2 行为克隆

A.3 Graph Shortest Path

A.3 图最短路

Appendix B Atari Task Scores

附录 B Atari 任务分数