Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
∗Equal contribution. Corresponding authors: ryan.lowe@cs.mcgill.ca, jxwuyi@gmail.com, mordatch@openai.com.
31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
第 31 届神经信息处理系统大会(NIPS 2017),美国加利福尼亚州长滩。
Abstract
摘要
We explore deep reinforcement learning methods for multi-agent domains. We begin by analyzing the difficulty of traditional algorithms in the multi-agent case: Q-learning is challenged by an inherent non-stationarity of the environment, while policy gradient suffers from a variance that increases as the number of agents grows. We then present an adaptation of actor-critic methods that considers action policies of other agents and is able to successfully learn policies that require complex multi-agent coordination. Additionally, we introduce a training regimen utilizing an ensemble of policies for each agent that leads to more robust multi-agent policies. We show the strength of our approach compared to existing methods in cooperative as well as competitive scenarios, where agent populations are able to discover various physical and informational coordination strategies.
我们探索面向多智能体领域的深度强化学习方法。我们首先分析传统算法在多智能体情形下的困难:Q 学习会受到环境固有非平稳性的挑战,而策略梯度则会随着智能体数目增加而出现更大的方差。随后,我们提出一种改进的演员–评论家方法,它会考虑其他智能体的动作策略,并能够成功学习需要复杂多智能体协调的策略。此外,我们引入一种训练安排,为每个智能体使用一组策略集成,从而得到更稳健的多智能体策略。我们表明,与已有方法相比,该方法在合作以及竞争场景中都更有优势,智能体群体能够发现多种身体上和信息上的协调策略。
1 Introduction
1 引言
Reinforcement learning (RL) has recently been applied to solve challenging problems, from game playing [23, 28] to robotics [18]. In industrial applications, RL is seeing use in large scale systems such as data center cooling [1]. Most of the successes of RL have been in single agent domains, where modelling or predicting the behaviour of other actors in the environment is largely unnecessary.
强化学习(RL)近来已被用来解决有挑战性的问题,从游戏博弈 [23, 28] 到机器人 [18]。在工业应用中,强化学习也开始用于数据中心冷却 [1] 这类大规模系统。强化学习的多数成功发生在单智能体领域,在那里对其他行动者的行为进行建模或预测,在很大程度上是不必要的。
However, there are a number of important applications that involve interaction between multiple agents, where emergent behavior and complexity arise from agents co-evolving together. For example, multi-robot control [20], the discovery of communication and language [29, 8, 24], multiplayer games [27], and the analysis of social dilemmas [17] all operate in a multi-agent domain. Related problems, such as variants of hierarchical reinforcement learning [6] can also be seen as a multi-agent system, with multiple levels of hierarchy being equivalent to multiple agents. Additionally, multi-agent self-play has recently been shown to be a useful training paradigm [28, 30]. Successfully scaling RL to environments with multiple agents is crucial to building artificially intelligent systems that can productively interact with humans and each other.
不过,也有许多重要应用涉及多个智能体之间的交互,涌现行为和复杂性来自智能体的共同演化。例如,多机器人控制 [20]、通信与语言的发现 [29, 8, 24]、多人游戏 [27] 以及社会困境的分析 [17],都运行在多智能体领域。相关问题,例如层次强化学习的若干变体 [6],也可以看成一个多智能体系统,多层次的层级相当于多个智能体。此外,多智能体自我对弈近来也被证明是一种有用的训练范式 [28, 30]。把强化学习成功扩展到有多个智能体的环境,对于构建能够与人类以及彼此有效交互的人工智能系统至关重要。
Unfortunately, traditional reinforcement learning approaches such as Q-Learning or policy gradient are poorly suited to multi-agent environments. One issue is that each agent’s policy is changing as training progresses, and the environment becomes non-stationary from the perspective of any individual agent (in a way that is not explainable by changes in the agent’s own policy). This presents learning stability challenges and prevents the straightforward use of past experience replay, which is crucial for stabilizing deep Q-learning. Policy gradient methods, on the other hand, usually exhibit very high variance when coordination of multiple agents is required. Alternatively, one can use model-based policy optimization which can learn optimal policies via back-propagation, but this requires a (differentiable) model of the world dynamics and assumptions about the interactions between agents. Applying these methods to competitive environments is also challenging from an optimization perspective, as evidenced by the notorious instability of adversarial training methods [11].
遗憾的是,Q 学习或策略梯度这类传统强化学习方法,并不适合多智能体环境。一个问题是,每个智能体的策略都随着训练推进而改变,从任何一个智能体的角度看,环境都变成非平稳的(而且这种变化不能用该智能体自身策略的改变来解释)。这会带来学习稳定性上的挑战,也使得不能直接使用对稳定深度 Q 学习至关重要的过往经验回放。另一方面,当需要多个智能体协调时,策略梯度方法通常会表现出非常高的方差。或者,也可以使用基于模型的策略优化,通过反向传播来学习最优策略,但这需要一个(可微的)世界动力学模型,以及对智能体之间交互的假设。把这些方法用到竞争环境中,从优化角度看也很困难,对抗训练方法出了名的不稳定就是一个例子 [11]。
In this work, we propose a general-purpose multi-agent learning algorithm that: (1) leads to learned policies that only use local information (i.e. their own observations) at execution time, (2) does not assume a differentiable model of the environment dynamics or any particular structure on the communication method between agents, and (3) is applicable not only to cooperative interaction but to competitive or mixed interaction involving both physical and communicative behavior. The ability to act in mixed cooperative-competitive environments may be critical for intelligent agents; while competitive training provides a natural curriculum for learning [30], agents must also exhibit cooperative behavior (e.g. with humans) at execution time.
在本工作中,我们提出一种通用的多智能体学习算法,它:(1)得到的策略在执行时只使用局部信息(也就是各自的观察);(2)不假设环境动力学是可微的,也不对智能体之间的通信方式做任何特定结构假设;(3)不仅适用于合作交互,也适用于包含身体行为和通信行为的竞争或混合交互。在合作与竞争混合的环境中行动的能力,对智能体可能至关重要;竞争性训练虽然为学习提供了一种自然的课程 [30],智能体在执行时也必须表现出合作行为(例如与人类合作)。
We adopt the framework of centralized training with decentralized execution, allowing the policies to use extra information to ease training, so long as this information is not used at test time. It is unnatural to do this with Q-learning without making additional assumptions about the structure of the environment, as the Q function generally cannot contain different information at training and test time. Thus, we propose a simple extension of actor-critic policy gradient methods where the critic is augmented with extra information about the policies of other agents, while the actor only has access to local information. After training is completed, only the local actors are used at execution phase, acting in a decentralized manner and equally applicable in cooperative and competitive settings. This is a natural setting for multi-agent language learning, as full centralization would not require the development of discrete communication protocols.
我们采用集中训练、分散执行的框架,允许策略在训练时使用额外信息以降低训练难度,只要这些信息在测试时不被使用。若不对环境结构做额外假设,用 Q 学习很难做到这一点,因为 Q 函数通常不能在训练和测试时包含不同的信息。因此,我们提出对演员–评论家策略梯度方法做一个简单扩展:评论家被补充以关于其他智能体策略的额外信息,而演员只使用局部信息。训练完成后,执行阶段只用局部演员,以分散方式行动,在合作和竞争设置中同样适用。这对多智能体语言学习是一种自然的设置,因为如果完全集中,就不需要发展离散的通信协议。
Since the centralized critic function explicitly uses the decision-making policies of other agents, we additionally show that agents can learn approximate models of other agents online and effectively use them in their own policy learning procedure. We also introduce a method to improve the stability of multi-agent policies by training agents with an ensemble of policies, thus requiring robust interaction with a variety of collaborator and competitor policies. We empirically show the success of our approach compared to existing methods in cooperative as well as competitive scenarios, where agent populations are able to discover complex physical and communicative coordination strategies.
由于集中式评论家函数显式使用了其他智能体的决策策略,我们还表明,智能体可以在线学习对其他智能体的近似模型,并有效地用在自己的策略学习过程中。我们还引入一种方法,通过用一组策略集成来训练智能体,从而提高多智能体策略的稳定性,这要求它们能与多种多样的合作者和竞争者策略进行稳健的交互。我们用实验表明,与已有方法相比,该方法在合作以及竞争场景中都能取得成功,智能体群体能够发现复杂的身体与通信协调策略。
2 Related Work
2 相关工作
The simplest approach to learning in multi-agent settings is to use independently learning agents. This was attempted with Q-learning in [34], but does not perform well in practice [22]. As we will show, independently-learning policy gradient methods also perform poorly. One issue is that each agent’s policy changes during training, resulting in a non-stationary environment and preventing the naïve application of experience replay. Previous work has attempted to address this by inputting other agent’s policy parameters to the Q function [35], explicitly adding the iteration index to the replay buffer, or using importance sampling [9]. Deep Q-learning approaches have previously been investigated in [33] to train competing Pong agents.
在多智能体设置中学习,最简单的做法是让智能体独立学习。这一点曾在 [34] 中用 Q 学习尝试过,但实践中表现并不好 [22]。我们也将表明,独立学习的策略梯度方法同样表现不佳。一个问题是,每个智能体的策略在训练过程中会改变,导致环境非平稳,从而无法朴素地使用经验回放。以往工作曾试图应对这一点,例如把其他智能体的策略参数输入 Q 函数 [35],在回放缓冲区中显式加入迭代编号,或使用重要性采样 [9]。深度 Q 学习方法此前也在 [33] 中被用来训练相互竞争的 Pong 智能体。
The nature of interaction between agents can either be cooperative, competitive, or both and many algorithms are designed only for a particular nature of interaction. Most studied are cooperative settings, with strategies such as optimistic and hysteretic Q function updates [15, 21, 25], which assume that the actions of other agents are made to improve collective reward. Another approach is to indirectly arrive at cooperation via sharing of policy parameters [12], but this requires homogeneous agent capabilities. These algorithms are generally not applicable in competitive or mixed settings. See [26, 4] for surveys of multi-agent learning approaches and applications.
智能体之间的交互可以是合作、竞争,或两者兼有,而许多算法只为某一种交互性质而设计。研究最多的是合作设置,所用策略包括乐观和滞后的 Q 函数更新 [15, 21, 25],它们假定其他智能体的动作是为了提高集体回报。另一种做法是通过共享策略参数间接达到合作 [12],但这要求智能体能力是同质的。这些算法一般不适用于竞争或混合设置。关于多智能体学习方法与应用的综述,可见 [26, 4]。
Concurrently to our work, [7] proposed a similar idea of using policy gradient methods with a centralized critic, and test their approach on a StarCraft micromanagement task. Their approach differs from ours in the following ways: (1) they learn a single centralized critic for all agents, whereas we learn a centralized critic for each agent, allowing for agents with differing reward functions including competitive scenarios, (2) we consider environments with explicit communication between agents, (3) they combine recurrent policies with feed-forward critics, whereas our experiments use feed-forward policies (although our methods are applicable to recurrent policies), (4) we learn continuous policies whereas they learn discrete policies.
与我们的工作同时,[7] 提出了类似的想法,使用带集中式评论家的策略梯度方法,并在《星际争霸》微操任务上测试。他们的方法与我们有以下不同:(1)他们为所有智能体学习一个共享的集中式评论家,而我们为每个智能体学习一个集中式评论家,从而允许智能体具有不同的奖励函数,也包括竞争场景;(2)我们考虑智能体之间有显式通信的环境;(3)他们把循环策略与前馈评论家结合起来,而我们的实验使用前馈策略(尽管我们的方法也适用于循环策略);(4)我们学习连续策略,而他们学习离散策略。
Recent work has focused on learning grounded cooperative communication protocols between agents to solve various tasks [29, 8, 24]. However, these methods are usually only applicable when the communication between agents is carried out over a dedicated, differentiable communication channel.
近期工作关注学习智能体之间有根基的合作通信协议,以解决各种任务 [29, 8, 24]。不过,这些方法通常只适用于智能体之间的通信是通过专用、可微的通信信道进行的情形。
Our method requires explicitly modeling decision-making process of other agents. The importance of such modeling has been recognized by both reinforcement learning [3, 5] and cognitive science communities [10]. [13] stressed the importance of being robust to the decision making process of other agents, as do others by building Bayesian models of decision making. We incorporate such robustness considerations by requiring that agents interact successfully with an ensemble of any possible policies of other agents, improving training stability and robustness of agents after training.
我们的方法需要显式建模其他智能体的决策过程。这种建模的重要性,强化学习 [3, 5] 和认知科学 [10] 两个社区都已认识到。[13] 强调了对其他智能体决策过程保持稳健的重要性,也有人通过建立决策的贝叶斯模型来做到这一点。我们纳入这种稳健性考虑的方式,是要求智能体能够与其他智能体任何可能策略的一个集成成功交互,从而提高训练稳定性以及训练后智能体的稳健性。
3 Background
3 背景
Markov Games. In this work, we consider a multi-agent extension of Markov decision processes (MDPs) called partially observable Markov games [19]. A Markov game for agents is defined by a set of states describing the possible configurations of all agents, a set of actions and a set of observations for each agent. To choose actions, each agent uses a stochastic policy , which produces the next state according to the state transition function . Each agent obtains rewards as a function of the state and agent’s action , and receives a private observation correlated with the state . The initial states are determined by a distribution . Each agent aims to maximize its own total expected return where is a discount factor and is the time horizon.
马尔可夫博弈。 在本工作中,我们考虑马尔可夫决策过程(MDP)的一种多智能体扩展,称为部分可观察马尔可夫博弈 [19]。 个智能体的马尔可夫博弈由以下内容定义:描述所有智能体可能配置的状态集合 ,以及每个智能体的动作集合 和观察集合 。为了选择动作,每个智能体 使用随机策略 ,下一状态由状态转移函数 产生。 每个智能体 按照状态与该智能体动作的函数 获得奖励,并收到与状态相关的私有观察 。初始状态由分布 决定。每个智能体 的目标是最大化自己的总期望回报 ,其中 是折扣因子, 是时间范围。
To minimize notation we will often omit from the subscript of .
为简化记号,我们常常会省略 下标中的 。
Q-Learning and Deep Q-Networks (DQN). Q-Learning and DQN [23] are popular methods in reinforcement learning and have been previously applied to multi-agent settings [8, 35]. Q-Learning makes use of an action-value function for policy as . This function can be recursively rewritten as . DQN learns the action-value function corresponding to the optimal policy by minimizing the loss:
Q 学习与深度 Q 网络(DQN)。 Q 学习和 DQN [23] 是强化学习中常用的方法,此前也已被用于多智能体设置 [8, 35]。Q 学习对策略 使用动作价值函数 。该 函数可以递归写成 。DQN 通过最小化如下损失,来学习与最优策略相对应的动作价值函数 :
where is a target function whose parameters are periodically updated with the most recent , which helps stabilize learning. Another crucial component of stabilizing DQN is the use of an experience replay buffer containing tuples .
其中 是目标 函数,其参数会周期性地用最新的 更新,这有助于稳定学习。稳定 DQN 的另一个关键组成部分,是使用包含元组 的经验回放缓冲区 。
Q-Learning can be directly applied to multi-agent settings by having each agent learn an independently optimal function [34]. However, because agents are independently updating their policies as learning progresses, the environment appears non-stationary from the view of any one agent, violating Markov assumptions required for convergence of Q-learning. Another difficulty observed in [9] is that the experience replay buffer cannot be used in such a setting since in general, when any .
通过让每个智能体 独立学习一个最优函数 ,可以把 Q 学习直接用到多智能体设置中 [34]。然而,由于智能体在学习过程中独立更新各自的策略,从任何一个智能体的角度看,环境都显得非平稳,从而违反了 Q 学习收敛所需的马尔可夫假设。[9] 中指出的另一困难是,在这种设置中一般不能使用经验回放缓冲区,因为通常当任一 时,有 。
Policy Gradient (PG) Algorithms. Policy gradient methods are another popular choice for a variety of RL tasks. The main idea is to directly adjust the parameters of the policy in order to maximize the objective by taking steps in the direction of . Using the function defined previously, the gradient of the policy can be written as [32]:
策略梯度(PG)算法。 策略梯度方法是各类强化学习任务的另一种常用选择。主要想法是直接调整策略参数 ,沿 的方向迈步,以最大化目标 。使用前面定义的 函数,策略的梯度可以写成 [32]:
where is the state distribution. The policy gradient theorem has given rise to several practical algorithms, which often differ in how they estimate . For example, one can simply use a sample return , which leads to the REINFORCE algorithm [37]. Alternatively, one could learn an approximation of the true action-value function by e.g. temporal-difference learning [31]; this is called the critic and leads to a variety of actor-critic algorithms [31].
其中 是状态分布。策略梯度定理衍生出若干实用算法,它们的差别往往在于如何估计 。例如,可以简单地使用采样回报 ,这就得到 REINFORCE 算法 [37]。或者,也可以通过例如时序差分学习 [31] 来学习对真实动作价值函数 的近似;这个 称为评论家,并引出各种演员–评论家算法 [31]。
Policy gradient methods are known to exhibit high variance gradient estimates. This is exacerbated in multi-agent settings; since an agent’s reward usually depends on the actions of many agents, the reward conditioned only on the agent’s own actions (when the actions of other agents are not considered in the agent’s optimization process) exhibits much more variability, thereby increasing the variance of its gradients. Below, we show a simple setting where the probability of taking a gradient step in the correct direction decreases exponentially with the number of agents.
策略梯度方法的梯度估计以方差高而著称。在多智能体设置中这一点更为严重:由于一个智能体的回报通常取决于许多智能体的动作,如果只以该智能体自己的动作为条件(也就是在其优化过程中不考虑其他智能体的动作),回报的变化会大得多,从而增大梯度的方差。下面我们给出一个简单设置,在其中沿正确方向迈出梯度步的概率会随智能体数目指数下降。
Proposition 1. Consider agents with binary actions: , where . We assume an uninformed scenario, in which agents are initialized to . Then, if we are estimating the gradient of the cost with policy gradient, we have:
where is the policy gradient estimator from a single sample, and is the true gradient.
Proof. See Appendix.
命题 1. 考虑 个具有二元动作的智能体:,其中 。我们假定一种无信息的情形,各智能体初始化为 。若用策略梯度估计代价 的梯度,则有:
其中 是来自单次采样的策略梯度估计量, 是真梯度。
证明. 见附录。
The use of baselines, such as value function baselines typically used to ameliorate high variance, is problematic in multi-agent settings due to the non-stationarity issues mentioned previously.
使用基线(例如通常用来减轻高方差的价值函数基线)在多智能体设置中会有问题,原因就是前面提到的非平稳性。
Deterministic Policy Gradient (DPG) Algorithms. It is also possible to extend the policy gradient framework to deterministic policies . In particular, under certain conditions we can write the gradient of the objective as:
确定性策略梯度(DPG)算法。 也可以把策略梯度框架扩展到确定性策略 。特别是,在一定条件下,目标 的梯度可以写成:
Since this theorem relies on , it requires that the action space (and thus the policy ) be continuous. Deep deterministic policy gradient (DDPG) is a variant of DPG where the policy and critic are approximated with deep neural networks. DDPG is an off-policy algorithm, and samples trajectories from a replay buffer of experiences that are stored throughout training. DDPG also makes use of a target network, as in DQN [23].
由于这一定理依赖 ,它要求动作空间 (从而策略 )是连续的。深度确定性策略梯度(DDPG)是 DPG 的一种变体,其中策略 和评论家 用深度神经网络来近似。DDPG 是一种离策略算法,从训练过程中存储的经验回放缓冲区中采样轨迹。与 DQN 一样,DDPG 也使用目标网络 [23]。
4 Methods
4 方法
4.1 Multi-Agent Actor Critic
4.1 多智能体演员评论家
图 1:我们提出的多智能体分散式演员、集中式评论家方法概览。
We have argued in the previous section that naïve policy gradient methods perform poorly in simple multi-agent settings, and this is supported in our experiments in Section 5. Our goal in this section is to derive an algorithm that works well in such settings. However, we would like to operate under the following constraints: (1) the learned policies can only use local information (i.e. their own observations) at execution time, (2) we do not assume a differentiable model of the environment dynamics, unlike in [24], and (3) we do not assume any particular structure on the communication method between agents (that is, we don’t assume a differentiable communication channel). Fulfilling the above desiderata would provide a general-purpose multi-agent learning algorithm that could be applied not just to cooperative games with explicit communication channels, but competitive games and games involving only physical interactions between agents.
我们在上一节已经论证,朴素的策略梯度方法在简单的多智能体设置中表现不佳,第 5 节的实验也支持这一点。本节的目标是推导一种在这类设置中表现良好的算法。不过,我们希望在以下约束下进行:(1)学到的策略在执行时只能使用局部信息(也就是各自的观察);(2)与 [24] 不同,我们不假设环境动力学是可微的;(3)我们对智能体之间的通信方式不做任何特定结构假设(也就是不假设存在可微的通信信道)。满足上述要求,将得到一种通用的多智能体学习算法,它不仅可以用于有显式通信信道的合作博弈,也可以用于竞争博弈,以及智能体之间只有身体交互的博弈。
Similarly to [8], we accomplish our goal by adopting the framework of centralized training with decentralized execution. Thus, we allow the policies to use extra information to ease training, so long as this information is not used at test time. It is unnatural to do this with Q-learning, as the Q function generally cannot contain different information at training and test time. Thus, we propose a simple extension of actor-critic policy gradient methods where the critic is augmented with extra information about the policies of other agents.
与 [8] 类似,我们通过采用集中训练、分散执行的框架来达成目标。因此,我们允许策略在训练时使用额外信息以降低训练难度,只要这些信息在测试时不被使用。用 Q 学习很难自然地做到这一点,因为 Q 函数通常不能在训练和测试时包含不同的信息。因此,我们提出对演员–评论家策略梯度方法做一个简单扩展:评论家被补充以关于其他智能体策略的额外信息。
More concretely, consider a game with agents with policies parameterized by , and let be the set of all agent policies. Then we can write the gradient of the expected return for agent , as:
更具体地说,考虑一个有 个智能体的博弈,策略参数为 ,并令 为所有智能体策略的集合。于是,智能体 的期望回报 的梯度可以写成:
Here is a centralized action-value function that takes as input the actions of all agents, , in addition to some state information , and outputs the -value for agent . In the simplest case, could consist of the observations of all agents, , however we could also include additional state information if available. Since each is learned separately, agents can have arbitrary reward structures, including conflicting rewards in a competitive setting.
这里 是一个集中式动作价值函数,它除了一些状态信息 之外,还把所有智能体的动作 作为输入,并输出智能体 的 值。最简单的情形下, 可以由所有智能体的观察组成,即 ,不过如果有额外的状态信息,也可以一并纳入。由于每个 是单独学习的,智能体可以具有任意的奖励结构,也包括竞争设置中相互冲突的奖励。
We can extend the above idea to work with deterministic policies. If we now consider continuous policies w.r.t. parameters (abbreviated as ), the gradient can be written as:
我们也可以把上述想法扩展到确定性策略。若现在考虑 个关于参数 的连续策略 (简记为 ),梯度可以写成:
Here the experience replay buffer contains the tuples , recording experiences of all agents. The centralized action-value function is updated as:
这里,经验回放缓冲区 包含元组 ,记录所有智能体的经验。集中式动作价值函数 按如下方式更新:
where is the set of target policies with delayed parameters . As shown in Section 5, we find the centralized critic with deterministic policies works very well in practice, and refer to it as multi-agent deep deterministic policy gradient, abbreviated as MADDPG. We provide the description of the full algorithm in the Appendix.
其中 是带有延迟参数 的目标策略集合。如第 5 节所示,我们发现带确定性策略的集中式评论家在实践中效果很好,并将其称为多智能体深度确定性策略梯度(multi-agent deep deterministic policy gradient,简称 MADDPG)。完整算法的描述见附录。
A primary motivation behind MADDPG is that, if we know the actions taken by all agents, the environment is stationary even as the policies change, since for any . This is not the case if we do not explicitly condition on the actions of other agents, as done for most traditional RL methods.
MADDPG 背后的一个主要动机是:如果我们知道所有智能体采取的动作,那么即使策略在变,环境也是平稳的,因为对任意 ,都有 。如果我们不像大多数传统强化学习方法那样,不对其他智能体的动作显式条件化,情况就不是这样。
Note that we require the policies of other agents to apply an update in Eq. 6. Knowing the observations and policies of other agents is not a particularly restrictive assumption; if our goal is to train agents to exhibit complex communicative behaviour in simulation, this information is often available to all agents. However, we can relax this assumption if necessary by learning the policies of other agents from observations — we describe a method of doing this in Section 4.2.
需要注意的是,要应用式 6 中的更新,我们需要其他智能体的策略。知道其他智能体的观察和策略,并不是一个特别强的假设;如果我们的目标是在仿真中训练智能体表现出复杂的通信行为,这类信息往往对所有智能体都是可得的。不过,如有必要,我们也可以通过从观察中学习其他智能体的策略来放宽这一假设——第 4.2 节将描述一种做法。
4.2 Inferring Policies of Other Agents
4.2 推断其他智能体的策略
To remove the assumption of knowing other agents’ policies, as required in Eq. 6, each agent can additionally maintain an approximation (where are the parameters of the approximation; henceforth ) to the true policy of agent , . This approximate policy is learned by maximizing the log probability of agent ’s actions, with an entropy regularizer:
为去掉式 6 中所需的“已知其他智能体策略”这一假设,每个智能体 还可以额外维护一个对智能体 真实策略 的近似 (其中 是近似的参数;以下记为 )。这一近似策略通过最大化智能体 动作的对数概率、并加上熵正则来学习:
where is the entropy of the policy distribution. With the approximate policies, in Eq. 6 can be replaced by an approximate value calculated as follows:
其中 是策略分布的熵。有了近似策略,式 6 中的 可以用如下计算的近似值 代替:
where denotes the target network for the approximate policy . Note that Eq. 7 can be optimized in a completely online fashion: before updating , the centralized function, we take the latest samples of each agent from the replay buffer to perform a single gradient step to update . We also input the action log probabilities of each agent directly into , rather than sampling.
其中 表示近似策略 的目标网络。需要注意,式 7 可以完全在线优化:在更新集中式 函数 之前,我们从回放缓冲区取出每个智能体 的最新样本,做一次梯度步进以更新 。我们还把每个智能体的动作对数概率直接输入 ,而不是再做采样。
4.3 Agents with Policy Ensembles
4.3 带策略集成的智能体
A recurring problem in multi-agent reinforcement learning is the environment non-stationarity due to the agents’ changing policies. This is particularly true in competitive settings, where agents can derive a strong policy by overfitting to the behavior of their competitors. Such policies are undesirable as they are brittle and may fail when the competitors alter their strategies.
多智能体强化学习中一个反复出现的问题,是智能体策略不断变化所导致的环境非平稳性。在竞争设置中尤其如此:智能体可能通过对手的行为过拟合,而得到看起来很强的策略。这样的策略并不理想,因为它们很脆弱,一旦对手改变策略就可能失效。
To obtain multi-agent policies that are more robust to changes in the policy of competing agents, we propose to train a collection of different sub-policies. At each episode, we randomly select one particular sub-policy for each agent to execute. Suppose that policy is an ensemble of different sub-policies with sub-policy denoted by . For agent , we are then maximizing the ensemble objective: .
为得到对竞争智能体策略变化更稳健的多智能体策略,我们提出训练 个不同的子策略。在每一回合,我们为每个智能体随机选择一个特定子策略来执行。假设策略 是 个不同子策略的集成,第 个子策略记为 。对智能体 ,我们最大化集成目标:。
Since different sub-policies will be executed in different episodes, we maintain a replay buffer for each sub-policy of agent . Accordingly, we can derive the gradient of the ensemble objective with respect to as follows:
由于不同子策略会在不同回合中执行,我们为智能体 的每个子策略 维护一个回放缓冲区 。据此,我们可以把集成目标相对于 的梯度写成:
5 Experiments
5 实验
5.1 Environments
5.1 环境
To perform our experiments, we adopt the grounded communication environment proposed in [24], which consists of agents and landmarks inhabiting a two-dimensional world with continuous space and discrete time. Agents may take physical actions in the environment and communication actions that get broadcasted to other agents. Unlike [24], we do not assume that all agents have identical action and observation spaces, or act according to the same policy . We also consider games that are both cooperative (all agents must maximize a shared return) and competitive (agents have conflicting goals). Some environments require explicit communication between agents in order to achieve the best reward, while in other environments agents can only perform physical actions. We provide details for each environment below.
为进行实验,我们采用 [24] 提出的有根基通信环境,其中 个智能体和 个地标位于一个连续空间、离散时间的二维世界中。 智能体可以在环境中采取身体动作,也可以采取会广播给其他智能体的通信动作。与 [24] 不同,我们不假设所有智能体具有相同的动作和观察空间,或按同一策略 行动。我们还同时考虑合作博弈(所有智能体必须最大化共享回报)和竞争博弈(智能体目标冲突)。有些环境需要智能体之间的显式通信才能获得最好的回报,而在另一些环境中智能体只能采取身体动作。各环境的细节如下。
Videos of our experimental results can be viewed at https://sites.google.com/site/multiagentac/
实验结果的视频可在 https://sites.google.com/site/multiagentac/ 观看。
The environments are publicly available: https://github.com/openai/multiagent-particle-envs
图 2:实验环境以及我们考虑的部分任务示意,包括 a) 合作通信 b) 猎食 c) 合作导航 d) 身体欺骗。全部实验结果的视频见项目网页。
Cooperative communication. This task consists of two cooperative agents, a speaker and a listener, who are placed in an environment with three landmarks of differing colors. At each episode, the listener must navigate to a landmark of a particular color, and obtains reward based on its distance to the correct landmark. However, while the listener can observe the relative position and color of the landmarks, it does not know which landmark it must navigate to. Conversely, the speaker’s observation consists of the correct landmark color, and it can produce a communication output at each time step which is observed by the listener. Thus, the speaker must learn to output the landmark colour based on the motions of the listener. Although this problem is relatively simple, as we show in Section 5.2 it poses a significant challenge to traditional RL algorithms.
合作通信。 这一任务包含两个合作智能体,一个说话者和一个听话者,他们被放在有三个不同颜色地标的环境中。在每一回合,听话者必须导航到某一特定颜色的地标,并根据到正确地标的距离获得回报。然而,虽然听话者能观察到各地标的相对位置和颜色,却不知道自己必须前往哪一个地标。相反,说话者的观察是正确的地标颜色,并且可以在每个时间步产生一个通信输出,由听话者观察。因此,说话者必须学会根据听话者的运动输出地标颜色。尽管这个问题相对简单,如第 5.2 节所示,它对传统强化学习算法构成了显著挑战。
Cooperative navigation. In this environment, agents must cooperate through physical actions to reach a set of landmarks. Agents observe the relative positions of other agents and landmarks, and are collectively rewarded based on the proximity of any agent to each landmark. In other words, the agents have to ‘cover’ all of the landmarks. Further, the agents occupy significant physical space and are penalized when colliding with each other. Our agents learn to infer the landmark they must cover, and move there while avoiding other agents.
合作导航。 在这一环境中,智能体必须通过身体动作合作,到达一组 个地标。智能体观察到其他智能体和地标的相对位置,并集体地根据任一智能体到每个地标的接近程度获得回报。换句话说,智能体必须“覆盖”所有地标。此外,智能体占据相当的物理空间,相互碰撞时会受到惩罚。我们的智能体学会推断自己必须覆盖的地标,并在避开其他智能体的同时移动到那里。
Keep-away. This scenario consists of landmarks including a target landmark, cooperating agents who know the target landmark and are rewarded based on their distance to the target, and adversarial agents who must prevent the cooperating agents from reaching the target. Adversaries accomplish this by physically pushing the agents away from the landmark, temporarily occupying it. While the adversaries are also rewarded based on their distance to the target landmark, they do not know the correct target; this must be inferred from the movements of the agents.
阻止靠近。 这一场景包含 个地标(其中包括一个目标地标)、 个知道目标地标并根据到目标的距离获得回报的合作智能体,以及 个必须阻止合作智能体到达目标的对抗智能体。对抗者通过在身体上把智能体推离地标、暂时占据该地标来做到这一点。虽然对抗者也根据到目标地标的距离获得回报,但他们并不知道正确目标,必须从智能体的运动中推断。
Physical deception. Here, agents cooperate to reach a single target landmark from a total of landmarks. They are rewarded based on the minimum distance of any agent to the target (so only one agent needs to reach the target landmark). However, a lone adversary also desires to reach the target landmark; the catch is that the adversary does not know which of the landmarks is the correct one. Thus the cooperating agents, who are penalized based on the adversary distance to the target, learn to spread out and cover all landmarks so as to deceive the adversary.
身体欺骗。 在这里, 个智能体合作,从总共 个地标中到达一个目标地标。他们根据任一智能体到目标的最小距离获得回报(因此只需一个智能体到达目标地标即可)。然而,还有一个单独的对抗者也想到达目标地标;关键在于,对抗者并不知道哪个地标是正确的。因此,这些会根据对抗者到目标的距离受到惩罚的合作智能体,学会散开并覆盖所有地标,以迷惑对抗者。
Predator-prey. In this variant of the classic predator-prey game, slower cooperating agents must chase the faster adversary around a randomly generated environment with large landmarks impeding the way. Each time the cooperative agents collide with an adversary, the agents are rewarded while the adversary is penalized. Agents observe the relative positions and velocities of the agents, and the positions of the landmarks.
猎食。 在这一经典猎食游戏的变体中, 个较慢的合作智能体必须在一个随机生成、有 个大地标挡路的环境中追逐较快的对抗者。每当合作智能体与对抗者发生碰撞,合作智能体获得回报,对抗者受到惩罚。智能体观察到各方的相对位置和速度,以及地标的位置。
Covert communication. This is an adversarial communication environment, where a speaker agent (‘Alice’) must communicate a message to a listener agent (‘Bob’), who must reconstruct the message at the other end. However, an adversarial agent (‘Eve’) is also observing the channel, and wants to reconstruct the message — Alice and Bob are penalized based on Eve’s reconstruction, and thus Alice must encode her message using a randomly generated key, known only to Alice and Bob. This is similar to the cryptography environment considered in [2].
隐蔽通信。 这是一个对抗性通信环境:说话者智能体(“Alice”)必须把一条消息传给听话者智能体(“Bob”),由后者在另一端重构该消息。然而,一个对抗智能体(“Eve”)也在观察信道,并想重构这条消息——Alice 和 Bob 会根据 Eve 的重构受到惩罚,因此 Alice 必须用一把只为 Alice 和 Bob 所知的随机密钥来编码消息。这与 [2] 中考虑的密码学环境类似。
5.2 Comparison to Decentralized Reinforcement Learning Methods
5.2 与分散式强化学习方法的比较
Figure 3: Comparison between MADDPG and DDPG (left), and between single policy MADDPG and ensemble MADDPG (right) on the competitive environments. Each bar cluster shows the 0-1 normalized score for a set of competing policies (agent v adversary), where a higher score is better for the agent. In all cases, MADDPG outperforms DDPG when directly pitted against it, and similarly for the ensemble against the single MADDPG policies. Full results are given in the Appendix.
图 3:在竞争环境中,MADDPG 与 DDPG 的比较(左),以及单一策略 MADDPG 与集成 MADDPG 的比较(右)。每组柱表示一组对抗策略(智能体对对抗者)的 0-1 归一化分数,分数越高对智能体越有利。在所有情形中,MADDPG 在直接对阵 DDPG 时都更优,集成策略相对单一 MADDPG 策略也同样如此。完整结果见附录。
图 4:在合作通信上训练 25000 回合后,MADDPG 相对传统强化学习方法的回报。
We implement MADDPG and evaluate it on the environments presented in Section 5.1. Unless otherwise specified, our policies are parameterized by a two-layer ReLU MLP with 64 units per layer. To support discrete communication messages, we use the Gumbel-Softmax estimator [14]. To evaluate the quality of policies learned in competitive settings, we pitch MADDPG agents against DDPG agents, and compare the resulting success of the agents and adversaries in the environment. We train our models until convergence, and then evaluate them by averaging various metrics for 1000 further iterations. We provide the tables and details of our results on all environments in the Appendix, and summarize them here.
我们实现了 MADDPG,并在第 5.1 节给出的环境上评估。除非另有说明,我们的策略用每层 64 个单元的两层 ReLU MLP 参数化。为支持离散通信消息,我们使用 Gumbel-Softmax 估计器 [14]。为评估在竞争设置中学到的策略质量,我们让 MADDPG 智能体对阵 DDPG 智能体,并比较双方在环境中的成功情况。我们把模型训练到收敛,再通过再运行 1000 次迭代并对各项指标取平均来评估。各环境上的表格和细节见附录,这里只做综述。
We first examine the cooperative communication scenario. Despite the simplicity of the task (the speaker only needs to learn to output its observation), traditional RL methods such as DQN, Actor-Critic, a first-order implementation of TRPO, and DDPG all fail to learn the correct behaviour (measured by whether the listener is within a short distance from the target landmark). In practice we observed that the listener learns to ignore the speaker and simply moves to the middle of all observed landmarks. We plot the learning curves over 25000 episodes for various approaches in Figure 4.
我们首先考察合作通信场景。尽管任务简单(说话者只需学会输出自己的观察),DQN、Actor-Critic、TRPO 的一阶实现以及 DDPG 等传统强化学习方法都未能学会正确行为(以听话者是否处于目标地标的较短距离之内来衡量)。实践中我们观察到,听话者会学会忽略说话者,只是走到所有观察到的地标的中间。图 4 画出了各种方法在 25000 个回合上的学习曲线。
We hypothesize that a primary reason for the failure of traditional RL methods in this (and other) multi-agent settings is the lack of a consistent gradient signal. For example, if the speaker utters the correct symbol while the listener moves in the wrong direction, the speaker is penalized. This problem is exacerbated as the number of time steps grows: we observed that traditional policy gradient methods can learn when the objective of the listener is simply to reconstruct the observation of the speaker in a single time step, or if the initial positions of agents and landmarks are fixed and evenly distributed. This indicates that many of the multi-agent methods previously proposed for scenarios with short time horizons (e.g. [16]) may not generalize to more complex tasks.
我们猜想,传统强化学习方法在这一(以及其他)多智能体设置中失败的一个主要原因,是缺乏一致的梯度信号。例如,如果说话者发出了正确符号,而听话者却走向错误方向,说话者也会受到惩罚。随着时间步增多,这一问题会加剧:我们观察到,当听话者的目标只是在单个时间步重构说话者的观察,或者智能体和地标的初始位置固定且均匀分布时,传统策略梯度方法是可以学习的。这表明,许多以往为短时间范围场景提出的多智能体方法(例如 [16])可能无法推广到更复杂的任务。
Conversely, MADDPG agents can learn coordinated behaviour more easily via the centralized critic. In the cooperative communication environment, MADDPG is able to reliably learn the correct listener and speaker policies, and the listener is often (84.0% of the time) able to navigate to the target.
相反,MADDPG 智能体可以借助集中式评论家更容易地学习协调行为。在合作通信环境中,MADDPG 能够可靠地学会正确的听话者和说话者策略,听话者常常(84.0% 的时间)能够导航到目标。
Figure 5: Comparison between MADDPG (left) and DDPG (right) on the cooperative communication (CC) and physical deception (PD) environments at , , and . Small dark circles indicate landmarks. In CC, the grey agent is the speaker, and the color of the listener indicates the target landmark. In PD, the blue agents are trying to deceive the red adversary, while covering the target landmark (in green). MADDPG learns the correct behavior in both cases: in CC the speaker learns to output the target landmark color to direct the listener, while in PD the agents learn to cover both landmarks to confuse the adversary. DDPG (and other RL algorithms) struggles in these settings: in CC the speaker always repeats the same utterance and the listener moves to the middle of the landmarks, and in PP one agent greedily pursues the green landmark (and is followed by the adversary) while the othe agent scatters. See video for full trajectories.
图 5:在合作通信(CC)和身体欺骗(PD)环境中,、 和 时 MADDPG(左)与 DDPG(右)的比较。小的深色圆圈表示地标。在 CC 中,灰色智能体是说话者,听话者的颜色表示目标地标。在 PD 中,蓝色智能体在覆盖目标地标(绿色)的同时试图迷惑红色对抗者。MADDPG 在两种情形中都学会了正确行为:在 CC 中,说话者学会输出目标地标的颜色以引导听话者;在 PD 中,智能体学会覆盖两个地标以迷惑对抗者。DDPG(以及其他强化学习算法)在这些设置中表现吃力:在 CC 中,说话者总是重复同一句话,听话者走到各地标中间;在 PP 中,一个智能体贪婪地追逐绿色地标(并被对抗者跟上),而另一个智能体则散开。完整轨迹见视频。
A similar situation arises for the physical deception task: when the cooperating agents are trained with MADDPG, they are able to successfully deceive the adversary by covering all of the landmarks around 94% of the time when (Figure 5). Furthermore, the adversary success is quite low, especially when the adversary is trained with DDPG (16.4% when ). This contrasts sharply with the behaviour learned by the cooperating DDPG agents, who are unable to deceive MADDPG adversaries in any scenario, and do not even deceive other DDPG agents when .
身体欺骗任务中也出现了类似情况:当合作智能体用 MADDPG 训练时,在 的情况下,它们约 94% 的时间能够通过覆盖全部地标成功迷惑对抗者(图 5)。此外,对抗者的成功率相当低,尤其是当对抗者用 DDPG 训练时( 时为 16.4%)。这与合作的 DDPG 智能体所学到的行为形成鲜明对比:它们在任何场景中都无法迷惑 MADDPG 对抗者,甚至在 时也无法迷惑其他 DDPG 智能体。
While the cooperative navigation and predator-prey tasks have a less stark divide between success and failure, in both cases the MADDPG agents outperform the DDPG agents. In cooperative navigation, MADDPG agents have a slightly smaller average distance to each landmark, but have almost half the average number of collisions per episode (when ) compared to DDPG agents due to the ease of coordination. Similarly, MADDPG predators are far more successful at chasing DDPG prey (16.1 collisions/episode) than the converse (10.3 collisions/episode).
合作导航和猎食任务在成功与失败之间的分野没有那么截然,但在这两种情形中,MADDPG 智能体都优于 DDPG 智能体。在合作导航中,MADDPG 智能体到各地标的平均距离略小,但由于更容易协调,每回合平均碰撞次数(当 时)几乎只有 DDPG 智能体的一半。类似地,MADDPG 捕食者追逐 DDPG 猎物要成功得多(每回合 16.1 次碰撞),反过来则只有每回合 10.3 次碰撞。
In the covert communication environment, we found that Bob trained with both MADDPG and DDPG out-performs Eve in terms of reconstructing Alice’s message. However, Bob trained with MADDPG achieves a larger relative success rate compared with DDPG (52.4% to 25.1%). Further, only Alice trained with MADDPG can encode her message such that Eve achieves near-random reconstruction accuracy.
在隐蔽通信环境中,我们发现,无论用 MADDPG 还是 DDPG 训练,Bob 在重构 Alice 的消息上都优于 Eve。不过,用 MADDPG 训练的 Bob 相对成功率更高(52.4% 对 25.1%)。此外,只有用 MADDPG 训练的 Alice 才能把消息编码到使 Eve 的重构准确率接近随机。
5.3 Effect of Learning Polices of Other Agents
5.3 学习其他智能体策略的效果
Figure 6: Effectiveness of learning by approximating policies of other agents in the cooperative communication scenario. Left: plot of the reward over number of iterations; MADDPG agents quickly learn to solve the task when approximating the policies of others. Right: KL divergence between the approximate policies and the true policies.
图 6:在合作通信场景中,通过近似其他智能体策略来学习的有效性。左:回报随迭代次数的变化;在近似他人策略时,MADDPG 智能体很快学会完成任务。右:近似策略与真实策略之间的 KL 散度。
We evaluate the effectiveness of learning the policies of other agents in the cooperative communication environment, following the same hyperparameters as the previous experiments and setting in Eq. 7. The results are shown in Figure 6. We observe that despite not fitting the policies of other agents perfectly (in particular, the approximate listener policy learned by the speaker has a fairly large KL divergence to the true policy), learning with approximated policies is able to achieve the same success rate as using the true policy, without a significant slowdown in convergence.
我们在合作通信环境中评估学习其他智能体策略的有效性,超参数与前面实验相同,并把式 7 中的 设为 。结果见图 6。我们观察到,尽管对其他智能体策略的拟合并不完美(特别是,说话者学到的近似听话者策略与真实策略之间有相当大的 KL 散度),用近似策略学习仍能达到与使用真实策略相同的成功率,而且收敛并没有明显变慢。
5.4 Effect of Training with Policy Ensembles
5.4 用策略集成训练的效果
We focus on the effectiveness of policy ensembles in competitive environments, including keep-away, cooperative navigation, and predator-prey. We choose sub-policies for the keep-away and cooperative navigation environments, and for predator-prey. To improve convergence speed, we enforce that the cooperative agents should have the same policies at each episode, and similarly for the adversaries. To evaluate the approach, we measure the performance of ensemble policies and single policies in the roles of both agent and adversary. The results are shown on the right side of Figure 3. We observe that agents with policy ensembles are stronger than those with a single policy. In particular, when pitting ensemble agents against single policy adversaries (second to left bar cluster), the ensemble agents outperform the adversaries by a large margin compared to when the roles are reversed (third to left bar cluster).
我们关注策略集成在竞争环境中的效果,包括阻止靠近、合作导航和猎食。在阻止靠近和合作导航环境中我们取 个子策略,在猎食中取 。为提高收敛速度,我们强制合作智能体在每一回合使用相同的策略,对抗者同样如此。为评估该方法,我们分别在智能体和对抗者的角色上衡量集成策略与单一策略的表现。结果见图 3 右侧。我们观察到,带策略集成的智能体强于只有单一策略的智能体。特别是,当集成智能体对阵单一策略对抗者时(从左数第二组柱),集成智能体对对抗者的优势,明显大于角色对调时(从左数第三组柱)。
6 Conclusions and Future Work
6 结论与未来工作
We have proposed a multi-agent policy gradient algorithm where agents learn a centralized critic based on the observations and actions of all agents. Empirically, our method outperforms traditional RL algorithms on a variety of cooperative and competitive multi-agent environments. We can further improve the performance of our method by training agents with an ensemble of policies, an approach we believe to be generally applicable to any multi-agent algorithm.
我们提出了一种多智能体策略梯度算法,其中智能体基于所有智能体的观察和动作学习一个集中式评论家。实验表明,该方法在多种合作和竞争多智能体环境上优于传统强化学习算法。通过用一组策略集成来训练智能体,还可以进一步提高该方法的表现;我们相信这一做法普遍适用于任何多智能体算法。
One downside to our approach is that the input space of grows linearly (depending on what information is contained in ) with the number of agents . This could be remedied in practice by, for example, having a modular function that only considers agents in a certain neighborhood of a given agent. We leave this investigation to future work.
我们方法的一个不足是, 的输入空间会(取决于 中包含什么信息)随智能体数目 线性增长。实践中可以通过例如使用模块化的 函数来补救,该函数只考虑给定智能体某个邻域内的智能体。我们把这一研究留给未来工作。
Acknowledgements
致谢
The authors would like to thank Jacob Andreas, Smitha Milli, Jack Clark, Jakob Foerster, and others at OpenAI and UC Berkeley for interesting discussions related to this paper, as well as Jakub Pachocki, Yura Burda, and Joelle Pineau for comments on the paper draft. We thank Tambet Matiisen for providing the code base that was used for some early experiments associated with this paper. Ryan Lowe is supported in part by a Vanier CGS Scholarship and the Samsung Advanced Institute of Technology. Finally, we’d like to thank OpenAI for fostering an engaging and productive research environment.
作者感谢 Jacob Andreas、Smitha Milli、Jack Clark、Jakob Foerster 以及 OpenAI 和加州大学伯克利分校的其他同事与本文相关的有趣讨论,并感谢 Jakub Pachocki、Yura Burda 和 Joelle Pineau 对论文草稿的意见。我们感谢 Tambet Matiisen 提供了本文早期部分实验所用的代码库。Ryan Lowe 的研究部分得到 Vanier CGS 奖学金和三星综合技术院的支持。最后,我们感谢 OpenAI 营造了一个富有吸引力且富有成效的研究环境。
References
参考文献
[1] DeepMind AI reduces google data centre cooling bill by 40. https://deepmind.com/blog/deepmind-ai-reduces-google-data-centre-cooling-bill-40/. Accessed: 2017-05-19.
[2] M. Abadi and D. G. Andersen. Learning to protect communications with adversarial neural cryptography. arXiv preprint arXiv:1610.06918, 2016.
[3] C. Boutilier. Learning conventions in multiagent stochastic domains using likelihood estimates. In Proceedings of the Twelfth international conference on Uncertainty in artificial intelligence, pages 106–114. Morgan Kaufmann Publishers Inc., 1996.
[4] L. Busoniu, R. Babuska, and B. De Schutter. A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems Man and Cybernetics Part C Applications and Reviews, 38(2):156, 2008.
[5] G. Chalkiadakis and C. Boutilier. Coordination in multiagent reinforcement learning: a bayesian approach. In Proceedings of the second international joint conference on Autonomous agents and multiagent systems, pages 709–716. ACM, 2003.
[6] P. Dayan and G. E. Hinton. Feudal reinforcement learning. In Advances in neural information processing systems, pages 271–271. Morgan Kaufmann Publishers, 1993.
[7] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. Counterfactual multi-agent policy gradients. arXiv preprint arXiv:1705.08926, 2017.
[8] J. N. Foerster, Y. M. Assael, N. de Freitas, and S. Whiteson. Learning to communicate with deep multi-agent reinforcement learning. CoRR, abs/1605.06676, 2016.
[9] J. N. Foerster, N. Nardelli, G. Farquhar, P. H. S. Torr, P. Kohli, and S. Whiteson. Stabilising experience replay for deep multi-agent reinforcement learning. CoRR, abs/1702.08887, 2017.
[10] M. C. Frank and N. D. Goodman. Predicting pragmatic reasoning in language games. Science, 336(6084):998–998, 2012.
[11] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in neural information processing systems, 2014.
[12] J. K. Gupta, M. Egorov, and M. Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. 2017.
[13] J. Hu and M. P. Wellman. Online learning about other agents in a dynamic multiagent system. In Proceedings of the Second International Conference on Autonomous Agents, AGENTS ’98, pages 239–246, New York, NY, USA, 1998. ACM.
[14] E. Jang, S. Gu, and B. Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016.
[15] M. Lauer and M. Riedmiller. An algorithm for distributed reinforcement learning in cooperative multi-agent systems. In In Proceedings of the Seventeenth International Conference on Machine Learning, pages 535–542. Morgan Kaufmann, 2000.
[16] A. Lazaridou, A. Peysakhovich, and M. Baroni. Multi-agent cooperation and the emergence of (natural) language. arXiv preprint arXiv:1612.07182, 2016.
[17] J. Z. Leibo, V. F. Zambaldi, M. Lanctot, J. Marecki, and T. Graepel. Multi-agent reinforcement learning in sequential social dilemmas. CoRR, abs/1702.03037, 2017.
[18] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. arXiv preprint arXiv:1504.00702, 2015.
[19] M. L. Littman. Markov games as a framework for multi-agent reinforcement learning. In Proceedings of the eleventh international conference on machine learning, volume 157, pages 157–163, 1994.
[20] L. Matignon, L. Jeanpierre, A.-I. Mouaddib, et al. Coordinated multi-robot exploration under communication constraints using decentralized markov decision processes. In AAAI, 2012.
[21] L. Matignon, G. J. Laurent, and N. Le Fort-Piat. Hysteretic q-learning: an algorithm for decentralized reinforcement learning in cooperative multi-agent teams. In Intelligent Robots and Systems, 2007. IROS 2007. IEEE/RSJ International Conference on, pages 64–69. IEEE, 2007.
[22] L. Matignon, G. J. Laurent, and N. Le Fort-Piat. Independent reinforcement learners in cooperative markov games: a survey regarding coordination problems. The Knowledge Engineering Review, 27(01), 2012.
[23] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
[24] I. Mordatch and P. Abbeel. Emergence of grounded compositional language in multi-agent populations. arXiv preprint arXiv:1703.04908, 2017.
[25] S. Omidshafiei, J. Pazis, C. Amato, J. P. How, and J. Vian. Deep decentralized multi-task multi-agent reinforcement learning under partial observability. CoRR, abs/1703.06182, 2017.
[26] L. Panait and S. Luke. Cooperative multi-agent learning: The state of the art. Autonomous Agents and Multi-Agent Systems, 11(3):387–434, Nov. 2005.
[27] P. Peng, Q. Yuan, Y. Wen, Y. Yang, Z. Tang, H. Long, and J. Wang. Multiagent bidirectionallycoordinated nets for learning to play starcraft combat games. CoRR, abs/1703.10069, 2017.
[28] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484 – 489, 2016.
[29] S. Sukhbaatar, R. Fergus, et al. Learning multiagent communication with backpropagation. In Advances in Neural Information Processing Systems, pages 2244–2252, 2016.
[30] S. Sukhbaatar, I. Kostrikov, A. Szlam, and R. Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv preprint arXiv:1703.05407, 2017.
[31] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press Cambridge, 1998.
[32] R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems, 2000.
[33] A. Tampuu, T. Matiisen, D. Kodelja, I. Kuzovkin, K. Korjus, J. Aru, J. Aru, and R. Vicente. Multiagent cooperation and competition with deep reinforcement learning. PloS one, 12(4):e0172395, 2017.
[34] M. Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning, pages 330–337, 1993.
[35] G. Tesauro. Extending q-learning to general adaptive multi-agent systems. In Advances in neural information processing systems, pages 871–878, 2004.
[36] P. S. Thomas and A. G. Barto. Conjugate markov decision processes. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 137–144, 2011.
[37] R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256, 1992.





