Generating Physically Stable and Buildable LEGO® Designs from Text

从文本生成物理稳定且可搭建的 LEGO® 设计

Ava Pun Thanks:  Indicates equal contribution.    Kangle Deng 1 1 footnotemark: 1    Ruixuan Liu 1 1 footnotemark: 1    Deva Ramanan    Changliu Liu    Jun-Yan Zhu Affiliation:  Carnegie Mellon University

Abstract

摘要

We introduce LegoGPT, the first approach for generating physically stable LEGO brick models from text prompts. To achieve this, we construct a large-scale, physically stable dataset of LEGO designs, along with their associated captions, and train an autoregressive large language model to predict the next brick to add via next-token prediction. To improve the stability of the resulting designs, we employ an efficient validity check and physics-aware rollback during autoregressive inference, which prunes infeasible token predictions using physics laws and assembly constraints. Our experiments show that LegoGPT produces stable, diverse, and aesthetically pleasing LEGO designs that align closely with the input text prompts. We also develop a text-based LEGO texturing method to generate colored and textured designs. We show that our designs can be assembled manually by humans and automatically by robotic arms. We also release our new dataset, StableText2Lego, containing over 47,000 LEGO structures of over 28,000 unique 3D objects accompanied by detailed captions, along with our code and models at the project website: https://avalovelace1.github.io/LegoGPT/.

我们提出 LegoGPT，这是首个从文本提示生成物理稳定 LEGO 积木模型的方法。为此，我们构建了大规模、物理稳定的 LEGO 设计数据集及其配套描述，并训练自回归大语言模型，以「下一词元预测」方式预测下一块积木。为提高生成结构的稳定性，我们在自回归推理阶段引入高效合法性检查与物理感知回滚，依据物理定律与装配约束剪除不可行的词元预测。实验表明，LegoGPT 能生成稳定、多样且美观、并与输入文本高度对齐的 LEGO 设计。我们还开发了基于文本的 LEGO 贴图方法，可生成上色与带纹理的设计。结果既可由人工手工搭建，也可由机械臂自动装配。我们同步发布新数据集 StableText2Lego：涵盖 28,000+ 独特三维物体的 47,000+ 个 LEGO 结构及详细描述，代码与模型见项目页 https://avalovelace1.github.io/LegoGPT/。

1 Introduction

1 引言

3D generative models have made remarkable progress, driven by advances in generative modeling [15, 65] and neural rendering [26, 52]. These models have enabled various applications in virtual reality, gaming, entertainment, and scientific computing. For example, several works have explored synthesizing 3D objects from text [58], adding texture to meshes [61, 9], and manipulating the shape and appearance of existing 3D objects and scenes [40, 19].

三维生成模型在生成建模 [15, 65] 与神经渲染 [26, 52] 的推动下取得显著进展，已在虚拟现实、游戏、娱乐与科学计算等场景落地。例如，已有工作探索从文本合成三维物体 [58]、为网格添加纹理 [61, 9]，以及编辑已有三维物体与场景的形状与外观 [40, 19]。

However, applying existing methods to create real-world objects remains challenging. Most approaches focus on generating diverse 3D objects with high-fidelity geometry and appearance [85, 21], but these digital designs often cannot be physically realized due to two key challenges [46]. First, the objects may be difficult to assemble or fabricate using standard components. Second, the resulting structure may be physically unstable even if assembly is possible. Without proper support, parts of the design can collapse, float, or remain disconnected.

然而，把现有方法用到真实世界物体仍很困难。多数工作侧重生成几何与外观保真度高的多样三维物体 [85, 21]，但这些数字设计往往无法落地，主要卡在两点 [46]。其一，用标准零件装配或制造可能很难；其二，即便能装，结构也可能物理不稳定。支撑不足时，局部会垮塌、悬空或彼此脱节。

In this work, we address the challenge of generating physically realizable objects and study this problem in the context of LEGO® design. LEGO is widely used in entertainment, education, artistic creation, and manufacturing prototyping. Additionally, it can serve as a reproducible research benchmark, as all standard components are readily available. Due to the significant effort required for manual design, recent studies have developed automated algorithms to streamline the process and generate compelling results. However, existing approaches primarily create LEGO designs from a given 3D object [45] or focus on a single object category [13, 12].

本文面向「可物理实现物体」的生成，并以 LEGO® 设计为研究对象。LEGO 广泛应用于娱乐、教育、艺术创作与制造原型；同时因标准零件易得，也适合作为可复现的研究基准。手工设计成本高，近年已有自动化算法简化流程并给出不错结果，但现有方法多半从给定三维物体生成 LEGO [45]，或只覆盖单一物体类别 [13, 12]。

Our goal is to develop a method for generating LEGO designs directly from freeform text prompts while ensuring physical stability and buildability. Specifically, we aim to train a generative model that produces designs that are:

我们的目标是直接从自由文本提示生成 LEGO 设计，并保证物理稳定与可搭建性。具体而言，希望生成的设计满足：

• • Physically stable: Built on a LEGO baseplate with strong structural integrity, without floating or collapsing bricks.

• • 物理稳定：建在底板之上，结构完整，无悬空或垮塌积木。

• • Buildable: Compatible with standard LEGO pieces and able to be assembled brick-by-brick by humans or robots.

• • 可搭建：兼容标准 LEGO 零件，可由人或机器人按块装配。

In this work, we introduce LegoGPT with the key insight of repurposing autoregressive large language models, originally trained for next-token prediction, for next-brick prediction. We formulate the problem of LEGO design as an autoregressive text generation task, where the next-brick dimension and placement are specified with a simple textual format. To ensure generated structures are both stable and buildable, we enforce physics-aware assembly constraints during both training and inference. During training, we construct a large-scale dataset of physically stable LEGO designs paired with captions. During autoregressive inference, we enforce feasibility with an efficient validity check and physics-aware rollback to ensure that the final tokens adhere to physics laws and assembly constraints.

本文提出 LegoGPT，核心想法是把原本做下一词元预测的自回归大语言模型，改造成「下一块积木」预测。我们把 LEGO 设计表述为自回归文本生成：用简洁文本格式指定下一块积木的尺寸与摆放。为同时保证稳定与可搭建，训练与推理都强制物理感知装配约束：训练侧构建带描述的大规模物理稳定 LEGO 数据集；推理侧用高效合法性检查与物理感知回滚，使最终词元服从物理定律与装配约束。

Our experiments show that the generated designs are stable, diverse, and visually appealing while adhering to input text prompts. Our method outperforms pre-trained LLMs with and without in-context learning, and previous approaches based on mesh-based 3D generation. Finally, we explore applications such as text-driven LEGO texturing, as well as manual assembly and automated robotic assembly of our designs. Our dataset, code, and models are available at the project website: https://avalovelace1.github.io/LegoGPT/.

实验表明，生成设计稳定、多样且视觉上美观，并能贴合输入文本。方法优于带/不带上下文学习的预训练 LLM，以及基于网格的三维生成路线。最后，我们还探索了文本驱动的 LEGO 贴图，以及人工与机器人自动装配。数据集、代码与模型见 https://avalovelace1.github.io/LegoGPT/。

2 Related Work

2 相关工作

Text-to-3D Generation.

Text-to-3D generation has seen remarkable progress in recent years, driven by advances in neural rendering and generative models. Dreamfusion [58] and Score Jacobian Chaining [76] pioneer zero-shot text-to-3D generation by optimizing neural radiance fields [52] with pre-trained diffusion models [62]. Subsequent work has explored alternative 3D representations [35, 3, 48, 66, 33, 42, 51] and improved loss functions [77, 83, 47, 74, 25, 44]. Rather than relying on iterative optimization, a promising alternative direction trains generative models directly on 3D asset datasets, with various backbones including diffusion models [63, 89, 32, 54, 85, 34, 20, 86, 60], large reconstruction models [21, 31, 81, 73], U-Nets [68, 36], and autoregressive models [55, 4, 67, 64, 5, 6, 18, 79].

文生三维。

得益于神经渲染与生成模型，文生三维近年进展很快。Dreamfusion [58] 与 Score Jacobian Chaining [76] 用预训练扩散模型 [62] 优化神经辐射场 [52]，开创了零样本文生三维。后续工作探索了替代三维表示 [35, 3, 48, 66, 33, 42, 51] 与改进损失 [77, 83, 47, 74, 25, 44]。另一条路线不再依赖迭代优化，而是直接在三维资产数据上训练生成模型，骨干包括扩散 [63, 89, 32, 54, 85, 34, 20, 86, 60]、大重建模型 [21, 31, 81, 73]、U-Net [68, 36] 与自回归模型 [55, 4, 67, 64, 5, 6, 18, 79]。

However, these existing methods cannot be directly applied to LEGO generation because they do not account for the unique physical constraints and assembly requirements of real-world object designs [46]. Our work bridges this gap by introducing a framework for generating physically stable and buildable LEGO designs directly from text prompts.

但这些方法不能直接用于 LEGO 生成：它们未考虑真实物体设计特有的物理约束与装配要求 [46]。本文通过从文本直接生成物理稳定、可搭建的 LEGO 设计来弥合这一缺口。

Autoregressive 3D Modeling.

Recent research has successfully used autoregressive models to generate 3D meshes [55, 64, 4, 67, 5, 6, 18, 79], often conditioned on input text or images. Most recently, LLaMA-Mesh [78] demonstrates that large language models (LLMs) can be fine-tuned to output 3D shapes in plain-text format, given a text prompt. However, most existing autoregressive methods focus on mesh generation. In contrast, we focus on generating LEGO designs from text prompts, leveraging LLMs’ reasoning capabilities.

自回归三维建模。

近期研究用自回归模型生成三维网格 [55, 64, 4, 67, 5, 6, 18, 79]，常以文本或图像为条件。最近 LLaMA-Mesh [78] 表明，对 LLM 微调后可按文本提示以纯文本格式输出三维形状。但现有自回归方法大多聚焦网格；本文则面向从文本生成 LEGO 设计，发挥 LLM 的推理能力。

Figure 2: StableText2Lego Dataset. (a) From a ShapeNetCore [2] mesh, we generate a LEGO design by voxelizing it onto a $20\times 20\times 20$ grid and applying legolization to determine the brick layout. (b) We augment each shape with multiple structural variations by randomizing the brick layout while preserving the overall shape. (c) Stability analysis [37] is performed on each variation to filter out physically unstable designs. (d) To obtain the corresponding captions for each shape, we render the LEGO design from 24 different viewpoints and use GPT-4o [1] to generate detailed geometric descriptions. (e) Data samples from 5 categories in our StableText2Lego dataset.

图 2：StableText2Lego 数据集。(a) 从 ShapeNetCore [2] 网格出发，体素化到 $20\times 20\times 20$ 网格并做 legolization，得到积木布局。(b) 在保持整体外形的前提下随机化积木布局，为每个形状做多种结构增广。(c) 对每种变体做稳定性分析 [37]，滤除物理不稳定设计。(d) 为获得每个形状的描述，从 24 个视角渲染 LEGO，并用 GPT-4o [1] 生成细致几何描述。(e) StableText2Lego 中 5 个类别的样本。

LEGO Assembly and Generation.

Creating LEGO® designs given a reference 3D shape has been widely studied [28]. Existing works [57, 69, 88] formulate the generation as an optimization problem guided by hand-crafted heuristic rules. Such heuristics can include ensuring that all bricks are interconnected, minimizing the number of bricks, and maximizing the number of brick orientation alternations. Luo et al. 2015 leverages structural stability estimation to find weak structural parts and rearrange the local brick layout to generate physically stable designs. Kim et al. 2020, Liu et al. 2025b formulate a planning problem to fill the target 3D model sequentially. However, these methods only generate designs given an input 3D shape, assuming a valid LEGO design exists, which is difficult to verify in practice.

LEGO 装配与生成。

给定参考三维形状生成 LEGO® 设计已被广泛研究 [28]。已有工作 [57, 69, 88] 把手写启发式规则做成优化问题，例如保证积木连通、尽量少用积木、尽量交替朝向等。Luo 等 2015 借助结构稳定性估计找薄弱局部并重排布局以得到物理稳定设计。Kim 等 2020、Liu 等 2025b 把填满目标三维模型表述为顺序规划。但这些方法都假定输入三维形状已存在且可做成合法 LEGO，而这一点实践中很难先验验证。

Few works have explored learning-based techniques to generate LEGO designs. Thompson et al. 2020 use a deep graph generative model in which the graph encodes brick connectivity. However, this method is limited to generating simple classes, such as walls and cuboids, using a single brick type. More recently, Ge et al. 2024b use a diffusion model to predict a semantic volume, which is then translated into a high-quality micro building. Their method produces impressive results for a single category. Zhou et al. 2022 and Ge et al. 2024a generate compelling figurine designs given an input portrait. They use machine learning to select from a pre-made set of components that best match an input photo. Although effective for faces, extending this selection-based approach to arbitrary objects is challenging. Zhou et al. 2023 formulate an optimization problem to create a LEGO model from an input image. While their output is a 2D LEGO mosaic, we focus on 3D structures in this work. Goldberg et al. 2025 queries a vision-language model (VLM) to generate diverse 3D assembly structures. However, they use regular building blocks instead of LEGO bricks, which do not have interlocking connections and thus have limited expressiveness.

用学习方法生成 LEGO 的工作仍少。Thompson 等 2020 用深度图生成模型编码积木连通，但仅限墙、长方体等简单类别，且只用单一砖型。较近的 Ge 等 2024b 用扩散预测语义体，再转成高质量微缩建筑，效果亮眼但限于单类。Zhou 等 2022 与 Ge 等 2024a 可根据肖像生成人偶：用机器学习从预制零件集中挑选最匹配照片的组件；对人脸有效，但难扩展到任意物体。Zhou 等 2023 从输入图像优化出 LEGO 模型，输出是二维马赛克，而我们关注三维结构。Goldberg 等 2025 查询视觉—语言模型生成多样三维装配结构，但用的是普通积木而非带互锁连接的 LEGO，表达力受限。

Our goal is closest to that of [30]. This work has three steps: (1) generating an image using a text-to-image model, (2) converting the image into voxels, and (3) using heuristics to create a physical LEGO build without considering physical constraints. In contrast, our method performs the text-to-LEGO task without requiring intermediate image or voxel representations.

目标上最接近 [30]。该工作分三步：(1) 文生图；(2) 图转体素；(3) 用启发式做物理 LEGO，且不考虑物理约束。相比之下，我们的文生 LEGO 不经过中间图像或体素表示。

Physics-Aware Generation.

Despite significant advances in 3D generation, incorporating physics constraints remains largely underexplored; most approaches focus primarily on geometry or appearance modeling. Physics-aware generation can be broadly categorized into two approaches: direct constraint enforcement and learned validation. Simple physical constraints, such as collision avoidance and contact requirements, can be incorporated directly through explicit penalty terms during optimization [53, 82, 84, 75, 41, 24, 16]. More complex physical properties, such as structural stability and dynamic behavior, typically require differentiable physics simulators [7, 56, 80, 50] or data-driven physics-aware assessment models [10, 49]. To our knowledge, our paper is the first attempt to incorporate physics-aware constraints into text-based LEGO generation.

物理感知生成。

三维生成进展很大，但引入物理约束仍相对少见；多数工作仍聚焦几何或外观。物理感知生成大致分两类：直接施加约束，与学习式校验。碰撞避免、接触要求等简单约束，可在优化中用显式惩罚项纳入 [53, 82, 84, 75, 41, 24, 16]。结构稳定性、动力学等更复杂性质，通常需要可微物理仿真 [7, 56, 80, 50] 或数据驱动的物理评估模型 [10, 49]。据我们所知，本文是首次把物理感知约束纳入基于文本的 LEGO 生成。

3 Dataset

3 数据集

Training a modern autoregressive model requires a large-scale dataset. Therefore, we introduce StableText2Lego, a new large-scale LEGO® dataset that contains $47{,}000+$ different LEGO structures, covering $28{,}000+$ unique 3D objects from $21$ common object categories of the ShapeNetCore dataset [2]. We select categories featuring diverse and distinctive 3D objects while excluding those resembling cuboids. Each structure is paired with a group of text descriptions and a stability score, which indicates its physical stability and buildability. Below, we describe the dataset construction, an overview of which is given in Figure 2.

训练现代自回归模型需要大规模数据。我们因此提出 StableText2Lego：含 $47{,}000+$ 种 LEGO® 结构，覆盖 ShapeNetCore [2] 中 $21$ 个常见类别的 $28{,}000+$ 独特三维物体。所选类别外形多样、有辨识度，并排除近似长方体的类别。每个结构配有一组文本描述与稳定性分数，刻画物理稳定与可搭建性。构建流程概览见图 2。

Figure 3: Method. (a) Our system tokenizes a LEGO design into a sequence of text tokens, ordered in a raster-scan manner from bottom to top. (b) We create an instruction dataset pairing brick sequences with descriptions to fine-tune LLaMA-3.2-Instruct-1B. (c) At inference time, LegoGPT generates LEGO designs incrementally by predicting one brick at a time given a text prompt. For each generated brick, we perform validity checks to ensure it is well-formatted, exists in our brick library, and does not collide with existing bricks. After completing the design, we verify its physical stability. If the structure is unstable, we roll back to the stable state by removing all the unstable bricks and their subsequent ones and resume generation from that point.

图 3：方法。(a) 系统将 LEGO 设计按自下而上光栅扫描顺序切成文本词元序列。(b) 构建积木序列与描述配对的指令数据，微调 LLaMA-3.2-Instruct-1B。(c) 推理时 LegoGPT 按文本提示一块一块增量生成。对每一块做合法性检查：格式正确、属于砖库、且不与已有积木碰撞。设计完成后核验物理稳定性；若不稳定，则回滚到稳定状态——去掉全部不稳定积木及其后续积木——再从该点继续生成。

LEGO Representation.

We consider LEGO structures built on a LEGO baseplate. Each LEGO structure in StableText2Lego is represented as $B=[b_{1},b_{2},\dots,b_{N}]$ with $N$ bricks, and each element denotes a brick’s state as $b_{i}=[h_{i},w_{i},x_{i},y_{i},z_{i}]$. Here, $h_{i}$ and $w_{i}$ indicate the brick length in the $X$ and $Y$ directions, respectively, and $x_{i}$, $y_{i}$, and $z_{i}$ denote the position of the stud closest to the origin. The position has $x_{i}\in[0,1,\dots,H-1]$, $y_{i}\in[0,1,\dots,W-1]$, $z_{i}\in[0,1,\dots,D-1]$, and $H$, $W$, and $D$ represent the dimensions of the discretized grid world.

LEGO 表示。

我们考虑建在底板上的 LEGO 结构。StableText2Lego 中每个结构表示为 $B=[b_{1},b_{2},\dots,b_{N}]$，含 $N$ 块积木；每块状态为 $b_{i}=[h_{i},w_{i},x_{i},y_{i},z_{i}]$。其中 $h_{i}$、$w_{i}$ 分别为 $X$、$Y$ 方向长度，$x_{i},y_{i},z_{i}$ 为最靠近原点的凸粒位置。位置满足 $x_{i}\in[0,1,\dots,H-1]$，$y_{i}\in[0,1,\dots,W-1]$，$z_{i}\in[0,1,\dots,D-1]$，$H,W,D$ 为离散网格世界尺寸。

Mesh-to-LEGO.

We construct the dataset by converting 3D shapes from ShapeNetCore [2] into LEGO structures as shown in Figure 2(a). Given a 3D mesh, we voxelize and downsample it to a $20\times 20\times 20$ grid world to ensure a consistent scale, i.e., $H=W=D=20$. The LEGO brick layout is generated by a split-and-remerge legolization algorithm. To improve data quality and diversity, we introduce randomness in legolization and generate multiple different LEGO structures for the same 3D object, as illustrated in Figure 2(b). This increases the chance of obtaining a stable LEGO structure and more diverse LEGO brick layouts. More details can be found in Appendix A. We use eight commonly available standard bricks: $1\times 1$, $1\times 2$, $1\times 4$, $1\times 6$, $1\times 8$, $2\times 2$, $2\times 4$, and $2\times 6$.

网格到 LEGO。

如图 2(a)，我们把 ShapeNetCore [2] 三维形状转成 LEGO。给定网格，体素化并下采样到 $20\times 20\times 20$ 网格以统一尺度，即 $H=W=D=20$。积木布局由 split-and-remerge legolization 得到。为提升质量与多样性，我们在 legolization 中引入随机性，同一三维物体生成多种不同 LEGO 结构，见图 2(b)，从而提高得到稳定结构与多样布局的机会。细节见附录 A。使用八种常见标准砖：$1\times 1$、$1\times 2$、$1\times 4$、$1\times 6$、$1\times 8$、$2\times 2$、$2\times 4$、$2\times 6$。

Stability Score.

We assess the physical stability of each structure, as illustrated in Figure 2(c), using the analysis method [37]. For a structure $B=[b_{1},b_{2},\dots,b_{N}]$, the stability score $S\in\mathbb{R}^{N}$ assigns each brick $b_{i}$ a value $s_{i}\in[0,1]$ that quantifies the internal stress at its connections. Higher scores $s_{i}$ indicate greater stability, while $s_{i}=0$ indicates an unstable brick that will cause structural failure. Calculating the stability score requires solving a nonlinear program to determine the forces acting on each brick to achieve a static equilibrium that prevents structural collapse, as detailed in Section 4.2. For typically-sized (i.e., $<200$ bricks) structures in Figure 2, it takes $\sim$0.35 seconds on average. We only include stable structures where all bricks have stability scores greater than 0.

稳定性分数。

如图 2(c)，我们用分析方法 [37] 评估物理稳定性。对结构 $B=[b_{1},b_{2},\dots,b_{N}]$，稳定性分数 $S\in\mathbb{R}^{N}$ 给每块 $b_{i}$ 赋 $s_{i}\in[0,1]$，刻画连接处内应力。$s_{i}$ 越高越稳，$s_{i}=0$ 表示该砖会导致结构失效。计算需解非线性规划，求使结构达到静力平衡、避免垮塌的各砖受力，细节见第 4.2 节。对图 2 中典型规模（即 $<200$ 块）结构，平均约需 $\sim$0.35 秒。我们只保留所有砖 $s_{i}>0$ 的稳定结构。

Caption Generation.

To obtain captions for each shape, we render the LEGO from 24 different viewpoints and combine them into a single multi-view image. We then prompt GPT-4o [1] to produce five descriptions for these renderings with various levels of detail. Importantly, we ask GPT-4o to omit color information and focus only on geometry. The complete GPT-4o prompt is provided in Appendix A.

描述生成。

为每个形状生成描述时，从 24 个视角渲染 LEGO 并拼成多视图，再提示 GPT-4o [1] 写出五条详略不同的描述。我们明确要求省略颜色、只谈几何。完整提示见附录 A。

Figure 2(e) shows several data samples in StableText2Lego. The rich variations within each category and the comprehensive text-LEGO pairs make it possible to train large-scale generative models. More insights on our StableText2Lego are discussed in Appendix A.

图 2(e) 展示 StableText2Lego 中若干样本。各类别内丰富变体以及完备的文本—LEGO 配对，使大规模生成模型训练成为可能。更多讨论见附录 A。

4 Method

4 方法

Here, we introduce LegoGPT, a method for generating physically stable LEGO designs from text prompts. Leveraging LLMs’ ability to model sequences and understand text, we fine-tune a pre-trained LLM for the LEGO generation task (Section 4.1). To increase the stability and buildability of our designs, we use brick-by-brick rejection sampling and physics-aware rollback during inference (Section 4.2). Figure 3 illustrates an overview of our method.

下面介绍 LegoGPT：从文本提示生成物理稳定 LEGO 设计的方法。利用 LLM 建模序列与理解文本的能力，我们在 LEGO 生成任务上微调预训练 LLM（第 4.1 节）。为提升稳定与可搭建性，推理时采用逐砖拒绝采样与物理感知回滚（第 4.2 节）。方法概览见图 3。

4.1 Model Fine-tuning

4.1 模型微调

Pre-trained LLMs excel at modeling sequences and understanding natural language, making them promising candidates for our task. We further fine-tune a pre-trained LLM on a custom instruction dataset containing text prompts and their corresponding LEGO structures from StableText2Lego.

预训练 LLM 擅长序列建模与自然语言理解，适合本任务。我们在自定义指令数据上进一步微调：数据来自 StableText2Lego 的文本提示及其对应 LEGO 结构。

Pre-trained Base Model.

We use LLaMA-3.2-1B-Instruct [11] as our base model. This model is fine-tuned to give coherent answers to instruction prompts, making it suitable for text-based LEGO design generation. As shown in Figure 5, the base model can generate LEGO-like designs through in-context learning, highlighting the promise of using pre-trained LLMs for our task. However, the generated designs often omit certain object parts and contain intersecting or disconnected bricks, making them physically unstable and unbuildable. To address these issues, we further fine-tune the pre-trained model using our dataset.

预训练基座。

基座选用 LLaMA-3.2-1B-Instruct [11]。该模型已针对指令提示做微调，适合基于文本的 LEGO 设计生成。如图 5，基座可通过上下文学习生成「像 LEGO」的设计，说明预训练 LLM 有潜力；但常缺部件、出现相交或断开的积木，物理上不稳定且难搭建。为此我们用自建数据继续微调。

Instruction Fine-tuning Dataset.

For each stable design and its corresponding captions, we construct an instruction in the following format: “(user) Create a LEGO model of {caption}. (assistant) {lego-design}.”

指令微调数据。

对每个稳定设计及其描述，构造如下指令：“(user) Create a LEGO model of {caption}. (assistant) {lego-design}.”

To simplify training and reuse LLaMA’s tokenizer, we represent LEGO designs in plain text. But what format should we use? The standard LEGO design format LDraw [29] has two main drawbacks. First, it does not directly include brick dimensions, which are crucial for assessing the structure and validating brick placements. Second, it contains unnecessary information, such as brick orientation and scale. This information is redundant, as each axis-aligned brick has only two valid orientations.

为简化训练并复用 LLaMA 分词器，我们用纯文本表示 LEGO 设计。该用什么格式？标准 LDraw [29] 有两点不足：一是不直接给出砖尺寸，而尺寸对结构理解与合法性至关重要；二是含朝向、尺度等冗余信息——轴对齐砖只有两种有效朝向。

Instead of using LDraw, we introduce a custom format to represent each LEGO design. Each line of our format represents one LEGO brick as “{$h$}$\times${$w$} ({$x$},{$y$},{$z$})”, where $h\times w$ are brick dimensions and $(x,y,z)$ are its coordinates. All bricks are 1-unit-tall, axis-aligned cuboids, and the order of $h$ and $w$ encodes the brick’s orientation about the vertical axis. This format significantly reduces the number of tokens required to represent a design while including brick dimension information essential for 3D reasoning. Bricks are ordered in a raster-scan manner from bottom to top.

我们因此改用自定义格式。每行一块砖：“{$h$}$\times${$w$} ({$x$},{$y$},{$z$})”，$h\times w$ 为尺寸，$(x,y,z)$ 为坐标。砖均为 1 单位高的轴对齐长方体，$h$ 与 $w$ 的顺序编码绕竖直轴朝向。该格式显著减少表示所需词元，同时保留三维推理必需的尺寸信息。砖按自下而上光栅扫描排序。

With our fine-tuned model LegoGPT $\theta$, we predict the bricks $b_{1},b_{2},...,b_{N}$ in an autoregressive manner:

微调后的 LegoGPT $\theta$ 按自回归方式预测砖序列 $b_{1},b_{2},...,b_{N}$：

$$\displaystyle p(b_{1},b_{2},...,b_{N}|\theta)=\prod_{i=1}^{N}p(b_{i}|b_{1},...,b_{i-1},\theta). \tag{1}$$

4.2 Integrating Physical Stability

4.2 融入物理稳定性

Although trained on physically stable data, our model sometimes generates designs that violate physics and assembly constraints. To address this issue, we further incorporate physical stability verification into autoregressive inference.

尽管在物理稳定数据上训练，模型仍偶尔生成违反物理与装配约束的设计。为此，我们在自回归推理中进一步引入物理稳定性校验。

Figure 4: Force Model. (a) We consider all forces exerted on a single brick, including gravity (black), vertical forces with the top brick (red/blue) and bottom brick (green/purple), and horizontal (shear) forces due to knob connections (cyan), and adjacent bricks (yellow). (b) The structural force model $\mathcal{F}$ extends the individual force model to multiple bricks. Solving for static equilibrium in $\mathcal{F}$ determines each brick’s stability score.

图 4：力模型。(a) 考虑单块积木所受全部力：重力（黑）、与上方砖的竖直力（红/蓝）、与下方砖的竖直力（绿/紫），以及凸粒连接引起的水平（剪切）力（青）和邻砖水平力（黄）。(b) 结构力模型 $\mathcal{F}$ 把单砖模型推广到多砖。在 $\mathcal{F}$ 上求解静力平衡，得到每块的稳定性分数。

A LEGO structure is considered physically stable and buildable if it does not collapse when built on a baseplate. To this end, we assess physical structural stability using the stability analysis method [37]. Here, we briefly overview this method below. Figure 4(a) illustrates all possible forces exerted on a single brick. We extend the single brick model and derive the structural force model $\mathcal{F}$, which consists of a set of candidate forces (e.g., pulling, pressing, supporting, dragging, normal, etc.), as demonstrated in Figure 4(b). For a LEGO structure $B=[b_{1},b_{2},\dots,b_{N}]$, each brick $b_{i}$ has $M_{i}$ candidate forces $F_{i}^{j}\in\mathcal{F}_{i},j\in[1,M_{i}]$. A structure is stable if all bricks can reach static equilibrium, i.e.,

若 LEGO 结构建在底板上不会垮塌，则视为物理稳定且可搭建。我们沿用稳定性分析方法 [37]，简述如下。图 4(a) 画出单砖可能受力；推广得到结构力模型 $\mathcal{F}$，包含一组候选力（拉、压、支撑、拖拽、法向等），见图 4(b)。对结构 $B=[b_{1},b_{2},\dots,b_{N}]$，每块 $b_{i}$ 有 $M_{i}$ 个候选力 $F_{i}^{j}\in\mathcal{F}_{i},j\in[1,M_{i}]$。若所有砖都能达到静力平衡，则结构稳定，即

$$\displaystyle\sum_{j}^{M_{i}}F_{i}^{j}=0,\qquad\sum_{j}^{M_{i}}{\tau}_{i}^{j}\dot{=}\sum_{j}^{M_{i}}L_{i}^{j}\times F_{i}^{j}=0, \tag{2}$$

where $L_{i}^{j}$ denotes the force lever corresponding to $F_{i}^{j}$. The stability analysis is formulated into a nonlinear program as

其中 $L_{i}^{j}$ 为对应 $F_{i}^{j}$ 的力臂。稳定性分析写成非线性规划：

$$\begin{split}\argmin_{\mathcal{F}}&\sum_{i}^{N}\Biggl\{|\sum_{j}^{M_{i}}F_{i}^{j}|+|\sum_{j}^{M_{i}}{\tau}_{i}^{j}|+\alpha\mathcal{D}_{i}^{\max}+\beta\sum\mathcal{D}_{i}\Biggl\},\end{split} \tag{3}$$

subject to three constraints: 1) all force candidates in $\mathcal{F}$ should take non-negative values; 2) certain forces exerted on the same brick cannot coexist, e.g., the pulling (red arrow) and pressing (blue arrow), the dragging (green arrow) and supporting (purple arrow); 3) Newton’s third law, e.g., at a given connection point, the supporting force on the upper brick should be equal to the pressing force on the bottom brick. $\mathcal{D}_{i}\subset\mathcal{F}_{i}$ is the set of candidate dragging forces (green arrow) on $b_{i}$. $\alpha$ and $\beta$ are hyperparameter weights.

约束有三：1) $\mathcal{F}$ 中候选力取值非负；2) 同一砖上某些力不能并存，例如拉（红）与压（蓝）、拖（绿）与支撑（紫）；3) 牛顿第三定律，例如连接点上，上方砖的支撑力应等于下方砖的压力。$\mathcal{D}_{i}\subset\mathcal{F}_{i}$ 是 $b_{i}$ 上候选拖拽力（绿）集合。$\alpha$、$\beta$ 为超参权重。

Solving the above nonlinear program in Eqn. 3 using Gurobi [17] finds a force distribution $\mathcal{F}$ that drives the structure to static equilibrium with the minimum required internal stress, suppressing the overall friction (i.e., $\sum\mathcal{D}_{i}$) as well as avoiding extreme values (i.e., $\mathcal{D}_{i}^{\max}$). From the force distribution $\mathcal{F}$, we obtain the per-brick stability score as

用 Gurobi [17] 求解式 3，得到使结构以最小所需内应力达到静力平衡的力分布 $\mathcal{F}$，同时抑制整体摩擦（即 $\sum\mathcal{D}_{i}$）并避免极值（即 $\mathcal{D}_{i}^{\max}$）。由 $\mathcal{F}$ 得到逐砖稳定性分数：

$$\begin{split}s_{i}=\begin{cases}0&\begin{array}[]{c}\sum_{j}^{M_{i}}F_{i}^{j}\neq 0\
\vee\quad\sum_{j}^{M_{i}}{\tau}{i}^{j}\neq 0\
\vee\quad\mathcal{D}{i}^{\max}>F_{T},\end{array}\
\frac{F_{T}-\mathcal{D}{i}^{\max}}{F{T}}&\text{otherwise},\end{cases}\end{split} \tag{4}$$

where $F_{T}$ is a measured constant friction capacity between brick connections. Higher scores $s_{i}$ indicate greater stability, while $s_{i}=0$ indicates an unstable brick that will cause structural failure: either $\mathcal{F}$ cannot reach static equilibrium ($\sum_{j}^{M_{i}}F_{i}^{j}\neq 0\vee\sum_{j}^{M_{i}}\tau_{i}^{j}\neq 0$) or the required friction exceeds the friction capacity of the material ($\mathcal{D}_{i}^{\max}>F_{T}$). Due to the equality constraints imposed by Newton’s third law, Eqn. 3 includes only the dragging forces and excludes pulling forces. For a physically stable structure, we need $s_{i}>0,\forall i\in[1,N]$.

其中 $F_{T}$ 为砖连接间测得的恒定摩擦容量。$s_{i}$ 越高越稳；$s_{i}=0$ 表示该砖会导致失效：要么 $\mathcal{F}$ 达不到静力平衡（$\sum_{j}^{M_{i}}F_{i}^{j}\neq 0\vee\sum_{j}^{M_{i}}\tau_{i}^{j}\neq 0$），要么所需摩擦超过材料容量（$\mathcal{D}_{i}^{\max}>F_{T}$）。因牛顿第三定律等式约束，式 3 只含拖拽力、不含拉力。物理稳定结构要求 $s_{i}>0,\forall i\in[1,N]$。

When to apply stability analysis?

Our model generates LEGO designs sequentially, one LEGO brick at a time. A straightforward approach to ensuring physical stability is to apply stability analysis to each step and resample a brick that would cause a collapse. However, this step-by-step validation, though efficient per check, could be time-consuming due to the large number of checks required. More importantly, many LEGO designs are unstable when partially constructed, yet become stable when fully assembled. Adding a stability check after each brick generation could overly constrain the model exploration space. Instead, we propose brick-by-brick rejection sampling combined with physics-aware rollback to balance stability and diversity.

何时做稳定性分析？

模型按块顺序生成。最直接做法是每步都做稳定性分析，并对会导致垮塌的砖重采样。单次检查虽快，但检查次数多、总耗时长；更重要的是，许多设计在半成品时不稳定，完工后才稳定。逐步检查会过度压缩探索空间。我们因此提出逐砖拒绝采样加物理感知回滚，在稳定与多样之间折中。

Brick-by-Brick Rejection Sampling

To improve inference speed and avoid overly constraining the model generation, we relax our constraints during inference. First, when the model generates a LEGO brick and its position, the brick should be well-formatted (e.g., available in the inventory) and not lie outside the workspace. Second, we ensure that newly added bricks do not collide with the existing structure. Formally, for each generated brick $b_{t}$, we have $\mathcal{V}_{t}\cap\mathcal{V}_{i}=\varnothing,\forall i\in[1,t-1]$, where $\mathcal{V}_{i}$ denotes the voxels occupied by $b_{i}$. These heuristics allow us to efficiently generate well-formatted LEGO structures without explicitly considering complex physical stability. To integrate these heuristics, we use rejection sampling: if a brick violates the heuristics, we resample a new brick from the model. Due to the relaxed constraints, most bricks are valid, and rejection sampling does not significantly affect inference time.

逐砖拒绝采样

为加快推理并避免过度约束，推理时放宽约束。第一，生成的砖及其位置须格式合法（如库存中有）且不越出工作空间；第二，新砖不得与已有结构碰撞。形式化地，对每块生成砖 $b_{t}$，要求 $\mathcal{V}_{t}\cap\mathcal{V}_{i}=\varnothing,\forall i\in[1,t-1]$，$\mathcal{V}_{i}$ 为 $b_{i}$ 占据的体素。这些启发式可高效得到格式正确的结构，而不显式处理复杂物理稳定性。实现上用拒绝采样：违规则从模型重采样。因约束较松，多数砖合法，对推理时间影响不大。

Physics-Aware Rollback

To ensure that the final design $B=[b_{1},b_{2},\dots,b_{N}]$ is physically stable, we calculate the stability score $S$. If the resulting design is unstable, i.e., $s_{i}=0,i\in\mathcal{I}$, we roll back the design to the state before the first unstable brick was generated, i.e., $B^{\prime}=[b_{1},b_{2},\dots,b_{\min{\mathcal{I}}-1}]$. Here, $\mathcal{I}$ is the set of the indices of all the unstable bricks. We repeat this process iteratively until we reach a stable structure $B^{\prime}$, and continue generation from the partial structure $B^{\prime}$. Note that we can use the per-brick stability score to efficiently find the collapsing bricks and their corresponding indices in the sequence. We summarize our inference sampling in Algorithm 1.

物理感知回滚

为保证最终设计 $B=[b_{1},b_{2},\dots,b_{N}]$ 物理稳定，计算稳定性分数 $S$。若不稳定，即 $s_{i}=0,i\in\mathcal{I}$，则回滚到第一块不稳定砖生成之前：$B^{\prime}=[b_{1},b_{2},\dots,b_{\min{\mathcal{I}}-1}]$，其中 $\mathcal{I}$ 为不稳定砖下标集合。迭代直至得到稳定结构 $B^{\prime}$，再从部分结构 $B^{\prime}$ 继续生成。逐砖分数可高效定位垮塌砖及其序列位置。推理采样见算法 1。

Input: Text prompt $c$; Autoregressive model $\theta$
Output: LEGO design following the text prompt
$B\leftarrow\text{empty LEGO structure}$; 1
loop 2
/* Predict next brick w/ rejection sampling */
for k=1,\ldots,\texttt{max_rejections} do 3
\texttt{context}\leftarrow T\oplus B.\texttt{to_text_format}(); 4
b\leftarrow\theta.\texttt{predict_tokens}(\texttt{context}) (Eqn. 1); 5
if $b$ is valid then break; 6
end for 7
B.\texttt{add_brick}(b); 8
if $b$ contains EOF then // Structure complete 9
if $B$ is stable or max rollbacks exceeded then return $B$; 10
while $B$ is unstable do // Rollback if unstable 11
$\mathcal{I}\leftarrow$ indices of unstable bricks in $B$; 12
$i\leftarrow\min\mathcal{I}$;
// idx of 1st unstable brick
$B\leftarrow[b_{1},\dots,b_{i-1}]$; 13
end while 14
end if 15
end 16
17

ALGORITHM 1 LegoGPT inference algorithm.

输入： 文本提示 $c$；自回归模型 $\theta$
输出： 符合文本提示的 LEGO 设计
$B\leftarrow$空 LEGO 结构；1
loop 2
/* 带拒绝采样地预测下一块砖 */
for k=1,\ldots,\texttt{max_rejections} do 3
\texttt{context}\leftarrow T\oplus B.\texttt{to_text_format}()；4
b\leftarrow\theta.\texttt{predict_tokens}(\texttt{context})（式 1）；5
if $b$ 合法 then break；6
end for 7
B.\texttt{add_brick}(b)；8
if $b$ 含 EOF then // 结构完成 9
if $B$ 稳定或已达最大回滚次数 then return $B$；10
while $B$ 不稳定 do // 不稳定则回滚 11
$\mathcal{I}\leftarrow B$ 中不稳定砖下标；12
$i\leftarrow\min\mathcal{I}$；
// 第一块不稳定砖下标
$B\leftarrow[b_{1},\dots,b_{i-1}]$；13
end while 14
end if 15
end 16
17

算法 1 LegoGPT 推理算法。

Method	% valid $\uparrow$	% stable $\uparrow$	mean brick stability $\uparrow$	min brick stability $\uparrow$	CLIP $\uparrow$
Pre-trained LLaMA (0-shot)	0.0%	0.0%	N/A	N/A	N/A
In-context learning (5-shot)	2.4%	1.2%	0.675	0.479	0.284
LLaMA-Mesh [78]	94.8%	50.8%	0.894	0.499	0.311
LGM [68]	100%	25.2%	0.942	0.231	0.299
XCube [60]	100%	75.2%	0.964	0.686	0.322
Hunyuan3D-2 [86]	100%	75.2%	0.973	0.704	0.324
Ours w/o rejection sampling or rollback	37.2%	12.8%	0.956	0.325	0.329
Ours w/o rollback	100%	24.0%	0.947	0.228	0.322
Ours (LegoGPT)	100%	98.8%	0.996	0.915	0.324

Table 1: Quantitative Analysis. We evaluate our method against several baselines on validity (no out-of-library, out-of-bounds, or colliding bricks), stability, and CLIP-based text similarity. Stability and CLIP score are computed over valid designs only. For LLaMA-Mesh [78], validity requires a well-formed OBJ file. Our method outperforms all baselines as well as the ablated setups on validity and stability using our proposed rejection sampling and rollback, while maintaining high text similarity.

Method	% valid $\uparrow$	% stable $\uparrow$	mean brick stability $\uparrow$	min brick stability $\uparrow$	CLIP $\uparrow$
Pre-trained LLaMA (0-shot)	0.0%	0.0%	N/A	N/A	N/A
In-context learning (5-shot)	2.4%	1.2%	0.675	0.479	0.284
LLaMA-Mesh [78]	94.8%	50.8%	0.894	0.499	0.311
LGM [68]	100%	25.2%	0.942	0.231	0.299
XCube [60]	100%	75.2%	0.964	0.686	0.322
Hunyuan3D-2 [86]	100%	75.2%	0.973	0.704	0.324
Ours w/o rejection sampling or rollback	37.2%	12.8%	0.956	0.325	0.329
Ours w/o rollback	100%	24.0%	0.947	0.228	0.322
Ours (LegoGPT)	100%	98.8%	0.996	0.915	0.324

表 1：定量分析。 在合法性（无库外、越界或碰撞砖）、稳定性与基于 CLIP 的文本相似度上对比若干基线。稳定性与 CLIP 仅在合法设计上计算。对 LLaMA-Mesh [78]，合法性要求 OBJ 文件格式正确。借助提出的拒绝采样与回滚，我们的方法在合法性与稳定性上优于全部基线及消融设置，同时保持较高文本相似度。

4.3 LEGO Texturing and Coloring

4.3 LEGO 贴图与上色

While our primary focus is on generating LEGO shapes, color and texture play a critical role in creative LEGO designs. Therefore, we propose a method that applies detailed UV textures or assigns uniform colors to individual bricks.

我们主要关注 LEGO 形状生成，但颜色与纹理对创意设计同样关键。因此提出方法：既可为结构施加细致 UV 纹理，也可为单砖赋统一颜色。

UV Texture Generation.

Given a structure $B$ and its corresponding mesh $\mathcal{M}$, we first identify the set of occluded bricks $B_{\text{occ}}$ that have all six faces covered by adjacent bricks, and remove $B_{\text{occ}}$ for efficiency. The remaining bricks $B_{\text{vis}}=B\setminus B_{\text{occ}}$ are merged into a single mesh $\mathcal{M}$ with cleaned overlapping vertices using ImportLDraw [72]. We generate a UV map $\text{UV}_{\mathcal{M}}$ by cube projection. The texture map $I_{\text{texture}}$ is then generated using FlashTex [9], a fast text-based mesh texturing approach:

UV 纹理生成。

给定结构 $B$ 及其网格 $\mathcal{M}$，先找出六面皆被邻砖遮挡的遮挡砖集合 $B_{\text{occ}}$ 并移除以提速。其余可见砖 $B_{\text{vis}}=B\setminus B_{\text{occ}}$ 用 ImportLDraw [72] 合并为单网格 $\mathcal{M}$ 并清理重叠顶点。用立方体投影生成 UV 图 $\text{UV}_{\mathcal{M}}$，再用快速文生网格贴图方法 FlashTex [9] 得到纹理图 $I_{\text{texture}}$：

$$I_{\text{texture}}=\text{FlashTex}(\mathcal{M},\text{UV}_{\mathcal{M}},c), \tag{5}$$

where text prompt $c$ describes the visual appearance. This texture can be applied through UV printing or stickers.

其中文本提示 $c$ 描述外观。该纹理可通过 UV 印刷或贴纸施加。

Uniform Brick Color Assignment.

We can also assign each brick a uniform color from the LEGO color library. Given a structure $B$, we convert it to a voxel grid $\mathcal{V}$ and then to a UV-unwrapped mesh $\mathcal{M}_{\mathcal{V}}$. For every voxel $v\in\mathcal{V}$, let $f^{v}_{i},i=1,\dots,N_{v}$ be its visible faces where $0\leq N_{v}\leq 6$. Each face $f^{v}_{i}$ is split into two triangles and mapped to a UV region $\mathcal{S}^{v}_{i}$, creating a mesh $\mathcal{M}_{\mathcal{V}}$ with UV map $\text{UV}_{\mathcal{V}}$. We apply FlashTex [9] to generate a texture $I_{\text{texture}}$:

统一砖色赋值。

也可从 LEGO 色库为每块砖赋统一色。给定 $B$，先转为体素网格 $\mathcal{V}$，再转为 UV 展开网格 $\mathcal{M}_{\mathcal{V}}$。对每个体素 $v\in\mathcal{V}$，设可见面为 $f^{v}_{i},i=1,\dots,N_{v}$，其中 $0\leq N_{v}\leq 6$。每个面 $f^{v}_{i}$ 拆成两个三角形并映射到 UV 区域 $\mathcal{S}^{v}_{i}$，得到带 UV 图 $\text{UV}_{\mathcal{V}}$ 的网格 $\mathcal{M}_{\mathcal{V}}$。用 FlashTex [9] 生成纹理 $I_{\text{texture}}$：

$$I_{\text{texture}}=\text{FlashTex}(\mathcal{M}_{\mathcal{V}},\text{UV}_{\mathcal{V}},c). \tag{6}$$

The color of each voxel $\mathcal{C}(v)\in\mathbb{R}^{3}$ is computed as:

每个体素颜色 $\mathcal{C}(v)\in\mathbb{R}^{3}$ 计算为：

$$\mathcal{C}(v)=\frac{1}{N_{v}}\sum_{i=1}^{N_{v}}\mathcal{C}(f^{v}_{i}),\quad\forall v\in\mathcal{V}, \tag{7}$$

where $\mathcal{C}(f^{v}_{i})=\frac{1}{|\mathcal{S}^{v}_{i}|}\sum_{(x,y)\in\mathcal{S}^{v}_{i}}I_{\text{texture}}(x,y)$ is the color of each visible face $f^{v}_{i}$, and $|\mathcal{S}^{v}_{i}|$ represents the number of pixels in region $\mathcal{S}^{v}_{i}$ in the UV map. For each brick $b_{t}$ and its constituent voxels $\mathcal{V}_{t}$, we compute the brick color $\mathcal{C}(b_{t})=\frac{1}{|\mathcal{V}_{t}|}\sum_{v\in\mathcal{V}_{t}}\mathcal{C}(v)$. Finally, we find the closest color in the color set. While UV texturing offers higher-fidelity details, uniform coloring allows us to use standard LEGO bricks.

其中 $\mathcal{C}(f^{v}_{i})=\frac{1}{|\mathcal{S}^{v}_{i}|}\sum_{(x,y)\in\mathcal{S}^{v}_{i}}I_{\text{texture}}(x,y)$ 为可见面 $f^{v}_{i}$ 的颜色，$|\mathcal{S}^{v}_{i}|$ 为 UV 区域像素数。对砖 $b_{t}$ 及其体素 $\mathcal{V}_{t}$，砖色 $\mathcal{C}(b_{t})=\frac{1}{|\mathcal{V}_{t}|}\sum_{v\in\mathcal{V}_{t}}\mathcal{C}(v)$，再映射到色库最近色。UV 贴图细节更高，统一上色则可用标准 LEGO 砖。

Figure 6: Ablation study. Brick-by-brick rejection sampling and physics-informed rollback help to ensure that the generated LEGO design is both valid and stable. Black indicates colliding bricks.

图 6：消融。逐砖拒绝采样与物理知情回滚有助于保证生成 LEGO 既合法又稳定。黑色表示碰撞砖。

Figure 7: Textured and Colored LEGO Generation. Our method can generate diverse textured (top two rows) and colored (bottom) LEGO models based on the same shape while using different appearance text prompts.

图 7：带纹理与上色的 LEGO 生成。同一形状下，换外观文本提示即可生成多样带纹理（上两行）与上色（下行）LEGO 模型。

Figure 8: Automated Assembly. Demonstrations of automated assembly of generated LEGO structures using robots.

图 8：自动装配。用机器人自动装配生成 LEGO 结构的演示。

5 Experiments

5 实验

5.1 Implementation Details

5.1 实现细节

Fine-tuning.

Our fine-tuning dataset contains 240k distinct prompts and 47k+ distinct LEGO designs. We use 90% of the data for training and 10% for evaluation. For efficiency, we include samples only up to 4096 tokens in length. Training details are provided in Appendix A.

微调。

微调数据含 24 万条不同提示与 4.7 万+ 不同 LEGO 设计，按 90%/10% 划分训练/评估。为效率只保留长度不超过 4096 词元的样本。训练细节见附录 A。

Inference.

To evaluate our method, we generate one LEGO design for each of 250 prompts randomly selected from the validation dataset. The nonlinear optimization in Eqn. 3 is solved using Gurobi [17]. We set $F_{T}=0.98N$ with $\alpha=10^{-3}$ and $\beta=10^{-6}$. We allow up to 100 physics-aware rollbacks before accepting the LEGO design. The median number of required rollbacks is 2, and the median time to generate one LEGO design is 40.8 seconds.

推理。

评估时从验证集随机抽 250 条提示，各生成一个 LEGO 设计。式 3 的非线性优化用 Gurobi [17] 求解。设 $F_{T}=0.98N$，$\alpha=10^{-3}$，$\beta=10^{-6}$。最多允许 100 次物理感知回滚。所需回滚次数中位数为 2，单次生成中位耗时 40.8 秒。

5.2 LEGO Generation Results

5.2 LEGO 生成结果

Figure 5 shows a gallery of diverse, high-quality LEGO designs that closely follow the input prompts.

图 5 展示多样、高质量且紧贴输入提示的 LEGO 设计画廊。

Baseline Comparisons.

As baselines, we use LLaMA-Mesh [78], LGM [68], XCube [60], and Hunyuan3D-2 [86] to generate a mesh from each prompt, and then convert the meshes to LEGO via legolization. Additionally, we compare our method with pre-trained models evaluated in both a zero-shot and few-shot manner. For few-shot evaluation, we provide the model with 5 examples of stable LEGO designs and their captions. We cannot run comparisons with Lennon et al. 2021 as the code has not been released.

基线对比。

基线包括 LLaMA-Mesh [78]、LGM [68]、XCube [60]、Hunyuan3D-2 [86]：先从提示生成网格，再经 legolization 转为 LEGO。另与零样本与少样本预训练模型对比；少样本提供 5 个稳定 LEGO 设计及描述。Lennon 等 2021 因未公开代码无法对比。

For each method, we compute the proportion of stable and valid structures among the generated designs. Additionally, for each valid structure, we compute its mean and minimum brick stability scores. To evaluate prompt alignment, we compute the CLIP score [59] between a rendered image of each valid structure and the text “A LEGO model of {prompt}”. As shown in Table 5, our method outperforms all baselines in these metrics. Figure 5 shows that our method generates LEGO structures of higher quality than the baselines.

对每种方法统计生成设计中稳定与合法结构比例；对每个合法结构计算平均与最小砖稳定性分数。提示对齐方面，计算合法结构渲染图与文本 “A LEGO model of {prompt}” 的 CLIP 分数 [59]。如表 5，我们的方法在这些指标上全面领先。图 5 也显示生成质量高于基线。

Ablation Study.

We demonstrate the importance of rejection sampling and physics-aware rollback. As seen in Figure 6, rejection sampling eliminates invalid bricks, such as those with collisions, while rollback helps to ensure the final model is physically stable. The quantitative results in Table 5 show that our full method generates a higher proportion of valid and stable LEGO designs, while closely following the text prompts.

消融。

我们验证拒绝采样与物理感知回滚的作用。如图 6，拒绝采样消除碰撞等非法砖，回滚帮助最终模型物理稳定。表 5 定量显示完整方法生成合法且稳定 LEGO 的比例更高，同时紧贴文本提示。

5.3 Extensions and Applications

5.3 扩展与应用

Robotic Assembly of Generated LEGO.

We demonstrate automated assembly using a dual-robot-arm system in Figure 8. The robots use the manipulation policy [38] and the asynchronous multi-agent planner [23] to manipulate LEGO bricks and construct the structure. More details are included in Appendix B. Since the generated structures are physically stable, efficient and automated assembly can be performed.

生成 LEGO 的机器人装配。

图 8 展示双臂机器人自动装配。机器人采用操作策略 [38] 与异步多智能体规划器 [23] 操作 LEGO 砖并搭建结构。细节见附录 B。因生成结构物理稳定，可进行高效自动装配。

Manual Assembly.

Our generated LEGO structures are physically valid and can be assembled by hand. See Appendix C for details.

人工装配。

生成结构物理合法，可手工搭建。详见附录 C。

Textured and Colored LEGO Model Generation.

Figure 7 shows both UV texturing and uniform coloring results of LEGO models, demonstrating our method’s ability to generate diverse styles while preserving the underlying geometry.

带纹理与上色的 LEGO 模型生成。

图 7 展示 UV 贴图与统一上色结果，说明方法能在保持底层几何的同时生成多样风格。

6 Discussion and Limitations

6 讨论与局限

In this work, we have introduced LegoGPT, an autoregressive model for generating LEGO designs from text prompts. Our method learns to predict the next brick sequentially while ensuring physical stability and buildability. We have shown that our method outperforms LLM backbone models and several recent text-to-3D generation methods.

本文提出 LegoGPT：从文本提示自回归生成 LEGO 设计的模型。方法学习顺序预测下一块砖，并保证物理稳定与可搭建性。结果表明优于 LLM 基座与若干近期文生三维方法。

Limitations.

Though our method outperforms existing methods, it still has several limitations. First, due to limited computational resources, we have not explored the largest 3D dataset. As a result, our method is restricted to generating designs within a $20\times 20\times 20$ grid across 21 categories. Future work includes scaling up model training on larger, more diverse datasets, such as Objaverse-XL [8] at higher grid resolutions. Training on large-scale datasets can also improve generalization to out-of-distribution text prompts.

局限。

尽管优于现有方法，仍有局限。首先，算力有限，未探索最大规模三维数据，因而目前限于 $20\times 20\times 20$ 网格、21 个类别。未来将在 Objaverse-XL [8] 等更大、更多样数据上以更高网格分辨率扩展训练，并改善对分布外文本提示的泛化。

Second, our method currently supports a fixed set of commonly used LEGO bricks. In future work, we plan to expand the brick library to include a broader range of dimensions and brick types, such as slopes and tiles, allowing for more diverse and intricate LEGO designs.

其次，当前仅支持一组常用 LEGO 砖。未来计划扩展砖库，纳入更多尺寸与类型（如斜坡、瓷砖），以支持更丰富精细的设计。

Acknowledgments.

We thank Minchen Li, Ken Goldberg, Nupur Kumari, Ruihan Gao, and Yihao Shi for their discussions and help. We also thank Jiaoyang Li, Philip Huang, and Shobhit Aggarwal for developing the bimanual robotic system. This work is partly supported by the Packard Foundation, Cisco Research Grant, and Amazon Faculty Award. This work is also in part supported by the Manufacturing Futures Institute, Carnegie Mellon University, through a grant from the Richard King Mellon Foundation. KD is supported by the Microsoft Research PhD Fellowship.

致谢。

感谢 Minchen Li、Ken Goldberg、Nupur Kumari、Ruihan Gao 与 Yihao Shi 的讨论与帮助；感谢 Jiaoyang Li、Philip Huang 与 Shobhit Aggarwal 开发双臂机器人系统。本工作部分得到 Packard Foundation、Cisco Research Grant 与 Amazon Faculty Award 支持，并部分得到卡内基梅隆大学 Manufacturing Futures Institute（Richard King Mellon Foundation 资助）支持。KD 获 Microsoft Research PhD Fellowship 资助。

References

参考文献

Achiam et al. [2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Chang et al. [2015] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.

Chen et al. [2023] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3D: Disentangling geometry and appearance for high-quality text-to-3D content creation. In IEEE International Conference on Computer Vision (ICCV) , 2023.

Chen et al. [2024a] Sijin Chen, Xin Chen, Anqi Pang, Xianfang Zeng, Wei Cheng, Yijun Fu, Fukun Yin, Yanru Wang, Zhibin Wang, Chi Zhang, Jingyi Yu, Gang Yu, Bin Fu, and Tao Chen. MeshXL: Neural coordinate field for generative 3D foundation models. arXiv preprint arXiv:2405.20853 , 2024a.

Chen et al. [2024b] Yiwen Chen, Tong He, Di Huang, Weicai Ye, Sijin Chen, Jiaxiang Tang, Xin Chen, Zhongang Cai, Lei Yang, Gang Yu, Guosheng Lin, and Chi Zhang. MeshAnything: Artist-created mesh generation with autoregressive transformers. arXiv preprint arXiv:2406.10163 , 2024b.

Chen et al. [2024c] Yiwen Chen, Yikai Wang, Yihao Luo, Zhengyi Wang, Zilong Chen, Jun Zhu, Chi Zhang, and Guosheng Lin. MeshAnything V2: Artist-created mesh generation with adjacent mesh tokenization. arXiv preprint arXiv:2408.02555 , 2024c.

Chen et al. [2024d] Yunuo Chen, Tianyi Xie, Zeshun Zong, Xuan Li, Feng Gao, Yin Yang, Ying Nian Wu, and Chenfanfu Jiang. Atlas3D: Physically constrained self-supporting text-to-3D for simulation and fabrication. arXiv preprint arXiv:2405.18515 , 2024d.

Deitke et al. [2023] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi, Carl Vondrick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, and Ali Farhadi. Objaverse-XL: A universe of 10M+ 3D objects. In Advances in Neural Information Processing Systems (NeurIPS) , pages 35799–35813. Curran Associates, Inc., 2023.

Deng et al. [2024] Kangle Deng, Timothy Omernick, Alexander Weiss, Deva Ramanan, Jun-Yan Zhu, Tinghui Zhou, and Maneesh Agrawala. FlashTex: Fast relightable mesh texturing with LightControlNet. In European Conference on Computer Vision (ECCV) , 2024.

Dong et al. [2024] Yuan Dong, Qi Zuo, Xiaodong Gu, Weihao Yuan, Zhengyi Zhao, Zilong Dong, Liefeng Bo, and Qixing Huang. GPLD3D: Latent diffusion of 3D shape generative models by enforcing geometric and physical priors. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Dubey et al. [2024] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024.

Ge et al. [2024a] Jiahao Ge, Mingjun Zhou, Wenrui Bao, Hao Xu, and Chi-Wing Fu. Creating LEGO figurines from single images. ACM Transactions on Graphics (TOG) , 43(4):153:1–153:16, 2024a.

Ge et al. [2024b] Jiahao Ge, Mingjun Zhou, and Chi-Wing Fu. Learn to create simple LEGO micro buildings. ACM Transactions on Graphics (TOG) , 43(6):249:1–249:13, 2024b.

Goldberg et al. [2025] Andrew Goldberg, Kavish Kondap, Tianshuang Qiu, Zehan Ma, Letian Fu, Justin Kerr, Huang Huang, Kaiyuan Chen, Kuan Fang, and Ken Goldberg. Blox-Net: Generative design-for-robot-assembly using VLM supervision, physics, simulation, and a robot with reset. In International Conference on Robotics and Automation (ICRA) . IEEE, 2025.

Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In Advances in Neural Information Processing Systems (NeurIPS) , 2014.

Guo et al. [2024] Minghao Guo, Bohan Wang, Pingchuan Ma, Tianyuan Zhang, Crystal Elaine Owens, Chuang Gan, Joshua B. Tenenbaum, Kaiming He, and Wojciech Matusik. Physically compatible 3D object modeling from a single image. arXiv preprint arXiv:2405.20510 , 2024.

Gurobi Optimization, LLC [2023] Gurobi Optimization, LLC. Gurobi Optimizer reference manual, 2023.

Hao et al. [2024] Zekun Hao, David W Romero, Tsung-Yi Lin, and Ming-Yu Liu. Meshtron: High-fidelity, artist-like 3D mesh generation at scale. arXiv preprint arXiv:2412.09548 , 2024.

Haque et al. [2023] Ayaan Haque, Matthew Tancik, Alexei Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct-NeRF2NeRF: Editing 3D scenes with instructions. In IEEE International Conference on Computer Vision (ICCV) , 2023.

Hong et al. [2024a] Fangzhou Hong, Jiaxiang Tang, Ziang Cao, Min Shi, Tong Wu, Zhaoxi Chen, Tengfei Wang, Liang Pan, Dahua Lin, and Ziwei Liu. 3DTopia: Large text-to-3D generation model with hybrid diffusion priors. arXiv preprint arXiv:2403.02234 , 2024a.

Hong et al. [2024b] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. LRM: Large reconstruction model for single image to 3D. In International Conference on Learning Representations (ICLR) , 2024b.

Hu et al. [2022] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR) , 2022.

Huang et al. [2025] Philip Huang, Ruixuan Liu, Shobhit Aggarwal, Changliu Liu, and Jiaoyang Li. Apex-mr: Multi-robot asynchronous planning and execution for cooperative assembly. In Robotics: Science and Systems , 2025.

Huang et al. [2023] Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, and Song-Chun Zhu. Diffusion-based generation, optimization, and planning in 3D scenes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Katzir et al. [2024] Oren Katzir, Or Patashnik, Daniel Cohen-Or, and Dani Lischinski. Noise-free score distillation. In International Conference on Learning Representations (ICLR) , 2024.

Kerbl et al. [2023] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D Gaussian splatting for real-time radiance field rendering. In ACM Transactions on Graphics (TOG) , 2023.

Kim et al. [2020] Jungtaek Kim, Hyunsoo Chung, Jinhwi Lee, Minsu Cho, and Jaesik Park. Combinatorial 3D shape generation via sequential assembly. arXiv preprint arXiv:2004.07414 , 2020.

Kim [2014] Jae Woo Kim. Survey on automated LEGO assembly construction. 2014.

LDraw.org [2025] LDraw.org. Ldraw.org homepage, 2025.

Lennon et al. [2021] Kyle Lennon, Katharina Fransen, Alexander O’Brien, Yumeng Cao, Matthew Beveridge, Yamin Arefeen, Nikhil Singh, and Iddo Drori. Image2Lego: Customized LEGO set generation from images. arXiv preprint arXiv:2108.08477 , 2021.

Li et al. [2024a] Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3D: Fast text-to-3D with sparse-view generation and large reconstruction model. In International Conference on Learning Representations (ICLR) , 2024a.

Li et al. [2023a] Muheng Li, Yueqi Duan, Jie Zhou, and Jiwen Lu. Diffusion-SDF: Text-to-shape via voxelized diffusion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023a.

Li et al. [2023b] Weiyu Li, Rui Chen, Xuelin Chen, and Ping Tan. SweetDreamer: Aligning geometric priors in 2D diffusion for consistent text-to-3D. arXiv preprint arXiv:2310.02596 , 2023b.

Li et al. [2024b] Weiyu Li, Jiarui Liu, Rui Chen, Yixun Liang, Xuelin Chen, Ping Tan, and Xiaoxiao Long. CraftsMan: High-fidelity mesh generation with 3D native generation and interactive geometry refiner. arXiv preprint arXiv:2405.14979 , 2024b.

Lin et al. [2023] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3D: High-resolution text-to-3D content creation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Liu et al. [2025a] Minghua Liu, Chong Zeng, Xinyue Wei, Ruoxi Shi, Linghao Chen, Chao Xu, Mengqi Zhang, Zhaoning Wang, Xiaoshuai Zhang, Isabella Liu, et al. MeshFormer: High-quality mesh generation with 3D-guided reconstruction model. Advances in Neural Information Processing Systems (NeurIPS) , 37:59314–59341, 2025a.

Liu et al. [2024a] Ruixuan Liu, Kangle Deng, Ziwei Wang, and Changliu Liu. StableLego: Stability analysis of block stacking assembly. IEEE Robotics and Automation Letters , 9(11):9383–9390, 2024a.

Liu et al. [2024b] Ruixuan Liu, Yifan Sun, and Changliu Liu. A lightweight and transferable design for robust LEGO manipulation. International Symposium on Flexible Automation , page V001T07A004, 2024b.

Liu et al. [2025b] Ruixuan Liu, Alan Chen, Weiye Zhao, and Changliu Liu. Physics-aware combinatorial assembly sequence planning using data-free action masking. IEEE Robotics and Automation Letters , 10(5):4882–4889, 2025b.

Liu et al. [2021] Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, and Bryan Russell. Editing conditional radiance fields. In IEEE International Conference on Computer Vision (ICCV) , pages 5773–5783, 2021.

Liu et al. [2023] Xueyi Liu, Bin Wang, He Wang, and Li Yi. Few-shot physically-aware articulated mesh generation via hierarchical deformation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Long et al. [2024] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al. Wonder3D: Single image to 3D using cross-domain diffusion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR) , 2019.

Lukoianov et al. [2024] Artem Lukoianov, Haitz Sáez de Ocáriz Borde, Kristjan Greenewald, Vitor Campagnolo Guizilini, Timur Bagautdinov, Vincent Sitzmann, and Justin Solomon. Score distillation via reparametrized DDIM. In Advances in Neural Information Processing Systems (NeurIPS) , 2024.

Luo et al. [2015] Sheng-Jie Luo, Yonghao Yue, Chun-Kai Huang, Yu-Huan Chung, Sei Imai, Tomoyuki Nishita, and Bing-Yu Chen. Legolization: optimizing LEGO designs. ACM Transactions on Graphics (TOG) , 34(6), 2015.

Makatura et al. [2023] Liane Makatura, Michael Foshey, Bohan Wang, Felix HähnLein, Pingchuan Ma, Bolei Deng, Megan Tjandrasuwita, Andrew Spielberg, Crystal Elaine Owens, Peter Yichen Chen, et al. How can large language models help humans in design and manufacturing? arXiv preprint arXiv:2307.14377 , 2023.

McAllister et al. [2024] David McAllister, Songwei Ge, Jia-Bin Huang, David W. Jacobs, Alexei A. Efros, Aleksander Holynski, and Angjoo Kanazawa. Rethinking score distillation as a bridge between image distributions. In Advances in Neural Information Processing Systems (NeurIPS) , 2024.

Metzer et al. [2023] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-NeRF for shape-guided generation of 3d shapes and textures. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Mezghanni et al. [2021] Mariem Mezghanni, Malika Boulkenafed, Andre Lieutier, and Maks Ovsjanikov. Physically-aware generative network for 3D shape modeling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Mezghanni et al. [2022] Mariem Mezghanni, Théo Bodrito, Malika Boulkenafed, and Maks Ovsjanikov. Physical simulation layer for accurate 3D modeling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Michel et al. [2022] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2Mesh: Text-driven neural stylization for meshes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Mildenhall et al. [2020] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV) , 2020.

Misra et al. [2025] Vihaan Misra, Peter Schaldenbrand, and Jean Oh. ShapeShift: Towards text-to-shape arrangement synthesis with content-aware geometric constraints. arXiv preprint arXiv:2503.14720 , 2025.

Nam et al. [2022] Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto Tono, Linqi Zhou, and Paul Guerrero. 3D-LDM: Neural implicit 3D shape generation with latent diffusion models. arXiv preprint arXiv:2212.00842 , 2022.

Nash et al. [2020] Charlie Nash, Yaroslav Ganin, S. M. Ali Eslami, and Peter Battaglia. PolyGen: An autoregressive generative model of 3D meshes. In International Conference on Machine Learning (ICML) , pages 7220–7229. PMLR, 2020.

Ni et al. [2024] Junfeng Ni, Yixin Chen, Bohan Jing, Nan Jiang, Bin Wang, Bo Dai, Puhao Li, Yixin Zhu, Song-Chun Zhu, and Siyuan Huang. PhyRecon: Physically plausible neural scene reconstruction. In Advances in Neural Information Processing Systems (NeurIPS) , 2024.

Ono et al. [2013] Sumiaki Ono, Alexis Andre, Youngha Chang, and Masayuki Nakajima. LEGO builder: Automatic generation of LEGO assembly manual from 3D polygon model. ITE Transactions on Media Technology and Applications , 1:354–360, 2013.

Poole et al. [2023] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D diffusion. In International Conference on Learning Representations (ICLR) , 2023.

Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML) , 2021.

Ren et al. [2024] Xuanchi Ren, Jiahui Huang, Xiaohui Zeng, Ken Museth, Sanja Fidler, and Francis Williams. XCube: Large-scale 3D generative modeling using sparse voxel hierarchies. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4209–4219, 2024.

Richardson et al. [2023] Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. Texture: Text-guided texturing of 3D shapes. In ACM SIGGRAPH , 2023.

Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Shue et al. [2023] J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3D neural field generation using triplane diffusion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Siddiqui et al. [2024] Yawar Siddiqui, Antonio Alliegro, Alexey Artemov, Tatiana Tommasi, Daniele Sirigatti, Vladislav Rosov, Angela Dai, and Matthias Nießner. MeshGPT: Generating triangle meshes with decoder-only transformers. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 19615–19625, 2024.

Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathain, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning (ICML) , 2015.

Sun et al. [2023] Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu. DreamCraft3D: Hierarchical 3D generation with bootstrapped diffusion prior. arXiv preprint arXiv:2310.16818 , 2023.

Tang et al. [2024] Jiaxiang Tang, Zhaoshuo Li, Zekun Hao, Xian Liu, Gang Zeng, Ming-Yu Liu, and Qinsheng Zhang. EdgeRunner: Auto-regressive auto-encoder for artistic mesh generation. arXiv preprint arXiv:2409.18114 , 2024.

Tang et al. [2025] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. LGM: Large multi-view gaussian model for high-resolution 3D content creation. In European Conference on Computer Vision (ECCV) , pages 1–18, Cham, 2025. Springer Nature Switzerland.

Testuz et al. [2013] Romain Pierre Testuz, Yuliy Schwartzburg, and Mark Pauly. Automatic generation of constructable brick sculptures. Technical report, École Polytechnique Fédérale de Lausanne, 2013.

Thompson et al. [2020] Rylee Thompson, Elahe Ghalebi, Terrance DeVries, and Graham W. Taylor. Building LEGO using deep generative models of graphs. arXiv preprint arXiv:2012.11543 , 2020.

Tian et al. [2022] Yunsheng Tian, Jie Xu, Yichen Li, Jieliang Luo, Shinjiro Sueda, Hui Li, Karl D.D. Willis, and Wojciech Matusik. Assemble them all: Physics-based planning for generalizable assembly by disassembly. ACM Transactions on Graphics (TOG) , 41(6), 2022.

TobyLobster [2025] TobyLobster. ImportLDraw - blender add-on for importing LDraw models. https://github.com/TobyLobster/ImportLDraw , 2025. Accessed: 2025-03-07.

Tochilkin et al. [2024] Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao. TripoSR: Fast 3D object reconstruction from a single image. arXiv preprint arXiv:2403.02151 , 2024.

Tran et al. [2024] Uy Dieu Tran, Minh Luu, Phong Ha Nguyen, Khoi Nguyen, and Binh-Son Hua. Diverse text-to-3D synthesis with augmented text embedding. In European Conference on Computer Vision (ECCV) , 2024.

Vilesov et al. [2023] Alexander Vilesov, Pradyumna Chari, and Achuta Kadambi. CG3D: Compositional generation for text-to-3D via Gaussian splatting. arXiv preprint arXiv:2311.17907 , 2023.

Wang et al. [2023a] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, and Greg Shakhnarovich. Score Jacobian chaining: Lifting pretrained 2D diffusion models for 3D generation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023a.

Wang et al. [2023b] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. ProlificDreamer: High-fidelity and diverse text-to-3D generation with variational score distillation. In Advances in Neural Information Processing Systems (NeurIPS) , 2023b.

Wang et al. [2024] Zhengyi Wang, Jonathan Lorraine, Yikai Wang, Hang Su, Jun Zhu, Sanja Fidler, and Xiaohui Zeng. LLaMA-Mesh: Unifying 3D mesh generation with language models. arXiv preprint arXiv:2411.09595 , 2024.

Weng et al. [2024] Haohan Weng, Zibo Zhao, Biwen Lei, Xianghui Yang, Jian Liu, Zeqiang Lai, Zhuo Chen, Yuhong Liu, Jie Jiang, Chunchao Guo, et al. Scaling mesh generation via compressive tokenization. arXiv preprint arXiv:2411.07025 , 2024.

Xu et al. [2024a] Qingshan Xu, Jiao Liu, Melvin Wong, Caishun Chen, and Yew-Soon Ong. Precise-physics driven text-to-3D generation. arXiv preprint arXiv:2403.12438 , 2024a.

Xu et al. [2024b] Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, and Kai Zhang. DMV3D: Denoising multi-view diffusion using 3D large reconstruction model. In International Conference on Learning Representations (ICLR) , 2024b.

Yang et al. [2024] Yandan Yang, Baoxiong Jia, Peiyuan Zhi, and Siyuan Huang. PhyScene: Physically interactable 3D scene synthesis for embodied AI. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Ye et al. [2024] Junliang Ye, Fangfu Liu, Qixiu Li, Zhengyi Wang, Yikai Wang, Xinzhou Wang, Yueqi Duan, and Jun Zhu. DreamReward: Text-to-3D generation with human preference. In European Conference on Computer Vision (ECCV) , 2024.

Yuan et al. [2023] Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. PhysDiff: Physics-guided human motion diffusion model. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Zhang et al. [2024] Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. CLAY: A controllable large-scale generative model for creating high-quality 3D assets. In ACM Transactions on Graphics (TOG) , 2024.

Zhao et al. [2025] Zibo Zhao, Zeqiang Lai, Qingxiang Lin, Yunfei Zhao, Haolin Liu, Shuhui Yang, Yifei Feng, Mingxin Yang, Sheng Zhang, Xianghui Yang, et al. Hunyuan3D 2.0: Scaling diffusion models for high resolution textured 3D assets generation. arXiv preprint arXiv:2501.12202 , 2025.

Zhou et al. [2022] Guyue Zhou, Liyi Luo, Hao Xu, Xinliang Zhang, Haole Guo, and Hao Zhao. Brick yourself within 3 minutes. In International Conference on Robotics and Automation (ICRA) , pages 6261–6267, 2022.

Zhou et al. [2019] J. Zhou, X. Chen, and Y. Xu. Automatic generation of vivid LEGO architectural sculptures. Computer Graphics Forum , 2019.

Zhou et al. [2021] Linqi Zhou, Yilun Du, and Jiajun Wu. 3D shape generation and completion through point-voxel diffusion. In IEEE International Conference on Computer Vision (ICCV) , 2021.

Zhou et al. [2023] Mingjun Zhou, Jiahao Ge, Hao Xu, and Chi-Wing Fu. Computational design of LEGO® sketch art. ACM Transactions on Graphics (TOG) , 42(6), 2023.

Appendix

附录

Appendix A LegoGPT Implementation Details

附录 A LegoGPT 实现细节

Captioning Details.

The complete prompt template used for GPT-4o caption generation is as follows:

描述细节。

用于 GPT-4o 描述生成的完整提示模板如下：

“This is a rendering of a 3D object built with LEGO bricks with 24 different views. The object belongs to the category of {CATEGORY_NAME}. You will generate five different captions for this {CATEGORY_NAME} that:

“这是用 LEGO 积木搭建的三维物体在 24 个不同视角下的渲染。物体属于类别 {CATEGORY_NAME}。请为该 {CATEGORY_NAME} 生成五条不同描述，要求：

1. Describes the core object/subject and its key geometric features

1. 描述核心物体/主体及其关键几何特征

2. Focuses on structure, geometry, and layout information

2. 聚焦结构、几何与布局信息

3. Uses confident, concrete, and declarative language

3. 使用自信、具体、陈述性语言

4. Omits color and texture information

4. 省略颜色与纹理信息

5. Excludes medium-related terms (model, render, design)

5. 排除媒介相关用语（模型、渲染、设计）

6. Do not describe or reference each view individually.

6. 不要逐个描述或引用各个视角。

7. Focus on form over function. Describe the physical appearance of components rather than their purpose.

7. 重形式轻功能。描述部件外观而非用途。

8. Describe components in detail, including size, shape, and position relative to other components.

8. 详细描述部件，包括相对其他部件的尺寸、形状与位置。

9. The five captions should be from coarse to fine, with the first one being the most coarse-grained (e.g., a general description of the object, within 10 words) and the last one being the most fine-grained (e.g., a detailed description of the object, within 50 words). The five captions should be different from each other. Do not include any ordering numbers (e.g., 1, a, etc.).

9. 五条描述由粗到细：第一条最粗（如物体概括，10 词内），最后一条最细（如详细描述，50 词内）；彼此不同；不要带序号（如 1、a 等）。

10. Describe the object using the category name "{CATEGORY_NAME}" or synonyms of the category name "{CATEGORY_NAME}".”

10. 用类别名 “{CATEGORY_NAME}” 或其同义词描述物体。”

StableText2Lego Details.

We generate StableText2Lego from the objects in ShapeNetCore [2]. While ShapeNetCore provides both voxel and mesh representations, we find that working directly with mesh data preserves better geometric details. We voxelize the 3D mesh into a $20\times 20\times 20$ grid representation and generate the LEGO structure using a split-and-remerge legolization algorithm. Split-and-remerge legolization has been studied in prior work [45]. But instead of initializing the voxel with $1\times 1$ unit bricks and randomly merging them into larger bricks [45], we initialize with heuristics by directly placing bricks to fill all the voxels, i.e., prioritizing placing 1) larger bricks and 2) bricks that connect multiple other bricks. After initializing the layout, we apply the new stability analysis [37] to identify weak parts in the structure and rearrange the local brick layout if necessary. In addition, to further improve data quality and diversity, we introduce randomness in the legolization process. Specifically, when initializing the LEGO structure, we randomly permute the order of brick placement within the same heuristic priority and generate two different structures for each object. With the mesh-to-LEGO and structure augmentation techniques, we generate $62{,}000+$ LEGO structures covering the $21$ categories with $31{,}218$ unique 3D objects from ShapeNetCore. Among the selected 3D objects, $\sim$92% (i.e., $28{,}822$) of them have at least one stable LEGO design, which offer $47{,}000+$ stable LEGO layouts. Our StableText2Lego dataset significantly expands upon the previous StableLego dataset [37] in several key aspects: it contains $>3\times$ more stable unique 3D objects with $>5\times$ more stable LEGO structures compared to $8{,}000+$ in StableLego, spans a diverse set of 21 object categories, and provides detailed geometric descriptions for each shape.

StableText2Lego 细节。

我们从 ShapeNetCore [2] 物体生成 StableText2Lego。虽同时提供体素与网格，直接用网格更能保留几何细节。将网格体素化为 $20\times 20\times 20$，再用 split-and-remerge legolization 生成结构。该算法已有研究 [45]；我们不先用 $1\times 1$ 砖随机合并 [45]，而是按启发式直接铺砖填满体素：优先 1) 更大砖；2) 连接多块其他砖的砖。初始化后用新稳定性分析 [37] 找薄弱处并必要时重排局部布局。为进一步提质增多样，legolization 引入随机性：同启发式优先级内随机置换放砖顺序，每物体生成两种结构。经网格转 LEGO 与结构增广，得到覆盖 $21$ 类、$31{,}218$ 个独特三维物体的 $62{,}000+$ LEGO 结构；其中约 $\sim$92%（即 $28{,}822$）至少有一种稳定设计，合计 $47{,}000+$ 稳定布局。相对此前 StableLego [37]，StableText2Lego 在稳定独特三维物体上多 $>3\times$、稳定结构多 $>5\times$（StableLego 为 $8{,}000+$），覆盖 21 类，并为每个形状提供细致几何描述。

Figure 9: Given a prompt and several example LEGO designs in text format, a pre-trained LLaMA model can generate LEGO designs with some structure.

图 9：给定提示与若干文本格式的示例 LEGO 设计，预训练 LLaMA 可生成具有一定结构的 LEGO 设计。

Figure 10: Novelty analysis. For each structure generated by LegoGPT, we find the closest structure in the training dataset as measured by Chamfer distance in voxel space. The generated structures are distinct from their nearest neighbors, indicating low memorization.

图 10：新颖性分析。对 LegoGPT 生成的每个结构，在训练数据中按体素空间 Chamfer 距离找最近邻。生成结构与最近邻明显不同，表明记忆程度低。

Training details.

The full prompt that we use to construct our instruction fine-tuning dataset is as follows:

训练细节。

构建指令微调数据的完整提示如下：

“*[SYSTEM]*You are a helpful assistant.

*[USER]*Create a LEGO model of the input. Format your response as a list of bricks: , where the brick position is (x,y,z).

Allowed brick dimensions are 2x4, 4x2, 2x6, 6x2, 1x2, 2x1, 1x4, 4x1, 1x6, 6x1, 1x8, 8x1, 1x1, 2x2. All bricks are 1 unit tall.

Input:

[INSERT CAPTION]”

We fine-tune using low-rank adaptation (LoRA) [22] with a rank of 32, alpha of 16, and dropout rate of 0.05. To prevent catastrophic forgetting and training instability, we apply LoRA to only the query and value matrices for a total of 3.4M tunable parameters. We train for three epochs on eight NVIDIA RTX A6000 GPUs. We use the AdamW optimizer [43] with a learning rate of $0.002$, using a cosine scheduler with 100 warmup steps and a global batch size of 64. The total training time is 12 hours.

我们用低秩适应（LoRA） [22] 微调：rank 32、alpha 16、dropout 0.05。为防止灾难性遗忘与训练不稳，LoRA 仅作用于 query 与 value 矩阵，共 340 万可调参数。在八张 NVIDIA RTX A6000 上训练三轮。优化器为 AdamW [43]，学习率 $0.002$，余弦调度、100 warmup、全局 batch 64。总训练约 12 小时。

Inference Details.

The sampling temperature is set to 0.6 for all experiments. During brick-by-brick rejection sampling, to mitigate repeated generation of rejected bricks, we increase the temperature by 0.01 each time the model generates a brick that has already been rejected.

推理细节。

所有实验采样温度 0.6。逐砖拒绝采样时，为减轻反复生成已被拒砖，每拒一次温度加 0.01。

To force each output brick to be in the format “{h}×{w} ({x},{y},{z})”, we sample only from the set of valid tokens during each step. For example, the first token must be a digit, the second an ‘x’, and so on. This has little to no effect on the output of LegoGPT, but helps force the baseline pre-trained models to output LEGO bricks in the specified format.

为强制输出格式 “{h}×{w} ({x},{y},{z})”，每步仅从合法词元集合采样。例如第一词元须为数字，第二为 ‘x’，以此类推。这对 LegoGPT 输出几乎无影响，但有助于迫使预训练基线按指定格式输出 LEGO 砖。

Only 1.2% of the generated designs exceeded the maximum number of rollbacks and failed to produce a stable final structure.

仅 1.2% 的生成设计超过最大回滚次数，未能得到稳定最终结构。

Novelty Analysis.

For each generated structure, we find its closest structure in the training dataset, measured by computing the Chamfer distance in voxel space. As seen in Figure 10, the generated structures are distinct from their nearest neighbors in the dataset, confirming that our model can create novel designs rather than simply memorizing the training data.

新颖性分析。

对每个生成结构，按体素空间 Chamfer 距离在训练数据中找最近邻。如图 10，生成结构与最近邻不同，说明模型能创造新设计而非简单记忆训练数据。

In-context Learning.

We use LLaMA-3.2-1B-Instruct [11] as our base model, chosen for its computational efficiency. The in-context learning pipeline is shown in Figure 9, where the base model can generate LEGO-like designs through in-context learning, while suffering from collisions and disconnectivity. We did not use rejection sampling or rollback when evaluating zero-shot or few-shot generation, as doing so resulted in an excessive number of rejections and a sharp increase in generation time.

上下文学习。

基座选用 LLaMA-3.2-1B-Instruct [11] 以兼顾算力。上下文学习流程见图 9：基座可生成「像 LEGO」的设计，但存在碰撞与不连通。评估零样本/少样本时未用拒绝采样或回滚，否则拒绝过多、生成时间急剧上升。

Appendix B Robotic Assembly

附录 B 机器人装配

We demonstrate automated assembly using a dual-robot-arm system as shown in Figure 8. The system consists of two Yaskawa GP4 robots, each equipped with an ATI force-torque sensor. A calibrated LEGO plate is placed between them, and the robots use the bricks initially placed on the plate to construct the LEGO structure. Given a $B=[b_{1},b_{2},\dots,b_{N}]$, we employ the action mask in [39] with assembly-by-disassembly search [71] to generate a physically executable assembly sequence for the robots, i.e., reordering the brick sequence so that 1) each intermediate structure is physically stable by itself, 2) each intermediate structure is stable under the robot operation, and 3) each assembly step is executable within the system’s dexterity. With the executable assembly sequence, an asynchronous planner [23] distributes the assembly tasks to the robots, plans the robots’ movements, and coordinates the bimanual system to construct the LEGO structure. The robots use the end-of-arm tool and the manipulation policy presented in [38, 23] with closed-loop force control to robustly manipulate LEGO bricks and construct the structure.

如图 8，我们用双臂机器人系统演示自动装配。系统含两台 Yaskawa GP4，各配 ATI 力/力矩传感器。标定后的 LEGO 板置于中间，机器人用板上初始砖搭建结构。给定 $B=[b_{1},b_{2},\dots,b_{N}]$，采用 [39] 的动作掩码与按拆卸搜索装配 [71]，生成机器人可执行的装配序列，即重排砖序使：1) 每个中间结构自身物理稳定；2) 在机器人操作下中间结构仍稳定；3) 每步装配在系统灵巧范围内可执行。有了可执行序列后，异步规划器 [23] 分配任务、规划运动并协调双臂搭建。机器人用末端工具与 [38, 23] 中的操作策略，配合闭环力控，稳健地操作 LEGO 砖并完成搭建。

Appendix C Manual Assembly

附录 C 人工装配

As shown in Figure 11, human users can assemble our generated structures, demonstrating their physical stability and buildability. Notably, since our method outputs a sequence of intermediate steps, it naturally serves as an intuitive assembly guide.

如图 11，用户可手工装配生成结构，说明其物理稳定与可搭建。由于方法输出中间步骤序列，本身也可作为直观装配指南。

Figure 11: Manual Assembly. Demonstrations of manual assembly of generated LEGO structures.

图 11：人工装配。手工装配生成 LEGO 结构的演示。