GimmBO: Interactive Generative Image Model Merging via Bayesian Optimization | AGIDaily
AGI Daily
资讯日报词条论文人物新创
返回论文列表

暂无章节

返回论文列表
目录

GimmBO: Interactive Generative Image Model Merging via Bayesian Optimization

收录于 SIGGRAPH 2026 Best Paper;ACM Transactions on Graphics 45(4):1–12,DOI 10.1145/3811293被引 1 次
打开论文来源

GimmBO: Interactive Generative Image Model Merging via Bayesian Optimization

CHENXI LIU, University of Toronto, Canada
SELENA LING, University of Toronto, Canada
ALEC JACOBSON, University of Toronto, Canada and Vector Institute, Canada

Authors' Contact Information: Chenxi Liu, University of Toronto, Toronto, Canada, liuchenxi0921@gmail.com; Selena Ling, University of Toronto, Toronto, Canada, selena.ling@mail.utoronto.ca; Alec Jacobson, University of Toronto, Toronto, Canada and Vector Institute, Toronto, Canada, jacobson@cs.toronto.edu.

Fig. 1.

Fig. 1. Given a prompt, default image generation may appear reasonable yet fail to match a user's creative intent (left). GimmBO enables users to explore weighted combinations of customization adapters (30 in this example) through preference feedback, producing an image that better reflects the desired style (right, corresponding adapter coefficients below). Readers are encouraged to zoom in on images for finer details throughout the paper.

图 1. 给定提示词后,默认生成的图像看起来也许还过得去,却对不上用户真正想要的效果(左)。GimmBO 让用户通过偏好反馈,探索定制适配器的加权组合(本例为 30 个),得到更符合目标风格的图像(右,下方为对应的适配器系数)。阅读本文其余插图时,建议放大查看细节。

Abstract

摘要

Fine-tuning-based adaptation is widely used to customize diffusion-based image generation, leading to large collections of community-created adapters that capture diverse subjects and styles. Adapters derived from the same base model can be merged linearly, enabling the synthesis of new visual results within a vast and continuous design space. To explore this space, current workflows rely on manual slider-based tuning, an approach that scales poorly and makes merging coefficient selection difficult, even when the candidate set is limited to 20–30 adapters. We propose GimmBO to support interactive exploration of adapter merging for image generation through Preferential Bayesian Optimization (PBO). Motivated by observations from real-world usage, including sparsity and constrained coefficient ranges, we introduce a two-stage BO backend that improves sampling efficiency and convergence in high-dimensional spaces. We evaluate our approach with simulated users and a user study, demonstrating improved convergence, high success rates, and consistent gains over BO and line-search baselines, and further show the flexibility of the framework through several extensions.

基于微调的适配,已经被广泛用来定制扩散图像生成,社区里因此积累了大量刻画不同主体与风格的适配器。从同一基底模型得到的适配器可以线性合并,从而在一个广阔而连续的设计空间里合成新的视觉结果。要探索这个空间,现有工作流仍主要靠手动拉滑条;这种方法很难扩展,即便候选只剩 20–30 个适配器,合并系数也不好看准。我们提出 GimmBO,用优先贝叶斯优化(Preferential Bayesian Optimization,PBO)支持对图像生成适配器合并的交互式探索。受真实使用中的稀疏性、系数范围受限等观察启发,我们引入两阶段贝叶斯优化后端,以改善高维空间中的采样效率与收敛。我们用模拟用户和一项用户研究做了评估,显示收敛更快、成功率更高,并相对贝叶斯优化与线搜索基线保持稳定优势;还通过若干扩展展示了这一框架的灵活性。

CCS Concepts: • Computing methodologies → Search methodologies; Image manipulation.

Additional Key Words and Phrases: Generative image models; model merging; Bayesian optimization; human-in-the-loop optimization.

1 Introduction

1 引言

Fine-tuning-based customization of diffusion models is widely used to synthesize subjects or styles underrepresented or missing from the original training data [Hu et al. 2022; Ruiz et al. 2023; Zheng et al. 2024]. These fine-tuned models, or adapters, are lightweight to train, store, and share, leading to large community-curated adapter libraries. Adapters derived from a common base model can be merged to combine visual characteristics, forming a vast and continuous design space (Figs. 1, 2), and are commonly used in multi-adapter form on image generation platforms (69% of adapter-based generations on Civitai[^1]). In practice, users explore this design space by manually tuning per-adapter slider values (e.g., α\alphaαs in inset); however, this approach scales poorly (with steep degradation beyond 5-10 sliders [Dang et al. 2022]) and makes coefficient selection difficult even after users limit the candidate set to 20–30 relevant adapters. Existing interactive design optimization systems are largely confined to low-dimensional design spaces (3-12D) in real applications [Koyama and Goto 2022; Koyama et al. 2020, 2017; Mo et al. 2024; Niwa et al. 2025], making 20-30D high-dimensional in this context.

Inset: two-coefficient slider examples on apple images.

基于微调来定制扩散模型,已经被广泛用来生成原训练数据中很少出现、甚至完全没有的主体或风格 [Hu et al. 2022; Ruiz et al. 2023; Zheng et al. 2024]。这些微调后的模型,也就是适配器,训练、存储和分享都相对轻便,于是形成了由社区维护的大型适配器库。来自同一基底模型的适配器可以合并,以组合不同的视觉特征,构成一个广阔而连续的设计空间(图 1、图 2);在图像生成平台上,它们也常常以多适配器的形式使用(Civitai[^1] 上基于适配器的生成中,有 69% 属于这种情况)。实践中,用户靠手动调节每个适配器的滑条取值(例如插图里的 α\alphaα)来探索这一空间;不过这种方法很难扩展(滑条超过 5–10 个就会明显变差 [Dang et al. 2022]),即便用户已经把候选收窄到 20–30 个相关适配器,系数仍然不好选。已有的交互式设计优化系统,在真实应用里大多局限于低维设计空间(3–12 维)[Koyama and Goto 2022; Koyama et al. 2020, 2017; Mo et al. 2024; Niwa et al. 2025],因此 20–30 维在这一语境下已经算高维。

[^1]: A community platform for image generation. Statistics collected in December 2025.

[^1]: 一个面向图像生成的社区平台。统计数据收集于 2025 年 12 月。

We propose GimmBO to allow users to explore and identify suitable coefficient combinations over 20–30 adapters during interactive design. This interactive human-in-the-loop setting raises two key challenges: user preferences define a black-box objective, and each evaluation is costly due to model inference and user interaction. We address these challenges using human-in-the-loop Bayesian optimization (BO), a sample-efficient approach commonly used in design tasks (Sec. 2). However, this setting lies outside the effective range of existing BO-based interactive design methods due to its high dimensionality. Surveying real-world practice, we observe that effective adapter merges are typically sparse, involving only a small subset of adapters, and that merging coefficients must lie within a limited magnitude range to avoid degraded or incoherent generations (Sec. 3). Motivated by these observations, we introduce a two-stage BO backend that first operates in a conservative search space for efficient sampling and then exploits sparsity and early variable selection to accelerate convergence (Sec. 4).

我们提出 GimmBO,让用户在交互式设计中探索并找出适合的系数组合,覆盖 20–30 个适配器。这种有人在环的交互设定带来两个关键困难:用户偏好定义的是一个黑盒目标,而每一次评估都因模型推理和用户交互而昂贵。我们用有人在环的贝叶斯优化(BO)来应对,这是设计任务中常用的样本高效方法(第 2 节)。不过,由于维度较高,这一设定已经超出了现有基于 BO 的交互设计方法的有效范围。考察真实用法后,我们看到有效的适配器合并通常是稀疏的,只用到一小部分适配器,而且合并系数必须落在有限的幅度范围内,否则生成结果容易退化、不连贯(第 3 节)。受这些观察启发,我们引入两阶段 BO 后端:先在较保守的搜索空间里高效采样,再借助稀疏性与早期变量选择加快收敛(第 4 节)。

Fig. 2.

Fig. 2. The adapter design space is vast. GimmBO assists users to produce substantially different yet equally satisfying results.

图 2. 适配器设计空间非常广阔。GimmBO 帮助用户得到彼此差异很大、却同样令人满意的结果。

We extensively evaluate our approach through quantitative experiments with both simulated and real users. Our primary experiments operate in a 20-dimensional design space of stylization adapters. Using simulated users on test sets constructed from realistic input distributions and real adapter collections, we demonstrate the practical usefulness of GimmBO under controlled and repeatable conditions. We then conduct a user study to validate effective support for human preference-driven interaction. Scalability is further assessed through stress tests with simulated users in 30D and 40D spaces. Across all evaluations, our approach exhibits faster convergence and higher-quality results, highlighting the benefits of incorporating domain-specific structure into the BO backend. Finally, we explore several extensions, including merging beyond style to incorporate content, as well as downstream applications. We also show that GimmBO integrates naturally with community model retrieval approaches [Luo et al. 2024], enabling an end-to-end workflow that automatically identifies a candidate set of 20–30 adapters and supports interactive exploration of their combinations. Our implementation and supplementary materials are available at https://gimmbo-project.github.io.

我们用模拟用户和真实用户的定量实验,对方法做了较充分的评估。主实验在风格适配器的 20 维设计空间中进行。用根据真实输入分布和真实适配器库构造的测试集,配合模拟用户,我们在可控、可重复的条件下展示了 GimmBO 的实际效用。随后我们进行用户研究,验证它能否有效支持由人类偏好驱动的交互。可扩展性则通过模拟用户在 30 维和 40 维空间中的压力测试进一步考查。在全部评估中,我们的方法都表现出更快的收敛和更高的结果质量,说明把领域特有的结构写进 BO 后端是有益的。最后,我们还探索了若干扩展,包括超越风格、把内容也纳入合并,以及下游应用。我们也表明,GimmBO 可以自然地接到社区模型检索方法 [Luo et al. 2024] 上,形成一套端到端流程:自动找出 20–30 个候选适配器,并支持对它们的组合做交互式探索。实现与补充材料见 https://gimmbo-project.github.io。

2 Related Work

2 相关工作

Customized image generation. Diffusion model customization adapts pretrained models to specific subjects and styles, enabling personalized image generation beyond the original training distribution. Early approaches bind new concepts via learned textual tokens, as in textual inversion [Gal et al. 2022], with extensions using multi-token or time-dependent embeddings [Voynov et al. 2023; Alaluf et al. 2023]. Complementary fine-tuning methods adapt model parameters from few reference images, including DreamBooth [Ruiz et al. 2023] and selective parameter updates to reduce overfitting and storage [Kumari et al. 2023]. To improve efficiency, parameter-efficient fine-tuning introduces lightweight trainable components such as style-specific modules [Sohn et al. 2023] and low-rank adaptation (LoRA) [Hu et al. 2022], now widely adopted for diffusion model customization. Beyond per-subject training, pre-trained adaptation methods enable fast one-shot personalization by conditioning on reference inputs [Li et al. 2023; Ye et al. 2023]. See [Zhang et al. 2025] for a comprehensive survey.

定制化图像生成。扩散模型的定制,是把预训练模型适配到特定主体和风格,从而在原始训练分布之外做个性化生成。早期方法用学到的文本 token 绑定新概念,如 textual inversion [Gal et al. 2022],后续工作则使用多 token 或随时间变化的嵌入 [Voynov et al. 2023; Alaluf et al. 2023]。与之互补的微调方法,则从少量参考图像出发调整模型参数,包括 DreamBooth [Ruiz et al. 2023],以及为减少过拟合和存储而做的选择性参数更新 [Kumari et al. 2023]。为了提高效率,参数高效微调引入了轻量可训练组件,例如面向风格的模块 [Sohn et al. 2023] 和低秩适配(LoRA)[Hu et al. 2022],后者现已广泛用于扩散模型定制。除了针对单个主体的训练,预训练好的适配方法还可以通过对参考输入做条件控制,实现快速的一次性个性化 [Li et al. 2023; Ye et al. 2023]。更全面的综述见 [Zhang et al. 2025]。

Model merging. In this setting, multiple adapters fine-tuned from a shared base model are combined to produce a merged effect [Yadav et al. 2024]. In the text-to-image domain, merging has been used to compose multiple concepts [Gu et al. 2023; Po et al. 2024; Dalva et al. 2025] or to combine content with style [Shah et al. 2024; Frenkel et al. 2024; Ouyang et al. 2025]. Recent works have also explored, in addition to adapter selection, generating the entire image-generation workflow [Gal et al. 2025; Gadot et al. 2025]. In contrast to these settings, which treat merging as a discrete operation or focus on fixed combinations, we focus on treating adapter merging as a continuous design problem, where the strength of each component can be adjusted (e.g., “+[Adapter]” vs. “+0.5 [Adapter]”). Our quantitative evaluation focuses on merging multiple style adapters (Sec. 5.1), reflecting common real-world usage and a setting that is comparatively less structured and under-explored than style-content composition. More broadly, GimmBO generalizes naturally to other forms of adapter composition, including subject- and content-based merging as demonstrated in Sec. 5.2. We adopt linear merging, a simple yet effective method that is widely used in practice, while remaining compatible with more advanced merging techniques. A related line of work studies community-model retrieval [Liu et al. 2024; Luo et al. 2024; Sonmezer et al. 2025], which selects and combines adapters shared online, e.g., via LLM-based textual retrieval. These approaches are complementary to ours, as they can help narrow the candidate set, but they do not explicitly address continuous control over merging weights; we demonstrate how GimmBO integrates naturally with such retrieval methods in Sec. 5.2.

模型合并。在这种设定下,从同一基底模型微调得到的多个适配器被组合在一起,产生合并后的效果 [Yadav et al. 2024]。在文本到图像领域,合并被用来组合多个概念 [Gu et al. 2023; Po et al. 2024; Dalva et al. 2025],或把内容与风格结合起来 [Shah et al. 2024; Frenkel et al. 2024; Ouyang et al. 2025]。近期工作除了适配器选择,还尝试生成整条图像生成工作流 [Gal et al. 2025; Gadot et al. 2025]。这些设定往往把合并看成离散操作,或只关注固定组合;与之不同,我们把适配器合并当作一个连续设计问题,每个组件的强度都可以调节(例如 “+[Adapter]” 与 “+0.5 [Adapter]”)。定量评估主要针对多个风格适配器的合并(第 5.1 节),这既更贴近常见用法,也比风格–内容组合更少结构化、更少被研究。更一般地,GimmBO 可以自然推广到其他形式的适配器组合,包括第 5.2 节展示的基于主体和内容的合并。我们采用线性合并:这种方法简单有效、在实践中使用广泛,同时仍可与更先进的合并技术兼容。另一类相关工作研究社区模型检索 [Liu et al. 2024; Luo et al. 2024; Sonmezer et al. 2025],用例如基于大语言模型的文本检索,来挑选并组合网上分享的适配器。这些方法与我们是互补的,它们可以帮助收窄候选集,但并没有显式处理合并权重的连续控制;我们在第 5.2 节展示了 GimmBO 如何与这类检索方法自然衔接。

Prompt-based customization. Existing methods explore customized image generation by adapting text prompts based on human interaction, often through visualization interfaces [Brade et al. 2023; Feng et al. 2023; Guo et al. 2024; Adamkiewicz et al. 2025] or conversational agents [Wen et al. 2026]. Such prompt-based approaches can be applied to scenarios like style mixing by iteratively refining textual descriptions. However, they implicitly assume that the desired styles are already encoded in the base model and can be elicited through prompting. As a result, they often struggle to capture out-of-distribution or highly specific styles compared to adapter-based approaches. To illustrate this limitation, we consider reproducing the style of an artist with only a small number of publicly available works, which are likely underrepresented in the base model’s training distribution (Fig. 3). In an adapter-based stylization setting (Stable Diffusion XL with LoRA adapters), fine-tuning on 30 images was sufficient to faithfully capture the target style. In contrast, prompt-based approaches using ChatGPT, either directly from reference images or via auto-captioning followed by SDXL generation, produced results with only partial similarity. We also evaluated PromptMagician [Feng et al. 2023], a visualization-based prompt exploration tool, but even careful prompt search did not achieve comparable results, suggesting that certain stylistic nuances are difficult to express purely through text.

基于提示词的定制。已有方法通过根据人的交互来改写文本提示,探索定制化图像生成,常见途径是可视化界面 [Brade et al. 2023; Feng et al. 2023; Guo et al. 2024; Adamkiewicz et al. 2025] 或对话式智能体 [Wen et al. 2026]。这类基于提示的方法,也可以通过反复改写文本描述的方式,用于风格混合等场景。不过,它们隐含地假定目标风格已经编码在基底模型里,并且可以通过提示词被诱发出来。因此,面对分布外或非常具体的风格时,它们往往不如基于适配器的方法。为说明这一局限,我们尝试复现一位公开作品很少的艺术家的风格,这些作品很可能在基底模型的训练分布中代表性不足(图 3)。在基于适配器的风格化设定下(Stable Diffusion XL 配 LoRA 适配器),用 30 张图像微调就足以忠实抓住目标风格。相比之下,用 ChatGPT 做基于提示的方法,无论是直接根据参考图,还是先自动生成说明再交给 SDXL 生成,都只能得到部分相似的结果。我们还评估了 PromptMagician [Feng et al. 2023] 这一基于可视化的提示探索工具,即便仔细搜索提示词,也无法达到可比的效果,说明有些风格细微之处很难只靠文字表达。

Fig. 3.

Fig. 3. Given 30 reference images by an artist with few publicly available works (top), we attempt to reproduce the style (bottom). Fine-tuning an SDXL LoRA adapter closely matches the target style, while prompt-based approaches, including ChatGPT image creation (a dark block indicates a censored word), auto-captioning followed by SDXL generation, and PromptMagician [Feng et al. 2023], achieve only partial similarity, highlighting the limitations of text-based elicitation for underrepresented styles.

图 3. 给定一位公开作品很少的艺术家的 30 张参考图(上),我们尝试复现其风格(下)。微调一个 SDXL LoRA 适配器能够贴近目标风格;而基于提示的方法,包括 ChatGPT 图像生成(深色色块表示被屏蔽的词)、自动配说明后再用 SDXL 生成,以及 PromptMagician [Feng et al. 2023],都只能达到部分相似,说明对代表性不足的风格而言,单靠文本诱发是有局限的。

Human-in-the-loop design optimization. Many design tasks are inherently subjective, making human judgment essential to the design process and typically expressed through iterative choices over design parameters. However, as the number of parameters grows, this exploration becomes difficult to perform manually, even with as few as 5–10 parameters [Dang et al. 2022]. Researchers have explored human-in-the-loop optimization to support efficient exploration of complex design spaces. Prior work includes interaction paradigms such as slider interfaces [Desai et al. 2019; Koyama et al. 2014; Shimizu et al. 2020; Shugrina et al. 2015] and design galleries [Marks et al. 2023; Ngan et al. 2006; Shapira et al. 2009] to reduce direct parameter manipulation, with Bayesian Optimization (BO) later adopted as a sample-efficient backend within these settings for visual design tasks [Koyama et al. 2017; Koyama et al. 2020]. BO has also been adapted to specific domains such as animation [Brochu et al. 2010], melody [Zhou et al. 2020], font [Kadner et al. 2021; Tatsukawa et al. 2025], and game levels [Khajah et al. 2016]. Our method falls within this line of domain-specific BO approaches, and we explicitly compare against previous BO-based interaction methods for general visual design tasks [Koyama et al. 2017; Koyama et al. 2020] (Sec. 5.1). While effective in lower-dimensional settings, these previous methods are ill suited to our 20–30D adapter-merging scenario: [Koyama et al. 2017] relies on single-direction updates that do not scale, and [Koyama et al. 2020] assumes interior solutions and ignores poor-quality regions, leading respectively to duplicated clamped samples and degraded configurations (Fig. 4, Sec. 3). Recently, researchers explore how to make the design process more cooperative [Koyama and Goto 2022; Mo et al. 2024; Niwa et al. 2025]. These methods emphasize interface design, whereas we contribute an optimization backend tailored to the model-merging problem. The two are complementary, and integrating them would be an interesting future direction.

有人在环的设计优化。许多设计任务本质上是主观的,因而人的判断对设计过程至关重要,通常通过对设计参数的反复选择来表达。不过,参数一多,即便只有 5–10 个,手工探索也会变得困难 [Dang et al. 2022]。研究者们探索用有人在环的优化,来支持对复杂设计空间的高效探索。先前工作包括滑条界面 [Desai et al. 2019; Koyama et al. 2014; Shimizu et al. 2020; Shugrina et al. 2015] 和设计画廊 [Marks et al. 2023; Ngan et al. 2006; Shapira et al. 2009] 等交互范式,以减少直接拨动参数;随后贝叶斯优化(BO)作为样本高效的后端,被引入这些设定,用于视觉设计任务 [Koyama et al. 2017; Koyama et al. 2020]。BO 也被适配到动画 [Brochu et al. 2010]、旋律 [Zhou et al. 2020]、字体 [Kadner et al. 2021; Tatsukawa et al. 2025] 和游戏关卡 [Khajah et al. 2016] 等具体领域。我们的方法属于这类面向特定领域的 BO 路径,并在一般视觉设计任务上,与先前基于 BO 的交互方法做了直接比较 [Koyama et al. 2017; Koyama et al. 2020](第 5.1 节)。这些方法在较低维设定中有效,却不适合我们 20–30 维的适配器合并场景:[Koyama et al. 2017] 依赖单方向更新,难以扩展;[Koyama et al. 2020] 假定解在区域内部,并忽略低质量区域,结果分别出现重复的被钳位样本和退化配置(图 4,第 3 节)。最近也有研究探索如何让设计过程更有协作性 [Koyama and Goto 2022; Mo et al. 2024; Niwa et al. 2025]。这些工作更强调界面设计,而我们贡献的是针对模型合并问题定制的优化后端。两者是互补的,把它们结合起来会是一个有意思的未来方向。

Most existing mechanisms for dealing with high-dimensionality in BO are difficult to apply directly in our setting: embedding methods [Wang et al. 2016; Raponi et al. 2020; Chiu et al. 2020] and trust regions [Diouane et al. 2023] do not account for solution sparsity without adaptation, and our human-preference objective is unlikely to be additive. We therefore apply BO solvers with built-in variable-selection capabilities [Eriksson and Jankowiak 2021; Liu et al. 2023], which are better matched to our problem.

现有大多数应对 BO 中高维问题的机制,都很难直接用在我们的设定里:嵌入方法 [Wang et al. 2016; Raponi et al. 2020; Chiu et al. 2020] 和信赖域 [Diouane et al. 2023] 若不加改造,就无法刻画解的稀疏性,而我们的人类偏好目标也不太可能具有可加性。因此,我们采用自带变量选择能力的 BO 求解器 [Eriksson and Jankowiak 2021; Liu et al. 2023],它们更契合这一问题。

Fig. 4.

Fig. 4. Previous BO-based interaction methods [Koyama et al. 2020], which do not account for the structure of the adapter-merging design space, may propose clamped or degraded samples in high-dimensional settings.

图 4. 先前基于 BO 的交互方法 [Koyama et al. 2020] 没有考虑适配器合并设计空间的结构,在高维设定下可能会提出被钳位或已经退化的样本。

3 How do people merge adapters today?

3 人们今天怎样合并适配器?

Given a generative image base model with weights W0\mathbf{W}_{0}W0​, fine-tuned adapters work by updating the weights (W0+ΔW\mathbf{W}_{0}+\Delta\mathbf{W}W0​+ΔW) via further loss optimization on a new, specialized dataset. For example, an adapter may be created by fine-tuning the SDXL base model to generate vector-art-style images. A different SDXL-based adapter may be fine-tuned to generate images of papercraft artwork. Sites like Civitai allow users to not only pick the strength of a given adapter but also blend multiple adapters linearly to combine their effects:

给定一个权为 W0\mathbf{W}_{0}W0​ 的生成图像基底模型,微调得到的适配器通过在新的专用数据上继续做损失优化,来更新权重(W0+ΔW\mathbf{W}_{0}+\Delta\mathbf{W}W0​+ΔW)。例如,可以把 SDXL 基底模型微调成一个生成矢量插画风格图像的适配器;另一个基于 SDXL 的适配器则可以微调成生成剪纸风格的图像。像 Civitai 这样的网站不仅允许用户选择单个适配器的强度,还可以把多个适配器线性混合,以组合它们的效果:

Wmerged=W0+∑i=1nαiΔWi,(1)\mathbf{W}_{\text{merged}}=\mathbf{W}_{0}+\sum_{i=1}^{n}\alpha_{i}\Delta\mathbf{W}_{i}, \tag{1}Wmerged​=W0​+i=1∑n​αi​ΔWi​,(1)

where αi\alpha_{i}αi​ is the merging coefficient controlling the strength of the iiith adapter’s weight update ΔWi\Delta\mathbf{W}_{i}ΔWi​. It is both well known in practice and well supported by linear mode connectivity theory [Ilharco et al. 2023; Frankle et al. 2020] that this simple linear combination of adapters of the same base model can result in meaningful combinations of individually learned abilities (e.g., image styles). In practice, most adapters are parameter-efficient fine-tunings such as LoRAs [Hu et al. 2022] (where weight updates are the product of two low rank matrices ΔW=BA\Delta\mathbf{W}=\mathbf{B}\mathbf{A}ΔW=BA), but this is not a requirement.

其中 αi\alpha_{i}αi​ 是控制第 iii 个适配器权重更新 ΔWi\Delta\mathbf{W}_{i}ΔWi​ 强度的合并系数。实践中已经很清楚,线性模式连通性理论也为此提供了支持 [Ilharco et al. 2023; Frankle et al. 2020]:把同一基底模型上的适配器做这种简单的线性组合,可以得到各自所学能力(例如图像风格)的有意义组合。实践中,大多数适配器是 LoRA [Hu et al. 2022] 这类参数高效微调(权重更新是两个低秩矩阵的乘积 ΔW=BA\Delta\mathbf{W}=\mathbf{B}\mathbf{A}ΔW=BA),但这并不是必要条件。

The combination of image styles is complex to describe, hard to predict, and exciting. On sites like Civitiai, users experiment with different selections of adapters and different merging coefficients to create and explore new styles that would otherwise be hard to achieve through prompt conditioning of the base model or with an individual existing adapter. The choice of merging coefficients αi\alpha_{i}αi​ is non-trivial. People typically rely on trial and error: pick new αi\alpha_{i}αi​ values with a slider or numeric entry, wait for a generated image, then iterate on their choice. Despite this painful process, there exist enough examples of successfully merged adapters to identify two common practices which guide our method’s design.

图像风格的组合很难描述、难以预知,却也很有吸引力。在 Civitai 这类网站上,用户尝试不同的适配器选择和不同的合并系数,去创造和探索那些单靠对基底模型做提示调节、或只用某一个现成适配器很难得到的新风格。合并系数 αi\alpha_{i}αi​ 的选择并不简单。人们通常只能试错:用滑条或数字输入挑一组新的 αi\alpha_{i}αi​,等图像生成出来,再继续改。尽管这个过程很折磨人,已经有足够多成功合并的例子,可以从中看出两种常见做法,并用来指导我们方法的设计。

Sparsity. We collected the generated images hosted on Civitai and analyzed their provenance. Of the 20,780 images with sufficient meta data, we found that a strong majority are created using at least one adapter (68%) and most of those are created by merging between two to nine adapters (69%) (see inset example of a generation using a merge of two adapters and a bar graph of adapter counts across the collected images). We speculate that this reflects both a preference for interpretability and avoidance of technical issues such as model interference. We will leverage this sparsity to guide our optimization from an initial large and possibly crudely collected candidate set (n=20−30n=20-30n=20−30) to a small active set of adapters (those with non-zero merging coefficient αi≠0\alpha_{i}\neq 0αi​=0). Despite the heavy-tailed distribution of contributions (out of 3,533 users, top 1% contribute 33.3% of works and top 5% 57.8%), statistics remain consistent after removing the top 1% and 5% of users, indicating no significant bias.

Inset: a typical sparse two-adapter merge and adapter-count distribution.

稀疏性。我们收集了 Civitai 上托管的生成图像,并分析它们的来源。在 20,780 张元数据足够完整的图像中,绝大多数至少使用了一个适配器(68%),而其中大多数又是由 2 到 9 个适配器合并得到的(69%)(见插图:一个由两个适配器合并得到的生成结果,以及所收集图像中适配器个数的条形图)。我们推测,这既反映了人们对可解释性的偏好,也是为了避开模型相互干扰一类技术问题。我们将利用这种稀疏性,把优化从最初较大、也可能收集得比较粗糙的候选集(n=20n=20n=20–303030)引导到一个很小的活跃适配器集合(即合并系数 αi≠0\alpha_{i}\neq 0αi​=0 的那些)。尽管贡献呈长尾分布(在 3,533 名用户中,前 1% 贡献了 33.3% 的作品,前 5% 贡献了 57.8%),去掉贡献最多的 1% 和 5% 用户后,统计仍然稳定,说明没有明显偏差。

Coefficient magnitudes. By construction, a successfully trained fine-tuned adapter works well (in isolation) with unit magnitude (α=1\alpha=1α=1). Linear mode connectivity supports that values within or near [0,1][0,1][0,1] will also behave well, diminishing the learned effect as α\alphaα approaches zero. When merging nnn adapters, not all combinations of merging coefficients in the entire [0,1]n[0,1]^{n}[0,1]n hypercube will result in meaningful generations. For example, the inset images generated by merging n=20n=20n=20 adapters produces a meaningful stylized image for a small sum of coefficients but loses coherence for a large sum. Analyzing the generated images from Civitai, we find that the sum of merging coefficients ∑i=1nαi\sum_{i=1}^{n}\alpha_{i}∑i=1n​αi​ falls off quickly, with most sums falling between 1.0 and 3.0 (inset histogram). We speculate that the true feasible set of merging coefficients is fuzzy, hard to precisely define and adapter dependent. We nevertheless leverage our observation by modeling this feasible set with a coarse proxy geometry. We will show that enforcing coefficients to stay within this set during early stages and loosening the restriction later will improve convergence speed and final accuracy.

Inset: generations at small and large coefficient sums, with a histogram of sums.

系数幅度。按构造,一个训练成功的微调适配器在单独使用、幅度取单位值(α=1\alpha=1α=1)时效果最好。线性模式连通性也支持:取值在 [0,1][0,1][0,1] 之内或附近时同样表现良好,而随着 α\alphaα 趋近零,所学效果会逐渐减弱。当合并 nnn 个适配器时,并非 [0,1]n[0,1]^{n}[0,1]n 超立方体中的所有系数组合都能得到有意义的生成。例如,插图中由 n=20n=20n=20 个适配器合并得到的图像,在系数之和较小时风格仍然成立,之和很大时就会失去连贯性。分析 Civitai 上的生成图像后,我们发现合并系数之和 ∑i=1nαi\sum_{i=1}^{n}\alpha_{i}∑i=1n​αi​ 下降得很快,大多数和落在 1.0 到 3.0 之间(见插图中的直方图)。我们推测,真正可行的合并系数集合是模糊的,很难精确界定,而且依赖于具体适配器。尽管如此,我们仍把这一观察写成一个粗糙的代理几何,用来刻画可行集。后面会看到,在早期阶段强制系数留在这个集合内,再在后期放宽限制,能够加快收敛并提高最终精度。

4 Our GimmBO Method

4 我们的 GimmBO 方法

Given a candidate set of nnn (typically 20-3020\text{-}3020-30) adapters, at the end of our interactive design process, our goal is to output a vector of merging coefficients α=[α1,…,αn]\bm{\alpha}=[\alpha_{1},\dots,\alpha_{n}]α=[α1​,…,αn​] which resulting in maximally satisfying image generations, as subjectively judged by the human user. We conceptually model this as an optimization problem:

给定一组 nnn 个(通常为 202020–303030 个)候选适配器,交互式设计过程结束时,我们的目标是输出一个合并系数向量 α=[α1,…,αn]\bm{\alpha}=[\alpha_{1},\dots,\alpha_{n}]α=[α1​,…,αn​],使生成的图像在人类用户的主观判断下尽可能令人满意。我们把这概念上看成一个优化问题:

argmaxα∈Af(g(α)),(2)\displaystyle\mathop{\text{argmax}}_{\bm{\alpha}\in\mathcal{\mathcal{A}}}f\left(g\left(\bm{\alpha}\right)\right), \tag{2}argmaxα∈A​f(g(α)),(2)

where A\mathcal{A}A is the merging coefficients design space, referred to as the search space in our optimization (a yet-to-be-defined subregion of [0,1]n[0,1]^{n}[0,1]n), ggg is the fixed function that maps a choice of merging coefficients to a generated image (eliding text prompts or other conditioning parameters), and fff models the user’s assigned utility of a given image. Our quest to find the best α\bm{\alpha}α values naturally relies on simultaneously learning a good surrogate approximation of fff from limited user feedback on generated images so far. We propose using human-in-the-loop Bayesian optimization.

其中 A\mathcal{A}A 是合并系数的设计空间,在优化里称为搜索空间(它是 [0,1]n[0,1]^{n}[0,1]n 中一块尚未具体定义的子区域),ggg 是把一组合并系数映射为生成图像的固定函数(略去文本提示或其他条件参数),fff 则刻画用户对给定图像赋予的效用。要找到最好的 α\bm{\alpha}α,自然需要同时根据目前有限的用户反馈,为 fff 学到一个好的代理近似。我们提出使用有人在环的贝叶斯优化。

Fig. 5.

Fig. 5. Starting from initial samples, GimmBO iteratively proposes new sample batches (pink stars and images), guided by an acquisition function (blue) over a surrogate model f^\hat{f}f^​ updated with accumulated samples (blue dots and images). In this 2D example, GimmBO rapidly converges to the target.

图 5. 从初始样本出发,GimmBO 迭代地提出新的样本批次(粉色星标与图像),由采集函数(蓝色)在代理模型 f^\hat{f}f^​ 上引导,而 f^\hat{f}f^​ 又随着累积样本(蓝色点与图像)更新。在这个二维例子里,GimmBO 很快收敛到目标。

Fig. 6.

Fig. 6. Batch samples at Steps 1 and 10 in a 20D matching task (q=8q=8q=8). Blue cells denote non-zero coefficients (darker = larger magnitude), white cells indicate zeros; green arrows mark ground-truth active axes. With identical initialization and random seed, B=2B=2B=2 performs best compared to B=1B=1B=1 and B=nB=nB=n.

图 6. 在一项 20 维匹配任务中,第 1 步和第 10 步的批次样本(q=8q=8q=8)。蓝色格子表示非零系数(颜色越深幅度越大),白色格子表示零;绿色箭头标出真实的活跃轴。在初始化和随机种子相同的情况下,B=2B=2B=2 优于 B=1B=1B=1 和 B=nB=nB=n。

GimmBO works iteratively (Fig. 5). Each iteration, we ask the user to rank a small number of images generated by different merging coefficient vectors. This ranking information is used to update: a prior over surrogates f^\hat{f}f^​ that aim to proportionally approximate the unknown user preference function fff; and an acquisition function whose (local) maxima represent informative points for reducing uncertainty that the maximizer of f^\hat{f}f^​ matches that of fff. The acquisition function is designed so that sampling it balances exploitation of regions predicted to yield high utility scores with exploration of uncertain regions that improve the surrogate model.

GimmBO 是迭代进行的(图 5)。每一轮,我们请用户对少量由不同合并系数向量生成的图像排序。这些排序信息用来更新两样东西:一组代理模型 f^\hat{f}f^​ 上的先验,它们旨在按比例逼近未知的用户偏好函数 fff;以及一个采集函数,其(局部)极大值对应那些有助于降低不确定性的点,也就是让 f^\hat{f}f^​ 的最大化者更可能与 fff 的最大化者一致。采集函数的设计,是为了在采样时平衡对预测高效用区域的利用,以及对能够改善代理模型的不确定区域的探索。

4.1 Preferential Surrogate

4.1 优先代理模型

We model the surrogate f^\hat{f}f^​ as a latent function over merging coefficients α\bm{\alpha}α equipped with a Gaussian process prior, conditioned on observed data D\mathcal{D}D:

我们把代理模型 f^\hat{f}f^​ 建为合并系数 α\bm{\alpha}α 上的潜在函数,并配以高斯过程先验,在已观测数据 D\mathcal{D}D 上做条件化:

f^(α∣D)∼GP(μ(α),k(α,α′)),(3)\displaystyle\hat{f}(\bm{\alpha}\mid\mathcal{D})\sim\mathcal{GP}\left(\mu(\bm{\alpha}),k(\bm{\alpha},\bm{\alpha}^{\prime})\right), \tag{3}f^​(α∣D)∼GP(μ(α),k(α,α′)),(3)

fully specified by a mean function μ\muμ and covariance function kkk.

由均值函数 μ\muμ 和协方差函数 kkk 完全确定。

Unlike classic Gaussian processes, we cannot expect users to provide data in the form of direct numeric utility values [Brochu et al. 2010; Tsukida and Gupta 2011]. Instead, we collect observed data in the form of ∣D∣|\mathcal{D}|∣D∣ pairwise comparisons:

与经典高斯过程不同,我们不能指望用户以直接的数值效用形式提供数据 [Brochu et al. 2010; Tsukida and Gupta 2011]。相反,我们以 ∣D∣|\mathcal{D}|∣D∣ 对成对比较的形式收集观测数据:

D={(αi,αj)  |  αi≻αj},(4)\displaystyle\mathcal{D}=\left\{(\bm{\alpha}_{i},\bm{\alpha}_{j})\;\middle|\;\bm{\alpha}_{i}\succ\bm{\alpha}_{j}\right\}, \tag{4}D={(αi​,αj​)∣αi​≻αj​},(4)

where αi≻αj\bm{\alpha}_{i}\succ\bm{\alpha}_{j}αi​≻αj​ encodes that the user preferred the generated image g(αi)g(\bm{\alpha}_{i})g(αi​) over g(αj)g(\bm{\alpha}_{j})g(αj​). We assume transitivity of user rankings, so presenting the user with NNN images and them to rank the top kkk results in up to k(k−1)/2+k(N−k)k(k-1)/2+k(N-k)k(k−1)/2+k(N−k) new inequalities in D\mathcal{D}D (all pairs in the top kkk + each top kkk versus the remaining). Given any update to D\mathcal{D}D, f^\hat{f}f^​ is relearned by first employing the preference learning approach of [Chu and Ghahramani 2005] to convert the pairwise inequalities into latent function values f^(αi)\hat{f}(\bm{\alpha}_{i})f^​(αi​) for each observed αi\bm{\alpha}_{i}αi​. Then the posterior mean and covariance of f^\hat{f}f^​ is fit to these values using a sparse axis-aligned subspace (SAAS) prior [Eriksson and Jankowiak 2021] (see supplemental for further details).

其中 αi≻αj\bm{\alpha}_{i}\succ\bm{\alpha}_{j}αi​≻αj​ 表示用户更喜欢 g(αi)g(\bm{\alpha}_{i})g(αi​) 生成的图像,而不是 g(αj)g(\bm{\alpha}_{j})g(αj​)。我们假定用户排序具有传递性,因此向用户展示 NNN 张图像并请他们排出前 kkk 名,最多会给 D\mathcal{D}D 增加 k(k−1)/2+k(N−k)k(k-1)/2+k(N-k)k(k−1)/2+k(N−k) 条新的不等式(前 kkk 名内部的所有成对比较,再加上前 kkk 名与其余图像的比较)。每当 D\mathcal{D}D 有更新,f^\hat{f}f^​ 都会重新学习:先采用 [Chu and Ghahramani 2005] 的偏好学习方法,把成对不等式转换成每个已观测 αi\bm{\alpha}_{i}αi​ 上的潜在函数值 f^(αi)\hat{f}(\bm{\alpha}_{i})f^​(αi​);再使用稀疏轴对齐子空间(SAAS)先验 [Eriksson and Jankowiak 2021],把 f^\hat{f}f^​ 的后验均值和协方差拟合到这些值上(更多细节见补充材料)。

4.2 Search space

4.2 搜索空间

We consider two choices for the search space of merging coefficients. The first is the entire unit hypercube:

关于合并系数的搜索空间,我们考虑两种选择。第一种是整个单位超立方体:

A=[0,1]n.(5)\displaystyle\mathcal{A}=[0,1]^{n}. \tag{5}A=[0,1]n.(5)

This choice works well when the size of the candidate set nnn is small, but when nnn is large sampling merging coefficient vectors α∈[0,1]n\bm{\alpha}\in[0,1]^{n}α∈[0,1]n often results in meaningless or poor quality images, wasting precious user feedback.

当候选集大小 nnn 较小时,这一选择效果很好;但当 nnn 较大时,从 α∈[0,1]n\bm{\alpha}\in[0,1]^{n}α∈[0,1]n 中采样得到的合并系数向量,往往产生无意义或质量很差的图像,浪费宝贵的用户反馈。

Instead, when the candidate set is large, we define the search space to model our observation that people tend to merge adapters with small total magnitude. In particular, we propose modeling the space with a BBB-capped simplex:

因此,在候选集较大时,我们让搜索空间体现这样一个观察:人们倾向于用较小的总幅度来合并适配器。具体来说,我们建议用 BBB-封顶单纯形 来刻画这一空间:

A=ΔB={α∈[0,1]n | ∑i=1nαi≤B}.(6)\displaystyle\mathcal{A}=\Delta_{B}=\left\{\bm{\alpha}\in[0,1]^{n}\ \middle|\ \sum_{i=1}^{n}\alpha_{i}\leq B\right\}. \tag{6}A=ΔB​={α∈[0,1]n ​ i=1∑n​αi​≤B}.(6)

Relative to the volume of [0,1]n[0,1]^{n}[0,1]n, this geometry has rapidly diminishing volume as nnn increases [Marichal and Mossinghoff 2006]:

相对于 [0,1]n[0,1]^{n}[0,1]n 的体积,这种几何的体积会随着 nnn 增大而迅速减小 [Marichal and Mossinghoff 2006]:

Vol⁡(n;ΔB)=1n!∑k=0⌊B⌋(−1)k(nk)(B−k)n.(7)\displaystyle\operatorname{Vol}(n;\Delta_{B})=\frac{1}{n!}\sum_{k=0}^{\lfloor B\rfloor}(-1)^{k}\binom{n}{k}(B-k)^{n}. \tag{7}Vol(n;ΔB​)=n!1​k=0∑⌊B⌋​(−1)k(kn​)(B−k)n.(7)

For example, Vol⁡(10;2)≈0.03%\operatorname{Vol}(10;2)\approx 0.03\%Vol(10;2)≈0.03% and Vol⁡(30;2)<10−21%\operatorname{Vol}(30;2)<10^{-21}\%Vol(30;2)<10−21%. We will show that choosing A=ΔB\mathcal{A}=\Delta_{B}A=ΔB​ for early iterations works well, while relaxing A=[0,1]n\mathcal{A}=[0,1]^{n}A=[0,1]n in later iterations allows polished accuracy.

例如,Vol⁡(10;2)≈0.03%\operatorname{Vol}(10;2)\approx 0.03\%Vol(10;2)≈0.03%,而 Vol⁡(30;2)<10−21%\operatorname{Vol}(30;2)<10^{-21}\%Vol(30;2)<10−21%。我们将表明,早期迭代取 A=ΔB\mathcal{A}=\Delta_{B}A=ΔB​ 效果很好,而在后期把 A\mathcal{A}A 放宽为 [0,1]n[0,1]^{n}[0,1]n,则可以精细提高精度。

4.3 Initialization

4.3 初始化

With our surrogate in place, we initialize samples of merging coefficient values α\bm{\alpha}α from A=ΔB\mathcal{A}=\Delta_{B}A=ΔB​, for which images are generated and shown to the user for ranking. Due to the diminishing volume of ΔB\Delta_{B}ΔB​ for large nnn, naive rejection sampling from [0,1]n[0,1]^{n}[0,1]n would be intractably slow. We propose to use a modified Dirichlet or “stick breaking” process. Given a uniformly random sample x∈[0,1]n\mathbf{x}\in[0,1]^{n}x∈[0,1]n, we sample α\bm{\alpha}α via an coordinate-dependent inductive process:

有了代理模型之后,我们从 A=ΔB\mathcal{A}=\Delta_{B}A=ΔB​ 初始化合并系数样本 α\bm{\alpha}α,生成图像并展示给用户排序。由于 nnn 较大时 ΔB\Delta_{B}ΔB​ 的体积很小,若从 [0,1]n[0,1]^{n}[0,1]n 做朴素的拒绝采样,会慢到无法使用。我们改用一种改进的 Dirichlet 过程,也就是“折棍”过程。给定均匀随机样本 x∈[0,1]n\mathbf{x}\in[0,1]^{n}x∈[0,1]n,我们通过一个与坐标有关的归纳过程来采样 α\bm{\alpha}α:

αi=min(xiRi−1,1) and Ri=Ri−1−αi with R0=B.(8)\displaystyle\alpha_{i}=\mathop{\text{min}}(x_{i}R_{i-1},1)\text{ and }R_{i}=R_{i-1}-\alpha_{i}\text{ with }R_{0}=B. \tag{8}αi​=min(xi​Ri−1​,1) and Ri​=Ri−1​−αi​ with R0​=B.(8)

This distribution is known to exhibit coordinate-ordering bias, which we mitigate by picking a random coordinate ordering for each sample. Finally, we further sparsify samples by zeroing entries below a threshold τ\tauτ.

这一分布已知会带有坐标顺序偏差,我们通过为每个样本随机挑选坐标顺序来减轻它。最后,再把低于阈值 τ\tauτ 的分量置零,使样本进一步稀疏。

4.4 Acquisition

4.4 采集函数

We adopt an upper confidence bound (UCB) acquisition function [Srinivas et al. 2009], which provides a simple and interpretable hyperparameter λ\lambdaλ to balance exploration and exploitation. First written as a function of a single α\bm{\alpha}α vector, UCB is a simple sum of mean and variance terms:

我们采用上置信界(UCB)采集函数 [Srinivas et al. 2009],它用一个简单、可解释的超参数 λ\lambdaλ 来平衡探索与利用。先把它写成单个 α\bm{\alpha}α 向量的函数,UCB 就是均值项与方差项的和:

ϕ(α∣f^)=μ(α)+λk(α,α).(9)\displaystyle\phi(\bm{\alpha}\mid\hat{f})=\mu(\bm{\alpha})+\lambda\sqrt{k(\bm{\alpha},\bm{\alpha})}. \tag{9}ϕ(α∣f^​)=μ(α)+λk(α,α)​.(9)

Each user-facing iteration we sample qqq merging coefficient vectors {α1,…,αq}\{\bm{\alpha}_{1},\dots,\bm{\alpha}_{q}\}{α1​,…,αq​} using a batched-version of ϕ\phiϕ maximizing the expectation of maximal single acquisition value [Wilson et al. 2017]:

在每一轮面向用户的迭代中,我们用 ϕ\phiϕ 的批处理版本采样 qqq 个合并系数向量 {α1,…,αq}\{\bm{\alpha}_{1},\dots,\bm{\alpha}_{q}\}{α1​,…,αq​},最大化单个采集值中最大者的期望 [Wilson et al. 2017]:

max{α1,…,αq}Eϕ[maxj=1,…,qϕ(αj∣f^)]αj∈A for j=1,…,q.(10)\displaystyle\mathop{\text{max}}_{\{\bm{\alpha}_{1},\dots,\bm{\alpha}_{q}\}} \\ \displaystyle\quad\mathbb{E}_{\phi}\left[\mathop{\text{max}}_{j=1,\dots,q}\phi(\bm{\alpha}_{j}\mid\hat{f})\right] \\ \displaystyle\quad\bm{\alpha}_{j}\in\mathcal{A}\text{ for }j=1,\dots,q. \tag{10}max{α1​,…,αq​}​Eϕ​[maxj=1,…,q​ϕ(αj​∣f^​)]αj​∈A for j=1,…,q.(10)

When A=[0,1]n\mathcal{A}=[0,1]^{n}A=[0,1]n, we can easily randomly sample starting positions and run L-BFGS-B with multiple random restarts to ascend to a set of local maxima.

当 A=[0,1]n\mathcal{A}=[0,1]^{n}A=[0,1]n 时,我们可以轻松随机抽样起始位置,并用多次随机重启的 L-BFGS-B 爬升到一组局部极大值。

When the search space is the BBB-capped simplex A=ΔB\mathcal{A}=\Delta_{B}A=ΔB​, we treat the stick-breaking process in the previous Section 4.3 as a parameterization of feasible merging coefficients as a function of points in the unit hypercube α(x)\bm{\alpha}(\mathbf{x})α(x). We draw starting positions by sampling x∈[0,1]n\mathbf{x}\in[0,1]^{n}x∈[0,1]n and multi-restart L-BFGS-B ascent treating x\mathbf{x}x as optimization parameters. Fig. 6 demonstrates the impact of different choices of BBB, and we set B=2B=2B=2 by default.

当搜索空间是 BBB-封顶单纯形 A=ΔB\mathcal{A}=\Delta_{B}A=ΔB​ 时,我们把上一节 4.3 中的折棍过程,看成可行合并系数关于单位超立方体中点的参数化 α(x)\bm{\alpha}(\mathbf{x})α(x)。起始位置通过对 x∈[0,1]n\mathbf{x}\in[0,1]^{n}x∈[0,1]n 采样得到,再以 x\mathbf{x}x 为优化变量,做多次随机重启的 L-BFGS-B 爬升。图 6 展示了不同 BBB 选择的影响,我们默认取 B=2B=2B=2。

Images are generated for each of the newly acquired αj\bm{\alpha}_{j}αj​ samples. These images, combined with those of the current top-ranked merge coefficient vector and a few previously seen samples, are presented to the user for the next round of ranking.

每个新得到的 αj\bm{\alpha}_{j}αj​ 样本都会生成图像。这些图像,连同当前排在最前面的合并系数向量所对应的图像,以及少量先前见过的样本,会一起交给用户,供下一轮排序。

4.5 Two-stage Optimization

4.5 两阶段优化

We assume that the candidate set could be large (n=20-30n=20\text{-}30n=20-30), but the final active set of non-zero merge coefficients will be small (see Sec. 3). We propose splitting iterations into two stages: an initial stage that searches all merging coefficients αi\alpha_{i}αi​ over A=ΔB\mathcal{A}=\Delta_{B}A=ΔB​ to encourage sparse, meaningful combinations, followed by a final, polishing stage that fixes the sparsity pattern of α\bm{\alpha}α to admit only a few non-zero entries (z<nz<nz<n) and optimizes over A=[0,1]z\mathcal{A}=[0,1]^{z}A=[0,1]z for unconstrained precision and exploration of this now limited space. After T1T_{1}T1​ initial-stage iterations, we extract the sparsity pattern of the current top-ranked α\bm{\alpha}α, then proceed with T2T_{2}T2​ final-stage iterations (we set T1=10,T2=9T_{1}=10,T_{2}=9T1​=10,T2​=9 for all experiments). The final-stage starts the entire process over, except we retain any existing samples with same zzz active coefficients. The initial stage tends to converge to a moderately successful solution with stable sparsity pattern, then the final stage further boosts performance (inset).

Inset: two-stage optimization curve.

我们假定候选集可能较大(n=20n=20n=20–303030),但最终非零合并系数的活跃集合会很小(见第 3 节)。我们建议把迭代分成两个阶段:初始阶段在 A=ΔB\mathcal{A}=\Delta_{B}A=ΔB​ 上搜索所有合并系数 αi\alpha_{i}αi​,以鼓励稀疏而有意义的组合;随后进入最后的精修阶段,把 α\bm{\alpha}α 的稀疏模式固定下来,只允许少数非零分量(z<nz<nz<n),并在 A=[0,1]z\mathcal{A}=[0,1]^{z}A=[0,1]z 上做不受约束的精细优化和探索。经过 T1T_{1}T1​ 轮初始阶段迭代后,我们提取当前排在最前的 α\bm{\alpha}α 的稀疏模式,再进行 T2T_{2}T2​ 轮最终阶段迭代(全部实验中取 T1=10T_{1}=10T1​=10,T2=9T_{2}=9T2​=9)。最终阶段会把整个过程重新来过,不过会保留那些活跃系数个数同为 zzz 的已有样本。初始阶段往往收敛到一个中等成功、稀疏模式已经稳定的解,最终阶段再进一步提升表现(见插图)。

4.6 Content control

4.6 内容控制

So far, we have described a human-in-the-loop preferential Bayesian optimization that places no requirements on the image generator ggg. However, style-content entanglement is a well-known challenge in image stylization, and fine-tuning-based methods are no exception: small parameter changes can unexpectedly alter content and confuse the user. To address this, we apply lightweight content control using SDEdit [Meng et al. 2021], with a reference image generated by the base model (no adapters), preserving content at minimal extra cost so users can focus on style. Compared to other methods [Zhang et al. 2023; Jones et al. 2024], SDEdit better preserves style, achieving a style similarity [Somepalli et al. 2024] of 0.77 (1 for a perfect match) at control strength 0.2 in our default setting (see supplemental).

到目前为止,我们描述的是一种有人在环的优先贝叶斯优化,它对图像生成器 ggg 没有任何要求。不过,风格与内容纠缠是图像风格化中的老问题,基于微调的方法也不例外:很小的参数变化就可能意外改变内容,让用户感到困惑。为缓解这一点,我们用 SDEdit [Meng et al. 2021] 做轻量的内容控制,参考图像由基底模型(不加适配器)生成,这样能以很小的额外代价保持内容,让用户把注意力放在风格上。与其他方法 [Zhang et al. 2023; Jones et al. 2024] 相比,SDEdit 能更好地保留风格:在我们的默认设定下,控制强度为 0.2 时,风格相似度 [Somepalli et al. 2024] 达到 0.77(完全匹配为 1)(见补充材料)。

4.7 Implementation Details

4.7 实现细节

We implemented the backend of GimmBO in python using BoTorch. We defer to our supplemental for detailed hyperparameter values.

GimmBO 的后端我们用 Python 和 BoTorch 实现。更详细的超参数取值放到补充材料中。

User Interface. Our frontend user interface is a web application (see inset screenshot). To reduce user fatigue, NNN generated images are presented to the user pre-sorted by corresponding surrogate estimate, and the user is only asked to identify and sort the top k≤Nk\leq Nk≤N. Whichever image is currently selected is shown in full resolution.

Inset: GimmBO web interface for top-k ranking.

用户界面。前端用户界面是一个 web 应用(见插图截图)。为减轻用户疲劳,呈现给用户的 NNN 张生成图像会先按对应的代理估计预排序,用户只需识别并排出前 k≤Nk\leq Nk≤N 名。当前选中的图像会以全分辨率显示。

5 Evaluations

5 评估

We evaluate our approach through quantitative experiments with both simulated and real users (Sec. 5.1) and further demonstrate several extensions (Sec. 5.2).

我们通过模拟用户和真实用户的定量实验评估这一方法(第 5.1 节),并进一步展示若干扩展(第 5.2 节)。

5.1 Quantitative Experiments

5.1 定量实验

We evaluate using a matching task, a standard protocol for assessing human-in-the-loop BO methods [Brochu et al. 2010; Koyama et al. 2017; Koyama et al. 2020]. The task measures how efficiently a method recovers a target solution under limited evaluation budgets, approximating practical scenarios where users have a clear goal, while enabling quantitative evaluation via known ground truth (GT).

我们采用匹配任务来评估,这是考查有人在环 BO 方法的标准协议 [Brochu et al. 2010; Koyama et al. 2017; Koyama et al. 2020]。该任务衡量在有限评估预算下,一种方法多快能找回目标解,既近似用户目标明确的实际场景,又因为有已知的真实答案(GT)而可以做定量评价。

Given an adapter collection of size nnn, test prompts {c(1),…,c(m)}\{c^{(1)},\dots,c^{(m)}\}{c(1),…,c(m)}, and corresponding GT coefficients {αGT(1),…,αGT(m)}\{\bm{\alpha}_{GT}^{(1)},\dots,\bm{\alpha}_{GT}^{(m)}\}{αGT(1)​,…,αGT(m)​}, the task is

给定大小为 nnn 的适配器集合、测试提示 {c(1),…,c(m)}\{c^{(1)},\dots,c^{(m)}\}{c(1),…,c(m)},以及对应的真实系数 {αGT(1),…,αGT(m)}\{\bm{\alpha}_{GT}^{(1)},\dots,\bm{\alpha}_{GT}^{(m)}\}{αGT(1)​,…,αGT(m)​},任务为

max⁡α∈Af(g(α,c(i)),IGT(i)),\displaystyle\max_{\bm{\alpha}\in\mathcal{A}}f(g(\bm{\alpha},c^{(i)}),\mathbf{I}_{GT}^{(i)}),α∈Amax​f(g(α,c(i)),IGT(i)​),

where IGT(i)=g(αGT(i),c(i))\mathbf{I}_{GT}^{(i)}=g(\bm{\alpha}_{GT}^{(i)},c^{(i)})IGT(i)​=g(αGT(i)​,c(i)) is not observed by our method and is available only to the simulated user. We allow 5 initial inferences for initialization, followed by 20 optimization iterations with 8 inferences per iteration, reflecting a realistic inference and interaction budget for human-in-the-loop settings.

其中 IGT(i)=g(αGT(i),c(i))\mathbf{I}_{GT}^{(i)}=g(\bm{\alpha}_{GT}^{(i)},c^{(i)})IGT(i)​=g(αGT(i)​,c(i)) 对我们的方法不可见,只提供给模拟用户。我们允许 5 次推理用于初始化,随后进行 20 轮优化、每轮 8 次推理,以反映有人在环设定下较为现实的推理与交互预算。

Fig. 7.

Fig. 7. Running-best similarity curves averaged over 30 prompt-weight combinations and 5 random seeds, together with success rates and sparsity recovery (F1; black horizontal lines indicate the median) of the final best result. Success rates are first reported overall and then broken down by the ground-truth number of active adapters zzz, measured at a similarity threshold of 0.9 (solid bars), with darker outlines indicating cases exceeding 0.95. Curves are shown for two individual test inputs, with images corresponding to the highlighted iterations (orange circles). CD: coordinate descent; DD: directional descent.

图 7. 在 30 组提示–权重组合、5 个随机种子上平均的运行最优相似度曲线,以及最终最优结果的成功率和稀疏模式恢复(F1;黑色横线表示中位数)。成功率先给出总体,再按真实活跃适配器个数 zzz 分组;以相似度阈值 0.9 计量(实心柱),更深的描边表示超过 0.95 的情形。还给出两个单独测试输入的曲线,并配上高亮迭代(橙色圆圈)对应的图像。CD:坐标下降;DD:方向下降。

Experiment setup. We construct an adapter collection, a set of test prompts, and corresponding target coefficients. The adapter collection is assembled from popular LoRA adapters on Civitai, manually selected to cover diverse styles and to produce images consistent with their advertised behavior under our local inference implementation. While GimmBO supports general fine-tuned models, including full-weight fine-tunes and other parameter-efficient adaptations, we use LoRAs due to their widespread adoption and straightforward integration. Our primary simulated and real-user experiments use collections of size n=20n=20n=20. To assess scalability, we additionally conduct simulated stress tests with n=30n=30n=30 and 404040. We evaluate m=30m=30m=30 prompts (five runs per prompt with different random seeds) of the form “a drawing of ” (or “a portrait of ” for person nouns), with nouns drawn from CIFAR-10 and CIFAR-100 super-classes. Target coefficients are sampled to match the empirical distribution observed in real usage statistics (Sec. 3); further details are provided in the supplemental.

实验设置。我们构造了一个适配器集合、一组测试提示,以及相应的目标系数。适配器集合来自 Civitai 上较流行的 LoRA,经人工挑选以覆盖多样风格,并在我们的本地推理实现下产生与其宣称行为相符的图像。虽然 GimmBO 支持一般的微调模型,包括全权重微调和其他参数高效适配,我们使用 LoRA 是因为它们采用广泛、接入也更直接。主要的模拟用户和真实用户实验使用 n=20n=20n=20 的集合。为考查可扩展性,我们另外做了 n=30n=30n=30 和 404040 的模拟压力测试。我们评估 m=30m=30m=30 条形如 “a drawing of ” 的提示(人物名词则用 “a portrait of ”),每条提示用不同随机种子跑五次,名词取自 CIFAR-10 和 CIFAR-100 的超类。目标系数按真实使用统计中观察到的经验分布采样(第 3 节);更多细节见补充材料。

Comparisons. We compare against both line-search- and BO-based approaches. For line search, we evaluate cyclic and random coordinate descent, mimicking the slider-based workflow where one coefficient is adjusted at a time, as well as random directional descent, which samples a random slider direction. Each slider spans the full coefficient hypercube, and line-search baselines use evenly spaced samples (including endpoints), as common line-search algorithms (e.g., Brent’s method) probe primarily near initial points under a limited budget of eight evaluations and often fail to escape local minima. The two prior general BO methods rely on oracle local search via a slider [Koyama et al. 2017] and on a design gallery [Koyama et al. 2020], which we set to use 3×3 grids (no zoom-ins) matching our budget of 8 inferences per iteration. We note that the gallery approach comprises both an optimization scheme and a dedicated interface design, which can offer benefits such as reduced cognitive load through structured layouts and zoom-in capabilities for detailed inspection. In our experiments, the zoom-in functionality is disabled to match a fixed, limited inference budget across conditions, which may diminish some of the advantages of the original interface design. Therefore, the results should be interpreted as a comparison of optimization schemes under a controlled interaction budget, rather than a full system-level comparison.

比较对象。我们同时与基于线搜索和基于 BO 的方法比较。对线搜索,我们评估循环与随机坐标下降,以模仿一次只调一个系数的滑条工作流,以及随机方向下降,即随机采样一个滑条方向。每条滑条覆盖整个系数超立方体;线搜索基线采用包含端点的均匀取样,因为常见的线搜索算法(如 Brent 方法)在只有八次评估的有限预算下,主要在初始点附近探测,往往难以跳出局部极小。先前两种一般 BO 方法,分别依赖借助滑条的 oracle 局部搜索 [Koyama et al. 2017],以及设计画廊 [Koyama et al. 2020];后者我们设为使用 3×33\times 33×3 网格(不允许放大),以匹配每轮 8 次推理的预算。需要说明的是,画廊方法既包含一套优化方案,也包含专门的界面设计,结构化布局和放大查看细部等能力有助于降低认知负担。在我们的实验中,为了在各条件下匹配固定、有限的推理预算,放大功能被关闭,这可能会削弱原界面设计的部分优势。因此,结果应理解为在受控交互预算下对优化方案的比较,而不是完整的系统级比较。

Performance metrics. We evaluate result quality using DreamSim [Fu et al. 2023], a perception-aligned image similarity model. Given an image output g(α)g(\bm{\alpha})g(α) and a target image IGT\mathbf{I}_{GT}IGT​, we define the objective as f(g(α),IGT)=DreamSim(g(α),IGT)f(g(\bm{\alpha}),\mathbf{I}_{GT})=\text{DreamSim}(g(\bm{\alpha}),\mathbf{I}_{GT})f(g(α),IGT​)=DreamSim(g(α),IGT​), normalized to [0,1][0,1][0,1] (111 indicates an exact match). As shown in Fig. 7, DreamSim aligns well with human judgments; empirically, scores of 0.90.90.9 and 0.950.950.95 correspond to rough and near-exact matching, respectively. Additionally, we evaluate recovery of the GT active adapters using F1, which penalizes both missing and redundant adapters. This metric complements image-based similarity by capturing whether high-quality results are achieved using the correct adapter subset. Qualitative examples are in the supplemental material.

评价指标。我们用与感知对齐的图像相似度模型 DreamSim [Fu et al. 2023] 评价结果质量。给定输出图像 g(α)g(\bm{\alpha})g(α) 和目标图像 IGT\mathbf{I}_{GT}IGT​,目标函数定义为 f(g(α),IGT)=DreamSim(g(α),IGT)f(g(\bm{\alpha}),\mathbf{I}_{GT})=\text{DreamSim}(g(\bm{\alpha}),\mathbf{I}_{GT})f(g(α),IGT​)=DreamSim(g(α),IGT​),并归一化到 [0,1][0,1][0,1](111 表示完全匹配)。如图 7 所示,DreamSim 与人类判断相当一致;经验上,0.9 和 0.95 分别对应大致匹配和接近精确匹配。此外,我们用 F1 评价对 GT 活跃适配器的恢复,既惩罚漏掉的适配器,也惩罚多余的适配器。这一指标是对基于图像的相似度的补充,用来衡量高质量结果是否是靠正确的适配器子集得到的。定性例子见补充材料。

5.1.1 Simulated user experiments. We simulate human pairwise preference feedback using DreamSim, defining αi≻αj\bm{\alpha}_{i}\succ\bm{\alpha}_{j}αi​≻αj​ whenever DreamSim(g(αi),IGT)>DreamSim(g(αj),IGT)\text{DreamSim}(g(\bm{\alpha}_{i}),\mathbf{I}_{GT})>\text{DreamSim}(g(\bm{\alpha}_{j}),\mathbf{I}_{GT})DreamSim(g(αi​),IGT​)>DreamSim(g(αj​),IGT​). The target image IGT\mathbf{I}_{GT}IGT​ is only observed by the simulated user, not by GimmBO.

5.1.1 模拟用户实验。我们用 DreamSim 模拟人类的成对偏好反馈:只要 DreamSim(g(αi),IGT)>DreamSim(g(αj),IGT)\text{DreamSim}(g(\bm{\alpha}_{i}),\mathbf{I}_{GT})>\text{DreamSim}(g(\bm{\alpha}_{j}),\mathbf{I}_{GT})DreamSim(g(αi​),IGT​)>DreamSim(g(αj​),IGT​),就记 αi≻αj\bm{\alpha}_{i}\succ\bm{\alpha}_{j}αi​≻αj​。目标图像 IGT\mathbf{I}_{GT}IGT​ 只对模拟用户可见,不对 GimmBO 可见。

We report running-best similarities across iterations, together with success rates and sparsity recovery (F1) of the final best result (Fig. 7). We compare three classes of methods: Batch BO-based includes our method (q=8q{=}8q=8) and Sequential Gallery [Koyama et al. 2020], which can be viewed as a small-batch BO variant (q=2q{=}2q=2); Coordinate-descent updates one adapter at a time, including cyclic and random coordinate descent; Direction-descent includes random direction descent and the BO-powered Sequential Slider [Koyama et al. 2017], which operates on a single sample per iteration. Direction-descent methods struggle in high-dimensional spaces due to the low probability of sampling informative directions, while coordinate-descent methods achieve moderate gains by explicitly activating or deactivating individual adapters but struggle as dimensionality increases. Batch BO-based methods achieve substantially better performance, demonstrating the benefits of informed exploration. GimmBO achieves rapid Stage 1 gains via the capped simplex constraint (Sec. 4.2), with further improvement at the onset of Stage 2. As the GT number of active adapters zzz increases, success rates decline, reflecting the greater difficulty of recovering more complex merges; yet GimmBO consistently achieves the highest sparsity recovery, with many cases reaching F1=1\text{F1}{=}1F1=1. Note that the two-stage strategy may miss the global optimum (i.e., F1≠1\text{F1}\neq 1F1=1) if Stage 1 fails to include all true active adapters. Finally, in simulated stress tests at higher dimensionalities (30D–40D), GimmBO degrades gracefully and consistently outperforms the comparison (Fig. 8).

我们报告各迭代的运行最优相似度,以及最终最优结果的成功率和稀疏恢复(F1)(图 7)。比较分为三类:基于批次 BO 的包括我们的方法(q=8q{=}8q=8)和 Sequential Gallery [Koyama et al. 2020],后者可看成小批次 BO 变体(q=2q{=}2q=2);坐标下降 一次更新一个适配器,包括循环与随机坐标下降;方向下降 包括随机方向下降,以及每轮只处理一个样本的、由 BO 驱动的 Sequential Slider [Koyama et al. 2017]。方向下降在高维空间中很吃力,因为采到有信息的方向概率很低;坐标下降通过显式激活或关闭单个适配器能获得中等收益,但随着维数升高也会吃力。基于批次 BO 的方法表现明显更好,说明有信息的探索是有益的。GimmBO 借助封顶单纯形约束(第 4.2 节)在第 1 阶段就快速提升,并在第 2 阶段开始时进一步改善。随着 GT 活跃适配器个数 zzz 增大,成功率下降,反映出更复杂的合并更难恢复;但 GimmBO 始终取得最高的稀疏恢复,许多情形下 F1=1\text{F1}{=}1F1=1。需要注意,若第 1 阶段没能包含全部真正的活跃适配器,两阶段策略可能错过全局最优(即 F1≠1\text{F1}\neq 1F1=1)。最后,在更高维(30–40 维)的模拟压力测试中,GimmBO 的退化较平缓,并始终优于比较方法(图 8)。

Fig. 8.

Fig. 8. Left: running-best similarity curves under increasing dimensionality (30D and 40D), comparing GimmBO to Sequential Gallery [Koyama et al. 2020]; GimmBO degrades gracefully and consistently outperforms Sequential Gallery. Right: ablations in our default 20D setting, varying the simplex constraint B∈{1,2,n}B\in\{1, 2, n\}B∈{1,2,n} (B=nB=nB=n means unconstrained), top-kkk selection (k=1k=1k=1 vs. 5), and inclusion of two past samples. The default configuration (B=2B=2B=2, top-5, +past) performs best. Importantly, even with a matched interaction budget (B=2B=2B=2, top-1, −-−past), GimmBO substantially outperforms Sequential Gallery.

图 8. 左:在维数升高(30 维和 40 维)时的运行最优相似度曲线,比较 GimmBO 与 Sequential Gallery [Koyama et al. 2020];GimmBO 退化平缓,并始终优于 Sequential Gallery。右:在默认 20 维设定下的消融,分别改变单纯形约束 B∈{1,2,n}B\in\{1, 2, n\}B∈{1,2,n}(B=nB=nB=n 表示不加约束)、top-kkk 选择(k=1k=1k=1 与 5),以及是否纳入两个历史样本。默认配置(B=2B=2B=2,top-5,+past)表现最好。重要的是,即便把交互预算对齐(B=2B=2B=2,top-1,−-−past),GimmBO 仍明显优于 Sequential Gallery。

Fig. 9.

Fig. 9. Running-best similarities averaged over 12 input-user combinations. For the slider interface, both running-best (solid) and explored intermediate results (dashed) are shown. We report success rates (>0.9>0.9>0.9 in solid and >0.95>0.95>0.95 outlined), sparsity recovery (F1; black lines indicate the median), and the number of active adapters, with ground truth indicated.

图 9. 在 12 组输入–用户组合上平均的运行最优相似度。对滑条界面,同时给出运行最优(实线)和实际探索过的中间结果(虚线)。我们报告成功率(>0.9>0.9>0.9 为实心,>0.95>0.95>0.95 为描边)、稀疏恢复(F1;黑线为中位数),以及活跃适配器个数,并标出真实值。

5.1.2 User study. We conduct a user study to evaluate GimmBO in a controlled human-in-the-loop setting. Participants perform the same matching task using a subset of our 20D simulated experiment test inputs and the same adapter collection. We compare three interaction interfaces: (1) a slider-based interface commonly used in current adapter-merging workflows, (2) a gallery-based interface implementing Sequential Gallery (gallery for short), and (3) our proposed top-kkk ranking interface. The slider interface serves as a more realistic baseline than coordinate-descent approaches, offering richer interactions (e.g., per-adapter examples, multi-slider updates, and history); it is initialized at α=0\bm{\alpha}=\mathbf{0}α=0, with performance evaluated from the first user-modified result. After a tutorial and practice session, participants complete all three interface conditions for 20 iterations, using different inputs and with order counterbalanced across participants (study details in the supplemental).

5.1.2 用户研究。我们进行一项用户研究,在受控的有人在环设定下评估 GimmBO。参与者完成同样的匹配任务,测试输入取自我们 20 维模拟实验的一个子集,适配器集合也相同。我们比较三种交互界面:(1)当前适配器合并工作流中常用的滑条界面,(2)实现 Sequential Gallery 的画廊界面(简称画廊),以及(3)我们提出的 top-kkk 排序界面。滑条界面作为比坐标下降更贴近现实的基线,提供更丰富的交互(例如每个适配器的示例、同时调节多个滑条,以及历史记录);它从 α=0\bm{\alpha}=\mathbf{0}α=0 初始化,表现从用户第一次改动后的结果开始计算。在教程和练习之后,参与者在全部三种界面条件下各完成 20 轮,输入不同,顺序在参与者之间做了平衡(研究细节见补充材料)。

We recruit 12 participants (9 male and 3 female; 8 aged 25-34 and 4 aged 18-24) and record per-step user interactions, analyzed using the same DreamSim metrics (Fig. 9). We report similarity curves averaged across users and target inputs. For gallery and GimmBO, curves reflect images explicitly selected by users, while for slider we report the running-best results (solid), which serve as an upper bound; the dashed curve shows the actual intermediate explorations and exhibits fluctuations, indicating difficulty in anticipating slider effects. All methods show a statistically significant increase in similarity by the end of the session (gallery p<0.05p<0.05p<0.05, slider p<0.05p<0.05p<0.05, ours p<10−3p<10^{-3}p<10−3), with our method achieving the highest performance (>0.9>0.9>0.9 on average). Consistent with simulation, our method achieves higher success rates and F1, accurately recovering the GT adapter set, while baselines activate redundant adapters. Performance typically saturates by ∼\sim∼15, suggesting the potential to reduce user fatigue with fewer iterations. Accounting for user interaction time only, and excluding image generation latency (∼\sim∼10–15 s per batch), the mean interaction time per step is 10.6 s (gallery), 34.7 s (slider), and 50.5 s (ours). To match user interaction time budgets, we take the total interaction time of 20 slider or gallery steps and truncate GimmBO’s trajectory to the number of steps that fit within the same budget, which is approximately 14 (slider) and 4 (gallery) steps. Under this setup, our method outperforms slider with statistical significance (p<0.05p{<}0.05p<0.05) and achieves higher median similarity than gallery, though the difference is not statistically significant (p=0.397p{=}0.397p=0.397), as the matched budget corresponds to an early BO stage. In practice, iteration time is often dominated by model inference rather than user interaction, motivating our focus on iteration budgets. Including inference time in the comparison (15 s per iteration in our setup), our method outperforms both baselines with statistical significance (p<0.05p{<}0.05p<0.05). This advantage is expected to be larger in more realistic settings with limited compute resources (e.g., 1-2 local GPUs or online job queues), where inference can easily exceed one minute per iteration: with 60s inference, our method achieves higher median similarity (0.956 vs. 0.898 for slider; 0.926 vs. 0.854 for gallery), with statistical significance. See Sec. 6 for a further discussion about interaction time.

我们招募了 12 名参与者(9 男 3 女;8 人 25–34 岁,4 人 18–24 岁),记录每一步的用户交互,并用同样的 DreamSim 指标分析(图 9)。我们报告在用户和目标输入上平均的相似度曲线。对画廊和 GimmBO,曲线对应的是用户明确选出的图像;对滑条,我们报告运行最优结果(实线),作为上界;虚线则给出实际的中间探索,波动较大,说明滑条效果并不容易预判。到会话结束时,所有方法的相似度都有统计显著的提升(画廊 p<0.05p<0.05p<0.05,滑条 p<0.05p<0.05p<0.05,我们的方法 p<10−3p<10^{-3}p<10−3),其中我们的方法表现最好(平均 >0.9>0.9>0.9)。与模拟结果一致,我们的方法成功率和 F1 更高,能够准确找回 GT 适配器集合,而基线会激活多余的适配器。表现通常在大约 15 轮时趋于饱和,说明减少轮数有望降低用户疲劳。若只计用户交互时间、不计图像生成延迟(每批约 10–15 秒),每步平均交互时间为:画廊 10.6 秒,滑条 34.7 秒,我们的方法 50.5 秒。为了对齐用户交互时间预算,我们取 20 步滑条或画廊的总交互时间,把 GimmBO 的轨迹截到能装进同一预算的步数,大约分别是 14 步(对滑条)和 4 步(对画廊)。在这一设定下,我们的方法以统计显著的优势优于滑条(p<0.05p{<}0.05p<0.05),中位相似度也高于画廊,但差异不具统计显著性(p=0.397p{=}0.397p=0.397),因为对齐后的预算只对应 BO 的较早阶段。实践中,每轮时间往往由模型推理而不是用户交互主导,这也是我们更关注迭代预算的原因。若把推理时间一并计入比较(在我们的设定下每轮 15 秒),我们的方法以统计显著的优势优于两个基线(p<0.05p{<}0.05p<0.05)。在计算资源更紧的现实设定中(例如 1–2 块本地 GPU,或在线排队),这一优势预计会更大,因为推理很容易每轮超过一分钟:当推理为 60 秒时,我们的方法中位相似度更高(0.956 对滑条的 0.898;0.926 对画廊的 0.854),且具有统计显著性。关于交互时间的更多讨论见第 6 节。

Fig. 10.

Fig. 10. Given a text prompt, 20-25 community-shared adapters are gathered via coarse text-based retrieval of Stylus [Luo et al. 2024]. Three methods are applied to determine the adapter merging configurations: trivial averaging all retrieved adapters, Stylus refinement, and GimmBO. While averaging often introduces artifacts and Stylus-picked configurations yield a single fixed result, our method produces multiple, diverse, prompt-aligned outcomes per prompt through user-guided exploration (each pair starts from the same initialization and differs only due to user interactions). Examples are generated by SD1.5.

图 10. 给定文本提示后,通过 Stylus [Luo et al. 2024] 的粗粒度文本检索,收集 20–25 个社区分享的适配器。再用三种方法确定适配器合并配置:对检索到的适配器做简单平均、Stylus 精修,以及 GimmBO。平均常常引入伪影,Stylus 选出的配置往往只给出一个固定结果;我们的方法则通过用户引导的探索,为每条提示产生多种、多样且与提示对齐的结果(每一对从同一初始化出发,差异只来自用户交互)。示例由 SD1.5 生成。

5.1.3 Ablations. We ablate using our simulated 20D matching task by varying three key factors: the simplex constraint parameter B∈{1,2,n}B\in\{1,2,n\}B∈{1,2,n} (nnn means simplex constraint off), top-kkk selection (k=1,5k=1,5k=1,5), and inclusion of two past samples (+++past vs. −-−past) (Fig. 8, right). Our default configuration (B=2B=2B=2, top-5, +++past) performs best overall. Specifically, the B-capped simplex constraint (B<nB<nB<n) improves performance, top-5 selection outperforms top-1 selection, and incorporating past samples provides additional gains. Importantly, even with a matched interaction budget to Sequential Gallery (B=2B=2B=2, top-1, −-−past), GimmBO substantially outperforms, with gains attributable to method design rather than increased user effort.

5.1.3 消融。我们在模拟的 20 维匹配任务上做消融,改变三个关键因素:单纯形约束参数 B∈{1,2,n}B\in\{1,2,n\}B∈{1,2,n}(nnn 表示关掉单纯形约束)、top-kkk 选择(k=1,5k=1,5k=1,5),以及是否纳入两个历史样本(+++past 与 −-−past)(图 8 右)。默认配置(B=2B=2B=2,top-5,+++past)总体最好。具体来说,BBB-封顶单纯形约束(B<nB<nB<n)能提高表现,top-5 优于 top-1,纳入历史样本还能带来额外收益。重要的是,即便把交互预算与 Sequential Gallery 对齐(B=2B=2B=2,top-1,−-−past),GimmBO 仍然明显更好,说明增益来自方法设计,而不是用户更费力。

Fig. 11.

Fig. 11. Applications of adapter merging discovered with GimmBO. Disclaimer: Image generators and style adapters may not preserve facial identities or perceived personal characteristics (e.g., race, gender); adapter merging configurations found via GimmBO are no exception.

图 11. 用 GimmBO 发现的适配器合并在应用中的结果。免责声明:图像生成器和风格适配器未必能保持面部身份或可被感知的个人特征(例如种族、性别);经 GimmBO 找到的适配器合并配置同样如此。

5.2 Extensions

5.2 扩展

We explore several extensions of GimmBO. All qualitative results demonstrating visual diversity start from the same initialization and differ only through user interactions (20 rounds).

我们探索了 GimmBO 的若干扩展。所有展示视觉多样性的定性结果,都从同一初始化出发,差异只来自用户交互(20 轮)。

Integration with coarse retrieval. We integrate GimmBO with Stylus’s [Luo et al. 2024] coarse adapter retrieval (Fig. 10). Given a text prompt, we configure Stylus to retrieve 25 candidate adapters via keyword retrieval, of which 19–23 remain after filtering. We test prompts containing both content and style attributes. Because Stylus only releases precomputed retrieval embeddings for Stable Diffusion 1.5 (SD1.5), we conduct this experiment on SD1.5, additionally demonstrating generalization beyond SDXL. We compare against two baselines: uniformly averaging retrieved adapters and Stylus-refined coefficients. Uniform averaging often introduces artifacts due to irrelevant or mis-retrieved adapters (e.g., goldfish for the octopus prompt), while Stylus refinement yields a single fixed solution. In contrast, even in this challenging setting where SD1.5 is less expressive and community-shared adapters vary widely in quality, GimmBO enables effective navigation of the retrieved adapter space, producing diverse, high-quality, and prompt-aligned results.

与粗检索结合。我们把 GimmBO 接到 Stylus [Luo et al. 2024] 的粗粒度适配器检索上(图 10)。给定文本提示,我们配置 Stylus 用关键词检索 25 个候选适配器,过滤后通常剩下 19–23 个。测试提示同时包含内容和风格属性。由于 Stylus 只公开了 Stable Diffusion 1.5(SD1.5)的预计算检索嵌入,本实验在 SD1.5 上进行,也额外说明了方法可以推广到 SDXL 之外。我们与两个基线比较:对检索到的适配器做均匀平均,以及 Stylus 精修后的系数。均匀平均往往会因不相关或检索错误的适配器而引入伪影(例如章鱼提示下出现金鱼),Stylus 精修则只给出一个固定解。相比之下,即便在 SD1.5 表达力较弱、社区分享的适配器质量参差不齐的困难设定中,GimmBO 仍能有效探索检索到的适配器空间,产生多样、高质量且与提示对齐的结果。

Content merging. We explore assembling and merging external appearance attributes, including hair, glasses, and expressions, for face generation (Fig. 11(a)). Using a 25D adapter set, we verify that our method generalizes from style to content through a small quantitative matching experiment on three faces with five seeds each, achieving a success rate of 67%. We further present qualitative results demonstrating diverse attribute combinations for two faces.

内容合并。我们探索为人脸生成组装并合并外部外观属性,包括发型、眼镜和表情(图 11(a))。使用 25 维适配器集合,我们用一个小型定量匹配实验,在三张人脸、每种五个种子上验证方法能从风格推广到内容,成功率为 67%。我们还给出定性结果,展示两张人脸上多样的属性组合。

Downstream applications. Beyond single image generation, an adapter merging configuration can be reused to consistently stylize new images. In a downstream content-creation scenario, users identify a desired configuration with GimmBO and apply it to generate related visual assets by using input images as SDEdit control inputs.

下游应用。除了单张图像生成,一组适配器合并配置还可以被复用,以一致地风格化新图像。在下游的内容创作场景中,用户用 GimmBO 找到想要的配置,再把输入图像当作 SDEdit 的控制输入,生成相关的视觉资产。

6 Limitations and Future Work

6 局限与未来工作

We focus on linear model merging (Eq. 1) for evaluation, though GimmBO does not rely on linearity and can apply to other merging methods [Matena and Raffel 2021; Yadav et al. 2023; Zheng and Wang 2024], left as future work. It would be interesting to examine how often preference transitivity is violated in our setting [cf. Tversky and Kahneman 1981] and how to address such cases if necessary. The Dirichlet process used for acquisition sampling is fast and continuous, but this reparameterization introduces a known coordinate-ordering bias. While alternatives such as uniform sampling the simplex [Jacobson 2023] and projection-based methods [Wang and Lu 2015] avoid this bias, our initial experiments suggest that it may be serendipitously beneficial in our setting. Understanding this effect more systematically is left for future work. DreamSim, used for our simulated user and performance metric, measures perceptual similarity rather than style specifically, and may conflate content and style. We mitigate this with content control and complementary metrics, and future work could explore more style-specific evaluation measures.

评估中我们聚焦线性模型合并(公式 1),但 GimmBO 并不依赖线性,也可以用于其他合并方法 [Matena and Raffel 2021; Yadav et al. 2023; Zheng and Wang 2024],这一点留作未来工作。我们设定中偏好传递性被违反的频率有多高 [cf. Tversky and Kahneman 1981],以及必要时如何处理,也值得考察。用于采集采样的 Dirichlet 过程又快又连续,但这种重新参数化会引入已知的坐标顺序偏差。均匀采样单纯形 [Jacobson 2023] 和基于投影的方法 [Wang and Lu 2015] 等替代方案可以避开这一偏差,不过我们的初步实验提示,它在我们的设定里或许反而有益。更系统地理解这一效应,留待未来工作。DreamSim 被用作模拟用户和评价指标,衡量的是感知相似度而不是专门的风格相似度,因此可能把内容和风格混在一起。我们用内容控制和互补指标来缓解这一点,未来工作也可以探索更针对风格的评价方式。

GimmBO with default hyperparameters incurs noticeably more user interaction time than the alternatives (Sec. 5.1.2). Interaction effort can be reduced by adjusting interface parameters such as using a smaller top-kkk and disabling past samples, which incurs only a small similarity drop while still outperforming baselines in matched-budget settings (Sec. 5.1.3). Additional real-user studies to evaluate these trade-offs and explore lower-interaction paradigms are left for future work. When users have limited access to compute (e.g., commercial cloud settings), per-iteration image generation will take significantly longer than user ranking. This invites extensions of our work to hide latency with anticipatory system designs.

使用默认超参数时,GimmBO 的用户交互时间明显高于其他方法(第 5.1.2 节)。交互负担可以通过调整界面参数来减轻,例如减小 top-kkk、关闭历史样本,这样只会带来很小的相似度下降,并在对齐预算的设定下仍优于基线(第 5.1.3 节)。用更多真实用户研究来评估这些权衡、探索更低交互的范式,留作未来工作。当用户可用的算力有限时(例如商业云环境),每轮图像生成会明显长于用户排序。这提示可以把工作扩展到用前瞻式的系统设计来隐藏延迟。

Beyond adapter merging, it remains an open question how our formulation extends to other increasingly popular customization regimes such as reference-image conditioning [Ye et al. 2023; Wang et al. 2024; Wang et al. 2025] and style-injection pipelines [Patashnik et al. 2025]. These approaches offer alternative mechanisms for controlling style, often operating in latent or feature spaces rather than explicit parameter combinations. It is unclear which aspects of our method, including preference-based search, sparsity structure, and staged exploration, would transfer directly. Investigating such generalizations is a promising direction for future work.

在适配器合并之外,我们的形式如何扩展到其他日益流行的定制方式,例如参考图像条件控制 [Ye et al. 2023; Wang et al. 2024; Wang et al. 2025] 和风格注入管线 [Patashnik et al. 2025],仍然是一个开放问题。这些方法提供了控制风格的另一类机制,往往在潜在空间或特征空间中操作,而不是显式组合参数。我们方法中的哪些部分,包括基于偏好的搜索、稀疏结构以及分阶段探索,能够直接迁移,目前并不清楚。研究这类推广,是一个有前景的未来方向。

Reinforcement learning from human feedback (RLHF) has been explored for improving generative models, including diffusion models, but typically relies on large-scale feedback datasets [Yang et al. 2024] or automatic reward models [Miao et al. 2024]. This differs from our setting: a single user with a limited interaction budget, where data efficiency is critical. As a result, existing RLHF approaches are less directly applicable, though making RLHF viable in such low-feedback settings is an interesting direction for future work.

从人类反馈中强化学习(RLHF)已被用来改进生成模型,包括扩散模型,但通常依赖大规模反馈数据 [Yang et al. 2024] 或自动奖励模型 [Miao et al. 2024]。这与我们的设定不同:这里是单个用户、交互预算有限,数据效率至关重要。因此,现有 RLHF 方法较难直接套用;不过,让 RLHF 在这种低反馈设定里变得可行,会是一个有意思的未来方向。

Generative image models continue to be used in unethical and harmful ways; adapter models contribute to and possibly amplify these uses. Many of the adapters on Civitai target the sexualization of women via pose, clothing, or explicitly graphic subject matter. Using these models or GimmBO mergers of them to manipulate photos of real people without their consent is clearly unethical. It is not clear how these problems are to be solved or alleviated with further technology research.

生成图像模型仍在被用于不道德和有害的用途;适配器模型参与其中,并可能放大这些用途。Civitai 上许多适配器以姿态、服装或露骨题材为目标,对女性做性化处理。用这些模型,或用 GimmBO 对它们做合并,去未经同意地改动真人照片,显然是不道德的。这类问题能否仅靠进一步的技术研究得到解决或缓解,目前并不清楚。

A recently released proprietary and closed-source application, Style Creator[^2], offers related capabilities to GimmBO, while differing in both interaction and underlying representation. We hope that our work inspires further academic research on this problem.

最近发布的一个专有、闭源应用 Style Creator[^2],提供了与 GimmBO 相关的能力,但在交互方式和底层表示上都有所不同。我们希望这项工作能激发对这一问题的更多学术研究。

[^2]: https://docs.midjourney.com/hc/en-us/articles/41308374558221-Style-Creator, released November 20, 2025.

[^2]: https://docs.midjourney.com/hc/en-us/articles/41308374558221-Style-Creator,发布于 2025 年 11 月 20 日。

7 Acknowledgment

7 致谢

We thank Yuki Koyama for clarifications on prior work, Leping Qiu for assistance with user study design, all anonymous participants in our pilot and final studies, and the DGP members at the University of Toronto for discussions and support. Our research is funded in part by NSERC Discovery (RGPIN–2022–04680), the Ontario Early Research Award program, the Canada Research Chairs Program, a Sloan Research Fellowship, the DSI Catalyst Grant program, Arts & Science Postdoctoral Fellowship at the University of Toronto, and gifts by Adobe Inc.

我们感谢 Yuki Koyama 对先前工作的澄清,感谢 Leping Qiu 协助设计用户研究,感谢所有参与预实验和正式研究的匿名受试者,以及多伦多大学 DGP 成员的讨论与支持。本研究部分得到 NSERC Discovery(RGPIN–2022–04680)、安大略早期研究奖、加拿大研究讲席计划、Sloan Research Fellowship、DSI Catalyst Grant、多伦多大学 Arts & Science 博士后基金,以及 Adobe Inc. 赠款的资助。

References

参考文献

Adamkiewicz et al. (2025) K. Adamkiewicz, P. W. Woźniak, J. Dominiak, A. Romanowski, J. Karolus, and S. Frolov PromptMap: an alternative interaction style for ai-based image generation. In Proceedings of the 30th International Conference on Intelligent User Interfaces, pp. 1162–1176.

Alaluf et al. (2023) Y. Alaluf, E. Richardson, G. Metzer, and D. Cohen-Or A neural space-time representation for text-to-image personalization. ACM Transactions on Graphics (TOG) 42 ( 6 ), pp. 1–10.

Balandat et al. (2020) M. Balandat, B. Karrer, D. Jiang, S. Daulton, B. Letham, A. G. Wilson, and E. Bakshy BoTorch: a framework for efficient monte-carlo bayesian optimization. Advances in neural information processing systems 33, pp. 21524–21538.

Brade et al. (2023) S. Brade, B. Wang, M. Sousa, S. Oore, and T. Grossman Promptify: text-to-image generation through interactive prompt exploration with large language models. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pp. 1–14.

Brochu et al. (2010) E. Brochu, T. Brochu, and N. De Freitas A bayesian interactive optimization approach to procedural animation design. In Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp. 103–112.

Chiu et al. (2020) C. Chiu, Y. Koyama, Y. Lai, T. Igarashi, and Y. Yue Human-in-the-loop differential subspace search in high-dimensional latent space. ACM Transactions on Graphics (TOG) 39 ( 4 ), pp. 85–1.

Chu and Ghahramani (2005) W. Chu and Z. Ghahramani Preference learning with gaussian processes. In Proceedings of the 22nd international conference on Machine learning, pp. 137–144.

Dalva et al. (2025) Y. Dalva, H. Yesiltepe, and P. Yanardag LoRAShop: training-free multi-concept image generation and editing with rectified flow transformers. arXiv preprint arXiv:2505.23758.

Dang et al. (2022) H. Dang, L. Mecke, and D. Buschek Ganslider: how users control generative models for images using multiple sliders with and without feedforward information. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1–15.

Desai et al. (2019) R. Desai, F. Anderson, J. Matejka, S. Coros, J. McCann, G. Fitzmaurice, and T. Grossman Geppetto: enabling semantic design of expressive robot behaviors. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–14.

Diouane et al. (2023) Y. Diouane, V. Picheny, R. L. Riche, and A. S. D. Perrotolo TREGO: a trust-region framework for efficient global optimization. Journal of Global Optimization 86 ( 1 ), pp. 1–23.

Eriksson and Jankowiak (2021) D. Eriksson and M. Jankowiak High-dimensional bayesian optimization with sparse axis-aligned subspaces. In Uncertainty in Artificial Intelligence, pp. 493–503.

Feng et al. (2023) Y. Feng, X. Wang, K. K. Wong, S. Wang, Y. Lu, M. Zhu, B. Wang, and W. Chen Promptmagician: interactive prompt engineering for text-to-image creation. IEEE Transactions on Visualization and Computer Graphics 30 ( 1 ), pp. 295–305.

Frankle et al. (2020) J. Frankle, G. K. Dziugaite, D. Roy, and M. Carbin Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3259–3269.

Frenkel et al. (2024) Y. Frenkel, Y. Vinker, A. Shamir, and D. Cohen-Or Implicit style-content separation using b-lora. In European Conference on Computer Vision, pp. 181–198.

Fu et al. (2023) S. Fu, N. Tamir, S. Sundaram, L. Chai, R. Zhang, T. Dekel, and P. Isola DreamSim: learning new dimensions of human visual similarity using synthetic data. In Advances in Neural Information Processing Systems, Vol. 36, pp. 50742–50768.

Gadot et al. (2025) U. Gadot, R. Gal, Y. Ziser, G. Chechik, and S. Mannor Policy optimized text-to-image pipeline design. arXiv preprint arXiv:2505.21478.

Gal et al. (2022) R. Gal, Y. Alaluf, Y. Atzmon, O. Patashnik, A. H. Bermano, G. Chechik, and D. Cohen-Or An image is worth one word: personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618.

Gal et al. (2025) R. Gal, A. Haviv, Y. Alaluf, A. H. Bermano, D. Cohen-Or, and G. Chechik ComfyGen: prompt-adaptive workflows for text-to-image generation. In ICLR 2025 Workshop on Modular, Collaborative and Decentralized Deep Learning,

Gu et al. (2023) Y. Gu, X. Wang, J. Z. Wu, Y. Shi, Y. Chen, Z. Fan, W. Xiao, R. Zhao, S. Chang, W. Wu, et al. Mix-of-show: decentralized low-rank adaptation for multi-concept customization of diffusion models. Advances in Neural Information Processing Systems 36, pp. 15890–15902.

Guo et al. (2024) Y. Guo, H. Shao, C. Liu, K. Xu, and X. Yuan PrompTHis: visualizing the process and influence of prompt editing during text-to-image creation. IEEE Transactions on Visualization and Computer Graphics.

Hoffman et al. (2014) M. D. Hoffman A. Gelman et al. The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo.. J. Mach. Learn. Res. 15 ( 1 ), pp. 1593–1623.

Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. Lora: low-rank adaptation of large language models.. ICLR 1 ( 2 ), pp. 3.

Ilharco et al. (2023) G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations,

Jacobson (2023) A. Jacobson Sampling the capped simplex. Note: https://alecjacobson.com/weblog/sampling-the-capped-simplex.html Blog post. Accessed: 2026-04-30

Jones et al. (2024) M. Jones, S. Wang, N. Kumari, D. Bau, and J. Zhu Customizing text-to-image models with a single image pair. In SIGGRAPH Asia 2024 Conference Papers, pp. 1–13.

Kadner et al. (2021) F. Kadner, Y. Keller, and C. Rothkopf Adaptifont: increasing individuals’ reading speed with a generative font model and bayesian optimization. In Proceedings of the 2021 chi conference on human factors in computing systems, pp. 1–11.

Karras et al. (2022) T. Karras, M. Aittala, T. Aila, and S. Laine Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems 35, pp. 26565–26577.

Khajah et al. (2016) M. M. Khajah, B. D. Roads, R. V. Lindsey, Y. Liu, and M. C. Mozer Designing engaging games using bayesian optimization. In Proceedings of the 2016 CHI conference on human factors in computing systems, pp. 5571–5582.

Koyama and Goto (2022) Y. Koyama and M. Goto Bo as assistant: using bayesian optimization for asynchronously generating design suggestions. In Proceedings of the 35th annual ACM symposium on user interface software and technology, pp. 1–14.

Koyama et al. (2014) Y. Koyama, D. Sakamoto, and T. Igarashi Crowd-powered parameter analysis for visual design exploration. In Proceedings of the 27th annual ACM symposium on User interface software and technology, pp. 65–74.

Koyama et al. (2020) Y. Koyama, I. Sato, and M. Goto Sequential gallery for interactive visual design optimization. ACM Transactions on Graphics (TOG) 39 ( 4 ), pp. 88–1.

Koyama et al. (2017) Y. Koyama, I. Sato, D. Sakamoto, and T. Igarashi Sequential line search for efficient visual design optimization by crowds. ACM Transactions on Graphics (TOG) 36 ( 4 ), pp. 1–11.

Kumar and Rathinam (2025) R. Kumar and M. Rathinam DPM-solver-2m: a fast multistep dpm-solver-based scheme for real-time mimo channel estimation. IEEE Open Journal of the Communications Society.

Kumari et al. (2023) N. Kumari, B. Zhang, R. Zhang, E. Shechtman, and J. Zhu Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1931–1941.

Li et al. (2023) D. Li, J. Li, and S. Hoi Blip-diffusion: pre-trained subject representation for controllable text-to-image generation and editing. Advances in Neural Information Processing Systems 36, pp. 30146–30166.

Liu et al. (2024) C. Liu, T. Takikawa, and A. Jacobson A lora is worth a thousand pictures.

Liu et al. (2023) S. Liu, Q. Feng, D. Eriksson, B. Letham, and E. Bakshy Sparse bayesian optimization. In International Conference on Artificial Intelligence and Statistics, pp. 3754–3774.

Luo et al. (2024) M. Luo, J. Wong, B. Trabucco, Y. Huang, J. E. Gonzalez, Z. Chen, R. Salakhutdinov, and I. Stoica Stylus: automatic adapter selection for diffusion models. Advances in Neural Information Processing Systems 37, pp. 32888–32915.

Marichal and Mossinghoff (2006) J. Marichal and M. J. Mossinghoff Slices, slabs, and sections of the unit hypercube. arXiv preprint math/0607715.

Marks et al. (2023) J. Marks, B. Andalman, P.A. Beardsley, W. Freeman, S. Gibson, J. Hodgins, T. Kang, B. Mirtich, H. Pfister, W. Ruml, K. Ryall, J. Seims, and S. Shieber Design galleries: a general approach to setting parameters for computer graphics and animation. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2,

Matena and Raffel (2021) M. Matena and C. Raffel Merging models with fisher-weighted averaging. CoRR abs/2111.09832.

Meng et al. (2021) C. Meng, Y. He, Y. Song, J. Song, J. Wu, J. Zhu, and S. Ermon Sdedit: guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073.

Miao et al. (2024) Y. Miao, W. Loh, S. Kothawade, P. Poupart, A. Rashwan, and Y. Li Subject-driven text-to-image generation via preference-based reinforcement learning. Advances in Neural Information Processing Systems 37, pp. 123563–123591.

Mo et al. (2024) G. Mo, J. Dudley, L. Chan, Y. Liao, A. Oulasvirta, and P. O. Kristensson Cooperative multi-objective bayesian design optimization. ACM Transactions on Interactive Intelligent Systems 14 ( 2 ), pp. 1–28.

Ngan et al. (2006) A. Ngan, F. Durand, and W. Matusik Image-driven navigation of analytical brdf models.. Rendering Techniques 2006, pp. 399–407.

Niwa et al. (2025) R. Niwa, S. Yoshida, Y. Koyama, and Y. Ushiku Cooperative design optimization through natural language interaction. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology, pp. 1–25.

Ouyang et al. (2025) Z. Ouyang, Z. Li, and Q. Hou K-lora: unlocking training-free fusion of any subject and style loras. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 13041–13050.

Patashnik et al. (2025) O. Patashnik, R. Gal, D. Ostashev, S. Tulyakov, K. Aberman, and D. Cohen-Or Nested attention: semantic-aware attention values for concept personalization. In Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers, SIGGRAPH Conference Papers ’25.

Po et al. (2024) R. Po, G. Yang, K. Aberman, and G. Wetzstein Orthogonal adaptation for modular customization of diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7964–7973.

Raponi et al. (2020) E. Raponi, H. Wang, M. Bujny, S. Boria, and C. Doerr High dimensional bayesian optimization assisted by principal component analysis. In International Conference on Parallel Problem Solving from Nature, pp. 169–183.

Ruiz et al. (2023) N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman Dreambooth: fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22500–22510.

Shah et al. (2024) V. Shah, N. Ruiz, F. Cole, E. Lu, S. Lazebnik, Y. Li, and V. Jampani Ziplora: any subject in any style by effectively merging loras. In European Conference on Computer Vision, pp. 422–438.

Shapira et al. (2009) L. Shapira, A. Shamir, and D. Cohen-Or Image appearance exploration by model-based navigation. In Computer Graphics Forum, Vol. 28, pp. 629–638.

Shimizu et al. (2020) E. Shimizu, M. Fisher, S. Paris, J. McCann, and K. Fatahalian Design adjectives: a framework for interactive model-guided exploration of parameterized design spaces. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, pp. 261–278.

Shugrina et al. (2015) M. Shugrina, A. Shamir, and W. Matusik Fab forms: customizable objects for fabrication with validity and geometry caching. ACM Transactions on Graphics (TOG) 34 ( 4 ), pp. 1–12.

Sohn et al. (2023) K. Sohn, N. Ruiz, K. Lee, D. C. Chin, I. Blok, H. Chang, J. Barber, L. Jiang, G. Entis, Y. Li, Y. Hao, I. Essa, M. Rubinstein, and D. Krishnan StyleDrop: text-to-image generation in any style. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA.

Somepalli et al. (2024) G. Somepalli, A. Gupta, K. Gupta, S. Palta, M. Goldblum, J. Geiping, A. Shrivastava, and T. Goldstein Measuring style similarity in diffusion models. arXiv preprint arXiv:2404.01292.

Sonmezer et al. (2025) M. Sonmezer, M. Zheng, and P. Yanardag LoRAverse: a submodular framework to retrieve diverse adapters for diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17879–17888.

Srinivas et al. (2009) N. Srinivas, A. Krause, S. M. Kakade, and M. W. Seeger Gaussian process bandits without regret: an experimental design approach. CoRR abs/0912.3995.

Tatsukawa et al. (2025) Y. Tatsukawa, I. Shen, M. D. Dogan, A. Qi, Y. Koyama, A. Shamir, and T. Igarashi FontCraft: multimodal font design using interactive bayesian optimization. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pp. 1–14.

Tsukida and Gupta (2011) K. Tsukida and M. R. Gupta How to analyze paired comparison data. Technical report

Tversky and Kahneman (1981) A. Tversky and D. Kahneman The framing of decisions and the psychology of choice. science 211 ( 4481 ), pp. 453–458.

Virtanen et al. (2020) P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, İ. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods 17, pp. 261–272.

Voynov et al. (2023) A. Voynov, Q. Chu, D. Cohen-Or, and K. Aberman P+: extended textual conditioning in text-to-image generation. arXiv preprint arXiv:2303.09522.

Wang et al. (2024) H. Wang, M. Spinelli, Q. Wang, X. Bai, Z. Qin, and A. Chen Instantstyle: free lunch towards style-preserving in text-to-image generation. arXiv preprint arXiv:2404.02733.

Wang and Lu (2015) W. Wang and C. Lu Projection onto the capped simplex. CoRR abs/1503.01002.

Wang et al. (2025) Z. Wang, X. Wang, L. Xie, Z. Qi, Y. Shan, W. Wang, and P. Luo Styleadapter: a unified stylized image generation model. International Journal of Computer Vision 133 ( 4 ), pp. 1894–1911.

Wang et al. (2016) Z. Wang, F. Hutter, M. Zoghi, D. Matheson, and N. De Feitas Bayesian optimization in a billion dimensions via random embeddings. Journal of Artificial Intelligence Research 55, pp. 361–387.

Wen et al. (2026) X. Wen, L. Hegemann, X. Jin, S. Ma, and A. Oulasvirta Adaptive prompt elicitation for text-to-image generation. arXiv preprint arXiv:2602.04713.

Wilson et al. (2017) J. T. Wilson, R. Moriconi, F. Hutter, and M. P. Deisenroth The reparameterization trick for acquisition functions. CoRR abs/1712.00424.

Yadav et al. (2024) P. Yadav, C. Raffel, M. Muqeeth, L. Caccia, H. Liu, T. Chen, M. Bansal, L. Choshen, and A. Sordoni A survey on model moerging: recycling and routing among specialized experts for collaborative learning. arXiv preprint arXiv:2408.07057.

Yadav et al. (2023) P. Yadav, D. Tam, L. Choshen, C. A. Raffel, and M. Bansal TIES-merging: resolving interference when merging models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023,

Yang et al. (2024) K. Yang, J. Tao, J. Lyu, C. Ge, J. Chen, W. Shen, X. Zhu, and X. Li Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8941–8951.

Ye et al. (2023) H. Ye, J. Zhang, S. Liu, X. Han, and W. Yang Ip-adapter: text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721.

Zhang et al. (2023) L. Zhang, A. Rao, and M. Agrawala Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3836–3847.

Zhang et al. (2025) X. Zhang, X. Wei, W. Hu, J. Wu, J. Wu, W. Zhang, Z. Zhang, Z. Lei, and Q. Li A survey on personalized content synthesis with diffusion models. Machine Intelligence Research 22 ( 5 ), pp. 817–848.

Zheng et al. (2024) M. Zheng, E. Simsar, H. Yesiltepe, F. Tombari, J. Simon, and P. Yanardag Delul Stylebreeder: exploring and democratizing artistic styles through text-to-image models. Advances in Neural Information Processing Systems 37, pp. 34098–34122.

Zheng and Wang (2024) S. Zheng and H. Wang FREE-merging: fourier transform for model merging with lightweight experts. CoRR abs/2411.16815.

Zhou et al. (2020) Y. Zhou, Y. Koyama, M. Goto, and T. Igarashi Generative melody composition with human-in-the-loop bayesian optimization. arXiv preprint arXiv:2010.03190.