Rich Human Feedback for Text-to-Image Generation

面向文生图的丰富人类反馈

Youwei Liang Thanks:  Co-first authors, equal technical contribution Thanks:  The work was done during an internship at Google. Email: youwei@ucsd.edu Affiliation:  University of California, San Diego    Junfeng He 1 1 footnotemark: 1 Thanks:  Corresponding authors, equal leading contribution. Email: {junfenghe,leebird}@google.com Affiliation:  Google Research    Gang Li 1 1 footnotemark: 1 † † footnotemark: Affiliation:  Google Research    Peizhao Li 2 2 footnotemark: 2 Affiliation:  Brandeis University    Arseniy Klimovskiy Affiliation:  Google Research    Nicholas Carolan Affiliation:  Google Research    Jiao Sun 2 2 footnotemark: 2 Thanks:  Currently affiliated with Google Gemini Team Affiliation:  University of Southern California    Jordi Pont-Tuset Affiliation:  Google Research    Sarah Young Affiliation:  Google Research    Feng Yang Affiliation:  Google Research    Junjie Ke Affiliation:  Google Research    Krishnamurthy Dj Dvijotham Affiliation:  Google Research    Katherine M. Collins 2 2 footnotemark: 2 Affiliation:  University of Cambridge    Yiwen Luo Affiliation:  Google Research    Yang Li Affiliation:  Google Research    Kai J Kohlhoff Affiliation:  Google Research    Deepak Ramachandran Affiliation:  Google Research    Vidhya Navalpakkam Affiliation:  Google Research

Abstract

摘要

Recent Text-to-Image (T2I) generation models such as Stable Diffusion and Imagen have made significant progress in generating high-resolution images based on text descriptions. However, many generated images still suffer from issues such as artifacts/implausibility, misalignment with text descriptions, and low aesthetic quality. Inspired by the success of Reinforcement Learning with Human Feedback (RLHF) for large language models, prior works collected human-provided scores as feedback on generated images and trained a reward model to improve the T2I generation. In this paper, we enrich the feedback signal by (i) marking image regions that are implausible or misaligned with the text, and (ii) annotating which words in the text prompt are misrepresented or missing on the image. We collect such rich human feedback on 18K generated images (RichHF-18K) and train a multimodal transformer to predict the rich feedback automatically. We show that the predicted rich human feedback can be leveraged to improve image generation, for example, by selecting high-quality training data to finetune and improve the generative models, or by creating masks with predicted heatmaps to inpaint the problematic regions. Notably, the improvements generalize to models (Muse) beyond those used to generate the images on which human feedback data were collected (Stable Diffusion variants). The RichHF-18K data set will be released in our GitHub repository: https://github.com/google-research/google-research/tree/master/richhf_18k.

近年来，Stable Diffusion、Imagen 等文生图（T2I）模型在按文本描述生成高分辨率图像上取得显著进展。但许多生成结果仍存在伪影/不合理、与文本描述不对齐、美学质量偏低等问题。受大语言模型中人类反馈强化学习（RLHF）成功的启发，已有工作收集人类对生成图像的打分作为反馈，并训练奖励模型以改进 T2I。本文进一步丰富反馈信号：（i）标出图像中不合理或与文本不对齐的区域；（ii）标注提示词中哪些词在图像上被误表示或缺失。我们在 1.8 万张生成图像上收集这类丰富人类反馈（RichHF-18K），并训练多模态 Transformer 自动预测这些反馈。我们表明，预测出的丰富人类反馈可用于改进图像生成，例如筛选高质量训练数据以微调生成模型，或用预测热图做掩码对问题区域做修复。值得注意的是，这些改进能泛化到与收集反馈所用模型（Stable Diffusion 变体）不同的模型（Muse）。RichHF-18K 将在 GitHub 仓库发布：https://github.com/google-research/google-research/tree/master/richhf_18k。

1 Introduction

1 引言

Text-to-image (T2I) generation models Saharia et al. 2022; Croitoru et al. 2023; Yang et al. 2022; Zhang et al. 2023a; Rombach et al. 2022; Yu et al. 2022; Gu et al. 2023 are rapidly becoming a key to content creation in various domains, including entertainment, art, design, and advertising, and are also being generalized to image editing Sheynin et al. 2023; Brooks et al. 2023; Kawar et al. 2023; Wang et al. 2023, video generation Ho et al. 2022; Ni et al. 2023; Xing et al. 2023, among many other applications. Despite significant recent advances, the outputs still usually suffer from issues such as artifacts/implausibility, misalignment with text descriptions, and low aesthetic quality Xu et al. 2023; Kirstain et al. 2023; Wu et al. 2023b. For example, in the Pick-a-Pic dataset Kirstain et al. 2023, which mainly consists of images generated by Stable Diffusion model variants , many images (e.g. Fig. 1) contain distorted human/animal bodies (e.g. human hands with more than five fingers), distorted objects and implausibility issues such as a floating lamp. Our human evaluation experiments find that only $\sim$10% of the generated images in the dataset are free of artifacts and implausibility. Similarly, text-image misalignment issues are common too, e.g., the prompt is “a man jumping into a river” but the generated image shows the man standing.

文生图（T2I）模型 Saharia et al. 2022; Croitoru et al. 2023; Yang et al. 2022; Zhang et al. 2023a; Rombach et al. 2022; Yu et al. 2022; Gu et al. 2023 正迅速成为娱乐、艺术、设计、广告等内容创作的关键手段，并推广到图像编辑 Sheynin et al. 2023; Brooks et al. 2023; Kawar et al. 2023; Wang et al. 2023、视频生成 Ho et al. 2022; Ni et al. 2023; Xing et al. 2023 等应用。尽管近期进展显著，输出仍常出现伪影/不合理、与文本描述不对齐、美学质量偏低等问题 Xu et al. 2023; Kirstain et al. 2023; Wu et al. 2023b。例如，以 Stable Diffusion 变体生成为主的 Pick-a-Pic 数据集 Kirstain et al. 2023 中，许多图像（如图 1）存在扭曲的人体/动物身体（如超过五指的人手）、扭曲物体，以及悬浮灯具等不合理现象。我们的人工评测发现，该数据集中仅约 $\sim$10% 的生成图像没有伪影与不合理问题。图文不对齐同样常见，例如提示是「一个人跳进河里」，生成图却是人站着。

Existing automatic evaluation metrics for generated images, however, including the well-known IS Salimans et al. 2016 and FID Heusel et al. 2017, are computed over distributions of images and may not reflect nuances in individual images. Recent research has collected human preferences/ratings to evaluate the quality of generated images and trained evaluation models to predict those ratings Xu et al. 2023; Kirstain et al. 2023; Wu et al. 2023b, notably ImageReward Xu et al. 2023 or Pick-a-Pic Kirstain et al. 2023. While more focused, these metrics still summarize the quality of one image into a single numeric score. In terms of prompt-image alignment, there are also seminal single-score metrics such as CLIPScore Hessel et al. 2022 and more recent question generation and answering pipelines Hu et al. 2023b; Yarom et al. 2023; Cho et al. 2023c; Cho et al. 2023a. While more calibrated and explainable, these are expensive and complex models that still do not localize the regions of misalignment in the image.

然而现有对生成图像的自动评测指标，包括广为人知的 IS Salimans et al. 2016 与 FID Heusel et al. 2017，多在图像分布上计算，难以反映单张图像上的细微问题。近期研究收集人类偏好/打分来评估生成图像质量，并训练模型预测这些打分 Xu et al. 2023; Kirstain et al. 2023; Wu et al. 2023b，代表性的有 ImageReward Xu et al. 2023 与 Pick-a-Pic Kirstain et al. 2023。这些指标虽更聚焦，仍把一张图的质量压成单个数值。在提示–图像对齐方面，也有 CLIPScore Hessel et al. 2022 等经典单分指标，以及更近的问答管线 Hu et al. 2023b; Yarom et al. 2023; Cho et al. 2023c; Cho et al. 2023a。后者更可校准、可解释，但模型昂贵复杂，仍无法定位图像上的不对齐区域。

In this paper, we propose a dataset and a model of fine-grained multi-faceted evaluations that are interpretable and attributable (e.g., to regions with artifacts/implausibility or image-text misalignments), which provide a much richer understanding of the image quality than single scalar scores. As a first contribution, we collect a dataset of Rich Human Feedback on 18K images (RichHF-18K), which contains (i) point annotations on the image that highlight regions of implausibility/artifacts, and text-image misalignment; (ii) labeled words on the prompts specifying the missing or misrepresented concepts in the generated image; and (iii) four types of fine-grained scores for image plausibility, text-image alignment, aesthetics, and overall rating. Equipped with RichHF-18K, we design a multimodal transformer model, which we coin as Rich Automatic Human Feedback (RAHF) to learn to predict these rich human annotations on generated images and their associated text prompt. Our model can therefore predict implausibility and misalignment regions, misaligned keywords, as well as fine-grained scores. This not only provides reliable ratings, but also more detailed and explainable insights about the quality of the generated images. To the best of our knowledge, this is the first rich feedback dataset and model for state-of-the-art text-to-image generation models, providing an automatic and explainable pipeline to evaluate T2I generation.

本文提出一套细粒度、多面向的评测数据与模型，可解释且可归因（例如定位伪影/不合理区域或图文不对齐区域），相对单一标量分数能更丰富地刻画图像质量。作为第一项贡献，我们收集了覆盖 1.8 万张图像的丰富人类反馈数据集 RichHF-18K，包含：（i）标出不合理/伪影以及图文不对齐区域的点标注；（ii）标出提示中缺失或被误表示概念的词；（iii）四类细粒度分数——图像合理性、图文对齐、美学与总体评分。基于 RichHF-18K，我们设计多模态 Transformer 模型 Rich Automatic Human Feedback（RAHF），学习预测生成图像及其关联文本提示上的这些丰富标注。模型可预测不合理区域、不对齐区域、不对齐关键词以及细粒度分数，既给出可靠评分，也提供更细致、可解释的质量洞察。据我们所知，这是面向当代文生图模型的首个丰富反馈数据集与模型，给出了可自动、可解释的 T2I 评测管线。

The main contributions of this paper are summarized below:

本文主要贡献概括如下：

1. The first Rich Human Feedback dataset (RichHF-18K) on generated images (consisting of fine-grained scores, implausibility(artifact)/misalignment image regions, and misalignment keywords), on 18K Pick-a-Pic images.
2. A multimodal Transformer model (RAHF) to predict rich feedback on generated images, which we show to be highly correlated with the human annotations on a test set.
3. We further demonstrate the usefulness of the predicted rich human feedback by RAHF to improve image generation: (i) by using the predicted heatmaps as masks to inpaint problematic image regions and (ii) by using the predicted scores to help finetune image generation models (like Muse Chang et al. 2023), e.g., via selecting/filtering finetuning data, or as reward guidance. We show that in both cases we obtain better images than with the original model.
4. The improvement on the Muse model, which differs from the models that generated the images in our training set, shows the good generalization capacity of our RAHF model.

1. 首个面向生成图像的丰富人类反馈数据集 RichHF-18K（含细粒度分数、不合理（伪影）/不对齐图像区域、不对齐关键词），基于 1.8 万张 Pick-a-Pic 图像。
2. 多模态 Transformer 模型 RAHF，用于预测生成图像上的丰富反馈；在测试集上与人类标注高度相关。
3. 进一步展示 RAHF 预测的丰富人类反馈对改进图像生成的用途：（i）用预测热图作掩码修复问题区域；（ii）用预测分数帮助微调图像生成模型（如 Muse Chang et al. 2023），例如筛选/过滤微调数据，或作奖励引导。两种情形下均得到优于原模型的图像。
4. 对 Muse 的改进——该模型与训练集中生成图像所用模型不同——表明 RAHF 具有良好泛化能力。

2 Related works

2 相关工作

Text-to-image generation

Text-to-image (T2I) generation models have evolved and iterated through several popular model architectures in the deep learning era. An early work is the Generative Adversarial Network (GAN) Goodfellow et al. 2014; Brock et al. 2018; Karras et al. 2019, which trains a generator for image generation and a discriminator to distinguish between real and generated images, in parallel (also see Zhang et al. 2017; Xu et al. 2018; Zhu et al. 2019; Qiao et al. 2019; Li et al. 2019; Tao et al. 2022 among others). Another category of generation models develops from variational auto-encoders (VAEs) Kingma and Welling 2013; Higgins et al. 2016; Van Den Oord et al. 2017, which optimize evidence lower bound (ELBO) for the likelihood of the image data.

文生图

深度学习时代，文生图（T2I）模型经历了多种主流架构迭代。早期代表是生成对抗网络（GAN） Goodfellow et al. 2014; Brock et al. 2018; Karras et al. 2019：训练生成器造图，同时训练判别器区分真假图（另见 Zhang et al. 2017; Xu et al. 2018; Zhu et al. 2019; Qiao et al. 2019; Li et al. 2019; Tao et al. 2022 等）。另一类由变分自编码器（VAE） Kingma and Welling 2013; Higgins et al. 2016; Van Den Oord et al. 2017 发展而来，优化图像数据似然的证据下界（ELBO）。

Most recently, Diffusion Models (DMs) Sohl-Dickstein et al. 2015; Ho et al. 2020; Nichol et al. 2021; Rombach et al. 2022 have emerged as the state-of-the-art (SOTA) for Image Generation Dhariwal and Nichol 2021. DMs are trained to generate images progressively from random noise, with the ability to capture more diversity than GANs and achieve good sample quality Dhariwal and Nichol 2021. Latent Diffusion Models Rombach et al. 2022 are a further refinement that performs the diffusion process in a compact latent space for more efficiency.

最近，扩散模型（DM） Sohl-Dickstein et al. 2015; Ho et al. 2020; Nichol et al. 2021; Rombach et al. 2022 成为图像生成的前沿 Dhariwal and Nichol 2021。DM 从随机噪声逐步生成图像，比 GAN 更能覆盖多样性，并达到良好样本质量 Dhariwal and Nichol 2021。潜扩散模型 Rombach et al. 2022 进一步在紧凑潜空间做扩散，以提升效率。

Text-to-image evaluation and reward models

There has been much recent work on evaluation of text-to-image models along many dimensions Xu et al. 2023; Kirstain et al. 2023; Huang et al. 2023; Wu et al. 2023b; Wu et al. 2023a; Otani et al. 2023; Lee et al. 2023; Cho et al. 2023b. Xu et al. 2023 collect a human preference dataset by requesting users to rank multiple images and rate them according to their quality. They trained a reward model ImageReward for human preference learning, and proposed Reward Feedback Learning (ReFL) for tuning diffusion models with the ImageReward model. Kirstain et al. 2023 built a web application to collect human preferences by asking users to choose the better image from a pair of generated images, resulting in a dataset called Pick-a-Pic with more than 500K examples generated by T2I models such as Stable Diffusion 2.1, Dreamlike Photoreal 2.05, and Stable Diffusion XL variants. They leveraged the human preference dataset to train a CLIP-based Radford et al. 2021 scoring function, called PickScore, to predict human preferences. Huang et al. 2023 proposed a benchmark called T2I-CompBench for evaluating text-to-image models, which consists of 6,000 text prompts describing attribute binding, object relationships, and complex compositions. They utilized multiple pretrained vision-language models such as CLIP Radford et al. 2021 and BLIP Li et al. 2022 to calculate multiple evaluation metrics. Wu et al. 2023b; Wu et al. 2023a collected a large scale dataset of human choices on generated images and utilized the dataset to train a classifier that outputs a Human Preference Score (HPS). They showed improvement in image generation by tuning Stable Diffusion with the HPS. Recently, Lee Lee et al. 2023 proposed a holistic evaluation for T2I models with multiple fine-grained metrics.

文生图评测与奖励模型

近期沿多种维度评测文生图模型的工作很多 Xu et al. 2023; Kirstain et al. 2023; Huang et al. 2023; Wu et al. 2023b; Wu et al. 2023a; Otani et al. 2023; Lee et al. 2023; Cho et al. 2023b。Xu et al. 2023 收集人类偏好数据：让用户对多张图排序并按质量打分，训练奖励模型 ImageReward，并提出用 ImageReward 做 Reward Feedback Learning（ReFL）以调优扩散模型。Kirstain et al. 2023 搭建网页应用，让用户在一对生成图中选更好的一张，得到 Pick-a-Pic 数据集，含逾 50 万例，来自 Stable Diffusion 2.1、Dreamlike Photoreal 2.05、Stable Diffusion XL 等。他们用该偏好数据训练基于 CLIP Radford et al. 2021 的打分函数 PickScore。Huang et al. 2023 提出 T2I-CompBench，含 6000 条描述属性绑定、物体关系与复杂构图的提示，并用 CLIP Radford et al. 2021、BLIP Li et al. 2022 等预训练视觉–语言模型计算多项指标。Wu et al. 2023b; Wu et al. 2023a 收集大规模人类选择数据，训练输出人类偏好分（HPS）的分类器，并用 HPS 调优 Stable Diffusion 以改进生成。Lee Lee et al. 2023 近期提出带多项细粒度指标的整体评测。

Despite these valuable contributions, most existing works only use binary human ratings or preference ranking for construction of feedback/rewards, and lack the ability to provide detailed actionable feedback such as implausible regions of the image, misaligned regions, or misaligned keywords on the generated images. One recent paper related to our work is Zhang et al. 2023b, which collected a dataset of artifact regions for image synthesis tasks, trained a segmentation-based model to predict artifact regions, and proposed a region inpainting method for those regions. However, the focus of their work is artifact region only, while in this paper, we collected rich feedback for T2I generation containing not only artifact regions, but also misalignment regions, misaligned keywords, and four fine-grained scores from multiple aspects. To the best of our knowledge, this is the first work on heterogeneous rich human feedback for text-to-image models.

尽管上述工作很有价值，多数仍只用二元人类打分或偏好排序来构造反馈/奖励，难以给出可行动的细节反馈，例如图像的不合理区域、不对齐区域或不对齐关键词。与本文较近的是 Zhang et al. 2023b：为图像合成收集伪影区域数据集，训练基于分割的模型预测伪影区，并提出区域修复。但其焦点仅在伪影区域；本文收集的 T2I 丰富反馈不仅含伪影区，还有不对齐区域、不对齐关键词以及多角度的四类细粒度分数。据我们所知，这是面向文生图模型的首项异构丰富人类反馈工作。

3 Collecting rich human feedback

3 收集丰富人类反馈

Figure 1: An illustration of our annotation UI. Annotators mark points on the image to indicate artifact/implausibility regions (red points) or misaligned regions (blue points) w.r.t the text prompt. Then, they click on the words to mark the misaligned keywords (underlined and shaded) and choose the scores for plausibility, text-image alignment, aesthetics, and overall quality (underlined).

图 1：标注界面示意。标注者在图像上点选，标出伪影/不合理区域（红点）或相对文本提示的不对齐区域（蓝点）；再点击词语标出不对齐关键词（下划线并着色），并选择合理性、图文对齐、美学与总体质量分数（下划线）。

3.1 Data collection process

3.1 数据收集流程

In this section, we discuss our procedure to collect the RichHF-18K dataset, which includes two heatmaps (artifact/implausibility and misalignment), four fine-grained scores (plausibility, alignment, aesthetics, and overall score), and one text sequence (misaligned keywords).

本节说明我们如何收集 RichHF-18K：包含两张热图（伪影/不合理与不对齐）、四类细粒度分数（合理性、对齐、美学、总体分），以及一个文本序列（不对齐关键词）。

For each generated image, the annotators are first asked to examine the image and read the text prompt used to generate it. Then, they mark points on the image to indicate the location of any implausibility/artifact or misalignment w.r.t the text prompt. The annotators are told that each marked point has an “effective radius” (1/20 of the image height), which forms an imaginary disk centering at the marked point. In this way, we can use a relatively small amount of points to cover the image regions with flaws. Lastly, annotators label the misaligned keywords and the four types of scores for plausibility, image-text alignment, aesthetic, and overall quality, respectively, on a 5-point Likert scale. Detailed definitions of image implausibility/artifact and misalignment can be found in the supplementary materials. We designed a web UI, as shown in Fig. 1, to facilitate data collection. More details about data collection process can be found in the supplementary materials.

对每张生成图，标注者先查看图像并阅读用于生成它的文本提示，再在图上点选标出不合理/伪影或相对提示的不对齐位置。我们告知标注者每个点有一个「有效半径」（图像高度的 1/20），形成以该点为中心的虚拟圆盘，从而用较少点数覆盖有瑕疵的区域。最后，标注者标出不对齐关键词，并分别在 5 点李克特量表上给出合理性、图文对齐、美学与总体质量四类分数。图像不合理/伪影与不对齐的详细定义见补充材料。我们设计了网页 UI（见图 1）以便收集数据。更多流程细节见补充材料。

Figure 2: Histograms of the average scores of image-text pairs in the training set.

图 2：训练集中图文对平均分数的直方图。

Figure 3: Architecture of our rich feedback model. Our model consists of two streams of computation: one vision and one text stream. We perform self-attention on the ViT-outputted image tokens and the Text-embed module-outputted text tokens to fuse the image and text information. The vision tokens are reshaped into feature maps and mapped to heatmaps and scores. The vision and text tokens are sent to a Transformer decoder to generate a text sequence.

图 3：丰富反馈模型架构。模型含视觉与文本两路计算。我们对 ViT 输出的图像 token 与 Text-embed 模块输出的文本 token 做自注意力，以融合图文信息。视觉 token 重排为特征图并映射到热图与分数；视觉与文本 token 送入 Transformer 解码器生成文本序列。

3.2 Human feedback consolidation

3.2 人类反馈汇合

To improve the reliability of the collected human feedback on generated images, each image-text pair is annotated by three annotators. We therefore need to consolidate the multiple annotations for each sample. For the scores, we simply average the scores from the multiple annotators for an image to obtain the final score. For the misaligned keyword annotations, we perform majority voting to get the final sequence of indicators of aligned/misaligned, using the most frequent label for the keywords. For the point annotations, we first convert them to heatmaps for each annotation, where each point is converted to a disk region (as discussed in the last subsection) on the heatmap, and then we compute the average heatmap across annotators. The regions with clear implausibility are likely to be annotated by all annotators and have a high value on the final average heatmap.

为提高收集到的反馈可靠性，每对图文由三名标注者标注，因此需对多样本标注做汇合。对分数，直接取多名标注者对该图分数的平均作为最终分。对不对齐关键词，按多数投票得到对齐/不对齐指示序列，关键词取最常见标签。对点标注，先把每次标注转为热图（每个点按上一小节所述变为圆盘区域），再对标注者求平均热图。明显不合理的区域往往被所有人标出，在最终平均热图上取值较高。

3.3 RichHF-18K: a dataset of rich human feedback

3.3 RichHF-18K：丰富人类反馈数据集

We select a subset of image-text pairs from the Pick-a-Pic dataset for data annotation. Although our method is general and applicable to any generated images, we choose the majority of our dataset to be photo-realistic images, due to its importance and wider applications. Moreover, we also want to have balanced categories across the images. To ensure balance, we utilized the PaLI visual question answering (VQA) model Chen et al. 2022 to extract some basic features from the Pick-a-Pic data samples. Specifically, we asked the following questions for each image-text pair in Pick-a-Pic. 1) Is the image photorealistic? 2) Which category best describes the image? Choose one in ‘human’, ‘animal’, ‘object’, ‘indoor scene’, ‘outdoor scene’. PaLI’s answers to these two questions are generally reliable under our manual inspection. We used the answers to sample a diverse subset from Pick-a-Pic, resulting in 17K image-text pairs. We randomly split the 17K samples into two subsets, a training set with 16K samples and a validation set with 1K samples. The distribution of the attributes of the 16K training samples is shown in the supplementary materials. Additionally, we collect rich human feedback on the unique prompts and their corresponding images from the Pick-a-Pic test set as our test set. In total, we collected rich human feedback on the 18K image-text pairs from Pick-a-Pic. Our RichHF-18K dataset consists of 16K training, 1K validation, and 1K test samples.

我们从 Pick-a-Pic 中选取部分图文对做标注。方法虽通用、适用于任意生成图，但因重要性与应用面更广，数据集主体选为写实照片风格。同时希望类别尽量均衡。为此用 PaLI 视觉问答（VQA）模型 Chen et al. 2022 从 Pick-a-Pic 样本抽取基本特征，对每对图文提问：1）图像是否写实？2）哪一类最能描述该图？在「人」「动物」「物体」「室内场景」「室外场景」中选一。人工抽查显示 PaLI 对这两问的回答大体可靠。据此从 Pick-a-Pic 采样多样子集，得到 1.7 万对图文；随机划分为 1.6 万训练与 1 千验证。1.6 万训练样本的属性分布见补充材料。另外，我们对 Pick-a-Pic 测试集中唯一提示及其对应图像收集丰富人类反馈作为测试集。合计在 Pick-a-Pic 的 1.8 万对图文上收集了丰富人类反馈。RichHF-18K 含 1.6 万训练、1 千验证与 1 千测试样本。

Figure 4: Counts of the samples with maximum differences of the scores in the training set.

图 4：训练集中分数最大差值的样本计数。

3.4 Data statistics of RichHF-18K

3.4 RichHF-18K 数据统计

In this section, we summarize the statistics of the scores and perform the annotator agreement analysis for the scores. We standardize the scores $s$ with the formula $(s-s_{\text{min}})/(s_{\text{max}}-s_{\text{min}})$ ($s_{\text{max}}=5$ and $s_{\text{min}}=1$) so that the scores lie in the range [0, 1].

本节汇总分数统计，并做标注者一致性分析。用公式 $(s-s_{\text{min}})/(s_{\text{max}}-s_{\text{min}})$（$s_{\text{max}}=5$，$s_{\text{min}}=1$）标准化分数 $s$，使其落在 [0, 1]。

The histogram plots of the scores are shown in Fig. 2. The distribution of the scores is similar to a Gaussian distribution, while the plausibility and text-image alignment scores have a slightly higher percentage of score 1.0. The distribution of the collected scores ensures that we have a reasonable number of negative and positive samples for training a good reward model.

分数直方图见图 2。分布近似高斯，而合理性与图文对齐分数中 1.0 略多。该分布保证训练奖励模型时负样本与正样本数量较合理。

To analyze the rating agreement among annotators for an image-text pair, we calculate the maximum difference among the scores: $\text{max}_{\text{diff}}=\max(\text{scores})-\min(\text{scores})$ where scores are the three score labels for an image-text pair. We plot the histogram of $\text{max}_{\text{diff}}$ in Fig. 4. We can see that around 25% of the samples have perfect annotator agreement and around 85% of the samples have good annotator agreement ($\text{max}_{\text{diff}}$ is less than or equal to 0.25 after the standardization or 1 in the 5-point Likert scale).

为分析一对图文上标注者打分一致性，计算分数最大差： $\text{max}_{\text{diff}}=\max(\text{scores})-\min(\text{scores})$，其中 scores 为该对图文的三个分数标签。图 4 画出 $\text{max}_{\text{diff}}$ 直方图。可见约 25% 样本标注者完全一致，约 85% 一致性良好（标准化后 $\text{max}_{\text{diff}}$ 不超过 0.25，或在 5 点李克特量表上不超过 1）。

4 Predicting rich human feedback

4 预测丰富人类反馈

4.1 Models

4.1 模型

4.1.1 Architecture

The architecture of our model is shown in Fig. 3. We adopt a vision-language model based on ViT Dosovitskiy et al. 2020 and T5X Roberts et al. 2022 models, inspired by the Spotlight model architecture Li and Li 2023, but modifying both the model and pretraining datasets to better suit our tasks. We use a self-attention module Vaswani et al. 2017 among the concatenated image tokens and text tokens, similar to PaLI Chen et al. 2022, as our tasks require bidirectional information propagation. The text information is propagated to image tokens for text misalignment score and heatmap prediction, while the vision information is propagated to text tokens for better vision-aware text encoding to decode the text misalignment sequence. To pretrain the model on more diverse images, we add the natural image captioning task on the WebLI dataset Chen et al. 2022 to the pretraining task mixture.

4.1.1 架构

模型架构见图 3。我们采用基于 ViT Dosovitskiy et al. 2020 与 T5X Roberts et al. 2022 的视觉–语言模型，灵感来自 Spotlight Li and Li 2023，但调整了模型与预训练数据以更好适配任务。因任务需要双向信息传播，我们在拼接后的图像 token 与文本 token 上使用自注意力 Vaswani et al. 2017，做法类似 PaLI Chen et al. 2022：文本信息传到图像 token，用于文本不对齐分数与热图预测；视觉信息传到文本 token，以便做视觉感知的文本编码并解码文本不对齐序列。为在更多样图像上预训练，我们在预训练任务混合中加入 WebLI Chen et al. 2022 上的自然图像描述任务。

Specifically, the ViT takes the generated image as input and outputs image tokens as high-level representations. The text prompt tokens are embedded into dense vectors. The image tokens and embedded text tokens are concatenated and encoded by the Transformer self-attention encoder in T5X. On top of the encoded fused text and image tokens, we use three kinds of predictors to predict different outputs. For heatmap prediction, the image tokens are reshaped into a feature map and sent through convolution layers, deconvolution layers, and sigmoid activation, and outputs implausibility and misalignment heatmaps. For score prediction, the feature map is sent through convolution layers, linear layers, and sigmoid activation, resulting in scalars as fine-grained scores.

具体地，ViT 以生成图像为输入，输出作为高层表示的图像 token；文本提示 token 嵌入为稠密向量。图像 token 与嵌入后的文本 token 拼接，经 T5X 中的 Transformer 自注意力编码器编码。在编码后的融合图文 token 之上，用三类预测器输出不同结果。热图预测：图像 token 重排为特征图，经卷积、反卷积与 sigmoid，输出不合理与不对齐热图。分数预测：特征图经卷积、线性层与 sigmoid，得到细粒度标量分数。

To predict the keyword misalignment sequence, the original prompt used to generate the image is used as text input to the model. A modified prompt is used as the prediction target for the T5X decoder. The modified prompt has a special suffix (‘$\_0$’) for each misaligned token, e.g., a yellow_0 cat if the generated image contains a black cat and the word yellow is misaligned with the image. During evaluation, we can extract the misaligned keywords using the special suffix.

预测关键词不对齐序列时，生成该图所用的原始提示作为模型文本输入；经改写的提示作为 T5X 解码器的预测目标。改写提示为每个不对齐 token 加特殊后缀（「$\_0$」），例如生成图是黑猫而词「yellow」与图像不对齐时写成 a yellow_0 cat。评测时可用该特殊后缀抽出不对齐关键词。

4.1.2 Model variants

We explore two model variants for the prediction heads of the heatmaps and scores.

4.1.2 模型变体

我们为热图与分数的预测头探索了两种变体。

Multi-head

A straightforward way to predict multiple heatmaps and scores is to use multiple prediction heads, with one head for each score and heatmap type. This will require seven prediction heads in total.

多头

预测多张热图与多项分数的直接做法是使用多个预测头，每种分数与热图类型各一头，共需七个预测头。

Augmented prompt

Another approach is to use a single head for each prediction type, i.e., three heads in total, for heatmap, score, and misalignment sequence, respectively. To inform the model of the fine-grained heatmap or score type, we augment the prompt with the output type. More specifically, we prepend a task string (e.g., ‘implausibility heatmap’) to the prompt for each particular task of one example and use the corresponding label as the training target. During inference, by augmenting the prompt with the corresponding task string, the single heatmap (score) head can predict different heatmaps (scores). As we show in the experiments, this augmented prompt approach can create task-specific vision feature maps and text encodings, which performs significantly better in some of the tasks.

增强提示

另一种做法是每种预测类型只用一个头，即热图、分数与不对齐序列共三个头。为告知模型细粒度热图或分数类型，我们用输出类型增强提示：具体地，为每个样本的特定任务在提示前加上任务字符串（如「implausibility heatmap」），并以对应标签为训练目标。推理时，用相应任务字符串增强提示，单个热图（分数）头即可预测不同热图（分数）。实验表明，这种增强提示能形成任务特异的视觉特征图与文本编码，在部分任务上明显更好。

4.1.3 Model optimization

We train the model with a pixel-wise mean squared error (MSE) loss for the heatmap prediction, and MSE loss for the score prediction. For misalignment sequence prediction, the model is trained with teacher-forcing cross-entropy loss. The final loss function is the weighted combination of the heatmap MSE loss, score MSE loss, and the sequence teacher-forcing cross-entropy loss.

4.1.3 模型优化

热图预测用逐像素均方误差（MSE），分数预测用 MSE；不对齐序列预测用 teacher-forcing 交叉熵。最终损失是热图 MSE、分数 MSE 与序列交叉熵的加权组合。

4.2 Experiments

4.2 实验

(a) Image

(a) 图像

(a) Image

(a) 图像

	Plausibility	Aesthetics	Text-image Alignment	Overall
	PLCC $\uparrow$	SRCC $\uparrow$	PLCC $\uparrow$	SRCC $\uparrow$	PLCC $\uparrow$	SRCC $\uparrow$	PLCC $\uparrow$	SRCC $\uparrow$
ResNet-50	0.495	0.487	0.370	0.363	0.108	0.119	0.337	0.308
PickScore (off-the-shelf)	0.0098	0.0280	0.131	0.140	0.346	0.340	0.202	0.226
CLIP (off-the-shelf)	$-$	$-$	$-$	$-$	0.185	0.130	$-$	$-$
CLIP (fine-tuned)	0.390	0.378	0.357	0.360	0.398	0.390	0.353	0.352
Our Model (multi-head)	0.666	0.654	0.605	0.591	0.487	0.500	0.582	0.561
Our Model (augmented prompt)	0.693	0.681	0.600	0.589	0.474	0.496	0.580	0.562

Table 1: Score prediction results on the test set.

	合理性	美学	图文对齐	总体
	PLCC $\uparrow$	SRCC $\uparrow$	PLCC $\uparrow$	SRCC $\uparrow$	PLCC $\uparrow$	SRCC $\uparrow$	PLCC $\uparrow$	SRCC $\uparrow$
ResNet-50	0.495	0.487	0.370	0.363	0.108	0.119	0.337	0.308
PickScore（现成）	0.0098	0.0280	0.131	0.140	0.346	0.340	0.202	0.226
CLIP（现成）	$-$	$-$	$-$	$-$	0.185	0.130	$-$	$-$
CLIP（微调）	0.390	0.378	0.357	0.360	0.398	0.390	0.353	0.352
本文模型（多头）	0.666	0.654	0.605	0.591	0.487	0.500	0.582	0.561
本文模型（增强提示）	0.693	0.681	0.600	0.589	0.474	0.496	0.580	0.562

表 1：测试集上的分数预测结果。

	All data	$GT=0$	$GT>0$
	MSE $\downarrow$	MSE $\downarrow$	CC $\uparrow$	KLD $\downarrow$	SIM $\uparrow$	NSS $\uparrow$	AUC-Judd $\uparrow$
ResNet-50	0.00996	0.00093	0.506	1.669	0.338	2.924	0.909
Ours (multi-head)	0.01216	0.00141	0.425	1.971	0.302	2.330	0.877
Ours (augmented prompt)	0.00920	0.00095	0.556	1.652	0.409	3.085	0.913

Table 2: Implausibility heatmap prediction results on the test set. $GT=0$ refers to empty implausibility heatmap, i.e., no artifacts/implausibility (69 out of 995 test samples are empty), for ground truth. $GT>0$ refers to heatmaps with artifacts/implausibility, for ground truth.

	全部数据	$GT=0$	$GT>0$
	MSE $\downarrow$	MSE $\downarrow$	CC $\uparrow$	KLD $\downarrow$	SIM $\uparrow$	NSS $\uparrow$	AUC-Judd $\uparrow$
ResNet-50	0.00996	0.00093	0.506	1.669	0.338	2.924	0.909
本文（多头）	0.01216	0.00141	0.425	1.971	0.302	2.330	0.877
本文（增强提示）	0.00920	0.00095	0.556	1.652	0.409	3.085	0.913

表 2：测试集上不合理热图预测结果。$GT=0$ 表示真实不合理热图为空，即无伪影/不合理（995 个测试样本中 69 个为空）。$GT>0$ 表示真实热图含伪影/不合理。

	All data	$GT=0$	$GT>0$
	MSE $\downarrow$	MSE $\downarrow$	CC $\uparrow$	KLD $\downarrow$	SIM $\uparrow$	NSS $\uparrow$	AUC-Judd $\uparrow$
CLIP gradient	0.00817	0.00551	0.015	3.844	0.041	0.143	0.643
Our Model (multi-head)	0.00303	0.00015	0.206	2.932	0.093	1.335	0.838
Our Model (augmented prompt)	0.00304	0.00006	0.212	2.933	0.106	1.411	0.841

Table 3: Text misalignment heatmap prediction results on the test set. $GT=0$ refers to empty misalignment heatmap, i.e., no misalignment (144 out of 995 test samples are empty), for ground truth. $GT>0$ refers to heatmaps with misalignment, for ground truth.

	全部数据	$GT=0$	$GT>0$
	MSE $\downarrow$	MSE $\downarrow$	CC $\uparrow$	KLD $\downarrow$	SIM $\uparrow$	NSS $\uparrow$	AUC-Judd $\uparrow$
CLIP 梯度	0.00817	0.00551	0.015	3.844	0.041	0.143	0.643
本文模型（多头）	0.00303	0.00015	0.206	2.932	0.093	1.335	0.838
本文模型（增强提示）	0.00304	0.00006	0.212	2.933	0.106	1.411	0.841

表 3：测试集上文本不对齐热图预测结果。$GT=0$ 表示真实不对齐热图为空，即无不对齐（995 个测试样本中 144 个为空）。$GT>0$ 表示真实热图含不对齐。

	Precision	Recall	F1 Score
Multi-head	62.9	33.0	43.3
Augmented prompt	61.3	34.1	43.9

Table 4: Text misalignment prediction results on the test set.

	精确率	召回率	F1
多头	62.9	33.0	43.3
增强提示	61.3	34.1	43.9

表 4：测试集上文本不对齐预测结果。

(a) Prompt: gamer playing league of legends at night. Plausibility score. GT: 0.333, Our model: 0.410 Overall score. GT: 0.417, Our model: 0.457

(a) 提示：gamer playing league of legends at night。合理性分。真值：0.333，本文模型：0.410。总体分。真值：0.417，本文模型：0.457

(a) Muse Chang et al. 2023 before finetuning

(a) Muse Chang et al. 2023 微调前

(a) Prompt: a baseball with the parthenon on its cover, sitting on the pitcher’s mound

(a) 提示：a baseball with the parthenon on its cover, sitting on the pitcher’s mound

Preference	$\gg$	$>$	$\approx$	$<$	$\ll$
Percentage	21.5%	30.33%	31.33%	12.67%	4.17%

Table 5: Human Evaluation Results: Finetuned Muse vs original Muse model preference: Percentage of examples where finetuned Muse is significantly better ($\gg$), slightly better ($>$), about the same ($\approx$), slightly worse ($<$), significantly worse ($\ll$) than original Muse. Data was collected from 6 individuals in a randomized survey.

偏好	$\gg$	$>$	$\approx$	$<$	$\ll$
占比	21.5%	30.33%	31.33%	12.67%	4.17%

表 5：人工评测结果：微调后 Muse 相对原 Muse 的偏好。微调 Muse 显著更好（$\gg$）、略好（$>$）、大致相同（$\approx$）、略差（$<$）、显著更差（$\ll$）的样本占比。数据来自 6 人随机问卷。

4.2.1 Experimental setup

Our model is trained on the 16K RichHF-18K training samples, and the hyperparameters were tuned using the model performance on the 1K RichHF-18K validation set. The hyperparameters setup can be found in supplementary material.

4.2.1 实验设置

模型在 RichHF-18K 的 1.6 万训练样本上训练，超参按 1 千验证集上的表现调优。超参设置见补充材料。

Evaluation metrics

For score prediction tasks, we report the Pearson linear correlation coefficient (PLCC) and Spearman rank correlation coefficient (SRCC), which are typical evaluation metrics for score predictions Ke et al. 2021. For heatmap prediction tasks, a straightforward way to evaluate the results would be to borrow standard saliency heatmap evaluation metrics such as NSS/KLD Bylinskii et al. 2018. However, these metrics cannot be applied directly in our case as all these metrics assume the ground truth heatmap is not empty; yet in our case, empty ground truth is possible (e.g., for artifact/implausibility heatmap, it means the image does not have any artifact/implausibility). As such, we report MSE on all samples and on those with empty ground truth, respectively, and report saliency heatmap evaluation metrics like NSS/KLD/AUC-Judd/SIM/CC Bylinskii et al. 2018 for the samples with non-empty ground truth. For the misaligned keyword sequence prediction, we adopt the token-level precision, recall, and F1-score. Specifically, the precision/recall/F1 scores are computed for the misaligned keywords over all the samples.

评测指标

分数预测报告 Pearson 线性相关系数（PLCC）与 Spearman 秩相关系数（SRCC），这是分数预测的常用指标 Ke et al. 2021。热图预测若直接借用 NSS/KLD Bylinskii et al. 2018 等显著图指标，前提是真实热图非空；而我们的情形允许真实热图为空（例如伪影/不合理热图为空表示图像无伪影/不合理）。因此我们分别报告全体样本与真实热图为空样本上的 MSE，并对真实热图非空样本报告 NSS/KLD/AUC-Judd/SIM/CC Bylinskii et al. 2018。不对齐关键词序列采用 token 级精确率、召回率与 F1：在全部样本上对不对齐关键词计算。

Baselines

For comparison, we finetune two ResNet-50 models He et al. 2016, with multiple fully connected layers and deconvolution heads to predict the scores and heatmaps, respectively. We also use the off-the-shelf PickScore model Kirstain et al. 2023 to compute the PickScores and calculate the metrics w.r.t each of our four ground truth scores. We use the off-the-shelf CLIP model Radford et al. 2021 as a baseline to compute the cosine similarity of the image and text embeddings and use it to calculate the text-image alignment metric, as the CLIP cosine similarity is designed to reflect the alignment between images and prompts. Besides, we also fine-tune a CLIP model to predict the four types of scores using our training dataset. For misalignment heatmap prediction, we use CLIP gradient Simonyan et al. 2014 map as a baseline.

基线

对照方面，我们微调两个 ResNet-50 He et al. 2016，分别接多层全连接与反卷积头预测分数与热图。也用现成 PickScore Kirstain et al. 2023 计算 PickScore，并相对我们四类真实分数算指标。用现成 CLIP Radford et al. 2021 的图文嵌入余弦相似度作为图文对齐基线（该相似度旨在反映图与提示对齐）。另在训练数据上微调 CLIP 以预测四类分数。不对齐热图预测以 CLIP 梯度图 Simonyan et al. 2014 为基线。

Quantitative analysis

The experimental results of our model prediction on the four fine-grained scores, the implausibility heatmap, misalignment heatmap, and misalignment keyword sequence on our RichHF-18K test set are presented in Table 1, Table 2, Table 3, and Table 4 respectively.

定量分析

模型在 RichHF-18K 测试集上对四类细粒度分数、不合理热图、不对齐热图与不对齐关键词序列的预测结果分别见表 1、表 2、表 3 与表 4。

In both Table 1 and Table 3, the two variants of our proposed model both significantly outperform ResNet-50 (or CLIP for text-image alignment score). Yet, in Table 2, the multi-head version of our model performs worse than ResNet-50, but our augmented prompt version outperforms ResNet-50. The main reason might be that in the multi-head version, without augmenting the prediction task in the prompt, the same prompt is used for all the seven prediction tasks, and hence the feature maps and text tokens will be the same for all tasks. It might not be easy to find a good tradeoff among these tasks, and hence the performance of some tasks such as artifact/implausibility heatmap became worse. However, after augmenting the prediction task into a prompt, the feature map and text token can be adapted to each particular task with better results. Additionally, we note that misalignment heatmap prediction generally has worse results than artifact/implausibility heatmap prediction, possibly because misalignment regions are less well-defined, and the annotations may therefore be noisier.

在表 1 与表 3 中，本文两种变体均显著优于 ResNet-50（图文对齐分数上优于 CLIP）。但在表 2 中，多头变体弱于 ResNet-50，而增强提示变体优于 ResNet-50。主因可能是：多头变体未在提示中增强预测任务时，七个预测任务共用同一提示，特征图与文本 token 对所有任务相同，难在任务间取得良好折中，以致伪影/不合理热图等变差。把预测任务增强进提示后，特征图与文本 token 可按任务适配，效果更好。另外，不对齐热图整体弱于伪影/不合理热图，可能因为不对齐区域更难界定，标注噪声更大。

Qualitative examples

We show some example predictions from our model for implausibility heatmap (Fig. 5), where our model identifies the regions with artifact/implausibility, and for misalignment heatmap (Fig. 6), where our model identifies the objects that don’t correspond to the prompt. Fig. 7 shows some example images and their ground-truth and predicted scores. More examples are in the supplementary material.

定性例子

图 5 给出不合理热图预测示例，模型标出伪影/不合理区域；图 6 给出不对齐热图示例，模型标出与提示不符的物体。图 7 展示部分图像及其真实与预测分数。更多例子见补充材料。

5 Learning from rich human feedback

5 从丰富人类反馈中学习

In this section, we investigate whether the predicted rich human feedback (e.g., scores and heatmaps) can be used to improve image generation. To ensure that the gains from our RAHF model generalize across generative model families, we mainly use Muse Chang et al. 2023 as our target model to improve, which is based on a masked transformer architecture and thus different from the Stable Diffusion model variants in our RichHF-18K dataset.

本节考察预测出的丰富人类反馈（如分数与热图）能否改进图像生成。为验证 RAHF 的收益能否跨生成模型族泛化，我们主要用 Muse Chang et al. 2023 作为改进目标——它基于掩码 Transformer，与 RichHF-18K 中的 Stable Diffusion 变体不同。

Finetuning generative models with predicted scores

We first illustrate that finetuning with RAHF scores can improve Muse. First, we generate eight images for each of the $12true564$ prompts (the prompt set is created via PaLM 2 Chowdhery et al. 2022; Anil et al. 2023 with some seed prompts) using the pre-trained Muse model. We predict RAHF scores for each image, and if the highest score for the images from each prompt is above a fixed threshold, it will be selected as part of our finetuning dataset. The Muse model is then finetuned with this dataset. This approach could be viewed as a simplified version of Direct Preference Optimization Fan et al. 2023.

用预测分数微调生成模型

我们先说明用 RAHF 分数微调可改进 Muse。对 $12\,564$ 条提示（用 PaLM 2 Chowdhery et al. 2022; Anil et al. 2023 与若干种子提示合成）各用预训练 Muse 生成八张图，对每张图预测 RAHF 分数；若每条提示下最高分超过固定阈值，则纳入微调数据集，再用该数据微调 Muse。这可视为直接偏好优化 Fan et al. 2023 的简化版。

In Fig. 8 (a)-(b), we show one example of finetuning Muse with our predicted plausibility score (threshold=0.8). To quantify the gain from Muse finetuning, we used 100 new prompts to generate images, and asked 6 annotators to perform side-by-side comparisons (for plausibility) between two images from the original Muse and the fine-tuned Muse respectively. The annotators choose from five possible responses (image A is significantly/slightly better than image B, about the same, image B is slightly/significantly better than image A), without knowledge of which model is used to generate the image A/B. The results in Table 5 demonstrate that the finetuned Muse with RAHF plausibility scores possesses significantly fewer artifacts/implausibility than the original Muse.

图 8 (a)–(b) 给出用预测合理性分（阈值=0.8）微调 Muse 的一例。为量化收益，用 100 条新提示生成图像，请 6 名标注者对原 Muse 与微调 Muse 各一张图做并排比较（比合理性）。标注者从五档回答中选择（A 显著/略好于 B、大致相同、B 略/显著好于 A），且不知 A/B 来自哪一模型。表 5 显示：用 RAHF 合理性分微调后的 Muse，伪影/不合理明显少于原 Muse。

Moreover, in Fig. 8 (c)-(d), we show an example of using the RAHF aesthetic score as Classifier Guidance to the Latent Diffusion model Rombach et al. 2022, similar to the approach in Bansal et al. 2023, demonstrating that each of the fine-grained scores can improve different aspects of the generative model/results.

此外，图 8 (c)–(d) 给出把 RAHF 美学分用作潜扩散模型 Rombach et al. 2022 的分类器引导的例子（做法类似 Bansal et al. 2023），说明各类细粒度分数可改进生成模型/结果的不同方面。

Region inpainting with predicted heatmaps and scores

We demonstrate that our model’s predicted heatmaps and scores can be used to perform region inpainting to improve the quality of generated images. For each image, we first predict implausibility heatmaps, then create a mask by processing the heatmap (using thresholding and dilating). Muse inpainting Chang et al. 2023 is applied within the masked region to generate new images that match the text prompt. Multiple images are generated, and the final image is chosen by the highest predicted plausibility score by our RAHF.

用预测热图与分数做区域修复

我们展示：可用模型预测的热图与分数做区域修复以提升生成质量。对每张图先预测不合理热图，再经阈值与膨胀等处理得到掩码；在掩码区域内用 Muse 修复 Chang et al. 2023 按文本提示生成新图。生成多张后，取 RAHF 预测合理性分最高者作为最终结果。

In Fig. 9, we show several inpainting results with our predicted implausibility heatmaps and plausibility scores. As shown, more plausible images with fewer artifacts are generated after inpainting. Again, this shows that our RAHF generalizes well to images from a generative model very different from the ones whose images are used to train RAHF. More details and examples can be found in the supplementary material.

图 9 给出若干用预测不合理热图与合理性分做修复的结果：修复后图像更合理、伪影更少。这再次表明 RAHF 能泛化到与训练所用生成模型差异很大的图像。更多细节与例子见补充材料。

6 Conclusions and limitations

6 结论与局限

In this work, we contributed RichHF-18K, the first rich human feedback dataset for image generation. We designed and trained a multimodal Transformer to predict the rich human feedback, and demonstrated some instances to improve image generation with our rich human feedback.

本文贡献了面向图像生成的首个丰富人类反馈数据集 RichHF-18K；设计并训练多模态 Transformer 预测这些反馈，并展示用丰富人类反馈改进图像生成的若干实例。

While some of our results are quite exciting and promising, there are several limitations to our work. First, the model performance on the misalignment heatmap is worse than that on the implausibility heatmaps, possibly due to the noise in the misalignment heatmap. It is somewhat ambiguous how to label some misalignment cases such as absent objects on the image. Improving the misalignment label quality is one of the future directions. Second, it would be helpful to collect more data on generative models beyond Pick-a-Pic (Stable Diffusion) and investigate their effect on the RAHF models. Moreover, while we present three promising ways to leverage our model to improve T2I generation, there is a myriad of other ways to utilize rich human feedback that can be explored, e.g., how to use the predicted heatmaps or scores as a reward signal to finetune generative models with reinforcement learning, and how to use the predicted heatmaps as a weighting map, or how to use the predicted misaligned sequences in learning from human feedback to help improve image generation, etc. We hope RichHF-18K and our initial models inspire quests to investigate these research directions in future work.

结果令人振奋，但工作仍有局限。第一，不对齐热图表现弱于不合理热图，可能因不对齐热图噪声更大；图像上缺失物体等情形如何标注也较含糊，提升不对齐标注质量是未来方向之一。第二，宜在 Pick-a-Pic（Stable Diffusion）之外的生成模型上收集更多数据，并研究其对 RAHF 的影响。此外，我们展示了三种利用模型改进 T2I 的途径，还有许多用法可探索，例如把预测热图或分数作强化学习微调的奖励信号、把热图作加权图、或在从人类反馈学习中使用预测的不对齐序列等。希望 RichHF-18K 与初步模型能启发后续沿这些方向推进。

References

参考文献

Supplementary Material

补充材料

Anil et al. (2023) Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403 , 2023.

Bansal et al. (2023) Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models, 2023.

Brock et al. (2018) Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 , 2018.

Brooks et al. (2023) Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18392–18402, 2023.

Bylinskii et al. (2018) Zoya Bylinskii, Tilke Judd, Aude Oliva, Antonio Torralba, and Frédo Durand. What do different evaluation metrics tell us about saliency models? IEEE Transactions on Pattern Analysis and Machine Intelligence , 2018.

Chang et al. (2023) Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704 , 2023.

Chen et al. (2022) Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali: A jointly-scaled multilingual language-image model, 2022.

Cho et al. (2023a) Jaemin Cho, Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian Scene Graph: Improving Reliability in Fine-Grained Evaluation for Text-to-Image Generation. In arXiv:2310.18235 , 2023a.

Cho et al. (2023b) Jaemin Cho, Abhay Zala, and Mohit Bansal. Visual programming for text-to-image generation and evaluation. In NeurIPS , 2023b.

Cho et al. (2023c) Jaemin Cho, Abhay Zala, and Mohit Bansal. Visual programming for text-to-image generation and evaluation. In NeurIPS , 2023c.

Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

Croitoru et al. (2023) Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Dhariwal and Nichol (2021) Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems , 34:8780–8794, 2021.

Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Fan et al. (2023) Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. In Advances in Neural Information Processing Systems , 2023.

Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems , 27, 2014.

Gu et al. (2023) Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Josh Susskind, and Navdeep Jaitly. Matryoshka diffusion models, 2023.

He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770–778, 2016.

Hessel et al. (2022) Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In EMNLP , 2022.

Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Higgins et al. (2016) Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations , 2016.

Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840–6851, 2020.

Ho et al. (2022) Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 , 2022.

Hu et al. (2023a) Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. TIFA: Accurate and interpretable text-to-image faithfulness evaluation with question answering. arXiv preprint arXiv:2303.11897 , 2023a.

Hu et al. (2023b) Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A. Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In ICCV , 2023b.

Huang et al. (2023) Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. arXiv preprint arXiv:2307.06350 , 2023.

Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401–4410, 2019.

Kawar et al. (2023) Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In CVPR , 2023.

Ke et al. (2021) Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021.

Kingma and Welling (2013) Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Kirstain et al. (2023) Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. arXiv preprint arXiv:2305.01569 , 2023.

Lee et al. (2023) Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Benita Teufel, Marco Bellagente, Minguk Kang, Taesung Park, Jure Leskovec, Jun-Yan Zhu, Li Fei-Fei, Jiajun Wu, Stefano Ermon, and Percy Liang. Holistic evaluation of text-to-image models. In Neural Information Processing Systems Datasets and Benchmarks Track , 2023.

Li et al. (2019) Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr. Controllable text-to-image generation. Advances in Neural Information Processing Systems , 32, 2019.

Li and Li (2023) Gang Li and Yang Li. Spotlight: Mobile ui understanding using vision-language models with a focus. In International Conference on Learning Representations , 2023.

Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning , pages 12888–12900. PMLR, 2022.

Ni et al. (2023) Haomiao Ni, Changhao Shi, Kai Li, Sharon X. Huang, and Martin Renqiang Min. Conditional image-to-video generation with latent flow diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 18444–18455, 2023.

Nichol et al. (2021) Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021.

Otani et al. (2023) Mayu Otani, Riku Togashi, Yu Sawai, Ryosuke Ishigami, Yuta Nakashima, Esa Rahtu, Janne Heikkilä, and Shin’ichi Satoh. Toward verifiable and reproducible human evaluation for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14277–14286, 2023.

Qiao et al. (2019) Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. Mirrorgan: Learning text-to-image generation by redescription. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1505–1514, 2019.

Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748–8763. PMLR, 2021.

Roberts et al. (2022) Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. Scaling up models and data with t5x and seqio . arXiv preprint arXiv:2203.17189 , 2022.

Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684–10695, 2022.

Saharia et al. (2022) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 35:36479–36494, 2022.

Salimans et al. (2016) Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems , 29, 2016.

Sheynin et al. (2023) Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks, 2023.

Simonyan et al. (2014) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings , 2014.

Sohl-Dickstein et al. (2015) Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pages 2256–2265. PMLR, 2015.

Tao et al. (2022) Ming Tao, Hao Tang, Fei Wu, Xiao-Yuan Jing, Bing-Kun Bao, and Changsheng Xu. Df-gan: A simple and effective baseline for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16515–16525, 2022.

Van Den Oord et al. (2017) Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems , 30, 2017.

Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Wang et al. (2023) S. Wang, C. Saharia, C. Montgomery, J. Pont-Tuset, S. Noy, S. Pellegrini, Y. Onoe, S. Laszlo, D. J. Fleet, R. Soricut, J. Baldridge, M. Norouzi, P. Anderson, and W. Chan. Imagen editor and editbench: Advancing and evaluating text-guided image inpainting. In CVPR , 2023.

Wu et al. (2023a) Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341 , 2023a.

Wu et al. (2023b) Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2096–2105, 2023b.

Xing et al. (2023) Zhen Xing, Qijun Feng, Haoran Chen, Qi Dai, Han Hu, Hang Xu, Zuxuan Wu, and Yu-Gang Jiang. A survey on video diffusion models. arXiv preprint arXiv:2310.10647 , 2023.

Xu et al. (2023) Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. In Neural Information Processing Systems , 2023.

Xu et al. (2018) Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1316–1324, 2018.

Yang et al. (2022) Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys , 2022.

Yarom et al. (2023) Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text-image alignment evaluation, 2023.

Yu et al. (2022) Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789 , 2(3):5, 2022.

Zhang et al. (2023a) Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon. Text-to-image diffusion model in generative ai: A survey. arXiv preprint arXiv:2303.07909 , 2023a.

Zhang et al. (2017) Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision , pages 5907–5915, 2017.

Zhang et al. (2023b) Lingzhi Zhang, Zhengjie Xu, Connelly Barnes, Yuqian Zhou, Qing Liu, He Zhang, Sohrab Amirghodsi, Zhe Lin, Eli Shechtman, and Jianbo Shi. Perceptual artifacts localization for image synthesis tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7579–7590, 2023b.

Zhu et al. (2019) Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5802–5810, 2019.

7 Ethical conduct

7 伦理规范

Our data collection has been approved by an Institutional Review Board.

我们的数据收集已通过机构审查委员会（IRB）批准。

8 Data collection details

8 数据收集细节

8.1 Image artifacts/implausibility definitions

8.1 图像伪影/不合理定义

1. Distorted human/animal bodies/faces (a) Distorted/combined faces and/or body parts (unless specified in the text caption) (b) Missing body parts (unless specified in the text caption) (c) Additional body parts (unless specified in the text caption)
2. Distorted objects (non human/animal) (a) Distorted objects (e.g., furniture, vehicles, buildings) (unless specified in the text caption)
3. Distorted/Nonsensical text (a) Text that is distorted, nonsensical, or misspelled (unless specified in the text caption)
4. Nonsensical Representations (a) Representations that are unrealistic/nonsensical (unless specified in the text caption), or difficult to understand
5. Excessive blurriness/lack of sharpness (a) The image contains excessive blurriness or quality that detracts from the overall image (focus on one part of the image is OK) (b) The image contains a lack of definition/sharpness that detracts from the overall image
6. Any other artifacts or implausibility not covered above

1. 扭曲的人体/动物身体/面部 (a) 扭曲/拼合的面部和/或身体部位（除非文本说明） (b) 缺失身体部位（除非文本说明） (c) 多余身体部位（除非文本说明）
2. 扭曲物体（非人/动物） (a) 扭曲的物体（如家具、车辆、建筑）（除非文本说明）
3. 扭曲/无意义文字 (a) 扭曲、无意义或拼写错误的文字（除非文本说明）
4. 无意义表征 (a) 不真实/无意义或难以理解的表征（除非文本说明）
5. 过度模糊/缺乏锐度 (a) 图像过度模糊或质量差到损害整体观感（局部对焦可以） (b) 图像缺乏清晰度/锐度以致损害整体观感
6. 上述未覆盖的其他伪影或不合理

8.2 Text-image misalignment definitions and what-to-do

8.2 图文不对齐定义与操作说明

Since we require the annotators to mark the misaligned words in the text prompt, we differentiate this part from Section 8.1 by including a what-to-do under each definition.

由于要求标注者标出提示中的不对齐词，本节在每条定义下附上操作说明，以区别于第 8.1 节。

1. Something is missing: a human/animal/object specified in the text caption is missing in the image • Click on that word of the human/animal/object in the text
2. Incorrect attributes: an attribute (e.g., color) of an object specified in the text is incorrect in the image • Click on that word of the attribute in the text and click on the region of the object on the image
3. Incorrect actions: an action specified in the text caption is not represented in the image • Click on that word of the action in the text and click on the region of the wrong actions on the image
4. Incorrect numbers: counts of humans/animals/objects in the image do not match those specified in the text • Click on the number in the text
5. Incorrect position: the spatial position of two entities in the image does not match that specified in the text • Click on the word of the position in the text
6. Other: any other inconsistency between text and image • Click on the word of the inconsistency in the text

1. 有东西缺失：文本中写明的人/动物/物体在图像中缺失 • 点击文本中该人/动物/物体对应的词
2. 属性错误：文本中物体的某属性（如颜色）在图像中不正确 • 点击文本中该属性词，并点击图像上该物体区域
3. 动作错误：文本中的动作未在图像中体现 • 点击文本中该动作词，并点击图像上错误动作区域
4. 数量错误：图像中人/动物/物体数量与文本不符 • 点击文本中的数量词
5. 位置错误：图像中两实体的空间位置与文本不符 • 点击文本中表示位置的词
6. 其他：图文之间的其他不一致 • 点击文本中体现该不一致的词

8.3 Additional details

8.3 其他细节

Annotation guideline

To ensure the annotators understand the above definitions, we provide 4-10 examples for each definition of the annotation terms in the guideline. All of our annotators can read English and thus understand the text prompts. In some of the prompts, there are concepts or person names in the text prompts that are uncommon and may cause confusion to the annotators. Therefore, we instruct the annotators to do a quick search on the internet regarding any unfamiliar concepts in the text prompts and skip samples with confusing prompts full of strange concepts.

标注指南

为保证标注者理解上述定义，指南中为每类标注术语提供 4–10 个例子。全体标注者均可读英语，能理解文本提示。部分提示含生僻概念或人名，可能造成困惑；我们要求标注者对不熟悉概念快速检索，并跳过充满怪异概念、令人困惑的提示。

Annotation interface

We designed a web UI to facilitate data collection with the following principles: 1) convenience for annotators to perform annotations, ideally within a short time for an image-text pair and, 2) allowing annotators to perform all annotations on the same UI, so that the fine-grained scores are based on the annotated regions and keywords. To this end, we created the interface as illustrated in Fig. 1. The main UI consists of an image displayed on the left and a panel on the right, where the text prompt is shown at the top of the panel. Annotators are asked to first click on the image to annotate the artifact/implausible regions and misalignment regions, and then select the misaligned keywords and the fine-grained scores on the right of the panel.

标注界面

我们设计网页 UI 以便收集数据，原则是：1）方便标注者操作，理想情况下一对图文可在短时间内完成；2）同一 UI 上完成全部标注，使细粒度分数建立在已标区域与关键词之上。界面如图 1：左侧显示图像，右侧面板顶部为文本提示。标注者先在图上点击标出伪影/不合理区域与不对齐区域，再在右侧选择不对齐关键词与细粒度分数。

More details

We created detailed annotation guidelines to instruct the annotators regarding the annotation steps, interactions with the web UI, examples of different types of implausibility, artifacts, and misalignment. All the annotators (27 in total) are trained with the annotation guidelines and calibrated, before they perform the annotation in order to reduce annotation discrepancy and improve quality. Our annotation took around 3,000 rater-hours in total. To improve the effectiveness of the collected dataset and control the time spent on annotation, we filter out any image-text pairs that have a text prompt with less than 3 words or more than 20 words. We also filter out non-English prompts or any prompts containing emoji.

更多细节

我们编写了详细标注指南，说明步骤、与网页 UI 的交互，以及各类不合理、伪影与不对齐的例子。全部标注者（共 27 人）先经指南培训并校准，再正式标注，以减少差异、提高质量。标注合计约 3000 人时。为提高数据有效性并控制耗时，过滤提示不足 3 词或超过 20 词的图文对，并过滤非英语提示或含 emoji 的提示。

Dataset size

Since the Pick-a-Pic v1 dataset contains some images and/or prompts that are inappropriate (e.g., containing nudity), we ask the annotators to mark these images with a special flag and skip the annotation. We filter out these inappropriate images and/or prompts during data post-processing. For this reason, the total number of images in our final training set is around 300 short of 16K.

数据集规模

Pick-a-Pic v1 中部分图像和/或提示不适宜（例如含裸露内容），我们请标注者用特殊标记标出并跳过。后处理时过滤这些不适宜样本。因此最终训练集图像数约比 1.6 万少 300 张。

Additional details of data collection

The distribution of the attributes of the 16K training samples is shown in Fig. 10. We can see a relatively balanced distribution of the types of content in the generated images in our dataset.

数据收集补充

1.6 万训练样本的属性分布见图 10。可见生成图像内容类型分布相对均衡。

Figure 10: Histograms of the PaLI attributes of the images in the training set.

图 10：训练集图像 PaLI 属性的直方图。

8.4 Discussions and limitations

8.4 讨论与局限

We choose points over bounding boxes in our region annotation because we find that points are much faster to mark and can have a reasonable coverage over image regions with various shapes when we specify an effective radius for each point as discussed in the main paper.

区域标注我们选择点而非边界框，因为点标记更快；在为每个点规定有效半径后（正文已述），对各种形状区域也能有合理覆盖。

As a limitation in our region/heatmap annotations, we notice there is an over-annotation issue in the artifacts/implausibility region annotation. Specifically, our annotators tend to annotate more human faces and hands on the images than necessary. One reason is that human faces and hands in the Pick-a-Pic dataset indeed have more artifacts/implausibility than other parts. Moreover, the annotators, as humans, may naturally pay more attention to human faces and hands, resulting in over-annotation of these parts. Nevertheless, the over-annotation issue is minor in our final dataset, as we strive to provide feedback to the annotators frequently to make them less nitpicking about human faces and hands.

区域/热图标注的一个局限是伪影/不合理区域存在过度标注：标注者往往对人脸与手标得过多。一是 Pick-a-Pic 中人脸与手确实比其他部位伪影/不合理更多；二是作为人类，标注者自然更关注脸与手。不过最终数据集中过度标注问题较轻——我们频繁向标注者反馈，减少对人脸与手的过度挑剔。

Another limitation is the diversity of the subjects in the prompts/images. The Pick-a-Pic dataset (and many others) is predominantly full of human, dog, and cat subjects. For this reason, it is challenging to find a very diverse dataset for annotation. We strive to make the subjects more diverse by using balanced categories as indicated by the PaLI attributes (as in Fig. 10). We didn’t choose more fine-grained categories for PaLI to test as there would be an endless list of subjects we could consider. Therefore, we leave the goal of annotating more diverse images/prompts in future works.

另一局限是提示/图像中主体的多样性。Pick-a-Pic（及许多其他数据集）以人、狗、猫为主，很难找到极多样的标注集。我们尽量用 PaLI 属性（图 10）做类别均衡。未对 PaLI 选用更细类别，因为可考虑的主体几乎无穷。把更多样图像/提示纳入标注留给未来工作。

9 Experimental details

9 实验细节

Hyperparameters

The main model components consist of a ViT B16 encoder for image encoding, a T5 base encoder for mixing image and text tokens, and three predictors for score, heatmap, and text misalignment, respectively. The ViT B16 encoder uses a 16x16 patch size, 12 layers with 12 heads with a hidden dimension of 768, wherein the MLP has a hidden dimension of 3072. The T5 base encoder uses 12 layers with 12 heads with a hidden dimension of 768, wherein the MLP has a hidden dimension of 2048. The score predictor consists of four convolutional layers with layer norm and ReLU activation, and the filter size, kernel size, and strides are $[768,384,128,64],[2,2,2,2],[1,1,1,1]$, respectively. Three dense layers of output sizes 2048, 1024, and 1, respectively, are used to generate a scalar with ReLU activation for the first two layers and sigmoid for the last. The heatmap predictor consists of two convolution layers with filter size, kernel size, and stride as $[768,384],[3,3],[1,1]$, respectively. It then uses four de-convolution layers to up-sample to the required output size, with the filter size, kernel size, and stride as $[768,384,384,192],[3,3,3,3],[2,2,2,2]$, respectively. Each de-convolution layer is with two read-out convolution layers of kernel size 3 and stride 1. Layer norm and ReLU are used for each layer. In the end, two read-out convolution layers and a final sigmoid activation are used to generate the heatmap prediction. The text predictor is implemented using a T5 base decoder with 12 layers of 12 heads, MLP dimension 2048, and hidden dimension 768. The output token length is 64.

超参数

主模型组件包括：用于图像编码的 ViT B16 编码器、混合图文 token 的 T5 base 编码器，以及分别用于分数、热图与文本不对齐的三个预测器。ViT B16 使用 16×16 patch，12 层、12 头、隐维 768，MLP 隐维 3072。T5 base 编码器为 12 层、12 头、隐维 768，MLP 隐维 2048。分数预测器含四层卷积（带 LayerNorm 与 ReLU），滤波数、核大小与步长分别为 $[768,384,128,64]$、$[2,2,2,2]$、$[1,1,1,1]$；再接输出维 2048、1024、1 的三层全连接，前两层 ReLU、末层 sigmoid，得到标量。热图预测器含两层卷积，滤波数、核与步长为 $[768,384]$、$[3,3]$、$[1,1]$；再经四层反卷积上采样到目标尺寸，滤波数、核与步长为 $[768,384,384,192]$、$[3,3,3,3]$、$[2,2,2,2]$；每层反卷积后接两层核为 3、步长 1 的读出卷积；各层用 LayerNorm 与 ReLU；最后两层读出卷积加 sigmoid 得到热图。文本预测器用 T5 base 解码器：12 层、12 头、MLP 维 2048、隐维 768，输出 token 长度 64。

We train the model on the datasets with a batch size of 256 for 20K iterations. We utilize the AdamW optimizer with a base learning rate of 0.015. We linearly increase the learning rate from 0 to the base learning rate in the first 2000 iterations, and then decrease the learning rate with a reciprocal square root scheduler w.r.t the number of iterations. We trained the model using 64 Google Cloud TPU v3 chips.

在数据集上以 batch size 256 训练 2 万步。优化器用 AdamW，基础学习率 0.015；前 2000 步从 0 线性升至基础学习率，其后按步数 reciprocal square root 调度衰减。训练使用 64 块 Google Cloud TPU v3。

Image augmentations

For each image, we randomly crop it by sampling a bounding box with 80%-100% width and 80%-100% height. The cropping is applied by 50% chance and otherwise the original image is used. Note that we also crop the corresponding part of the implausibility heatmap and misalignment heatmap to match the cropped image. We then create an augmented version of the image by applying several random augmentations including random brightness (max delta 0.05), random contrast (random contrast factor between 0.8 and 1), random hue (max delta 0.025), random saturation (random saturation factor between 0.8 and 1) and random jpeg noise (jpeg quality between 70 and 100). By 10% chance the augmented version is used instead of the original image. We convert the image to grayscale by 10% probability as the final image.

图像增强

对每张图，以 80%–100% 宽与 80%–100% 高随机采样边界框做裁剪；50% 概率应用裁剪，否则用原图。不合理热图与不对齐热图同步裁剪以匹配。再以随机亮度（最大 delta 0.05）、随机对比度（因子 0.8–1）、随机色调（最大 delta 0.025）、随机饱和度（因子 0.8–1）与随机 JPEG 噪声（质量 70–100）得到增强版；10% 概率用增强版替代原图。最终以 10% 概率转为灰度。

Finetuning generative models with predicted scores

To generate the training prompt set, we provide five hand-crafted seed prompts as examples and then ask PaLM 2 Anil et al. 2023 to generate similar textual prompts. We include additional instructions that specify the prompt length and the object category. We then explain why we do not use existing benchmark datasets for training. Theoretically, we can get an infinite number of prompts using the prompt synthesis technique we proposed above. Existing datasets are 1) relatively small (e.g., TIFA Hu et al. 2023a has 4k prompts, Davidsonian Scene Graph (DSG) Cho et al. 2023c has only 1k prompts), or 2) containing prompts that are simple and not diverse enough, for example, only measuring single objects in Parti benchmark Yu et al. 2022. This motivates us to synthesize a larger set of diverse prompts for training purposes. For the 100 prompts for our human evaluation, they are sampled from the existing benchmark: TIFA Hu et al. 2023a. We only did our evaluation on 100 prompts due to the high cost of the human annotation.

用预测分数微调生成模型

为合成训练提示集，我们提供五条手工种子提示为例，请 PaLM 2 Anil et al. 2023 生成相似文本提示，并附加关于提示长度与物体类别的说明。下面说明为何不用现有基准做训练。理论上上述合成可得到近乎无限提示；现有数据集则 1）规模偏小（如 TIFA Hu et al. 2023a 约 4k 提示，Davidsonian Scene Graph (DSG) Cho et al. 2023c 仅约 1k），或 2）提示简单、不够多样，例如 Parti Yu et al. 2022 主要测单物体。这促使我们合成更大、更多样的训练提示集。人工评测用的 100 条提示采自现有基准 TIFA Hu et al. 2023a；因人工标注成本高，仅评测 100 条。

10 Additional qualitative examples

10 更多定性例子

Fig. 11 provides more examples of artifacts/implausibility heatmaps. We can see that our RAHF model can more accurately locate the positions of artifacts/implausibility on various subjects such as human hands, animals, vehicles, and concept arts.

图 11 给出更多伪影/不合理热图例子。可见 RAHF 能在人手、动物、车辆、概念艺术等多种主体上较准确定位伪影/不合理位置。

(a) Image

(a) 图像

Fig. 12 provides more examples of misalignment heatmaps. We can see that our RAHF model can more accurately locate the positions of misalignment on various subjects such as animals, objects, and different outdoor scenes. For example, our model can identify the subtle difference between the real handlebar of a Segway and the one depicted in the image.

图 12 给出更多不对齐热图例子。可见 RAHF 能在动物、物体与不同室外场景上较准确定位不对齐；例如能区分赛格威真实车把与图像中画出的车把之间的细微差别。

Image          GT          Our model         CLIP gradient

(a) Prompt: Photo of a cat eating a burger like a person

图像        真值        本文模型       CLIP 梯度

(a) 提示：Photo of a cat eating a burger like a person

Fig. 13 provides more examples of score predictions, where our RAHF model predicts scores that are quite close to the ground truth score from human evaluation.

图 13 给出更多分数预测例子；RAHF 预测分与人类评测真值相当接近。

Fig. 14 provides examples for the misaligned keywords prediction, which shows that our RAHF model can predict the majority of the misaligned keywords marked by human annotators.

图 14 给出不对齐关键词预测例子；RAHF 能预测人类标注者标出的多数不对齐关键词。

Fig. 15 provides more examples of the comparison before and after finetuning Muse with examples selected based on the predicted scores by our RAHF model and examples of using RAHF model predicted overall score as Classifier Guidance. We can see enhanced image quality of the generation from the finetuned Muse model and the Latent Diffusion model, which highlights the potential of improving T2I generation with our reward model.

图 15 给出更多对比：按 RAHF 预测分数筛选样本微调 Muse 前后，以及用 RAHF 预测总体分作分类器引导。可见微调后 Muse 与潜扩散模型的生成质量提升，凸显用本奖励模型改进 T2I 的潜力。

Fig. 16 provides more examples of Muse inpainting with the predicted masks (converted from heatmaps) by our RAHF model, where the inpainted regions are significantly improved in plausibility.

图 16 给出更多用 RAHF 预测掩码（由热图转换）做 Muse 修复的例子；被修复区域合理性明显提升。

(a) Prompt: Computer science students fighting with computer keyboards. Plausibility score. GT: 0.25, Our model: 0.236 Overall score. GT: 0.5, Our model: 0.341

(a) 提示：Computer science students fighting with computer keyboards。合理性分。真值：0.25，本文模型：0.236。总体分。真值：0.5，本文模型：0.341

(a) The prompt is: Two cats watering roses in a greenhouse. The ground truth labels two, watering, greenhouse as misaligned keywords and our model predicts two, greenhouse as misaligned keywords. This text will be invisible, just to occupy some spaces.

(a) 提示为：Two cats watering roses in a greenhouse。真值将 two, watering, greenhouse 标为不对齐关键词，本文模型预测 two, greenhouse。本段文字不可见，仅占位。

(a) Muse before finetuning

(a) Muse 微调前

(a) Prompt: A 3d printed sculpture of a cat made of iron and plastic, with arabic translation and ic gradients.

(a) 提示：A 3d printed sculpture of a cat made of iron and plastic, with arabic translation and ic gradients.