User preference prediction requires a comprehensive and accurate understanding of individual tastes. This includes both surface-level attributes, such as color and style, and deeper content-related aspects, such as themes and composition. However, existing methods typically rely on general human preferences or assume static user profiles, often neglecting individual variability and the dynamic, multifaceted nature of personal taste. To address these limitations, we propose an approach built upon Multimodal Large Language Models, introducing contrastive preference loss and preference tokens to learn personalized user preferences from historical interactions. The contrastive preference loss is designed to effectively distinguish between user ''likes'' and ''dislikes'', while the learnable preference tokens capture shared interest representations among existing users, enabling the model to activate group-specific preferences and enhance consistency across similar users. Extensive experiments demonstrate our model outperforms other methods in preference prediction accuracy, effectively identifying users with similar aesthetic inclinations and providing more precise guidance for generating images that align with individual tastes. The project page is \texttt{this https URL}.
用户偏好预测需要对个人品味进行全面而准确的理解,这包括表层属性(如颜色和风格)以及更深层次的内容相关方面(如主题和构图)。然而,现有的方法通常依赖于一般的人类偏好或假设静态的用户档案,往往忽视了个体差异和个人喜好的动态、多面性。为了解决这些限制,我们提出了一种基于多模态大型语言模型的方法,并引入对比偏好损失和偏好标记来从历史交互中学习个性化的用户偏好。对比偏好损失旨在有效地区分用户的“喜欢”和“不喜欢”,而可学习的偏好标记则捕获现有用户之间的共享兴趣表示,使模型能够激活特定群体的偏好并增强类似用户的一致性。广泛的实验表明,我们的模型在偏好预测准确性方面优于其他方法,可以有效地识别具有相似审美倾向的用户,并为生成与个人品味相符的图像提供更精确的指导。 项目页面位于:\[此URL\]
https://arxiv.org/abs/2508.08220
Diffusion models (DMs) have demonstrated an unparalleled ability to create diverse and high-fidelity images from text prompts. However, they are also well-known to vary substantially regarding both prompt adherence and quality. Negative prompting was introduced to improve prompt compliance by specifying what an image must not contain. Previous works have shown the existence of an ideal negative prompt that can maximize the odds of the positive prompt. In this work, we explore relations between negative prompting and classifier-free guidance (CFG) to develop a sampling procedure, {\it Adaptive Negative Sampling Without External Resources} (ANSWER), that accounts for both positive and negative conditions from a single prompt. This leverages the internal understanding of negation by the diffusion model to increase the odds of generating images faithful to the prompt. ANSWER is a training-free technique, applicable to any model that supports CFG, and allows for negative grounding of image concepts without an explicit negative prompts, which are lossy and incomplete. Experiments show that adding ANSWER to existing DMs outperforms the baselines on multiple benchmarks and is preferred by humans 2x more over the other methods.
扩散模型(DM)已经展示了从文本提示生成多样且高质量图像的无与伦比的能力。然而,它们在遵循提示和质量方面也表现出相当大的差异性。为了解决这一问题,引入了负向提示来明确说明图像中不应包含的内容,从而提高对正向提示的遵从度。先前的研究表明存在一个理想的负向提示可以最大化正向提示的成功率。 在这项工作中,我们探讨了负向提示与无分类器指导(CFG)之间的关系,并开发了一种采样程序——“无需外部资源的自适应负样本生成”(ANSWER),该程序能够在单个提示下同时考虑正面和负面条件。此方法利用扩散模型对否定的理解能力来提高图像忠实于提示的概率。ANSWER是一种不需要训练的技术,适用于任何支持CFG的模型,并且可以在没有显式负向提示的情况下实现图像概念的负向锚定,而显式的负向提示往往是低效且不完整的。 实验结果表明,在现有DM中加入ANSWER在多个基准测试上超越了基线方法,并且人类更偏爱该方法2倍于其他方法。
https://arxiv.org/abs/2508.02973
Cooking process visualization is a promising task in the intersection of image generation and food analysis, which aims to generate an image for each cooking step of a recipe. However, most existing works focus on generating images of finished foods based on the given recipes, and face two challenges to visualize the cooking process. First, the appearance of ingredients changes variously across cooking steps, it is difficult to generate the correct appearances of foods that match the textual description, leading to semantic inconsistency. Second, the current step might depend on the operations of previous step, it is crucial to maintain the contextual coherence of images in sequential order. In this work, we present a cooking process visualization model, called Chain-of-Cooking. Specifically, to generate correct appearances of ingredients, we present a Dynamic Patch Selection Module to retrieve previously generated image patches as references, which are most related to current textual contents. Furthermore, to enhance the coherence and keep the rational order of generated images, we propose a Semantic Evolution Module and a Bidirectional Chain-of-Thought (CoT) Guidance. To better utilize the semantics of previous texts, the Semantic Evolution Module establishes the semantical association between latent prompts and current cooking step, and merges it with the latent features. Then the CoT Guidance updates the merged features to guide the current cooking step remain coherent with the previous step. Moreover, we construct a dataset named CookViz, consisting of intermediate image-text pairs for the cooking process. Quantitative and qualitative experiments show that our method outperforms existing methods in generating coherent and semantic consistent cooking process.
烹饪过程可视化是在图像生成和食品分析交叉领域的一项有前景的任务,旨在为食谱中的每一步骤生成一张图像。然而,大多数现有的工作主要集中在根据给定的食谱生成完成食物的图片上,并且在可视化烹饪过程时面临着两个挑战:首先,食材外观会随着步骤的变化而发生变化,因此很难根据文本描述生成匹配的食物外观,从而导致语义不一致;其次,当前步骤可能依赖于前一操作的结果,所以保持图像序列中的上下文连贯性至关重要。为此,在这项工作中我们提出了一种名为“Cooking Chain”的烹饪过程可视化模型。 具体而言,为了生成正确的食材外观,我们设计了一个动态补丁选择模块来检索先前生成的与当前文本内容最相关的图像片段作为参考。此外,为增强图像的一致性和保持合理顺序,我们提出了语义演变模块和双向思考链(CoT)引导策略。为了更好地利用之前文本的语义信息,语义演变模块建立了隐式提示和当前烹饪步骤之间的语义关联,并将其与潜在特征进行融合。随后,CoT 引导更新这些合并的特征以指导当前步骤保持连贯性,使其与前一步骤一致。 此外,我们构建了一个名为CookViz的数据集,该数据集中包含了用于描述整个烹饪过程中的中间图像-文本对。定量和定性的实验表明,在生成连贯且语义一致的烹饪过程中,我们的方法优于现有的其他方法。
https://arxiv.org/abs/2507.21529
When synthesizing identities as face recognition training data, it is generally believed that large inter-class separability and intra-class attribute variation are essential for synthesizing a quality dataset. % This belief is generally correct, and this is what we aim for. However, when increasing intra-class variation, existing methods overlook the necessity of maintaining intra-class identity consistency. % To address this and generate high-quality face training data, we propose Vec2Face+, a generative model that creates images directly from image features and allows for continuous and easy control of face identities and attributes. Using Vec2Face+, we obtain datasets with proper inter-class separability and intra-class variation and identity consistency using three strategies: 1) we sample vectors sufficiently different from others to generate well-separated identities; 2) we propose an AttrOP algorithm for increasing general attribute variations; 3) we propose LoRA-based pose control for generating images with profile head poses, which is more efficient and identity-preserving than AttrOP. % Our system generates VFace10K, a synthetic face dataset with 10K identities, which allows an FR model to achieve state-of-the-art accuracy on seven real-world test sets. Scaling the size to 4M and 12M images, the corresponding VFace100K and VFace300K datasets yield higher accuracy than the real-world training dataset, CASIA-WebFace, on five real-world test sets. This is the first time a synthetic dataset beats the CASIA-WebFace in average accuracy. In addition, we find that only 1 out of 11 synthetic datasets outperforms random guessing (\emph{i.e., 50\%}) in twin verification and that models trained with synthetic identities are more biased than those trained with real identities. Both are important aspects for future investigation.
在将身份合成用作面部识别训练数据时,通常认为大类间分离性和类内属性变化对于生成高质量的数据集至关重要。这种看法通常是正确的,这也是我们的目标所在。然而,在增加类内变化的同时,现有方法往往忽略了保持类内身份一致性的重要性。为了应对这一挑战并生成高质量的面部训练数据,我们提出了一种名为Vec2Face+的生成模型,该模型可以直接从图像特征创建图像,并允许对人脸身份和属性进行连续且易于控制的操作。通过使用Vec2Face+,我们采用三种策略获得了具备适当类间分离性和类内变化及一致性身份的数据集:1)采样足够不同于其他样本的向量以生成具有较好隔离性的身份;2)提出了一种AttrOP算法来增加一般属性的变化;3)提出了基于LoRA的姿态控制方法,用于生成带有侧脸姿态的图像,这种方法比AttrOP更高效且更能保持身份一致性。 我们的系统产生了VFace10K,这是一个拥有10,000个不同身份的人工合成面部数据集,在七个真实世界的测试集中使面部识别模型达到了最先进的精度水平。当我们扩展该数据集规模至4M和12M图像时,对应的VFace100K和VFace300K数据集在五个实际测试集合上比现实世界训练数据集CASIA-WebFace的平均准确度更高。这是首次合成数据集在平均精度上超越了CASIA-WebFace。 此外,我们发现只有11个合成数据集中的一项超过了随机猜测(即50%)的表现,在双胞胎验证任务中。另外,使用合成身份训练出的模型比使用真实身份训练出的模型更加偏颇。这两者都是未来研究的重要方面。
https://arxiv.org/abs/2507.17192
Most sign language handshape datasets are severely limited and unbalanced, posing significant challenges to effective model training. In this paper, we explore the effectiveness of augmenting the training data of a handshape classifier by generating synthetic data. We use an EfficientNet classifier trained on the RWTH German sign language handshape dataset, which is small and heavily unbalanced, applying different strategies to combine generated and real images. We compare two Generative Adversarial Networks (GAN) architectures for data generation: ReACGAN, which uses label information to condition the data generation process through an auxiliary classifier, and SPADE, which utilizes spatially-adaptive normalization to condition the generation on pose information. ReACGAN allows for the generation of realistic images that align with specific handshape labels, while SPADE focuses on generating images with accurate spatial handshape configurations. Our proposed techniques improve the current state-of-the-art accuracy on the RWTH dataset by 5%, addressing the limitations of small and unbalanced datasets. Additionally, our method demonstrates the capability to generalize across different sign language datasets by leveraging pose-based generation trained on the extensive HaGRID dataset. We achieve comparable performance to single-source trained classifiers without the need for retraining the generator.
大多数手语手势数据集在规模和平衡性上都存在严重限制,这给有效的模型训练带来了重大挑战。本文探讨了通过生成合成数据来增强手势分类器训练数据有效性的方法。我们使用了一个基于RWTH德国手语手势数据集(该数据集较小且不平衡)训练的EfficientNet分类器,并应用不同的策略将生成的数据与真实图像结合。我们比较了两种用于数据生成的生成对抗网络(GAN)架构:ReACGAN,通过辅助分类器利用标签信息来调节数据生成过程;以及SPADE,它采用空间自适应归一化来根据姿势信息调整生成过程。ReACGAN能够生成与特定手势标签相符的真实图像,而SPADE则专注于生成具有准确手部姿态配置的图像。 我们提出的方法在RWTH数据集上将当前最先进的准确性提高了5%,解决了小规模和不平衡数据集的问题。此外,我们的方法还展示了跨不同手语数据集泛化的能力,通过基于姿势的生成技术,在庞大的HaGRID数据集上进行训练后实现这一点。我们在无需重新训练生成器的情况下,达到了与单一来源训练分类器相当的表现。 综上所述,本文提出的方法为解决手势识别中数据集限制问题提供了一种有效途径,并展示了其在不同手语数据集中泛化的潜力。
https://arxiv.org/abs/2507.17008
AutoRegressive (AR) models have made notable progress in image generation, with Masked AutoRegressive (MAR) models gaining attention for their efficient parallel decoding. However, MAR models have traditionally underperformed when compared to standard AR models. This study refines the MAR architecture to improve image generation quality. We begin by evaluating various image tokenizers to identify the most effective one. Subsequently, we introduce an improved Bidirectional LLaMA architecture by replacing causal attention with bidirectional attention and incorporating 2D RoPE, which together form our advanced model, MaskGIL. Scaled from 111M to 1.4B parameters, MaskGIL achieves a FID score of 3.71, matching state-of-the-art AR models in the ImageNet 256x256 benchmark, while requiring only 8 inference steps compared to the 256 steps of AR models. Furthermore, we develop a text-driven MaskGIL model with 775M parameters for generating images from text at various resolutions. Beyond image generation, MaskGIL extends to accelerate AR-based generation and enable real-time speech-to-image conversion. Our codes and models are available at this https URL.
自回归(AutoRegressive,AR)模型在图像生成领域取得了显著进展,而遮蔽自回归(Masked AutoRegressive,MAR)模型由于其高效的并行解码能力受到了广泛关注。然而,与标准的AR模型相比,传统的MAR模型性能较差。本研究对MAR架构进行了改进,以提高图像生成质量。我们首先评估了各种图像标记器,以确定最有效的标记器。随后,我们通过将因果注意力替换为双向注意力,并引入2D RoPE来改进Bidirectional LLaMA架构,从而形成我们的先进模型MaskGIL。从1.11亿参数扩展到14亿参数规模的MaskGIL,在ImageNet 256x256基准测试中达到了3.71的FID评分,与最先进的AR模型相匹配,同时仅需8个推理步骤,而相比之下AR模型需要256个步骤。 此外,我们开发了一个具有7.75亿参数、基于文本驱动的MaskGIL模型,用于在各种分辨率下从文本生成图像。除了图像生成之外,MaskGIL还扩展了基于自回归的生成加速,并实现了实时语音转图像转换。我们的代码和模型可在以下网址获取:[此处插入链接](请将实际URL填入)。
https://arxiv.org/abs/2507.13032
Recently, personalized portrait generation with a text-to-image diffusion model has significantly advanced with Textual Inversion, emerging as a promising approach for creating high-fidelity personalized images. Despite its potential, current Textual Inversion methods struggle to maintain consistent facial identity due to semantic misalignments between textual and visual embedding spaces regarding identity. We introduce ID-EA, a novel framework that guides text embeddings to align with visual identity embeddings, thereby improving identity preservation in a personalized generation. ID-EA comprises two key components: the ID-driven Enhancer (ID-Enhancer) and the ID-conditioned Adapter (ID-Adapter). First, the ID-Enhancer integrates identity embeddings with a textual ID anchor, refining visual identity embeddings derived from a face recognition model using representative text embeddings. Then, the ID-Adapter leverages the identity-enhanced embedding to adapt the text condition, ensuring identity preservation by adjusting the cross-attention module in the pre-trained UNet model. This process encourages the text features to find the most related visual clues across the foreground snippets. Extensive quantitative and qualitative evaluations demonstrate that ID-EA substantially outperforms state-of-the-art methods in identity preservation metrics while achieving remarkable computational efficiency, generating personalized portraits approximately 15 times faster than existing approaches.
最近,使用文本到图像扩散模型进行个性化肖像生成在引入Textual Inversion技术后取得了显著进展,成为创建高保真个性化图像的一种有前景的方法。尽管具有巨大潜力,现有的Textual Inversion方法却难以保持面部身份的一致性,因为文本和视觉嵌入空间之间关于身份的语义不匹配导致了这个问题。我们提出了ID-EA(Identity-Driven Embedding Alignment),这是一种新的框架,它引导文本嵌入与视觉身份嵌入对齐,从而在个性化生成中提高身份保存的效果。 ID-EA包括两个关键组成部分:由身份驱动的增强器(ID-Enhancer)和根据身份条件适配器(ID-Adapter)。首先,ID-Enhancer 将身份嵌入与文本 ID 锚点集成起来,并使用来自面部识别模型的视觉身份嵌入对代表性文本嵌入进行细化。然后,ID-Adapter 利用增强后的身份嵌入来调整文本条件,在预训练的 UNet 模型中通过调节交叉注意力模块确保身份保存。这一过程鼓励文本特征找到前景片段中的最相关视觉线索。 广泛的定量和定性评估表明,与现有方法相比,ID-EA 在身份保持度量方面显著超越了最先进的技术,并且实现了令人瞩目的计算效率,在生成个性化肖像时比现有的方法快大约15倍。
https://arxiv.org/abs/2507.11990
With recent advancements in text-to-image (T2I) models, effectively generating multiple instances within a single image prompt has become a crucial challenge. Existing methods, while successful in generating positions of individual instances, often struggle to account for relationship discrepancy and multiple attributes leakage. To address these limitations, this paper proposes the relation-aware disentangled learning (RaDL) framework. RaDL enhances instance-specific attributes through learnable parameters and generates relation-aware image features via Relation Attention, utilizing action verbs extracted from the global prompt. Through extensive evaluations on benchmarks such as COCO-Position, COCO-MIG, and DrawBench, we demonstrate that RaDL outperforms existing methods, showing significant improvements in positional accuracy, multiple attributes consideration, and the relationships between instances. Our results present RaDL as the solution for generating images that consider both the relationships and multiple attributes of each instance within the multi-instance image.
随着文本到图像(T2I)模型的近期进展,有效地在一个图片提示中生成多个实例已经成为一个重要的挑战。现有的方法虽然在生成单个实例的位置方面取得了成功,但在处理关系差异和多重属性泄露时往往力不从心。为了应对这些限制,本文提出了一种基于关系感知解耦学习(RaDL)框架。通过可学习的参数,RaDL增强了每个实例特有的属性,并通过使用从全局提示中提取的动作动词来生成具有关系意识的图像特征。利用Relation Attention机制,该方法能够更好地理解和处理实例之间的关系以及多重属性。 在COCO-Position、COCO-MIG和DrawBench等基准测试上的广泛评估表明,RaDL优于现有的方法,在位置准确性、考虑多重属性及实例间的关系方面均有显著改进。我们的结果将RaDL视为生成同时考虑每个实例之间关系及其多重属性的多实例图像的最佳解决方案。
https://arxiv.org/abs/2507.11947
The advancements in the domain of LLMs in recent years have surprised many, showcasing their remarkable capabilities and diverse applications. Their potential applications in various real-world scenarios have led to significant research on their reliability and effectiveness. On the other hand, multimodal LLMs and Text-to-Image models have only recently gained prominence, especially when compared to text-only LLMs. Their reliability remains constrained due to insufficient research on assessing their performance and robustness. This paper aims to establish a comprehensive evaluation framework for Text-to-Image models, concentrating particularly on their adherence to prompts. We created a novel dataset that aimed to assess the robustness of these models in generating images that conform to the specified factors of variation in the input text prompts. Our evaluation studies present findings on three variants of Stable Diffusion models: Stable Diffusion 3 Medium, Stable Diffusion 3.5 Large, and Stable Diffusion 3.5 Large Turbo, and two variants of Janus models: Janus Pro 1B and Janus Pro 7B. We introduce a pipeline that leverages text descriptions generated by the gpt-4o model for our ground-truth images, which are then used to generate artificial images by passing these descriptions to the Text-to-Image models. We then pass these generated images again through gpt-4o using the same system prompt and compare the variation between the two descriptions. Our results reveal that these models struggle to create simple binary images with only two factors of variation: a simple geometric shape and its location. We also show, using pre-trained VAEs on our dataset, that they fail to generate images that follow our input dataset distribution.
近年来,大型语言模型(LLM)领域取得了许多令人惊讶的进步,展示了它们在多种场景中的显著能力和广泛应用。由于这些模型在实际应用场景中显示出巨大的潜力,因此对其可靠性和有效性进行了大量的研究。相比之下,多模态的LLM和文本到图像模型直到最近才开始受到关注,特别是在与纯文本LLM相比时。尽管如此,由于缺乏对它们性能和鲁棒性的评估研究,其可靠性仍然受限。 本文旨在为文本到图像(Text-to-Image)模型建立一个全面的评价框架,特别关注这些模型在生成符合输入文本提示中指定因素变化的图像方面的表现能力。为此,我们创建了一个新的数据集,用于评估这些模型在上述条件下的稳健性。我们的研究对三种Stable Diffusion模型进行了评测:Stable Diffusion 3 Medium、Stable Diffusion 3.5 Large以及Stable Diffusion 3.5 Large Turbo;同时还有两种Janus模型:Janus Pro 1B和Janus Pro 7B。 为了进行这项评估,我们引入了一个管道系统,该系统利用gpt-4o生成的文本描述作为真实图像的基础,并将这些描述传递给Text-to-Image模型以生成人工图像。然后,我们将生成的图像再次通过相同的系统提示传递给gpt-4o,并比较两个描述之间的变化。 我们的结果表明,在只有两种因素(一个简单的几何形状及其位置)发生二元变化的情况下,这些模型在创建简单二元图象方面遇到困难。此外,我们还使用预训练的变分自编码器(VAE),并在我们的数据集上展示了它们无法生成符合输入数据分布特性的图像。 总的来说,这项研究揭示了现有Text-to-Image模型的一些局限性,并为进一步改善这些模型提供了有价值的见解和建议。
https://arxiv.org/abs/2507.08039
Remarkable progress in text-to-image diffusion models has brought a major concern about potentially generating images on inappropriate or trademarked concepts. Concept erasing has been investigated with the goals of deleting target concepts in diffusion models while preserving other concepts with minimal distortion. To achieve these goals, recent concept erasing methods usually fine-tune the cross-attention layers of diffusion models. In this work, we first show that merely updating the cross-attention layers in diffusion models, which is mathematically equivalent to adding \emph{linear} modules to weights, may not be able to preserve diverse remaining concepts. Then, we propose a novel framework, dubbed Concept Pinpoint Eraser (CPE), by adding \emph{nonlinear} Residual Attention Gates (ResAGs) that selectively erase (or cut) target concepts while safeguarding remaining concepts from broad distributions by employing an attention anchoring loss to prevent the forgetting. Moreover, we adversarially train CPE with ResAG and learnable text embeddings in an iterative manner to maximize erasing performance and enhance robustness against adversarial attacks. Extensive experiments on the erasure of celebrities, artistic styles, and explicit contents demonstrated that the proposed CPE outperforms prior arts by keeping diverse remaining concepts while deleting the target concepts with robustness against attack prompts. Code is available at this https URL
文本到图像的扩散模型取得了显著的进步,但也引发了人们对生成不适当或受商标保护的概念图片的担忧。概念擦除技术的研究目标是在删除特定概念的同时尽可能地保留其他概念而不造成重大失真。最近的方法通常通过微调扩散模型中的交叉注意力层来实现这一目的。 然而,在这项工作中,研究者们首先展示了仅仅更新扩散模型中的交叉注意力层(这在数学上等同于向权重中添加线性模块)可能不足以保护多样化存在的其他概念。为了解决这个问题,他们提出了一种新的框架——概念定点擦除器(Concept Pinpoint Eraser, CPE)。该框架通过添加非线性的残差注意门(Residual Attention Gates, ResAGs),选择性地删除目标概念,同时利用注意力锚定损失来防止遗忘,从而保护其他广泛分布的概念。此外,研究者们还采用对抗训练方法迭代式地优化CPE及其可学习的文本嵌入,以最大化擦除性能并增强对对抗攻击的鲁棒性。 通过在名人、艺术风格和显性内容上的大量实验验证了所提出的CPE框架,在删除目标概念的同时能够保持多样化存在的其他概念,并且具有较强的抗干扰能力。代码可在提供的链接处获取。
https://arxiv.org/abs/2506.22806
Standard datasets often present limitations, particularly due to the fixed nature of input data sensors, which makes it difficult to compare methods that actively adjust sensor parameters to suit environmental conditions. This is the case with Automatic-Exposure (AE) methods, which rely on environmental factors to influence the image acquisition process. As a result, AE methods have traditionally been benchmarked in an online manner, rendering experiments non-reproducible. Building on our prior work, we propose a methodology that utilizes an emulator capable of generating images at any exposure time. This approach leverages BorealHDR, a unique multi-exposure stereo dataset, along with its new extension, in which data was acquired along a repeated trajectory at different times of the day to assess the impact of changing illumination. In total, BorealHDR covers 13.4 km over 59 trajectories in challenging lighting conditions. The dataset also includes lidar-inertial-odometry-based maps with pose estimation for each image frame, as well as Global Navigation Satellite System (GNSS) data for comparison. We demonstrate that by using images acquired at various exposure times, we can emulate realistic images with a Root-Mean-Square Error (RMSE) below 1.78% compared to ground truth images. Using this offline approach, we benchmarked eight AE methods, concluding that the classical AE method remains the field's best performer. To further support reproducibility, we provide in-depth details on the development of our backpack acquisition platform, including hardware, electrical components, and performance specifications. Additionally, we share valuable lessons learned from deploying the backpack over more than 25 km across various environments. Our code and dataset are available online at this link: this https URL BorealHDR
标准数据集通常存在一些限制,特别是由于输入数据传感器的固有特性难以改变,这使得比较那些能够根据环境条件调整传感器参数的方法变得困难。这种情况尤其体现在自动曝光(AE)方法中,这些方法依赖于环境因素来影响图像获取过程。因此,传统的AE方法在线进行基准测试,导致实验不可重复。 基于我们之前的工作,我们提出了一种利用模拟器生成任意曝光时间图像的方法。这种方法使用了BorealHDR数据集,这是一个独特的多曝光立体视觉数据集,以及其新的扩展版本,在这个版本中,数据是在一天中的不同时间段沿着相同的轨迹收集的,以评估光照变化的影响。总体而言,BorealHDR涵盖了59条轨迹、13.4公里的距离,并在挑战性的照明条件下进行了采集。该数据集还包括基于激光雷达-惯性里程计的地图以及每帧图像的姿态估计,还提供了全球导航卫星系统(GNSS)数据进行比较。 我们展示了通过使用不同曝光时间获取的图像,可以生成与真实情况相比误差低于1.78%的逼真图像。利用这种方法,我们在离线状态下对八种AE方法进行了基准测试,并得出结论:经典的AE方法仍然是表现最佳的方法。为了进一步支持可重复性研究,我们提供了有关背包采集平台开发的详细信息,包括硬件、电气组件和性能规范。此外,我们也分享了在超过25公里的各种环境下部署背包所获得的经验教训。 我们的代码和数据集可通过以下链接获取:[BorealHDR](https://this https URL)
https://arxiv.org/abs/2506.18844
We consider the problem of disentangling 3D from large vision-language models, which we show on generative 3D portraits. This allows free-form text control of appearance attributes like age, hair style, and glasses, and 3D geometry control of face expression and camera pose. In this setting, we assume we use a pre-trained large vision-language model (LVLM; CLIP) to generate from a smaller 2D dataset with no additional paired labels and with a pre-defined 3D morphable model (FLAME). First, we disentangle using canonicalization to a 2D reference frame from a deformable neural 3D triplane representation. But another form of entanglement arises from the significant noise in the LVLM's embedding space that describes irrelevant features. This damages output quality and diversity, but we overcome this with a Jacobian regularization that can be computed efficiently with a stochastic approximator. Compared to existing methods, our approach produces portraits with added text and 3D control, where portraits remain consistent when either control is changed. Broadly, this approach lets creators control 3D generators on their own 2D face data without needing resources to label large data or train large models.
我们考虑从大型视觉-语言模型中分离出三维信息的问题,并通过生成的三维肖像展示了这一过程。这使得我们可以用自由形式的文字控制外观属性(如年龄、发型和眼镜),并通过3D几何学控制面部表情和相机姿态。在这个设置下,假设我们使用一个预训练的大规模视觉-语言模型(LVLM;CLIP)从一个小的2D数据集中生成结果,并且该数据集没有额外配对标签,同时定义了一个预设的三维可变形模型(FLAME)。首先,我们通过将神经3D三平面表示规范到二维参考帧来实现分离。然而,另一种纠缠形式来自于LVLM嵌入空间中的大量噪声,这些噪声描述了无关特征。这种噪声会损害输出质量和多样性,但我们可以通过计算效率高的随机近似器进行雅可比正则化的方法克服这一问题。 与现有方法相比,我们的方法可以在生成的肖像中添加文本和3D控制,当改变任一控制时,肖像仍然保持一致性。总体而言,这种方法让创作者能够在其自己的2D面部数据上控制三维生成模型,并且无需为标注大量数据或训练大型模型投入资源。
https://arxiv.org/abs/2506.14015
We present LatentCSI, a novel method for generating images of the physical environment from WiFi CSI measurements that leverages a pretrained latent diffusion model (LDM). Unlike prior approaches that rely on complex and computationally intensive techniques such as GANs, our method employs a lightweight neural network to map CSI amplitudes directly into the latent space of an LDM. We then apply the LDM's denoising diffusion model to the latent representation with text-based guidance before decoding using the LDM's pretrained decoder to obtain a high-resolution image. This design bypasses the challenges of pixel-space image generation and avoids the explicit image encoding stage typically required in conventional image-to-image pipelines, enabling efficient and high-quality image synthesis. We validate our approach on two datasets: a wide-band CSI dataset we collected with off-the-shelf WiFi devices and cameras; and a subset of the publicly available MM-Fi dataset. The results demonstrate that LatentCSI outperforms baselines of comparable complexity trained directly on ground-truth images in both computational efficiency and perceptual quality, while additionally providing practical advantages through its unique capacity for text-guided controllability.
我们提出了LatentCSI,这是一种利用预训练的潜在扩散模型(LDM)从WiFi CSI测量生成物理环境图像的新方法。与以往依赖复杂且计算密集的技术如GAN的方法不同,我们的方法采用轻量级神经网络直接将CSI幅度映射到LDM的潜在空间中。接着,我们使用带有文本引导的LDM去噪扩散模型对潜在表示进行处理,并通过预训练解码器解码以获得高分辨率图像。这种设计绕过了像素空间图像生成的挑战,并避免了传统图像到图像工作流通常需要的显式图像编码阶段,从而实现了高效且高质量的图像合成。 我们在两个数据集上验证了我们的方法:一个是使用现成的WiFi设备和摄像头收集的宽带CSI数据集;另一个是公开可用的MM-Fi数据子集。实验结果表明,在计算效率和感知质量方面,LatentCSI优于直接在真实图像上训练的复杂度相当的基础模型,同时通过其独特的文本引导可控性提供了实用优势。
https://arxiv.org/abs/2506.10605
Generating images in a consistent reference visual style remains a challenging computer vision task. State-of-the-art methods aiming for style-consistent generation struggle to effectively separate semantic content from stylistic elements, leading to content leakage from the image provided as a reference to the targets. To address this challenge, we propose Only-Style: a method designed to mitigate content leakage in a semantically coherent manner while preserving stylistic consistency. Only-Style works by localizing content leakage during inference, allowing the adaptive tuning of a parameter that controls the style alignment process, specifically within the image patches containing the subject in the reference image. This adaptive process best balances stylistic consistency with leakage elimination. Moreover, the localization of content leakage can function as a standalone component, given a reference-target image pair, allowing the adaptive tuning of any method-specific parameter that provides control over the impact of the stylistic reference. In addition, we propose a novel evaluation framework to quantify the success of style-consistent generations in avoiding undesired content leakage. Our approach demonstrates a significant improvement over state-of-the-art methods through extensive evaluation across diverse instances, consistently achieving robust stylistic consistency without undesired content leakage.
在一致的参考视觉风格下生成图像仍然是一个具有挑战性的计算机视觉任务。最先进的旨在实现风格一致性生成的方法难以有效地区分语义内容和风格元素,导致从提供的参考图片中向目标泄露不必要的内容信息。为了解决这一挑战,我们提出了Only-Style:一种方法,旨在以语义连贯的方式减少内容泄漏,同时保持风格的一致性。Only-Style通过在推理过程中定位内容泄漏来工作,并允许自适应调整控制样式对齐过程的参数(特别是在参考图像中包含主体的图块内)。这种自适应处理最能平衡风格一致性与泄漏消除之间的关系。此外,定位内容泄露的功能可以作为独立组件存在,给定一个参考目标图片对,允许任何特定方法的参数进行自适应调节,从而控制样式参照的影响。 我们还提出了一种新的评估框架,以量化风格一致生成在避免不需要的内容泄漏方面的成功程度。通过广泛的测试和各种实例,我们的方法明显优于最先进的方法,在保持强大风格一致性的同时,始终能有效地防止不需要的内容泄露。
https://arxiv.org/abs/2506.09916
Recently, Large language models (LLMs) have shown great promise across a diversity of tasks, ranging from generating images to reasoning spatially. Considering their remarkable (and growing) textual reasoning capabilities, we investigate LLMs' potency in conducting analyses of an individual's preferences in music (based on playlist metadata, personal write-ups, etc.) and producing effective prompts (based on these analyses) to be passed to Suno AI (a generative AI tool for music production). Our proposition of a novel LLM-based textual representation to music model (which we call TuneGenie) and the various methods we develop to evaluate & benchmark similar models add to the increasing (and increasingly controversial) corpus of research on the use of AI in generating art.
最近,大型语言模型(LLM)在多种任务中展现了巨大的潜力,从生成图像到进行空间推理。鉴于它们卓越且不断增长的文本推理能力,我们研究了这些模型在分析个人音乐偏好方面的效力(基于播放列表元数据、个人评论等),并开发有效提示(基于这些分析)以传递给Suno AI(一种用于音乐制作的生成式AI工具)。我们提出了一种新颖的LLM基文本表示到音乐模型(称之为TuneGenie),以及我们为评估和基准测试类似模型而发展的各种方法,都增加了关于在艺术创作中使用AI的研究日益增长且日益具有争议性的文献。
https://arxiv.org/abs/2506.12083
Text-to-image (T2I) models have garnered significant attention for generating high-quality images aligned with text prompts. However, rapid T2I model advancements reveal limitations in early benchmarks, lacking comprehensive evaluations, for example, the evaluation on reasoning, text rendering and style. Notably, recent state-of-the-art models, with their rich knowledge modeling capabilities, show promising results on the image generation problems requiring strong reasoning ability, yet existing evaluation systems have not adequately addressed this frontier. To systematically address these gaps, we introduce OneIG-Bench, a meticulously designed comprehensive benchmark framework for fine-grained evaluation of T2I models across multiple dimensions, including prompt-image alignment, text rendering precision, reasoning-generated content, stylization, and diversity. By structuring the evaluation, this benchmark enables in-depth analysis of model performance, helping researchers and practitioners pinpoint strengths and bottlenecks in the full pipeline of image generation. Specifically, OneIG-Bench enables flexible evaluation by allowing users to focus on a particular evaluation subset. Instead of generating images for the entire set of prompts, users can generate images only for the prompts associated with the selected dimension and complete the corresponding evaluation accordingly. Our codebase and dataset are now publicly available to facilitate reproducible evaluation studies and cross-model comparisons within the T2I research community.
文本到图像(T2I)模型因其能生成高质量并与文本提示相匹配的图像而备受关注。然而,随着T2I模型的快速进步,早期基准测试中出现了一些局限性,例如缺乏全面评估,特别是在推理、文本渲染和风格方面。值得注意的是,最近最先进的模型在需要强大推理能力的图像生成问题上展现了令人鼓舞的结果,但现有的评估系统尚未充分解决这一领域的问题。为了系统地填补这些空白,我们引入了OneIG-Bench,这是一个精心设计的基准框架,旨在对T2I模型进行细致全面的多维度评估,包括提示-图像匹配度、文本渲染精度、推理生成内容、风格化和多样性等方面。通过结构化的评估方法,该基准能够深入分析模型性能,帮助研究人员和实践者识别整个图像生成流程中的优势与瓶颈。具体而言,OneIG-Bench允许用户灵活地进行特定评估子集的测试,而非为所有提示生成图像,使用者只需针对选定维度的相关提示生成图像,并完成相应的评估即可。我们的代码库和数据集现已公开发布,便于T2I研究社区内的可重复性研究及跨模型比较。
https://arxiv.org/abs/2506.07977
Domain generalization (DG) for object detection aims to enhance detectors' performance in unseen scenarios. This task remains challenging due to complex variations in real-world applications. Recently, diffusion models have demonstrated remarkable capabilities in diverse scene generation, which inspires us to explore their potential for improving DG tasks. Instead of generating images, our method extracts multi-step intermediate features during the diffusion process to obtain domain-invariant features for generalized detection. Furthermore, we propose an efficient knowledge transfer framework that enables detectors to inherit the generalization capabilities of diffusion models through feature and object-level alignment, without increasing inference time. We conduct extensive experiments on six challenging DG benchmarks. The results demonstrate that our method achieves substantial improvements of 14.0% mAP over existing DG approaches across different domains and corruption types. Notably, our method even outperforms most domain adaptation methods without accessing any target domain data. Moreover, the diffusion-guided detectors show consistent improvements of 15.9% mAP on average compared to the baseline. Our work aims to present an effective approach for domain-generalized detection and provide potential insights for robust visual recognition in real-world scenarios. The code is available at this https URL.
领域泛化(Domain Generalization,DG)在目标检测中的应用旨在提高检测器在未见过场景下的表现。由于实际应用场景中存在复杂多变的情况,这一任务依然充满挑战。最近,扩散模型展示了其在生成多样化场景方面的卓越能力,这激发了我们将它们用于改善DG任务的潜力的研究兴趣。与直接生成图像不同,我们的方法通过在扩散过程中提取多步中间特征来获取跨域不变性特征,从而实现泛化检测。 此外,我们提出了一种高效的知识转移框架,使检测器能够继承扩散模型的泛化能力,而无需增加推理时间,并且该框架是通过特征级和对象级对齐完成的。我们在六个具有挑战性的DG基准数据集上进行了广泛的实验,结果显示我们的方法在不同域和损坏类型上比现有DG方法提高了14.0% mAP(平均精度)。值得注意的是,即使不访问目标域数据,我们提出的方法也超越了大多数领域适应方法的表现。 此外,由扩散模型指导的检测器相比基准线,在所有测试场景中平均mAP值提升达到了15.9%。我们的工作旨在提供一种有效的跨领域泛化检测解决方案,并为在实际场景中的稳健视觉识别提供潜在见解。代码可以在以下链接获取:[此链接](https://this https URL)。 请注意,最后一个URL应替换为您实际提供的代码仓库地址或其他相关资源的正确链接。
https://arxiv.org/abs/2503.02101
Text-to-image (T2I) generation model has made significant advancements, resulting in high-quality images aligned with an input prompt. However, despite T2I generation's ability to generate fine-grained images, it still faces challenges in accurately generating images when the input prompt contains complex concepts, especially human pose. In this paper, we propose PointT2I, a framework that effectively generates images that accurately correspond to the human pose described in the prompt by using a large language model (LLM). PointT2I consists of three components: Keypoint generation, Image generation, and Feedback system. The keypoint generation uses an LLM to directly generate keypoints corresponding to a human pose, solely based on the input prompt, without external references. Subsequently, the image generation produces images based on both the text prompt and the generated keypoints to accurately reflect the target pose. To refine the outputs of the preceding stages, we incorporate an LLM-based feedback system that assesses the semantic consistency between the generated contents and the given prompts. Our framework is the first approach to leveraging LLM for keypoints-guided image generation without any fine-tuning, producing accurate pose-aligned images based solely on textual prompts.
文本到图像(T2I)生成模型已经取得了显著进展,能够根据输入提示生成高质量的图像。然而,尽管T2I生成模型能够在某些方面产生细节丰富的图像,当输入提示包含复杂的概念时,尤其是人体姿态时,仍然面临准确生成图像的挑战。 在本文中,我们提出了PointT2I框架,该框架利用大规模语言模型(LLM)有效地生成与提示中描述的人体姿态相匹配的高质量图像。PointT2I包括三个组成部分:关键点生成、图像生成和反馈系统。 1. **关键点生成**:这部分使用一个大模型直接根据输入提示生成对应人体姿势的关键点,而不依赖于外部参考。 2. **图像生成**:该部分基于文本提示以及由前面步骤产生的关键点来生成图像,以准确反映目标姿态。 3. **反馈系统**:为了优化前两个阶段的输出,我们引入了一个基于大模型的反馈机制,它评估生成内容与给定提示之间的语义一致性。 我们的框架是首个不进行微调即可利用LLM引导关键点并据此生成精确对齐的人体姿势图像的方法。仅通过文本提示,PointT2I就能产生准确的姿态对齐图像。
https://arxiv.org/abs/2506.01370
Although chain-of-thought reasoning and reinforcement learning (RL) have driven breakthroughs in NLP, their integration into generative vision models remains underexplored. We introduce ReasonGen-R1, a two-stage framework that first imbues an autoregressive image generator with explicit text-based "thinking" skills via supervised fine-tuning on a newly generated reasoning dataset of written rationales, and then refines its outputs using Group Relative Policy Optimization. To enable the model to reason through text before generating images, We automatically generate and release a corpus of model crafted rationales paired with visual prompts, enabling controlled planning of object layouts, styles, and scene compositions. Our GRPO algorithm uses reward signals from a pretrained vision language model to assess overall visual quality, optimizing the policy in each update. Evaluations on GenEval, DPG, and the T2I benchmark demonstrate that ReasonGen-R1 consistently outperforms strong baselines and prior state-of-the-art models. More: this http URL.
尽管基于链式思维推理和强化学习(RL)在自然语言处理(NLP)领域取得了突破性进展,但它们在生成视觉模型中的整合仍处于探索阶段。我们引入了ReasonGen-R1框架,这是一个两阶段的架构:首先通过在新生成的推理数据集上进行监督微调,赋予自回归图像生成器显式的基于文本的“思考”技能;然后利用群体相对策略优化(GRPO)来改进其输出结果。为了使模型能够在生成图像之前通过文本进行推理,我们自动创建并发布了一套由模型创作的论述文集,这些文档与视觉提示配对,从而能够控制物体布局、风格和场景构图的设计。我们的GRPO算法使用预训练的视觉语言模型提供的奖励信号来评估整体视觉质量,并在每次更新中优化策略。在GenEval、DPG以及T2I基准测试上的评估结果表明,ReasonGen-R1始终优于强大的基线模型及先前最先进的模型。更多信息,请参见:[此链接](http://this.http.url)。 注意:原文中的“this http URL”是一个占位符或伪链接,并非实际链接地址,在正式发布时应替换为有效的链接地址。
https://arxiv.org/abs/2505.24875
The diffusion models (DMs) have demonstrated the remarkable capability of generating images via learning the noised score function of data distribution. Current DM sampling techniques typically rely on first-order Langevin dynamics at each noise level, with efforts concentrated on refining inter-level denoising strategies. While leveraging additional second-order Hessian geometry to enhance the sampling quality of Langevin is a common practice in Markov chain Monte Carlo (MCMC), the naive attempts to utilize Hessian geometry in high-dimensional DMs lead to quadratic-complexity computational costs, rendering them non-scalable. In this work, we introduce a novel Levenberg-Marquardt-Langevin (LML) method that approximates the diffusion Hessian geometry in a training-free manner, drawing inspiration from the celebrated Levenberg-Marquardt optimization algorithm. Our approach introduces two key innovations: (1) A low-rank approximation of the diffusion Hessian, leveraging the DMs' inherent structure and circumventing explicit quadratic-complexity computations; (2) A damping mechanism to stabilize the approximated Hessian. This LML approximated Hessian geometry enables the diffusion sampling to execute more accurate steps and improve the image generation quality. We further conduct a theoretical analysis to substantiate the approximation error bound of low-rank approximation and the convergence property of the damping mechanism. Extensive experiments across multiple pretrained DMs validate that the LML method significantly improves image generation quality, with negligible computational overhead.
扩散模型(DM)通过学习数据分布的噪声得分函数,展示了生成图像的强大能力。目前的DM采样技术通常依赖于每个噪声水平上的第一阶Langevin动力学,并且努力集中在改进跨层级去噪策略上。尽管在马尔可夫链蒙特卡洛(MCMC)方法中利用额外的二阶Hessian几何来增强Langevin采样的质量是一种常见做法,但在高维DM中直接使用Hessian几何会导致二次复杂度的计算成本,使其无法扩展。 在这项工作中,我们引入了一种新的莱文伯格-马夸尔特-兰热姆(Levenberg-Marquardt-Langevin, LML)方法,该方法以无训练的方式近似扩散Hessian几何,借鉴了著名的Levenberg-Marquardt优化算法。我们的方法提出了两项关键创新: 1. 一种对扩散Hessian的低秩近似,利用DM固有的结构,并避免直接进行二次复杂度计算。 2. 一个阻尼机制来稳定近似的Hessian。 这种LML近似Hessian几何使扩散采样能够执行更准确的步骤并提高图像生成质量。我们进一步进行了理论分析以验证低秩近似误差界和阻尼机制的收敛性。在多个预训练DM上的广泛实验验证了LML方法显著提高了图像生成的质量,且计算开销几乎可以忽略不计。
https://arxiv.org/abs/2505.24222