Style_Transfer
-
Text to Sketch Generation with Multi-Styles
2025-11-06 07:13:56Tengjie Li, Shikui Tu, Lei XuAbstract
Recent advances in vision-language models have facilitated progress in sketch generation. However, existing specialized methods primarily focus on generic synthesis and lack mechanisms for precise control over sketch styles. In this work, we propose a training-free framework based on diffusion models that enables explicit style guidance via textual prompts and referenced style sketches. Unlike previous style transfer methods that overwrite key and value matrices in self-attention, we incorporate the reference features as auxiliary information with linear smoothing and leverage a style-content guidance mechanism. This design effectively reduces content leakage from reference sketches and enhances synthesis quality, especially in cases with low structural similarity between reference and target sketches. Furthermore, we extend our framework to support controllable multi-style generation by integrating features from multiple reference sketches, coordinated via a joint AdaIN module. Extensive experiments demonstrate that our approach achieves high-quality sketch generation with accurate style alignment and improved flexibility in style control. The official implementation of M3S is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2511.04123
PDF
-
Cross-fluctuation phase transitions reveal sampling dynamics in diffusion models
2025-10-31 09:40:59Sai Niranjan Ramachandran, Manish Krishan Lal, Suvrit SraAbstract
We analyse how the sampling dynamics of distributions evolve in score-based diffusion models using cross-fluctuations, a centered-moment statistic from statistical physics. Specifically, we show that starting from an unbiased isotropic normal distribution, samples undergo sharp, discrete transitions, eventually forming distinct events of a desired distribution while progressively revealing finer structure. As this process is reversible, these transitions also occur in reverse, where intermediate states progressively merge, tracing a path back to the initial distribution. We demonstrate that these transitions can be detected as discontinuities in $n^{\text{th}}$-order cross-fluctuations. For variance-preserving SDEs, we derive a closed-form for these cross-fluctuations that is efficiently computable for the reverse trajectory. We find that detecting these transitions directly boosts sampling efficiency, accelerates class-conditional and rare-class generation, and improves two zero-shot tasks--image classification and style transfer--without expensive grid search or retraining. We also show that this viewpoint unifies classical coupling and mixing from finite Markov chains with continuous dynamics while extending to stochastic SDEs and non Markovian samplers. Our framework therefore bridges discrete Markov chain theory, phase analysis, and modern generative modeling.
Abstract (translated)
URL
https://arxiv.org/abs/2511.00124
PDF
-
Security Risk of Misalignment between Text and Image in Multi-modal Model
2025-10-30 03:31:20Xiaosen Wang, Zhijin Ge, Shaokang WangAbstract
Despite the notable advancements and versatility of multi-modal diffusion models, such as text-to-image models, their susceptibility to adversarial inputs remains underexplored. Contrary to expectations, our investigations reveal that the alignment between textual and Image modalities in existing diffusion models is inadequate. This misalignment presents significant risks, especially in the generation of inappropriate or Not-Safe-For-Work (NSFW) content. To this end, we propose a novel attack called Prompt-Restricted Multi-modal Attack (PReMA) to manipulate the generated content by modifying the input image in conjunction with any specified prompt, without altering the prompt itself. PReMA is the first attack that manipulates model outputs by solely creating adversarial images, distinguishing itself from prior methods that primarily generate adversarial prompts to produce NSFW content. Consequently, PReMA poses a novel threat to the integrity of multi-modal diffusion models, particularly in image-editing applications that operate with fixed prompts. Comprehensive evaluations conducted on image inpainting and style transfer tasks across various models confirm the potent efficacy of PReMA.
Abstract (translated)
URL
https://arxiv.org/abs/2510.26105
PDF
-
GeoDiffusion: A Training-Free Framework for Accurate 3D Geometric Conditioning in Image Generation
2025-10-25 15:40:34Phillip Mueller, Talip Uenlue, Sebastian Schmidt, Marcel Kollovieh, Jiajie Fan, Stephan Guennemann, Lars MikelsonsAbstract
Precise geometric control in image generation is essential for engineering \& product design and creative industries to control 3D object features accurately in image space. Traditional 3D editing approaches are time-consuming and demand specialized skills, while current image-based generative methods lack accuracy in geometric conditioning. To address these challenges, we propose GeoDiffusion, a training-free framework for accurate and efficient geometric conditioning of 3D features in image generation. GeoDiffusion employs a class-specific 3D object as a geometric prior to define keypoints and parametric correlations in 3D space. We ensure viewpoint consistency through a rendered image of a reference 3D object, followed by style transfer to meet user-defined appearance specifications. At the core of our framework is GeoDrag, improving accuracy and speed of drag-based image editing on geometry guidance tasks and general instructions on DragBench. Our results demonstrate that GeoDiffusion enables precise geometric modifications across various iterative design workflows.
Abstract (translated)
URL
https://arxiv.org/abs/2510.22337
PDF
-
GuitarFlow: Realistic Electric Guitar Synthesis From Tablatures via Flow Matching and Style Transfer
2025-10-23 13:31:41Jackson Loth, Pedro Sarmento, Mark Sandler, Mathieu BarthetAbstract
Music generation in the audio domain using artificial intelligence (AI) has witnessed steady progress in recent years. However for some instruments, particularly the guitar, controllable instrument synthesis remains limited in expressivity. We introduce GuitarFlow, a model designed specifically for electric guitar synthesis. The generative process is guided using tablatures, an ubiquitous and intuitive guitar-specific symbolic format. The tablature format easily represents guitar-specific playing techniques (e.g. bends, muted strings and legatos), which are more difficult to represent in other common music notation formats such as MIDI. Our model relies on an intermediary step of first rendering the tablature to audio using a simple sample-based virtual instrument, then performing style transfer using Flow Matching in order to transform the virtual instrument audio into more realistic sounding examples. This results in a model that is quick to train and to perform inference, requiring less than 6 hours of training data. We present the results of objective evaluation metrics, together with a listening test, in which we show significant improvement in the realism of the generated guitar audio from tablatures.
Abstract (translated)
URL
https://arxiv.org/abs/2510.21872
PDF
-
Adaptive Divergence Regularized Policy Optimization for Fine-tuning Generative Models
2025-10-20 19:46:02Jiajun Fan, Tong Wei, Chaoran Cheng, Yuxin Chen, Ge LiuAbstract
Balancing exploration and exploitation during reinforcement learning fine-tuning of generative models presents a critical challenge, as existing approaches rely on fixed divergence regularization that creates an inherent dilemma: strong regularization preserves model capabilities but limits reward optimization, while weak regularization enables greater alignment but risks instability or reward hacking. We introduce Adaptive Divergence Regularized Policy Optimization (ADRPO), which automatically adjusts regularization strength based on advantage estimates-reducing regularization for high-value samples while applying stronger regularization to poor samples, enabling policies to navigate between exploration and aggressive exploitation according to data quality. Our implementation with Wasserstein-2 regularization for flow matching generative models achieves remarkable results on text-to-image generation, achieving better semantic alignment and diversity than offline methods like DPO and online methods with fixed regularization like ORW-CFM-W2. ADRPO enables a 2B parameter SD3 model to surpass much larger models with 4.8B and 12B parameters in attribute binding, semantic consistency, artistic style transfer, and compositional control while maintaining generation diversity. ADRPO generalizes to KL-regularized fine-tuning of both text-only LLMs and multi-modal reasoning models, enhancing existing online RL methods like GRPO. In LLM fine-tuning, ADRPO demonstrates an emergent ability to escape local optima through active exploration, while in multi-modal audio reasoning, it outperforms GRPO through superior step-by-step reasoning, enabling a 7B model to outperform substantially larger commercial models including Gemini 2.5 Pro and GPT-4o Audio, offering an effective plug-and-play solution to the exploration-exploitation challenge across diverse generative architectures and modalities.
Abstract (translated)
在强化学习（Reinforcement Learning，RL）中对生成模型进行微调时，平衡探索和利用是一个关键挑战。现有的方法依赖于固定的发散正则化（divergence regularization），这种固定的方法产生了固有的两难局面：强烈的正则化可以保持模型的能力但限制了奖励优化的范围；而较弱的正则化虽然能更好地对齐目标，却可能带来不稳定或奖励操纵的风险。为此，我们提出了自适应发散正则化的策略优化（Adaptive Divergence Regularized Policy Optimization, ADRPO），该方法可以根据优势估计自动调整正则化强度——对于高价值样本减少正则化，而对于低质量的样本施加强正则化，从而使得策略能够在探索与积极利用之间根据数据的质量灵活切换。 我们的实现方案中使用了Wasserstein-2正则化的流匹配生成模型，在文本到图像生成的任务上取得了显著成果，其在语义对齐和多样性方面超越了离线方法（如DPO）以及固定正则化在线方法（例如ORW-CFM-W2）。ADRPO允许一个参数量为20亿的SD3模型在属性绑定、语义一致性、艺术风格转换及组合控制等任务上优于具有48亿和120亿参数的更大规模模型，同时保持生成多样性的平衡。 此外，ADRPO适用于KL正则化微调仅限于文本的大语言模型（Large Language Models, LLMs）以及多模态推理模型。它增强了现有在线RL方法（如GRPO）。在LLM微调过程中，ADRPO展示了通过积极探索逃离局部最优解的能力；而在处理多模态音频推理时，相比GRPO，它能凭借更出色的逐步推理能力击败包括Gemini 2.5 Pro和GPT-4o Audio在内的大型商业模型。这使得一个具有70亿参数的模型能够超越这些商业模型，并为各种生成架构及模态中探索与利用之间的挑战提供了一个有效的即插即用解决方案。 综上所述，ADRPO通过自适应调整正则化强度的方式，在多个场景下提升了生成模型的效果和性能，特别是在平衡探索与利用方面表现出色。
URL
https://arxiv.org/abs/2510.18053
PDF
-
REALM: An MLLM-Agent Framework for Open World 3D Reasoning Segmentation and Editing on Gaussian Splatting
2025-10-18 08:53:08Changyue Shi, Minghao Chen, Yiping Mao, Chuxiao Yang, Xinyuan Hu, Jiajun Ding, Zhou YuAbstract
Bridging the gap between complex human instructions and precise 3D object grounding remains a significant challenge in vision and robotics. Existing 3D segmentation methods often struggle to interpret ambiguous, reasoning-based instructions, while 2D vision-language models that excel at such reasoning lack intrinsic 3D spatial understanding. In this paper, we introduce REALM, an innovative MLLM-agent framework that enables open-world reasoning-based segmentation without requiring extensive 3D-specific post-training. We perform segmentation directly on 3D Gaussian Splatting representations, capitalizing on their ability to render photorealistic novel views that are highly suitable for MLLM comprehension. As directly feeding one or more rendered views to the MLLM can lead to high sensitivity to viewpoint selection, we propose a novel Global-to-Local Spatial Grounding strategy. Specifically, multiple global views are first fed into the MLLM agent in parallel for coarse-level localization, aggregating responses to robustly identify the target object. Then, several close-up novel views of the object are synthesized to perform fine-grained local segmentation, yielding accurate and consistent 3D masks. Extensive experiments show that REALM achieves remarkable performance in interpreting both explicit and implicit instructions across LERF, 3D-OVS, and our newly introduced REALM3D benchmarks. Furthermore, our agent framework seamlessly supports a range of 3D interaction tasks, including object removal, replacement, and style transfer, demonstrating its practical utility and versatility. Project page: this https URL.
Abstract (translated)
在视觉和机器人技术领域，将复杂的指令与精确的三维物体定位相结合仍然是一个重大挑战。现有的三维分割方法通常难以解释那些基于推理的模糊指令，而二维视觉语言模型虽然擅长处理此类推理任务，却缺乏对三维空间的理解。为此，在这篇论文中，我们提出了REALM框架，这是一种创新的多模态大规模预训练语言模型（MLLM）与代理相结合的方法，它能够在无需进行大量特定于3D数据的后期训练的情况下执行基于推理的开放世界分割。我们在3D高斯光束表示上直接进行分割，利用其能够渲染非常逼真的新视角的特点，这些视角非常适合MLLM的理解能力。 然而，将一个或多个渲染视图直接输入到MLLM中可能会导致对视点选择的高度敏感性。因此，我们提出了一个新颖的全局到局部空间定位策略：首先并行地向代理中的MLLM提供多个全局视图以进行粗略级别的本地化，并通过聚合这些响应来稳健地识别目标物体；然后，生成该对象的一系列特写的新视角来进行精细分割，从而获得精确且一致的3D掩膜。 经过广泛的实验表明，REALM在理解LERF、3D-OVS以及我们新引入的REALM3D基准测试中的明确和隐含指令方面表现出卓越性能。此外，我们的代理框架还能够无缝支持一系列三维交互任务，包括物体移除、替换及风格转移等操作，这展示了其实用性和灵活性。 项目主页: [链接](this https URL)
URL
https://arxiv.org/abs/2510.16410
PDF
-
Generative human motion mimicking through feature extraction in denoising diffusion settings
2025-10-16 14:41:54Alexander Okupnik, Johannes Schneider, Kyriakos FlourisAbstract
Recent success with large language models has sparked a new wave of verbal human-AI interaction. While such models support users in a variety of creative tasks, they lack the embodied nature of human interaction. Dance, as a primal form of human expression, is predestined to complement this experience. To explore creative human-AI interaction exemplified by dance, we build an interactive model based on motion capture (MoCap) data. It generates an artificial other by partially mimicking and also "creatively" enhancing an incoming sequence of movement data. It is the first model, which leverages single-person motion data and high level features in order to do so and, thus, it does not rely on low level human-human interaction data. It combines ideas of two diffusion models, motion inpainting, and motion style transfer to generate movement representations that are both temporally coherent and responsive to a chosen movement reference. The success of the model is demonstrated by quantitatively assessing the convergence of the feature distribution of the generated samples and the test set which serves as simulating the human performer. We show that our generations are first steps to creative dancing with AI as they are both diverse showing various deviations from the human partner while appearing realistic.
Abstract (translated)
URL
https://arxiv.org/abs/2511.00011
PDF
-
LLM one-shot style transfer for Authorship Attribution and Verification
2025-10-15 08:43:24Pablo Miralles-Gonz\'alez, Javier Huertas-Tato, Alejandro Mart\'in, David CamachoAbstract
Computational stylometry analyzes writing style through quantitative patterns in text, supporting applications from forensic tasks such as identity linking and plagiarism detection to literary attribution in the humanities. Supervised and contrastive approaches rely on data with spurious correlations and often confuse style with topic. Despite their natural use in AI-generated text detection, the CLM pre-training of modern LLMs has been scarcely leveraged for general authorship problems. We propose a novel unsupervised approach based on this extensive pre-training and the in-context learning capabilities of LLMs, employing the log-probabilities of an LLM to measure style transferability from one text to another. Our method significantly outperforms LLM prompting approaches of comparable scale and achieves higher accuracy than contrastively trained baselines when controlling for topical correlations. Moreover, performance scales fairly consistently with the size of the base model and, in the case of authorship verification, with an additional mechanism that increases test-time computation; enabling flexible trade-offs between computational cost and accuracy.
Abstract (translated)
计算性文体测量通过文本中的定量模式分析写作风格，支持从法医任务（如身份链接和抄袭检测）到人文学科中文学归属等各类应用。虽然监督学习和对比方法依赖于带有伪相关的数据，并且常常将写作风格与主题混淆，但现代大型语言模型（LLM）的CLM预训练在一般作者识别问题上却鲜少被利用。我们提出了一种基于广泛预训练及LLM上下文学习能力的新颖无监督方法，通过使用LLM的日志概率来测量一种文本向另一种文本的写作风格可转移性。我们的方法显著优于规模相当的提示式LLM方法，并且在控制主题相关性的前提下，比对比训练基线获得了更高的准确率。此外，性能随着基础模型大小的增长而相对一致地提升；而在作者验证的情况下，通过增加测试时计算量的额外机制，使计算成本与准确性之间的灵活权衡成为可能。
URL
https://arxiv.org/abs/2510.13302
PDF
-
Uncertainty-Aware ControlNet: Bridging Domain Gaps with Synthetic Image Generation
2025-10-13 12:41:28Joshua Niemeijer, Jan Ehrhardt, Heinz Handels, Hristina UzunovaAbstract
Generative Models are a valuable tool for the controlled creation of high-quality image data. Controlled diffusion models like the ControlNet have allowed the creation of labeled distributions. Such synthetic datasets can augment the original training distribution when discriminative models, like semantic segmentation, are trained. However, this augmentation effect is limited since ControlNets tend to reproduce the original training distribution. This work introduces a method to utilize data from unlabeled domains to train ControlNets by introducing the concept of uncertainty into the control mechanism. The uncertainty indicates that a given image was not part of the training distribution of a downstream task, e.g., segmentation. Thus, two types of control are engaged in the final network: an uncertainty control from an unlabeled dataset and a semantic control from the labeled dataset. The resulting ControlNet allows us to create annotated data with high uncertainty from the target domain, i.e., synthetic data from the unlabeled distribution with labels. In our scenario, we consider retinal OCTs, where typically high-quality Spectralis images are available with given ground truth segmentations, enabling the training of segmentation networks. The recent development in Home-OCT devices, however, yields retinal OCTs with lower quality and a large domain shift, such that out-of-the-pocket segmentation networks cannot be applied for this type of data. Synthesizing annotated images from the Home-OCT domain using the proposed approach closes this gap and leads to significantly improved segmentation results without adding any further supervision. The advantage of uncertainty-guidance becomes obvious when compared to style transfer: it enables arbitrary domain shifts without any strict learning of an image style. This is also demonstrated in a traffic scene experiment.
Abstract (translated)
生成模型是用于控制高质量图像数据创建的宝贵工具。受控扩散模型，如ControlNet，允许创建标记分布。这样的合成数据集可以在训练辨别性模型（例如语义分割）时扩充原始训练分布。然而，这种增强效果有限，因为ControlNets倾向于复制原训练数据分布。这项工作引入了一种方法，利用未标注领域的数据来训练ControlNets，并通过将不确定性概念引入控制机制中实现这一目标。不确定性表示给定的图像并非下游任务（如分割）原始训练数据集的一部分。因此，在最终网络中有两种类型的控制：一种是从未标记数据集中来的不确定性控制和一种是从标记数据集来的语义控制。由此产生的ControlNet使我们能够从目标领域创建具有高不确定性的注释数据，即带有标签的来自未标注分布的合成图像。 在我们的场景中考虑视网膜OCT（光学相干断层扫描），通常有高质量的Spectralis图像及其给定的真实分割结果，这可以用来训练分割网络。然而，最近家用OCT设备的发展带来了较低质量和域偏移较大的视网膜OCT数据，使得现成的分割网络无法应用于此类数据。利用所提出的方法从Home-OCT领域生成注释图像填补了这一空白，并在不添加额外监督的情况下显著提高了分割结果。 与风格转移相比，不确定性指导的优势变得明显：它可以实现任意域偏移而不严格学习图像样式。这一点也在交通场景实验中得到了验证。
URL
https://arxiv.org/abs/2510.11346
PDF
-
SceneTextStylizer: A Training-Free Scene Text Style Transfer Framework with Diffusion Model
2025-10-13 02:11:57Honghui Yuan, Keiji YanaiAbstract
With the rapid development of diffusion models, style transfer has made remarkable progress. However, flexible and localized style editing for scene text remains an unsolved challenge. Although existing scene text editing methods have achieved text region editing, they are typically limited to content replacement and simple styles, which lack the ability of free-style transfer. In this paper, we introduce SceneTextStylizer, a novel training-free diffusion-based framework for flexible and high-fidelity style transfer of text in scene images. Unlike prior approaches that either perform global style transfer or focus solely on textual content modification, our method enables prompt-guided style transformation specifically for text regions, while preserving both text readability and stylistic consistency. To achieve this, we design a feature injection module that leverages diffusion model inversion and self-attention to transfer style features effectively. Additionally, a region control mechanism is introduced by applying a distance-based changing mask at each denoising step, enabling precise spatial control. To further enhance visual quality, we incorporate a style enhancement module based on the Fourier transform to reinforce stylistic richness. Extensive experiments demonstrate that our method achieves superior performance in scene text style transformation, outperforming existing state-of-the-art methods in both visual fidelity and text preservation.
Abstract (translated)
随着扩散模型的快速发展，风格迁移取得了显著进展。然而，场景文本中的灵活且局部化的风格编辑仍然是一个未解决的挑战。尽管现有的场景文本编辑方法已经实现了文字区域编辑，但它们通常局限于内容替换和简单的样式更改，并缺乏自由式风格转移的能力。在本文中，我们介绍了SceneTextStylizer，这是一种新颖的、无需训练的基于扩散模型的框架，用于对场景图像中的文本进行灵活且高保真的风格转换。 与先前的方法要么执行全局风格迁移，要么仅专注于文本内容修改不同，我们的方法通过使用提示引导的方式实现了针对文字区域的特定风格变换，并同时保持了文本可读性和风格一致性。为了实现这一点，我们设计了一个特征注入模块，该模块利用扩散模型逆向工程和自注意力机制有效地转移风格特征。此外，我们引入了一种基于距离变化掩码的区域控制机制，在每次去噪步骤中应用以实现精确的空间控制。为了进一步提高视觉质量，我们还集成了一个基于傅立叶变换的风格增强模块，用于加强风格丰富性。 大量的实验表明，我们的方法在场景文本风格转换方面表现出了优越性能，在视觉保真度和文本保留方面均优于现有的最先进的方法。
URL
https://arxiv.org/abs/2510.10910
PDF
-
Jigsaw3D: Disentangled 3D Style Transfer via Patch Shuffling and Masking
2025-10-12 08:22:57Yuteng Ye, Zheng Zhang, Qinchuan Zhang, Di Wang, Youjia Zhang, Wenxiao Zhang, Wei Yang, Yuan LiuAbstract
Controllable 3D style transfer seeks to restyle a 3D asset so that its textures match a reference image while preserving the integrity and multi-view consistency. The prevalent methods either rely on direct reference style token injection or score-distillation from 2D diffusion models, which incurs heavy per-scene optimization and often entangles style with semantic content. We introduce Jigsaw3D, a multi-view diffusion based pipeline that decouples style from content and enables fast, view-consistent stylization. Our key idea is to leverage the jigsaw operation - spatial shuffling and random masking of reference patches - to suppress object semantics and isolate stylistic statistics (color palettes, strokes, textures). We integrate these style cues into a multi-view diffusion model via reference-to-view cross-attention, producing view-consistent stylized renderings conditioned on the input mesh. The renders are then style-baked onto the surface to yield seamless textures. Across standard 3D stylization benchmarks, Jigsaw3D achieves high style fidelity and multi-view consistency with substantially lower latency, and generalizes to masked partial reference stylization, multi-object scene styling, and tileable texture generation. Project page is available at: this https URL
Abstract (translated)
可控的3D风格转换旨在重新设计一个3D资产，使其纹理与参考图像相匹配，并同时保持其完整性和多视角一致性。目前流行的方法要么依赖于直接引用样式令牌注入或从2D扩散模型进行评分蒸馏，这需要大量的场景级优化，且经常将风格与语义内容纠缠在一起。 我们引入了Jigsaw3D，这是一种基于多视图扩散的流水线，它可以将风格和内容分离，并实现快速、视角一致性的风格化。我们的核心思想是利用拼图操作（参考补丁的空间打乱和随机掩码）来抑制物体语义并隔离样式统计信息（调色板、笔触、纹理）。我们通过视图交叉注意力机制将这些样式提示集成到多视图扩散模型中，生成以输入网格为条件的视角一致化的风格化渲染。然后将这些渲染中的样式烘焙到表面，以产生无缝的纹理。 在标准的3D风格化基准测试中，Jigsaw3D实现了高风格保真度和多视角一致性，并且延迟明显降低。此外，该方法还能推广至部分遮挡参考样式的风格化、多个物体场景的风格化以及可平铺纹理生成等应用场景。项目页面可在以下链接访问：[此URL](this%20https%20URL)（请将占位符替换为实际链接）。
URL
https://arxiv.org/abs/2510.10497
PDF
-
ReMix: Towards a Unified View of Consistent Character Generation and Editing
2025-10-11 10:31:56Benjia Zhou, Bin Fu, Pei Cheng, Yanru Wang, Jiayuan Fan, Tao ChenAbstract
Recent advances in large-scale text-to-image diffusion models (e.g., FLUX.1) have greatly improved visual fidelity in consistent character generation and editing. However, existing methods rarely unify these tasks within a single framework. Generation-based approaches struggle with fine-grained identity consistency across instances, while editing-based methods often lose spatial controllability and instruction alignment. To bridge this gap, we propose ReMix, a unified framework for character-consistent generation and editing. It constitutes two core components: the ReMix Module and IP-ControlNet. The ReMix Module leverages the multimodal reasoning ability of MLLMs to edit semantic features of input images and adapt instruction embeddings to the native DiT backbone without fine-tuning. While this ensures coherent semantic layouts, pixel-level consistency and pose controllability remain challenging. To address this, IP-ControlNet extends ControlNet to decouple semantic and layout cues from reference images and introduces an {\epsilon}-equivariant latent space that jointly denoises the reference and target images within a shared noise space. Inspired by convergent evolution and quantum decoherence,i.e., where environmental noise drives state convergence, this design promotes feature alignment in the hidden space, enabling consistent object generation while preserving identity. ReMix supports a wide range of tasks, including personalized generation, image editing, style transfer, and multi-condition synthesis. Extensive experiments validate its effectiveness and efficiency as a unified framework for character-consistent image generation and editing.
Abstract (translated)
最近在大规模文本到图像扩散模型（例如FLUX.1）方面的进展，极大地提升了连贯人物生成和编辑的视觉保真度。然而，现有的方法很少将这些任务整合在一个单一框架内完成。基于生成的方法难以实现实例间的细微身份一致性，而基于编辑的方法则往往失去空间控制性和指令对齐性。为解决这一问题，我们提出了ReMix框架，旨在统一处理人物连贯性的生成和编辑任务。该框架包含两个核心组件：ReMix模块和IP-ControlNet。 - **ReMix模块**利用多模态推理能力来编辑输入图像的语义特征，并将指令嵌入到DiT骨干网络中，而无需进行微调。尽管这确保了连贯的语义布局，但像素级一致性以及姿态控制仍然具有挑战性。 - **IP-ControlNet**则在ControlNet的基础上进行了扩展，以从参考图像中解耦出语义和布局线索，并引入了一个{\epsilon}-等变隐空间，该空间能在共享噪声空间内同时对齐参考图和目标图的去噪过程。这种设计灵感来源于趋同进化与量子退相干现象，即环境中的噪声驱动状态收敛。这一设计在隐藏空间中促进了特征的对齐，在保持身份一致性的前提下实现了物体生成的一致性。 ReMix框架支持多种任务，包括个性化生成、图像编辑、风格迁移和多条件合成等，并通过广泛的实验验证了其作为统一框架的有效性和效率，在连贯人物图像生成和编辑领域表现卓越。
URL
https://arxiv.org/abs/2510.10156
PDF
-
UniVideo: Unified Understanding, Generation, and Editing for Videos
2025-10-09 16:01:30Cong Wei, Quande Liu, Zixuan Ye, Qiulin Wang, Xintao Wang, Pengfei Wan, Kun Gai, Wenhu ChenAbstract
Unified multimodal models have shown promising results in multimodal content generation and editing but remain largely limited to the image domain. In this work, we present UniVideo, a versatile framework that extends unified modeling to the video domain. UniVideo adopts a dual-stream design, combining a Multimodal Large Language Model (MLLM) for instruction understanding with a Multimodal DiT (MMDiT) for video generation. This design enables accurate interpretation of complex multimodal instructions while preserving visual consistency. Built on this architecture, UniVideo unifies diverse video generation and editing tasks under a single multimodal instruction paradigm and is jointly trained across them. Extensive experiments demonstrate that UniVideo matches or surpasses state-of-the-art task-specific baselines in text/image-to-video generation, in-context video generation and in-context video editing. Notably, the unified design of UniVideo enables two forms of generalization. First, UniVideo supports task composition, such as combining editing with style transfer, by integrating multiple capabilities within a single instruction. Second, even without explicit training on free-form video editing, UniVideo transfers its editing capability from large-scale image editing data to this setting, handling unseen instructions such as green-screening characters or changing materials within a video. Beyond these core capabilities, UniVideo also supports visual-prompt-based video generation, where the MLLM interprets visual prompts and guides the MMDiT during synthesis. To foster future research, we will release our model and code.
Abstract (translated)
统一的多模态模型在多模态内容生成和编辑方面显示出了令人鼓舞的结果，但仍然主要局限于图像领域。在这项工作中，我们介绍了UniVideo，这是一个灵活的框架，它将统一建模扩展到了视频领域。UniVideo采用了双流设计，结合了用于指令理解的多模态大型语言模型（MLLM）与用于视频生成的多模态DiT（MMDiT）。这种设计能够准确解释复杂的多模态指令，并保持视觉一致性。基于这一架构，UniVideo将多样化的视频生成和编辑任务统一在一个单一的多模态指令范式下，并在这些任务之间进行联合训练。大量的实验表明，UniVideo在文本/图像到视频生成、上下文视频生成以及上下文视频编辑方面达到了或超越了最先进的特定任务基线。 值得注意的是，UniVideo的统一设计能够实现两种形式的泛化：首先，UniVideo支持任务组合，如将编辑与风格迁移相结合，在单一指令中集成多种能力。其次，即使没有在自由形式视频编辑上进行明确训练，UniVideo也能将其编辑能力从大规模图像编辑数据迁移到这种环境中，处理诸如绿幕人物或改变视频中材质等未见过的指令。 除了这些核心功能外，UniVideo还支持基于视觉提示的视频生成，在这种模式下，MLLM解释视觉提示并指导MMDiT在合成过程中。为了促进未来的研究，我们将发布我们的模型和代码。
URL
https://arxiv.org/abs/2510.08377
PDF
-
PickStyle: Video-to-Video Style Transfer with Context-Style Adapters
2025-10-08 21:02:55Soroush Mehraban, Vida Adeli, Jacob Rommann, Babak Taati, Kyryl TruskovskyiAbstract
We address the task of video style transfer with diffusion models, where the goal is to preserve the context of an input video while rendering it in a target style specified by a text prompt. A major challenge is the lack of paired video data for supervision. We propose PickStyle, a video-to-video style transfer framework that augments pretrained video diffusion backbones with style adapters and benefits from paired still image data with source-style correspondences for training. PickStyle inserts low-rank adapters into the self-attention layers of conditioning modules, enabling efficient specialization for motion-style transfer while maintaining strong alignment between video content and style. To bridge the gap between static image supervision and dynamic video, we construct synthetic training clips from paired images by applying shared augmentations that simulate camera motion, ensuring temporal priors are preserved. In addition, we introduce Context-Style Classifier-Free Guidance (CS-CFG), a novel factorization of classifier-free guidance into independent text (style) and video (context) directions. CS-CFG ensures that context is preserved in generated video while the style is effectively transferred. Experiments across benchmarks show that our approach achieves temporally coherent, style-faithful, and content-preserving video translations, outperforming existing baselines both qualitatively and quantitatively.
Abstract (translated)
我们研究了使用扩散模型进行视频风格迁移的任务，目标是在保持输入视频上下文的同时将其渲染为由文本提示指定的目标风格。一个主要的挑战是缺乏配对视频数据以供监督训练。为此，我们提出了PickStyle框架，这是一种视频到视频的风格迁移框架，它在预先训练好的视频扩散主干网络中加入了风格适配器，并利用带有源风格对应关系的成对静止图像数据进行训练。PickStyle通过在条件模块的自我注意层中插入低秩适配器来实现高效的运动-风格传输专业化，同时保持视频内容和风格之间的一致性。 为了弥合静态图像监督与动态视频之间的差距，我们通过施加模拟摄像机移动的共享增强处理，从成对静止图象构建合成训练片段，以确保时间先验得以保留。此外，我们引入了上下文-样式无分类器引导（Context-Style Classifier-Free Guidance, CS-CFG），这是一种新颖的将无分类器引导分解为独立的文字（风格）和视频（上下文）方向的方法。CS-CFG确保在生成的视频中保持内容一致性的同时有效迁移风格。 跨基准实验表明，我们的方法实现了时间上连贯、风格忠实且内容保留的视频转换，在定性和定量评估中均优于现有基线方法。
URL
https://arxiv.org/abs/2510.07546
PDF
-
Emotional Text-To-Speech Based on Mutual-Information-Guided Emotion-Timbre Disentanglement
2025-10-02 07:03:50Jianing Yang, Sheng Li, Takahiro Shinozaki, Yuki Saito, Hiroshi SaruwatariAbstract
Current emotional Text-To-Speech (TTS) and style transfer methods rely on reference encoders to control global style or emotion vectors, but do not capture nuanced acoustic details of the reference speech. To this end, we propose a novel emotional TTS method that enables fine-grained phoneme-level emotion embedding prediction while disentangling intrinsic attributes of the reference speech. The proposed method employs a style disentanglement method to guide two feature extractors, reducing mutual information between timbre and emotion features, and effectively separating distinct style components from the reference speech. Experimental results demonstrate that our method outperforms baseline TTS systems in generating natural and emotionally rich speech. This work highlights the potential of disentangled and fine-grained representations in advancing the quality and flexibility of emotional TTS systems.
Abstract (translated)
当前的情感文本到语音（TTS）和风格转换方法依赖于参考编码器来控制全局风格或情感向量，但无法捕捉参考语音中的细微声学细节。为此，我们提出了一种新的情感TTS方法，该方法能够在分离参考语音的内在属性的同时，在音素级别上进行精细的情感嵌入预测。所提出的方法采用了一种风格解耦方法，指导两个特征提取器减少音色和情感特征之间的相互信息，并有效地从参考语音中区分出不同的风格成分。实验结果表明，与基线TTS系统相比，我们的方法在生成自然且情感丰富的语音方面表现更佳。这项工作强调了在提高情感TTS系统的质量和灵活性方面，解耦的细粒度表示法的巨大潜力。
URL
https://arxiv.org/abs/2510.01722
PDF
-
PyramidStyler: Transformer-Based Neural Style Transfer with Pyramidal Positional Encoding and Reinforcement Learning
2025-10-02 06:54:52Raahul Krishna Durairaju (California State University, Fullerton), K. Saruladha (Puducherry Technological University)Abstract
Neural Style Transfer (NST) has evolved from Gatys et al.'s (2015) CNN-based algorithm, enabling AI-driven artistic image synthesis. However, existing CNN and transformer-based models struggle to scale efficiently to complex styles and high-resolution inputs. We introduce PyramidStyler, a transformer framework with Pyramidal Positional Encoding (PPE): a hierarchical, multi-scale encoding that captures both local details and global context while reducing computational load. We further incorporate reinforcement learning to dynamically optimize stylization, accelerating convergence. Trained on Microsoft COCO and WikiArt, PyramidStyler reduces content loss by 62.6% (to 2.07) and style loss by 57.4% (to 0.86) after 4000 epochs--achieving 1.39 s inference--and yields further improvements (content 2.03; style 0.75) with minimal speed penalty (1.40 s) when using RL. These results demonstrate real-time, high-quality artistic rendering, with broad applications in media and design.
Abstract (translated)
神经风格迁移（NST）源自 Gatys 等人于2015年提出的基于 CNN 的算法，该技术使得 AI 驱动的艺术图像合成成为可能。然而，现有的基于 CNN 和变压器的模型在处理复杂风格和高分辨率输入时难以高效扩展。我们引入了 PyramidStyler，这是一种采用金字塔位置编码（PPE）的变压器框架：这种分层、多尺度编码方法能够在减少计算负担的同时捕捉局部细节与全局背景信息。此外，我们还结合了强化学习来动态优化图像美化效果，从而加快收敛速度。PyramidStyler 在训练过程中使用了微软 COCO 和 WikiArt 数据集，在经过4000个周期的训练后，其内容损失降低了62.6%（降至2.07），风格损失则减少了57.4%（降至0.86），并在实现1.39秒推理时间的同时达到了上述效果。当采用强化学习时，PyramidStyler 还能在几乎不增加运行速度的情况下进一步优化结果（内容损耗为2.03；风格损耗为0.75，且仅需额外的0.01秒）。这些成果表明了实时高质量艺术渲染的可能性，并在媒体和设计领域有着广泛的应用前景。
URL
https://arxiv.org/abs/2510.01715
PDF
-
Multi-level Dynamic Style Transfer for NeRFs
2025-10-01 07:19:27Zesheng Li, Shuaibo Li, Wei Ma, Jianwei Guo, Hongbin ZhaAbstract
As the application of neural radiance fields (NeRFs) in various 3D vision tasks continues to expand, numerous NeRF-based style transfer techniques have been developed. However, existing methods typically integrate style statistics into the original NeRF pipeline, often leading to suboptimal results in both content preservation and artistic stylization. In this paper, we present multi-level dynamic style transfer for NeRFs (MDS-NeRF), a novel approach that reengineers the NeRF pipeline specifically for stylization and incorporates an innovative dynamic style injection module. Particularly, we propose a multi-level feature adaptor that helps generate a multi-level feature grid representation from the content radiance field, effectively capturing the multi-scale spatial structure of the scene. In addition, we present a dynamic style injection module that learns to extract relevant style features and adaptively integrates them into the content patterns. The stylized multi-level features are then transformed into the final stylized view through our proposed multi-level cascade decoder. Furthermore, we extend our 3D style transfer method to support omni-view style transfer using 3D style references. Extensive experiments demonstrate that MDS-NeRF achieves outstanding performance for 3D style transfer, preserving multi-scale spatial structures while effectively transferring stylistic characteristics.
Abstract (translated)
随着神经辐射场（NeRFs）在各种三维视觉任务中的应用不断扩展，已经开发出了多种基于NeRF的风格转换技术。然而，现有方法通常将风格统计信息整合到原始的NeRF流程中，这往往导致内容保留和艺术风格化方面的结果不尽如人意。本文介绍了一种名为多级动态风格传输（MDS-NeRF）的新方法，该方法专门重新设计了用于风格化的NeRF管道，并引入了一个创新性的动态样式注入模块。 具体而言，我们提出了一个多级特征适配器，它有助于从内容辐射场中生成一个多级特征网格表示，有效捕捉场景的多尺度空间结构。此外，我们还介绍了一个动态样式注入模块，该模块能够学习提取相关风格特征，并将它们自适应地集成到内容模式中。通过我们的提出的多级级联解码器，这些经过风格化的多级特性随后被转换为最终的风格化视图。 另外，我们将3D风格传输方法扩展以支持使用3D样式参考进行全方位风格传输。大量的实验表明，MDS-NeRF在保持多尺度空间结构的同时能够有效地转移艺术特征，在三维风格传输方面取得了卓越的表现。
URL
https://arxiv.org/abs/2510.00592
PDF
-
Stylos: Multi-View 3D Stylization with Single-Forward Gaussian Splatting
2025-09-30 16:09:13Hanzhou Liu, Jia Huang, Mi Lu, Srikanth Saripalli, Peng JiangAbstract
We present Stylos, a single-forward 3D Gaussian framework for 3D style transfer that operates on unposed content, from a single image to a multi-view collection, conditioned on a separate reference style image. Stylos synthesizes a stylized 3D Gaussian scene without per-scene optimization or precomputed poses, achieving geometry-aware, view-consistent stylization that generalizes to unseen categories, scenes, and styles. At its core, Stylos adopts a Transformer backbone with two pathways: geometry predictions retain self-attention to preserve geometric fidelity, while style is injected via global cross-attention to enforce visual consistency across views. With the addition of a voxel-based 3D style loss that aligns aggregated scene features to style statistics, Stylos enforces view-consistent stylization while preserving geometry. Experiments across multiple datasets demonstrate that Stylos delivers high-quality zero-shot stylization, highlighting the effectiveness of global style-content coupling, the proposed 3D style loss, and the scalability of our framework from single view to large-scale multi-view settings.
Abstract (translated)
我们介绍了Stylos，这是一个单一前向的3D高斯框架，用于在未布置的内容上进行三维风格转换，从单张图像到多视角集合，并以单独的参考风格图像为条件。Stylos 无需场景优化或预计算姿态即可合成具有几何感知和视图一致性的风格化3D高斯场景，从而能够泛化至未知类别、场景和风格。在核心方面，Stylos 采用了一个Transformer骨干网，并包含两个路径：几何预测保留自注意力以保持几何精度，而风格则通过全局跨注意力注入以确保视图间的一致性。随着基于体素的3D风格损失的加入，该损失将聚合场景特征与风格统计对齐，Stylos 在保持几何结构的同时实现了视图一致性的风格化。多数据集上的实验表明，Stylos 能够提供高质量的零样本风格化效果，这突显了全局风格-内容耦合的有效性、提出的3D风格损失以及我们的框架从单视角扩展到大规模多视角设置的能力。
URL
https://arxiv.org/abs/2509.26455
PDF
-
Personalized Scientific Figure Caption Generation: An Empirical Study on Author-Specific Writing Style Transfer
2025-09-30 05:52:15Jaeyoung Kim, Jongho Lee, Hongjun Choi, Sion JangAbstract
We study personalized figure caption generation using author profile data from scientific papers. Our experiments demonstrate that rich author profile data, combined with relevant metadata, can significantly improve the personalization performance of multimodal large language models. However, we also reveal a fundamental trade-off between matching author style and maintaining caption quality. Our findings offer valuable insights and future directions for developing practical caption automation systems that balance both objectives. This work was conducted as part of the 3rd SciCap challenge.
Abstract (translated)
我们研究了利用科学论文中的作者个人资料数据进行个性化图像描述生成的方法。我们的实验表明，丰富的作者个人资料数据与相关元数据相结合，可以显著提高多模态大型语言模型的个性化性能。然而，我们也揭示了在匹配作者风格和保持描述质量之间存在一个基本的权衡。我们的研究发现为开发既能平衡这两个目标又能实际应用的图像描述自动化系统提供了宝贵的见解和未来发展方向。这项工作是作为第三届SciCap挑战赛的一部分完成的。
URL
https://arxiv.org/abs/2509.25817
PDF
