Abstract
Semantic segmentation takes pivotal roles in various applications such as autonomous driving and medical image analysis. When deploying segmentation models in practice, it is critical to test their behaviors in varied and complex scenes in advance. In this paper, we construct an automatic data generation pipeline Gen4Seg to stress-test semantic segmentation models by generating various challenging samples with different attribute changes. Beyond previous evaluation paradigms focusing solely on global weather and style transfer, we investigate variations in both appearance and geometry attributes at the object and image level. These include object color, material, size, position, as well as image-level variations such as weather and style. To achieve this, we propose to edit visual attributes of existing real images with precise control of structural information, empowered by diffusion models. In this way, the existing segmentation labels can be reused for the edited images, which greatly reduces the labor costs. Using our pipeline, we construct two new benchmarks, Pascal-EA and COCO-EA. We benchmark a wide variety of semantic segmentation models, spanning from closed-set models to open-vocabulary large models. We have several key findings: 1) advanced open-vocabulary models do not exhibit greater robustness compared to closed-set methods under geometric variations; 2) data augmentation techniques, such as CutOut and CutMix, are limited in enhancing robustness against appearance variations; 3) our pipeline can also be employed as a data augmentation tool and improve both in-distribution and out-of-distribution performances. Our work suggests the potential of generative models as effective tools for automatically analyzing segmentation models, and we hope our findings will assist practitioners and researchers in developing more robust and reliable segmentation models.
Abstract (translated)
语义分割在自动驾驶和医学图像分析等多种应用中扮演着关键角色。当将分割模型部署到实际场景时，提前测试它们在各种复杂环境中的表现是至关重要的。在这篇论文中，我们构建了一个自动数据生成管道Gen4Seg，通过生成具有不同属性变化的各种挑战性样本来压力测试语义分割模型。与以往只关注全局天气和风格转换的评估范式不同，我们研究了对象级和图像级别上外观和几何属性的变化情况。这些变化包括物体的颜色、材质、尺寸、位置以及图像级别的天气和风格等。为了实现这一目标，我们提出利用扩散模型对现有真实图像的视觉属性进行编辑，并且能够精确控制结构信息。这样可以重用原有的分割标签来编辑后的图片，大大减少了人工劳动成本。 使用我们的管道，我们构建了两个新的基准测试集：Pascal-EA和COCO-EA。我们将各种语义分割模型进行了广泛的评估，这些模型涵盖了从封闭集合模型到开放词汇大模型的不同类型。我们有以下几个主要发现： 1. 在几何变化的情况下，先进的开放词汇模型并不比封闭集合方法表现出更强的鲁棒性。 2. 数据增强技术（如CutOut和CutMix）在对抗外观变化时有限于提升模型的鲁棒性。 3. 我们的管道也可以用作数据增强工具，并且能够改善分布内和分布外的表现。 我们的工作展示了生成模型作为自动分析分割模型的有效工具的巨大潜力，希望我们的研究发现能帮助从业者和研究人员开发出更加稳健可靠的分割模型。
URL
https://arxiv.org/abs/2603.01535
