Semantic_Segmentation
-
CroCoMAE: Cross-modal self-supervised collaborative representation learning for SAR-optical remote sensing data
2026-11-10 14:10:41Vito Recchia, Giuseppe Guarino, C\'assio Fraga Dantas, Giuseppina Andresini, Annalisa Appice, Dino IencoAbstract
Remote sensing increasingly captures the Earth's surface through complementary sensors such as SAR and optical, yet ground-truth annotations remain scarce, limiting supervised approaches. Self-supervised learning (SSL) addresses the label scarcity, and cross-modal SSL further exploits the complementarity between co-located sensors, but existing cross-modal frameworks align modalities mostly through global image-level representations or treat them as largely redundant, leaving their fine-grained complementary structure underexploited. We propose CroCoMAE, a cross-modal SSL framework for SAR-optical remote sensing that unifies contrastive learning and masked image modeling within a ViT-based architecture, operating at two complementary levels. At the image level, an intra-modality contrastive loss on the class token yields globally discriminative, augmentation-invariant representations. At the patch level, masked image modeling drives fine-grained per-modality representations, while a cross-modal contrastive objective aligns a shared subspace across SAR and optical patch tokens, leaving the complementary subspace unconstrained to retain modality-specific information. On two SAR-optical benchmarks covering scene classification and semantic segmentation, CroCoMAE improves over uni-modal SSL baselines and outperforms state-of-the-art cross-modal frameworks in most configurations, with gains in both single-modality and multi-modal evaluation across varying label regimes and the largest improvements on semantic segmentation.
Abstract (translated)
URL
https://arxiv.org/abs/2611.14124
PDF
-
HiPFuse: Real-Time LiDAR Semantic Segmentation for Robotic Perception on Edge Devices
2026-11-10 12:57:21Haiyang Wu, George Vosselman, Ville LehtolaAbstract
Real-time robotic perception requires accurate LiDAR semantics under limited resources. Bird's-eye-view (BEV) aggregation is efficient but can mix differently labeled surfaces in columns. We present HiPFuse, a point-BEV fusion framework that exchanges point and height-partitioned BEV features across horizontal scales, combining point-wise detail with 2D spatial context from one LiDAR scan. Fixed height assignments preserve separate vertical aggregation groups, with auxiliary supervision of initial BEV features using targets from the same 3D cells. A dual-scale BEV context module adds broader context from a coarser grid. On SemanticKITTI and MCD, HiPFuse achieves 63.7% and 62.5% mIoU at 14.2 and 22.4 FPS on Jetson AGX Orin, respectively, exceeding the 10 Hz scan rate. Separate training and evaluation show indoor applicability. Replacing only the baseline semantic front end with HiPFuse in fixed robotic systems reduces odometry absolute trajectory error RMSE by 10.6% on average across two outdoor sequences and increases accepted-loop recall by 8.7 percentage points on SemanticKITTI. Qualitative semantic maps show more continuous road labeling. Our code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2611.14025
PDF
-
Sparse All-Layer Connector for Domain Generalised Semantic Segmentation
2026-11-06 03:53:55Ruoyu Guo, Xin Kun Lin, Maurice Pagnucco, Yang SongAbstract
Domain generalised semantic segmentation (DGSS) has recently benefited from combining multiple foundation models, such as vision-language and vision foundation models, to leverage their diverse representations for generalisation. Existing multi-model approaches share a common design choice: cross-model fusion is restricted to depth-aligned layers, implicitly assuming that mutually beneficial information resides at the same depth across models. However, foundation models pretrained under different objectives may develop distinct representation hierarchies, making the optimality of depth-aligned fusion questionable. Moreover, we observe that the foundation models exhibit different depth-wise domain sensitivity. Motivated by these observations, we propose a Sparse All-Layer Connector (SALC) that enables each layer to access information from preceding layers of another model. As the number of accessible layers grows with depth, dense aggregation may hurt generalisation. SALC therefore learns to adaptively select a sparse subset of informative layers. We further introduce a candidate dropout regularisation that strengthens sparsity and encourages SALC to explore diverse layers, leading to more robust selection. Across four foundation model combinations and three DGSS evaluation settings, the proposed design demonstrates stronger generalisation than conventional depth-aligned fusion. The code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2610.15660
PDF
-
From Geometry to Function: A Multi-Task LiDAR Benchmark Dataset for Semantic and Functional Parsing of Urban Facades
2026-11-05 19:48:16Abdiaziz Muse, Tooba Javed, Tao Pu, Jiju Poovvancheri, Dong ChenAbstract
Understanding building facades is central to urban digital twins, heritage modelling, and sustainable city analysis, yet existing LiDAR benchmarks stop at geometry and semantic segmentation and offer little support for reasoning about the functional role of each architectural element. We introduce HFX3D, a large-scale urban facade benchmark of approximately 492 million LiDAR points from 26 buildings in Halifax, Canada, captured across four seasons, day and night, and varied weather. Each facade is annotated at three tiers, semantic, instance, and per-instance functional attributes, making it, to our knowledge, the first facade dataset to pair geometry and semantics with functional labels. Annotations are produced by a semi-automatic pipeline that combines geometry-guided heuristics with expert validation. We benchmark representative point-based, graph-based, convolutional, and transformer architectures and find that semantic segmentation is already challenging under severe class imbalance, while instance separation of repetitive structures and functional reasoning from geometry are substantially harder. The benchmark establishes a new foundation for function-aware 3D urban understanding. Project page: this https URL ; code: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2611.08645
PDF
-
MCUFlowNet: Hardware-Aware Neural Architecture Search for Optical Flow on Microcontrollers
2026-11-05 15:06:14Enmin Lin, Jona BeysensAbstract
Dense optical flow allows small drones to avoid obstacles. However, few dense optical flow networks are designed for microcontrollers (MCUs) with neural accelerators, whose on-chip SRAM limits the input resolution. We present PACE-NAS, a macro-to-micro neural architecture search framework that decouples memory from latency: a macro design fixes the backbone and its peak SRAM, and a micro design searches operators and depths to trade accuracy against compiler-estimated latency. We apply it to EdgeFlowNet on a Grove Vision AI V2 board (Ethos-U55), where profiling locates the peak SRAM at the full-resolution output head. Replacing transposed convolutions with bilinear upsampling and convolutions lowers the peak SRAM by 18%, and two lightweight context modules lower the endpoint error (EPE) without raising the peak. All searched subnets share this peak, so the search needs no memory constraint. We name the Pareto-optimal models MCUFlowNet; the fastest variant, MCUFlowNet-S, runs 1.76x faster than a retrained EdgeFlowNet at a 1.6% higher EPE and, in simulated forest flights at 2 m/s, reaches the goal without collision in 51.5% of trials, against 41.8%. PACE-NAS also applies to semantic segmentation, where a searched PIDNet-S variant runs 1.17x faster at similar accuracy. Code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2611.08147
PDF
-
LiSA: Linear Stochastic Attention for Long-Range Visual Modeling
2026-11-04 23:12:45Sahar Dastani, Fereshteh Shakeri, Samuel Barbeau, Kunal Mahatha, Ismail Ben Ayed, Herve Lombaert, Christian DesrosiersAbstract
Self-attention has become the standard for modern sequence modeling, enabling Transformers to model long-range dependencies across a wide range of modalities. Despite these advances, its quadratic cost with respect to sequence length remains a significant computational bottleneck. Recent work addresses this limitation by introducing linear variants of self-attention, namely with kernel approximations of the softmax operator. However, their first-order aggregation restricts expressivity and struggles to capture higher-order structure. We propose Linear Stochastic Attention (LiSA), a linear attention plug-in module that enables higher-order token interactions via a stochastic diffusion process based on truncated random walks. LiSA incorporates adaptive locality, favoring semantically similar neighbors to preserve fine-grained structures, while maintaining stable and efficient training. We conduct extensive experiments on classification, semantic segmentation, and object detection tasks and show improved results over state-of-the-art methods.
Abstract (translated)
URL
https://arxiv.org/abs/2611.04216
PDF
-
Agent-Driven Generative Priors for Pathology Image Synthesis and Segmentation
2026-11-04 16:02:29Sicheng Yang, Rui Tang, Haipeng Zhou, Hongqiu Wang, Zhaohu Xing, Jianda Ma, Lie Dai, Guang Yang, Lei ZhuAbstract
Deep learning methods for pathology are constrained by the limited availability and high cost of large-scale pixel-level annotations. Synthetic data can alleviate this limitation, but existing pipelines often lack reliable quality control and remain weakly connected to downstream analysis. To address these issues, we propose a three-stage framework that integrates automated data screening, generative representation learning, and efficient adaptation to downstream tasks. First, we develop a pathology agent system to assess image quality and filter samples, resulting in TCGA-200K, a dataset containing 217,030 pathology patches selected from the public TCGA repository. Second, we train a generative model on the screened data. In addition to image synthesis, its backbone learns structural and semantic representations that can be transferred to downstream tasks. Third, we adapt the pretrained generative backbone to semantic segmentation using a lightweight task-specific head. We further introduce PathSeg-380K, a semantic segmentation dataset with expert annotations for approximately 380,000 cells across 12 cell types. Experiments on multiple pathology datasets show that the proposed framework achieves state-of-the-art segmentation performance. Our code is publicly available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2611.06701
PDF
-
Rohbau3D-Semantics: Dense Semantic and Instance Labels for Point Clouds of Indoor Shell Construction Scenes
2026-11-04 08:59:39Lukas Rauch, Thomas BramlAbstract
We present the Rohbau3D-Semantics dataset, which adds dense semantic and instance-level labels to laser scans of the previously released Rohbau3D domain data from shell construction sites. The dataset comprises 504 point cloud scenes from 14 survey sites, each captured with a terrestrial laser scanner and stored in a scanner-centered local coordinate system. Unlike point clouds sampled from reconstructed meshes or synthetic building models, the data preserve the characteristic properties of real laser scanning, including non-uniform sampling, measurement artifacts, reflections, and fine geometric detail. Combined with manual labels, expert oversight during pointwise annotation, and full-scene review, this yields a high-quality benchmark for semantic scene understanding in complex built environments. The dataset includes 17 class labels of building components and one background class, along with instance annotations for object-level analysis. In addition to the 3D point clouds, the release also provides scripts for deriving geometrically aligned panoramic and cube-map representations of color, LiDAR reflection intensity, depth, surface normals, semantic labels, and instance labels from the 3D arrays. We describe the acquisition pipeline, annotation protocol, quality-control process, and panorama-generation procedure, and provide benchmark splits and baseline experiments for semantic segmentation. By combining precise terrestrial laser scans, dense annotations, and a wide variety of shell-construction scenes, Rohbau3D-Semantics bridges the gap among synthetic indoor datasets, mesh-derived benchmarks, and smaller construction-specific resources. The dataset is intended to support research on 3D scene understanding, construction-site analytics, and multimodal scene analysis in the built environment. Access and download the full dataset via this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2611.06180
PDF
-
Feature Puzzles: Single-Image Supervision for Multi-View 3D-Aware Segmentation
2026-11-04 03:07:40Jiahao Ma, Qiang Zhang, Miaomiao Liu, David Ahmedt-Aristizabal, Chuong NguyenAbstract
Humans can easily recognize the same objects across different viewpoints of a 3D scene, whereas teaching models such multi-view-consistent perception typically requires videos or calibrated multi-view captures, which are considerably more expensive and less diverse than large-scale monocular data.~In this paper, we introduce Feature Puzzles, a novel framework for learning multi-view-consistent segmentation using only single-image semantic segmentation supervision. Specifically, we propose a simple yet effective pipeline that constructs multi-view-consistent segmentation pseudo-labels through image cropping and geometric transformations. We further introduce a lightweight, plug-and-play network that leverages the multi-view-consistent image features learned by existing pretrained 3D foundation models and the generated pseudo-labels to perform multi-view-consistent segmentation. Importantly, our flexible framework supports various types of prompts provided from one or multiple views and can be readily extended to open-vocabulary 3D segmentation. Under this single-image supervision, our feed-forward model achieves the best mIoU on NVOS, remains competitive on SPIn-NeRF and ScanNet++ against baselines using per-scene optimization or in-domain multi-view training, and demonstrates zero-shot transfer to open-vocabulary 3D segmentation. Visual comparisons are available on our project page: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2611.05874
PDF
-
Role-based Effectiveness of an Epistemic Uncertainty Estimate in Training Models for Semantic Segmentation of 3D LiDAR Point Clouds
2026-11-03 20:01:52Krishnakumar N, Jaya Sreevalsan-NairAbstract
Point cloud segmentation is a fundamental task in 3D data analysis, with applications across robotics, augmented and virtual reality, and remote sensing. However, the inherent sparsity, noise, and unordered nature of point clouds introduce significant challenges, particularly in achieving reliable classification. While the state-of-the-art rely on training models for explicit modeling of uncertainty, there is a gap in exploring different roles that estimated uncertainty using generative approaches can play. This work investigates the impact of deterministic uncertainty in point cloud segmentation by explicitly modeling epistemic uncertainty and using it in different roles, namely, as a feature and in loss function. We evaluate a baseline artificial neural network (ANN) and two uncertainty-aware variants (uncertainty as an additional input feature and as a per-sample confidence weight in the training loss) on four large-scale LiDAR benchmarks spanning airborne and mobile acquisition, namely ISPRS Vaihingen, AHN3, Paris-Lille-3D, and Toronto-3D, together with PointNet++ and RandLA-Net references and their uncertainty-augmented counterparts under canonical spatially disjoint train and test splits. Under this protocol, uncertainty-aware modeling improves mean Intersection over Union, recovers minority classes that the baseline ANN never predicts at all, and in half of the configurations improves the alignment between prediction error and estimated uncertainty as measured by Patch Accuracy versus Patch Uncertainty. Our findings indicate that uncertainty can play multiple roles in the segmentation pipeline rather than as a device for improving accuracy alone.
Abstract (translated)
URL
https://arxiv.org/abs/2611.05569
PDF
-
FoAViS: Analyzing Learned Fixation Strategies in a Foveated Active Vision System
2026-10-30 16:34:59Valentin Hassler, Jonathan Henrich, Yoha\"i-Eliel Berreby, Haoze Zhu, Alexander EckerAbstract
Human vision is foveated and active: resolution is highest at the fovea and falls off towards the periphery, and fixations are actively directed to locations of interest. Building a machine system that processes information in the same way lets us study gaze behavior in a controlled setting where the system's components and objectives are known. The emergence of gaze characteristics can therefore be examined systematically, which may in turn inform our understanding of biological vision. This requires a model that can sample like the fovea, handle various tasks from classification to dense prediction, and be trained concurrently with a fixation policy. In our study, we present such a system and study its capabilities and the fixation policies that emerge from it. Our system, FoAViS, samples visual input by adopting the established log-polar foveation and accumulates information in a 2D spatial memory. Due to its 2D structure, the same memory can be read out globally for classification or densely for semantic segmentation, without any architectural change. Fixations are drawn from a 2D map by a policy trained concurrently with the task. We find that learned fixation policies select more informative, task-specific locations than non-learned heuristics. These policies also shape the task representations in a beneficial way: for both classification and segmentation, models trained concurrently with a policy outperform models trained with random locations, even when the latter are provided with learned scanpaths. Moreover, these scanpaths exhibit similarities to human gaze behavior by capturing aspects of empirical scanpath characteristics. Our work marks an important step away from standard passive, single-pass architectures, towards biologically grounded models of vision. It opens a path for studying goal-directed gaze behavior and information representation in the visual system.
Abstract (translated)
URL
https://arxiv.org/abs/2610.41987
PDF
-
Vision Transformer Ensembles for Panoramic Street Segmentation
2026-10-30 13:13:43Yunus Serhat B{\i}\c{c}ak\c{c}{\i}Abstract
Semantic segmentation of street panoramas can support detailed descriptions of urban environments, yet small datasets and unequal training costs make model selection difficult. This paper documents the two stages of a submission to the PalmCity challenge. In the first stage, nine pretrained segmentation systems are compared using approximately equal computation budgets, and the two leading candidates are trained with three random seeds. An equal probability average of three Encoder only Mask Transformer models with DINOv3 features, evaluated at three image scales with horizontal reflection, reaches 60.95% mean intersection over union on the 84 image validation split and 57.08% on the hidden test leaderboard, first in the snapshot of 5 October 2026. In the second stage, color and crop augmentation, rare class sampling, larger inputs, further architectures, and post processing are tested under the same update budget, and candidates are compared through a paired image bootstrap with selection rules fixed before the results. Training at 768 by 1536 and 896 by 1792 pixels gives the largest gains. The selected ensemble of four models scores 64.53% on validation when trained on the training images alone. Retrained on the training and validation images, it receives 61.54% mean intersection over union and 71.93% mean F1 on the hidden test leaderboard, first among five entries in the snapshot of 30 October 2026. Segment Anything voting, instance mask overlays, and ensembles with other architectures all reduce the validation score. Because the validation images guided many choices and the final models were trained on them, the validation scores are optimistic and the test score is the only independent measurement. The study reports all runs, comparisons, costs, and limits, and releases the code and models.
Abstract (translated)
URL
https://arxiv.org/abs/2610.06063
PDF
-
Inter-Frame Predictive Contour Coding for Semantic Map Compression
2026-10-29 19:07:28Runyu Yang, Hyomin Choi, Fabien Racap\'e, Ivan V. Baji\'cAbstract
Dynamic semantic maps have become an important intermediate representation for many computer vision applications. Efficient lossless compression of such maps is essential for reducing storage and transmission costs. Existing contour-based semantic map compression methods mainly exploit spatial redundancy within individual frames while largely ignoring the strong temporal correlation between adjacent semantic maps. In this paper, we propose an inter-frame predictive contour coding framework for lossless semantic map compression. The proposed framework first predicts blob positions and label indexes from the reference frame and then performs contour prediction using corresponding chain-code units in the reference contours. A mode selection mechanism is further employed to choose between intra and inter coding according to the estimated coding cost. By exploiting temporal contour redundancy while preserving the advantages of contour-based representation, the proposed framework significantly improves coding efficiency. Experimental results on the VSPW dynamic semantic segmentation dataset demonstrate that the proposed method consistently outperforms state-of-the-art lossless semantic map compression methods. Compared with the recent context-adaptive extended chain coding framework, the proposed inter-frame coding achieves an average bitrate reduction of 3.0\% without significant increase in the average encoding time, demonstrating an effective trade-off between compression performance and computational complexity. The source code is made publicly available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2610.40887
PDF
-
Spectral Invariant Fusion and Refinement for Domain Generalized Semantic Segmentation
2026-10-29 12:29:41Yansong Bai, Yunzhi Zhuge, Xinzhuo Yu, Lu Zhang, Jin Zhan, Pingping Zhang, Huchuan LuAbstract
Domain generalized semantic segmentation is challenging because it requires accurate dense prediction on unseen domains without any access to target-domain data during training. Existing methods improve transferability through data diversification, invariant learning, or direct use of foundation models, but they often struggle to simultaneously suppress domain-sensitive interference and preserve fine-grained spatial cues. In this work, we propose SIFR, a frequency-aware dual-stream framework that tackles domain shift at both the fusion and decoding stages. SIFR uses spectral modeling as a unified principle to regulate both heterogeneous prior interaction and dense decoder refinement. Instead of directly fusing frozen CLIP and DINOv2 features, SIFR performs spectral-guided invariant fusion to filter out domain-sensitive responses while preserving complementary semantic and structural information. In addition, SIFR incorporates a spectral decoder refinement module that stabilizes multi-scale decoder features through low-frequency statistics-based normalization and adaptive frequency selection, thereby reducing the propagation of residual domain bias in dense prediction. This design enables SIFR to jointly improve domain robustness and segmentation precision under severe distribution shifts. Extensive experiments on multiple DGSS benchmarks, including synthetic-to-real, real-to-real, and adverse-weather settings, demonstrate consistent improvements over previous state-of-the-art methods.
Abstract (translated)
URL
https://arxiv.org/abs/2610.39926
PDF
-
LiDAR-SHIELD: A Fully Dockerized Adversarial Robustness Assessment Framework for LiDAR-Based 3D Perception Models
2026-10-29 11:27:53Alexandros Gkillas, Ioulia Kapsali, Christos Anagnostopoulos, Aris S. LalosAbstract
LiDAR-based 3D perception underpins autonomous driving and robotics, yet deploying these models safely requires stress-testing them against adversarial inputs before real-world use. Although state-of-the-art adversarial attacks exist, their implementations are often fragmented across individual repositories, each tied to a specific architecture or task and requiring manual environment reconstruction and model-specific integration code. We present LIDARSHIELD, a unified and fully Dockerized framework that reverses this workflow: rather than installing the target model inside the toolkit, LIDARSHIELD exposes any model through three lightweight HTTP endpoints, namely /info, /forward, and /gradient As a result, connecting a new model or dataset requires only a single loader function or adapter. The framework supports both 3D semantic segmentation and 3D object detection through the same engine-service protocol, integrating a task-aware suite of state-of-the-art attacks, including PGD and Square Attack for segmentation, and point perturbation, point removal, point injection, and LiDAttack for detection. This enables systematic exposure of corner cases and vulnerabilities across heterogeneous LiDAR backbones. The LiDAR-Shield codebase is publicly available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2610.39848
PDF
-
HDVS: semi-supervised semantic segmentation via heterogeneous dual-branch voting supervision
2026-10-27 13:42:08Yongqi Shan, Yunzhi Zhuge, Huchuan LuAbstract
Semi-supervised semantic segmentation, which aims to utilize large volumes of unlabeled images to achieve accurate segmentation results with fewer human annotations, has attracted increasing attention. Prior methods were primarily based on the assumption that the pseudo-labels generated by each branch were sufficiently reliable to supervise the remaining branches. However, there is a concern that the errors in pseudo-labels could accumulate during co-training, potentially resulting in suboptimal performance. To this end, we present a novel framework called heterogeneous dual-branch voting supervision (HDVS), which is designed to enhance the reliability of pseudo-labels and mitigate the issues arising from pseudo-labeling. Specifically, based on the cross-supervision framework, we introduce a voting mechanism that correlates pseudo-labels from heterogeneous branches to produce pseudo-labels with enhanced reliability. Concurrently, we employ a feature communication module to introduce perturbations at the feature level in each branch to maximize prediction diversity across the dual branches while maintaining convergence. Comprehensive evaluations of the proposed HDVS on two benchmark datasets (PASCAL VOC 2012 and Cityscapes) demonstrate its superiority to the state-of-the-art approaches.
Abstract (translated)
URL
https://arxiv.org/abs/2610.37236
PDF
-
RePAIR: Inverting Backdoor Attacks in Semantic Segmentation
2026-10-27 10:53:47Arnold Brosch, Kira MaagAbstract
Despite significant advances in computer vision, semantic segmentation models remain vulnerable to backdoor attacks, posing a security risk. These attacks redirect pixels of a chosen victim class to an attacker-specified target class when a trigger is present, while preserving clean predictions. Existing defenses are largely derived from image classification, while dedicated segmentation defenses remain scarce and typically require retraining or model modification. We introduce RePAIR, a post-hoc, label-free, attack-unaware defense based on the observation that backdoor-induced shifts can be stronger in the classifier head than in internal representations, leaving recoverable victim-class semantics. RePAIR infers the victim-target relation from an unlabeled, potentially contaminated image pool, constructs a safety-approved semantic reference, and localizes suspicious pixels by comparing head predictions with internal semantic evidence. It then applies feature-space and output corrections toward the inferred victim semantics, while conservative localization and output constraints limit changes to benign predictions. We model the attack as adversarial probability transport and repair as its approximate reversal, deriving bounds and sufficient conditions for localization, residual attack success, clean-side changes, defense-induced target predictions, and attack-unaware pair identification. Experiments across segmentation architectures and attack settings show that RePAIR substantially reduces attack success while largely preserving clean performance.
Abstract (translated)
URL
https://arxiv.org/abs/2610.37015
PDF
-
SAGE: Saliency-Annealed Gated Encoder for Universal Vision Recognition
2026-10-27 06:34:00Le Yang, Yucheng Huang, Junyu Chen, Qi Dang, Xin Kong, Haifeng Huo, Qiang ZhaoAbstract
When adapting self-supervised Vision Transformers (ViTs) to dense prediction, existing methods often treat the ViT as a passive feature extractor and may underuse the rich priors encoded in its representations, including saliency, global context, and high-frequency details. We propose SAGE (Saliency-Annealed Gated Encoder), an enhanced ViT-Adapter backbone that explicitly exploits these priors through three complementary innovations: (1) a saliency-guided architecture that extracts DINO-style class-token (CLS) saliency to guide feature gating and high-frequency enhancement, together with dynamic multi-layer aggregation and a local-detail branch; (2) an annealing-based training paradigm designed to improve convergence and feature quality during early training while reducing feature drift later in training; and (3) a two-stage, quality-aware training mechanism that smoothly transitions from focal loss to Varifocal Loss (VFL) within a single training run. Across the evaluated tasks (Fig.~\ref{fig:fig_com}), SAGE achieves state-of-the-art (SOTA) or competitive results on COCO (62.8 box AP and 55.3 mask AP), COCO keypoint estimation (78.5 AP), ADE20K (60.3 mIoU for semantic segmentation and 44.9 mask AP for instance segmentation), TinyImageNet classification (94.76\% Top-1 accuracy), and DIOR-R oriented object detection (78.8 AP$_{50}$). The same backbone is used across tasks, with task-specific decoders. To the best of our knowledge, SAGE is the first framework to achieve multi-task state-of-the-art results on Huawei Ascend NPUs. Code and pretrained models will be released after internal review.
Abstract (translated)
URL
https://arxiv.org/abs/2610.36719
PDF
-
S$^2$-RL: Sample-Set Dual Reinforcement Learning for Generative Semantic Segmentation Dataset Distillation
2026-10-26 14:21:31Haoyu Wang, Fei Zhou, Qingqing Qiu, Lei Zhang, Wei Wei, Chen DingAbstract
While dataset distillation has witnessed significant advances in image classification, its extension to dense prediction tasks such as semantic segmentation is plagued by two core bottlenecks: bi-level optimization-based approaches suffer from prohibitive computational costs stemming from pixel-wise gradient unrolling, whereas proxy-based generative methods merely infuse visual textures into fixed semantic masks derived from real data, thus failing to condense rich semantic knowledge into a compact set of informative, novel scene generations. To address both bottlenecks simultaneously, we propose a novel Sample-Set Reinforcement Learning (S$^2$-RL) framework for generative semantic segmentation dataset distillation (SSDD). Specifically, S$^2$-RL formulates SSDD as a diffusion model based text-to-image generation paradigm, enabling flexible generation of images with informative, novel semantic distributions via a concise text prompt. Furthermore, we fine-tune a diffusion policy via Group Relative Policy Optimization (GRPO), leveraging a sample-set dual reward paradigm: for individual samples, it enforces semantic alignment with input text prompts and intra-class feature diversity; for sample sets, it maximizes inter-sample semantic distribution diversity by solving a contextual multi-armed bandit problem. These design choices enable S$^2$-RL to distill large-scale semantic segmentation datasets into a compact set of generated samples that encapsulate diverse, informative semantic knowledge without the need of pixel-wise gradient unrolling. Extensive experiments on the ADE20K and COCO datasets demonstrate that S$^2$-RL achieves substantial and consistent improvements over state-of-the-art baselines, establishing a new benchmark for SSDD.
Abstract (translated)
URL
https://arxiv.org/abs/2610.35491
PDF
-
Rethinking Data Imbalance in Class Incremental Surgical Instrument Segmentation
2026-10-26 06:52:40Shifang Zhao, Long Bai, Kun Yuan, Feng Li, Jieming Yu, Wenzhen Dong, Guankun Wang, Mobarakol Islam, Nicolas Padoy, Nassir Navab, Hongliang RenAbstract
In surgical instrument segmentation, the increasing variety of instruments over time poses a significant challenge for existing neural networks, as they are unable to effectively learn such incremental tasks and suffer from catastrophic forgetting. When learning new data, the model experiences a sharp performance drop on previously learned data. Although several continual learning methods have been proposed for incremental understanding tasks in surgical scenarios, the issue of data imbalance often leads to a strong bias in the segmentation head, resulting in poor performance. Data imbalance can occur in two forms: (i) class imbalance between new and old data, and (ii) class imbalance within the same time point of data. Such imbalances often cause the dominant classes to take over the training process of continual semantic segmentation (CSS). To address this issue, we propose SurgCSS, a novel plug-and-play CSS framework for surgical instrument segmentation under data imbalance. Specifically, we generate realistic surgical backgrounds through inpainting and blend instrument foregrounds with the generated backgrounds in a class-aware manner to balance the data distribution in various scenarios. We further propose the Class Desensitization Loss by employing contrastive learning to correct edge biases caused by data imbalance. Moreover, we dynamically fuse the weight parameters of the old and new models to achieve a better trade-off between the biased and unbiased model weights. To investigate the data imbalance problem in surgical scenarios, we construct a new benchmark for surgical instrument CSS by integrating datasets. Extensive experiments demonstrate the effectiveness of the proposed framework, achieving significant performance improvement against existing baselines.
Abstract (translated)
URL
https://arxiv.org/abs/2610.34925
PDF
