arXiv_AI
-
Agentic Computational Algebra
2026-11-11 22:32:22Ryan A. Anderson, Matthew Trager, Ben Bowman, Wei Xia, Stefano SoattoAbstract
We explore the use of AI agents for solving computational problems in algebra. For the problems we consider, the main challenge is not correctness but \emph{time}: general-purpose algorithms are guaranteed to provide correct solutions but can be prohibitively slow, while reformulations that exploit special structure often allow solving a problem much faster. We propose to use this setting to evaluate how effectively agents discover and exploit structure within and across tasks to reduce computational costs. More broadly, computational algebra supplies infinitely many problems with independently verifiable answers and systematically controllable structure and difficulty. To illustrate these possibilities, we use benchmark families of systems with controlled shared structure and evaluate agents in three settings: solving independent instances, solving online streams of related instances using persistent memory, and compiling a fast reusable solver for a known problem family. Our experiments show that (i) agents can extend the range of systems solvable within a fixed budget, although reasoning overhead can sometimes outweigh computational savings; (ii) memory can reduce the cost of repeated solving but may preserve suboptimal strategies; (iii) compilation produces faster reusable solvers but at substantial upfront cost. Overall, computational algebra offers concrete, measurable targets for improving how agents discover and reuse structure.
Abstract (translated)
URL
https://arxiv.org/abs/2611.16073
PDF
-
Tokenized Reward Systems for Federated Learning: The Privacy vs. Fairness Tradeoff
2026-11-11 22:21:23Andreas Sendros, Nikolaos Pavlidis, Vasilios Perifanis, Pavlos S. EfraimidisAbstract
Federated learning (FL) systems increasingly rely on blockchain based reward mechanisms to incentivize participation while preserving data privacy. We design and empirically evaluate two complementary Solidity smart contracts for FL reward distribution: the first one that allocates rewards proportionally to a publicly disclosed quality score, and a privacy preserving variant that hides individual quality scores behind a zero-knowledge (ZK) proof of a threshold condition. Both contracts mint a non transferable, use-and-burn token maintaining a trustless incentive structure. We evaluate the two designs through MNIST training with adversarial behavior, including Sybil attacks, zero gradient and random noise, across three independent random seeds. Our results reveal a fundamental tradeoff between cryptographic privacy and incentive fairness. The privacy preserving architecture strictly degrades system robustness, doubling the adversary's average reward capture while introducing severe outcome variance. This vulnerability is driven by a structural self-inclusion bias within cosine-similarity scoring: as the global model converges, legitimate gradient updates naturally diminish, rendering them statistically indistinguishable from bounded noise. Adversaries exploit this dynamic to systematically breach the ZK threshold, triggering a late stage failure cascade that culminates in absolute adversary monopoly over the reward pool and the total exclusion of honest contributors. Alongside a closed-form theorem formalizing this privacy--Sybil resistance tradeoff, we map strict mainnet deployment boundaries via linear gas scaling models, explicitly incorporating the post-Fusaka EIP-7825 per transaction gas cap. Ultimately, while cryptographically secure, static threshold ZK gates fail as incentive layer fairness mechanisms under realistic FL convergence dynamics on standard benchmarks.
Abstract (translated)
URL
https://arxiv.org/abs/2611.16065
PDF
-
Human Capital in Context: What Employment Survival Reveals About Worker-Firm Match
Abstract
There is no best worker in general, only a best worker for a particular firm. We develop a measure of worker-firm match quality that treats fit as a property of the pair, learned from the survival of employment relationships. A two-tower neural network embeds workers, from their career histories and current positions, and firms, from their characteristics and business descriptions, into a common space, and match quality is the cosine similarity of the two vectors. The measure takes its functional form from the skill-weights model of firm-specific human capital, and its only estimated objective, a separation hazard, from the view of the job match as an experience good. Estimated on more than a billion person-periods covering 101 million workers and 8.4 million firms, a 0.1 increase in match quality lowers the odds of separation by 48%. About a quarter of the variance of match quality is specific to the worker-firm pair, which worker-, firm-, and occupation-level measures set to zero by construction. Workers earn more when better matched, especially early in a relationship, and firms whose workforces score higher grow faster, become more profitable, and are issued more, and more valuable, patents over the next five years than their industry peers. Because it scores pairs that never formed, the measure also predicts where movers go, and acquirers record a larger share of the price as goodwill when the target's workers would fit the acquirer better than they fit the target.
Abstract (translated)
URL
https://arxiv.org/abs/2611.16058
PDF
-
Pyramidal Columns Acting as Attention Heads
2026-11-11 22:13:12Fabrizia Giulia Garavaglia, Marco GiuntiAbstract
Can the functional architecture of a Transformer attention head be given a biological construction? More specifically, could a suitable group of pyramidal columns in posterior temporal language regions (PTLRs) constitute the biological construction of a neuromorphic mechanical realization of an attention head, under a mathematical instantiation that need not coincide with its standard Transformer instantiation? This paper investigates this hypothesis by examining possible correspondences between the functional organization of attention heads and neurophysiological mechanisms operating within and across pyramidal columns involved in human language processing. Our analysis builds on a mechanical-structural description of Transformer attention heads as systems of simple neural networks linked through output-input and output-weight interconnections. We examine the functional organization of an attention head and the input-dependent transformations it produces, and ask how the corresponding processing relations could be constructed in biological neural tissue. We then identify candidate neural mechanisms of signal transmission and modulation within and across pyramidal columns that may jointly support such a construction. On this basis, we propose that a suitably organized group of pyramidal columns in PTLRs could constitute a biological construction of a neuromorphic mechanical realization of an attention head. Whether the required functional organization actually exists in PTLRs remains an open empirical question. The hypothesis does not require the biological system to reproduce the standard mathematical instantiation of Transformer attention: the same functional architecture may be mechanically realized and mathematically instantiated in a different way, consistent with its biological construction.
Abstract (translated)
URL
https://arxiv.org/abs/2611.16057
PDF
-
Scaling public input on AI policy with pluralistic agents
2026-11-11 21:35:06Jacy Reese Anthis, Ken Burke, David Madras, Crystal Qian, Meredith Ringel Morris, Logan Cross, Caroline Pantofaru, Utsav PrabhuAbstract
AI is being deployed across a vast range of application domains, cultural environments, and human stakeholders. Safe and responsible AI development requires continuously accounting for diverse perspectives. We develop and test pluralistic agents, AI agent representatives that deliberate on behalf of individuals to align AI systems with diverse human perspectives. Pluralistic agents could help people contribute to many more discussions than they could directly participate in, on technical topics that would require training, and on policy choices that rely on private information that corporations and governments cannot readily share. Our experiments identify opportunities, risks, and perceptions of pluralistic agents. Agents were able to reliably shift policies in the direction of their represented persona's preferences, but the extent of each agent's influence was sensitive to the instructions from its persona, such as having more influence when instructed to be more verbose or less cooperative. This suggests that influence in AI-assisted deliberation could vary based on how participants prompt their agents and how the system is designed. In a preregistered study, 746 humans participated in the pluralistic agents process. Participants viewed pluralistic agents as useful and fair, preferring agents over the baseline of providing input to a single AI reasoning model. Most participants preferred participating themselves over either pluralistic agents or the baseline, but pluralistic agents recovered 68% of the baseline-direct gap in reported satisfaction and 82% in fairness. Because pluralistic agents can be used in many real-world contexts where direct discussion is not, they could be a promising tool to scale AI safety policy development. We encourage cautious exploration of this method, playing to its strengths as an amplifier of human input rather than a replacement.
Abstract (translated)
URL
https://arxiv.org/abs/2611.16028
PDF
-
Runtime Analysis of Cartesian Genetic Programming on Max: A Proven Exponential Speedup
Abstract
Genetic Programming (GP) is a search paradigm inspired from natural evolution for the automated discovery of expressions, functions, and computer programs. Cartesian GP (CGP) is a flavor of GP that uses a graph-based model to encode candidate programs, as contrasted to the conventional Tree-based GP (TGP). Since its inception two decades ago, CGP has predominantly been analyzed empirically, thus little is known regarding its theoretical performance guarantees. This paper analyzes CGP for the Max problem, which has been only rigorously studied for TGP with a proven expected runtime superlinear in the optimal program length $n$. We prove that the expected time for the ($1$+$1$)~CGP employing common mutation operators to evolve an equivalent optimal of the same output is polylogarithmic in $n$, specifically at most $O(\log^9{n})$. This is remarkable, as it shows an exponential speedup by switching to CGP. Our results shed light on the benefit and the compactness of the graph-based representation for programs, and on how CGP can navigate its search and fitness spaces efficiently. This is a first proven performance guarantee for CGP on an established benchmark problem. Experiments complement our theoretical findings.
Abstract (translated)
URL
https://arxiv.org/abs/2611.16013
PDF
-
Iris-Z: Scaling One Generative Model Past Specialist Geometry Models
Abstract
Language moved from many task-specific models to one unified model: a single generative model with a single generative training task that handles all of them within the same backbone, and keeps improving with data, parameters and compute. We believe the bitter lesson applies just as much to computer vision, 3D and spatial understanding, in the end to visual intelligence: the effort should go into general architectures that solve all these tasks in a unified way, and into scaling them, rather than into task-specific designs. We test this on monocular geometry, a task that has belonged to specialised architectures for years. Iris-Z is a general image generator that produces appearance and geometry together. A single backbone denoises, reasons about and understands the image and its depth at the same time, and one set of weights generates images with their depth, estimates depth from photographs and paints images onto a given depth. At enough scale it surpasses state-of-the-art specialists on established depth benchmarks while keeping the generative abilities of the backbone it started from without specialized layers. We release two models for the community.
Abstract (translated)
URL
https://arxiv.org/abs/2611.16010
PDF
-
JustRUA: A Controlled Study of GPT-6 Robot-Use Agents
2026-11-11 21:08:31Zefan Wang, Yufan Dang, Xuanwei Shi, Mingqi Gao, Cheng Yin, SikYuen Tam, Hanyu Liu, Yasheng Wang, Yaxi Lu, Songyang Zhang, Xin Cong, Yuan YaoAbstract
GPT-6 Astra can drive a robot arm directly through a fixed harness, and scores of such robot-use agents on RoboDojo have risen quickly since. Reported systems differ in harness and test subset and often change several designs at once, so it is hard to tell which choices produce the gains. We take these designs apart and test them in one framework, along the harness around the model, the action interface, the reuse of experience and the scene geometry given to the model. A lean harness that keeps the full episode history and stops each motion early goes a long way. Reusing experience changes how the model acts but is not a free lunch, and scene geometry helps most, even when depth is predicted from RGB alone. JustRUA starts from the official L3 harness and keeps only what matters most. Even with no learned policy, self-improvement loop or task-specific tuning, this simple harness raises the score on the RoboDojo simulation benchmark substantially, from 29.7 to 50.3.
Abstract (translated)
URL
https://arxiv.org/abs/2611.16009
PDF
-
Fit-LRP: Fitting Linear Relaxations to Polytopes for Neural Network Verification over Non-Convex Regions
2026-11-11 21:05:13Brian Yarbrough, Jesse Wayment, Jingbo Wang, Philip E. Par\'{e}, Shreyas SundaramAbstract
State-of-the-art neural network verifiers evaluate linear relaxations over axis-aligned bounding boxes. However, verification domains in cyber-physical systems are rarely boxes. Reachability specifications, obstacle-avoidance constraints, and neural Lyapunov certificates typically yield oblique or non-convex polyhedral regions, over which box-based bounds become extremely loose. This paper proposes Fit-LRP (Fitting Linear Relaxations to Polytopes), a framework the implements exact optimization of affine bounds over non-convex domains (represented by unions of polytopes) by solving a sequence of small, low-dimensional linear programs. Our approach seamlessly enhances the final optimization step of LiRPA and $\alpha,\! \beta$-CROWN branch-and-bound (BaB) algorithms. In BaB verification, approximating oblique constraints with axis-aligned splits causes severe combinatorial blowup and risks non-termination. By optimizing the final affine bound of a nonlinear function over the exact polyhedral state-space constraints our method eliminates box-induced over-approximations, enabling earlier search-tree pruning. On ARCH-COMP and neural Lyapunov benchmarks, Fit-LRP reduces branching on the hardest region by $47\times$, cuts worst-case verification time by $15\times$, and with region-wise partitioning, verifies on a single CPU $6.1\times$ faster than $\alpha,\! \beta$-CROWN on a GPU.
Abstract (translated)
URL
https://arxiv.org/abs/2611.16003
PDF
-
What Roles Do System Prompts Really Assign? Validating and Steering Occupational Role Vectors in LLMs
2026-11-11 20:43:45Jose L. Sampedro Mazon, Ilmi Yoon, Nadim SaadAbstract
Applications built on large language models assign the model a role through its system prompt, intending to shape how it responds for their use case. We introduce a validation framework for extracting and injecting occupational role vectors, activation-level directions that capture how someone in a given occupational role would distinctively respond to a task or question. We validate these vectors through two complementary lenses, on Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct. The interpretability lens asks whether each vector is a stable, separable, domain-consistent representation of its role: extracted vectors are highly reproducible across repeated runs and sampling temperatures, remain separable even for very similar roles, and agree more with themselves across in-domain and out-of-domain questions than with other roles' signal. The control lens asks whether injecting a role vector during generation makes the model express the role, and sustain it over a conversation, as well as a role prompt does. On Qwen, injecting the vector alone makes the model express the intended role substantially more often than unsteered, but at best matches a role prompt; on both models, adding the vector to the prompt strengthens the role in single answers but did not slow the role's decline over 30 turns. Together, the two lenses show why both are needed: out of domain the role is expressed only subtly, yet the activations still encode it; and a vector that is stable and separable can still barely steer the model's behavior.
Abstract (translated)
URL
https://arxiv.org/abs/2611.15989
PDF
-
DuoJEPA: Cross-Modal Translation Between Unaligned Self-Supervised Embedding Spaces
2026-11-11 20:10:52Mahmut S. Gokmen, Mitchell A. Klusty, Evan W. Damron, Caroline N. Leach, Samuel E. Armstrong, V. K. Cody BumgardnerAbstract
Learning from paired modalities is commonly equated with aligning them: two encoders are trained until the embeddings of a pair coincide in a shared space. We study whether cross-modal retrieval and classification can instead be obtained while the two encoders' own embedding spaces remain directly unaligned. DuoJEPA trains one encoder per modality with the self-supervised LeJEPA objective and appends to each encoder a single transformer layer that translates its embedding into the other encoder's space, supervised by a contrastive objective computed entirely within that target space. No loss term compares the two own spaces directly, although the encoders are optimised jointly through the translation objective. In our experiments, the translator's residual connections determine where the cross-modal pressure is absorbed: with a residual translator the two encoders become directly aligned, whereas with a skipless (residual-free) translator a paired-cosine test detects no alignment in the final models, and at most a weak one in the ablations, at comparable cross-modal performance. The learned class-token translation is well approximated by a single scaled rotation of the source space. We evaluate DuoJEPA on two modality pairs that differ in nearly every respect. On chest radiographs and reports from CheXpert Plus, it attains a zero-shot AUC of 86.6 and a cross-modal retrieval score of 78.5, compared with 60.0 and 64.2 for our reimplementation of LeVLJEPA with the same encoders and data. On Sentinel-2 optical and Sentinel-1 radar imagery, the frozen encoders fall within the range of published Sentinel foundation models under our reproduction of the Copernicus-Bench linear-probe protocol, the translator retrieves the paired image of the other sensor among 4,000 candidates in 77-79% of cases, and a patch-level extension transfers the within-image similarity structure of the scene.
Abstract (translated)
URL
https://arxiv.org/abs/2611.15968
PDF
-
Spec-Driven PBT: Deriving Semantic Invariants from Code Specifications for Agentic Property-based Testing
2026-11-11 19:16:05Aditya Shukla, Kartik Gupta, Ali Alavi, Pashootan VaezipoorAbstract
Software is increasingly written by AI agents that produce an implementation from a natural language specification, a practice known as agentic coding. Code is now produced faster than it can be reviewed, and has been shown to exhibit its own characteristic failure patterns. Property-based testing (PBT), a long-standing technique in which invariant properties of a program are checked against many generated inputs, has recently been revisited with LLM agents authoring the properties, thereby removing the high-effort derivation of invariants that has long hindered its adoption. Existing methods, however, derive these properties from the implementation alone (Code-driven PBT). At the same time, a thorough specification such as a product requirements document has become routine within agentic coding, yet its role in testing the resulting code remains underexplored. We study the value of the specification as a source of properties for testing agent-written code. Our key insight is that the specification, unlike the implementation, records what the code was meant to do, and invariants derived from it are not biased by particular implementation choices. We propose Spec-Driven PBT, an agent that first expresses invariants in natural language from the specification and only then inspects the implementation to translate them into concrete tests. Across 25 applications from ViBench, Spec-Driven PBT finds a novel set of bugs that Code-driven PBT misses, while writing 2.4x fewer property tests. Though the two approaches surface a roughly equivalent number of bugs, there is little overlap: nearly 60% of the 169 total bugs are found by only one. Qualitatively, we find that the bugs unique to Spec-Driven PBT tend to violate documented product workflows whereas those to Code-driven PBT instead concentrate on implementation robustness, offering complementary value for bug discovery in agent-written code.
Abstract (translated)
URL
https://arxiv.org/abs/2611.15920
PDF
-
HeriKGQA: Grounded Knowledge-Graph Question Answering over Cultural Heritage Sensor Data
2026-11-11 19:09:13Lingxiao Kong, Vasileios Arampatzakis, Vasileios Sevetlidis, George Pavlidis, Zeyd BoukhersAbstract
Cultural heritage sites are increasingly monitored by sensor networks, but querying their data requires database expertise. Large Language Models (LLMs) offer a natural-language interface, yet over raw sensor records they lose track of which sensor, site, and time each value belongs to, and silently miscompute. We present HeriKGQA, a knowledge-graph question-answering method that confines the LLM to interpreting questions and phrasing answers, leaving computation to SPARQL and deterministic code. Questions are parsed into intents constrained by a heritage ontology aligning W3C SOSA/SSN with CIDOC-CRM, compiled into SPARQL, and answered from query results and temporal, spatial, and anomaly facts derived by a grounding layer, with error correction guarding each LLM step. On 500 questions over real data from five heritage sites, HeriKGQA reaches 99.0% accuracy, 18.2 points above the strongest of three baselines using the same LLM and receiving the same anomaly rules and answer conventions, with comparable answer quality while running almost four times faster. Without grounding, accuracy drops by 12.2 points and on anomaly questions from 99% to 66%. We report lessons from development and live deployment.
Abstract (translated)
URL
https://arxiv.org/abs/2611.15904
PDF
-
Reward-Driven Collider Physics Challenges for Self-Improving Agentic AI via Reinforcement Learning
2026-11-11 19:00:41Alexander Yue, Qibin Liu, Julia GonskiAbstract
Recent advances in agentic AI have demonstrated the potential to automate complex high-energy physics (HEP) workflows. We focus on the regime of HEP tasks where frontier models still struggle to provide correct and complete solutions, and introduce an open-ended optimization task format with continuous, automatically evaluated rewards. Unlike conventional benchmarks focused on task completion, these tasks target reinforcement learning (RL) environments to generate agentic improvement on state- of-the-art HEP challenges. We benchmark frontier reasoning models on several HEP tasks and demonstrate an initial proof of concept in which a small language model improves through RL using automated feedback without human intervention. By connecting challenging HEP problems with verifiable scientific rewards and RL, we aim to advance agentic AI from reproducing established workflows to autonomously discovering better scientific solutions, paving the way toward self-improving scientific agents.
Abstract (translated)
URL
https://arxiv.org/abs/2611.15885
PDF
-
Food Critic: An Agentic Visualization System for Auditable Restaurant-Safety Reasoning
2026-11-11 18:57:07Arleth Salinas, Ishrat Jahan Eliza, Valerio PascucciAbstract
Generative AI agents are increasingly being embedded into visualization systems that can search data, invoke tools, summarize evidence, and produce recommendations. However, when agents rely solely on their own knowledge, the visualization problem is not only whether the agent provides a useful answer, but also whether users can inspect how that answer is grounded. We present Food Critic, an agentic visualization system for diners' restaurant decision support that integrates public health inspection records with crowd review data. The system grounds agents under three conditions: a database-only ReAct agent, a retrieval- augmented agent with access to food-safety regulations, and an ontology-guided agent that produces structured hypotheses and typed evidence. Food Critic externalizes the agent's reasoning within these groundings as inspectable visual evidence. We evaluate LLM models with several use cases under each grounding of our agentic system. Our work positions restaurant choice as a concrete case for studying human-agent collaboration, visual provenance of agent reasoning, and the design of visual interfaces for grounded GenAI decision support.
Abstract (translated)
URL
https://arxiv.org/abs/2611.15843
PDF
-
Before AI Agents Negotiate for Us: A Competitive Tournament of Rule-Based, Search, Optimization, Learned, and Language-Model Negotiators Under Scarcity
2026-11-11 18:48:20Eduardo de Avila-Armenta, Ra\'ul Alejandro Aquino-Ibarra, Jorge Octavio Nicol\'as-D\'iaz, Umberto Leon-DominguezAbstract
Automated bilateral negotiation underlies two-sided matching markets such as freight brokering, where many shippers and carriers negotiate price before a matching mechanism decides who is paired with whom. Prior strategy comparisons typically use isolated one-on-one duels or a single fixed market, so they cannot show how a strategy behaves against several live rivals under a changing balance of supply and demand. We built nine negotiation architectures spanning rule-based, search-based, optimization-based, imitation-learning, and reinforcement-learning families, and ran all nine simultaneously on both sides of a Gale-Shapley matching market under a balanced condition and two scarcity conditions rotated across all 36 exclusion pairs. In the balanced carrier market, an imitation-learned engine (ML2), a reinforcement-learning policy trained only on isolated negotiations (RL), and an expected-utility optimization engine (OPT) formed a statistically tied top group, and no architecture led in every condition and role. RL beat all eight carrier rivals when carriers were scarce, but its match rate collapsed from 62% to 16% when shippers were scarce and it had to compete with the other carrier agents for demand. A second reinforcement-learning policy trained against rival agents (RL2) kept its match rate above 91% in every condition, yet lost on realized payoff to all eight shipper-side rivals in two of the three conditions, traced to a confirmed train/deployment feature mismatch. We also tested nine large language models (LLMs), from 1 to 72 billion parameters, under the same protocol: they lost significantly to the two strongest engines (OPT2 and ML2) in 32 of 36 comparisons and won none. These results show that isolated or single-condition evaluation can hide failure modes that a many-sided, scarcity-varying, statistically corrected tournament exposes directly.
Abstract (translated)
URL
https://arxiv.org/abs/2611.15838
PDF
-
The BEATS Dataset: Towards Understanding Music-Physiology Dynamics in Real-World Scenarios
2026-11-11 18:47:52Benjamin Gutierrez Serafin, Tanaya Guha, Fahim KawsarAbstract
We introduce the BEATS dataset, a robust compilation of psycho-physiological data collected in the wild from 30 participants while they listened to curated playlists intended to promote well-being. An acronym for Biomarkers, Experience and Affect from Therapeutic Soundscapes, the BEATS dataset aims to support research on wellbeing, music psychology, affective computing and signal processing. More specifically, this dataset includes: (1) sensor data collected during music listening sessions using a research-grade wearable device in free-movement environments; (2) music listening activity records, stimulus metadata, acoustic features, and self-reports associated with the listening sessions (3) individual characteristics, such as demographics, music preference, musical skills, and personality traits; and (4) self-reported psychological measures before and after their experimental participation spanning for a number of weeks. By situating data collection outside of lab settings, the BEATS dataset offers the opportunity to study real-world music-listening scenarios, allowing for a richer exploration of the physiological, subjective and therapeutic effects of music under real-world conditions.
Abstract (translated)
URL
https://arxiv.org/abs/2611.15837
PDF
-
Controlling the Probability of Unknown Unknowns in Pre-Deployment Testing of AI Systems
2026-11-11 18:46:36Rachel Li, Chris Hays, Aaditya Ramdas, Manish RaghavanAbstract
In pre-deployment testing of AI systems such as AI agents, a developer can often deal with known failure modes (e.g., leaking API keys, deleting important databases, or installing a malicious package) by implementing fixes or making contingency plans for when the same issue might occur again. What is more challenging are the unknown unknowns: the unseen failure modes that haven't surfaced during testing. The decision to stop testing and deploy therefore faces an inherent tradeoff: continue testing to observe more behaviors, at the cost of time and resources, or stop testing and deploy, at the risk of failure modes remaining undiscovered. How then does the developer decide when they have tested enough? To answer this, we develop an algorithm that certifies when the probability of observing an unseen failure mode is sufficiently small. The algorithm constructs an anytime-valid upper confidence sequence on this quantity, so the developer can repeatedly test a system, inspect the bound, and decide to stop when it falls below a chosen threshold without invalidating the statistical guarantee. Our problem can be interpreted as a sequential variant of the missing-mass estimation problem which gave rise to the classic Good-Turing estimator (Good, 1953). On SWE-Chat, a dataset of real-world coding agent traces (Baumann et al., 2026), we implement our procedure and show that it achieves tighter upper bounds than a simple multiple-testing corrected Good-Turing baseline, and even dominates a fixed-horizon (non-multiple-testing-corrected) Good-Turing upper confidence bound, which is not anytime valid. Additionally, to better match realistic evaluation pipelines, we extend our method to handle sampling from a different task distribution and imperfect automated failure labeling.
Abstract (translated)
URL
https://arxiv.org/abs/2611.15835
PDF
-
NIBench: benchmarking autonomous AI agents for execution, skill evolution and scientific judgment in neuroimaging
2026-11-11 18:44:56Lujia Zhong, Yihao Xia, Jianwei Zhang, Shuo Huang, Xinyu Nie, Mingyang Xia, Zhiwei Deng, Jiaxin Yue, Xinkai Wang, Junhan Zhao, Yonggang ShiAbstract
Autonomous AI agents are increasingly used to automate biomedical research, creating a growing need to benchmark their capabilities on real scientific data and workflows. Their reliability in neuroimaging remains insufficiently characterized, where plausible analytical outputs can conceal errors that affect scientific conclusions. Existing studies have advanced scientific question answering and tool-based automation, but provide limited evidence on workflow reconstruction, acquisition of procedural expertise and evidence-based scientific judgment. Here we present NIBench, a neuroimaging benchmark comprising three complementary evaluations of autonomous agents. First, we benchmark execution and workflow reconstruction using 56 tasks across seven tool ecosystems, covering structural, diffusion and functional magnetic resonance imaging, positron emission tomography and multimodal analyses. Second, we examine whether agents can evolve their procedural skills through distillation of execution trajectories, evaluating the resulting skills on previously encountered tasks, unseen tasks and other models. Third, we assess scientific judgment using 36 claims from published Alzheimer's disease studies, requiring agents to analyse raw cohort data and provide reproducible evidence. We evaluate 19 language models within a shared agent framework, with selected models undergoing skill distillation and claim evaluation. Leading agents execute supplied analysis protocols nearly perfectly, whereas withholding procedural guidance widens performance differences. Distilled skills improve performance on acquisition tasks and transfer between models, but benefits on unseen tasks are inconsistent. Agents reproduce their reported results yet sometimes reach conclusions inconsistent with expert-derived answers, and access to data does not always improve judgment.
Abstract (translated)
URL
https://arxiv.org/abs/2611.15831
PDF
-
Characterizing Scanpath Decoding Pipelines with Perturbation Profiles
Abstract
When a scanpath descriptor supports a classification, it is rarely clear what the classification rests on. We propose perturbation profiles. Five transformations each target one stated property of every scanpath (fixation order, the coupling of durations to positions, the duration distribution, orientation and handedness, and fine spatial structure) while preserving a listed set of invariants. The classifier is refit, and the change in participant-held-out AUC is the pipeline's sensitivity to that property. Intervals come from a participant bootstrap that refits the classifier; in simulations they were conservative (99.7% coverage; standard errors about 1.8 times the empirical spread). We profiled nine descriptors on thirteen binary problems from three public datasets, with a within-fold AUC and class-balanced folds and a stated rule for which sensitivities to interpret. Of 565 intervals, 95 excluded zero; 80 were robust to the pooled AUC, 20 fold partitions and longer runs (36 of them within +/-0.05 AUC), and 45 also held with boosted trees. Most came from GazeBase (322 participants). Permuting fixation order raised the AUC of a descriptor based on mean saccade amplitude, because permuted amplitude has as its expectation the mean pairwise distance, a measure of spread. A "comprehension" contrast pooled over two programs was a contrast between their layouts: it vanished among trials of the same program, and synthetic gaze drawn from the layout alone reproduced the program contrast beneath it. For native versus non-native readers, spatial jitter at the scale of text lines reduced recurrence's discrimination. A Python package is released.
Abstract (translated)
URL
https://arxiv.org/abs/2611.11632
PDF
