AI agents in real workflows: the toughest test
Benchmarks on real SaaS, agents that design themselves and multi-agent coordination: what AI can — and can't — do today.
Today’s most important AI news
The day was dominated by a question that recurs with more urgency every week: can AI agents work in real conditions, not lab environments? Three research developments point in the same direction and, together, tell a more nuanced story than headlines usually admit. Agents are progressing, but their failures remain systemic and predictable.
SaaS-Bench: when agents face the software you use every day
For years, AI agent benchmarks have been built on simplified tasks: fictional forms, trimmed-down interfaces, single-step flows. SaaS-Bench changes that. The benchmark presented on arXiv this week puts agents to work directly inside real SaaS applications — the kind of software an operations, sales or support team uses daily — and asks them to complete multi-step professional workflows.
The results are revealing. So-called Computer-Using Agents (CUAs), which combine language models with the ability to interact with browsers and graphical interfaces, perform notably worse as the professional context grows in complexity. It’s not that they fail at individual tasks: they fail at coordinating between steps, at maintaining state throughout a session, and at adapting when an interface doesn’t behave exactly as they expected.
For a product manager or operations leader evaluating whether to adopt agents in their stack, this benchmark has direct implications. The promise of “automate your workflows in tools like Salesforce, Notion or Jira” is still real in bounded cases, but any flow that requires conditional decisions or multiple chained tools still needs human supervision. The gap isn’t in the model’s intelligence: it’s in operational robustness.
AIRA: agents that design the AI models of the future
If agents struggle to navigate a CRM, what happens when the task is designing neural network architectures from scratch? That’s exactly what the AIRA project, also presented this week, investigates. The system uses two complementary frameworks: AIRA-Compose, which searches architectures at a high level by exploring computational primitives with 11 agents in parallel for 24 hours, and AIRA-Design, which implements the mechanical details of the candidate architectures.
What makes AIRA different from earlier Neural Architecture Search (NAS) approaches is its focus on models beyond the standard Transformer. The agents evaluate candidates with millions of parameters, extrapolate the most promising designs and refine them iteratively, all without direct human intervention. The term the authors use is “recursive self-improvement”: AI improving the design of AI itself.
The immediate practical scope is limited — this kind of research takes years to translate into products — but the direction matters. If agents manage to automate parts of architecture design, the bottleneck for creating specialized models will stop being ML research talent and start being compute capacity and clarity of objective. For companies that today depend on generic foundation models, that could open the door to vertical models designed by agents at a fraction of the current cost.
TeamTR: why teams of agents are sometimes worse than a single model
There’s a paradox circulating among teams building multi-agent systems: frequently, a single well-trained model outperforms a team of specialized agents. TeamTR, a CS.LG paper published this week, identifies the exact mechanism behind that phenomenon and proposes a solution.
The problem is called “compounding occupancy shift”. When a team of agents is trained sequentially — that is, one agent is tuned, then another, and so on — each update shifts the distribution of shared context. Subsequent agents are evaluated on data that no longer represents the system’s current state, and the error accumulates. The result is a team that in theory should collaborate better, but in practice generates less reliable answers than a single model.
The solution TeamTR proposes is based on fine-tuning with trust-region constraints applied to the whole team, not to each agent separately. In complex reasoning tests, the approach recovers the lost performance and in some cases beats the single-agent baseline. For teams building multi-agent pipelines today with LangChain, CrewAI or similar frameworks, this work is a concrete technical warning: how you train the system matters as much as the architecture you choose.
In a nutshell
What ties today’s news together isn’t a spectacular launch or a controversy, but something more sober and more useful: AI is being subjected to more honest exams. SaaS-Bench measures agents in the real world and finds cracks. TeamTR dissects why multi-agent systems fail structurally. AIRA pushes the limits of what agents can design autonomously. The pattern is clear: the research community is leaving convenience benchmarks behind and starting to measure what really matters. That’s good for those making adoption decisions, because it means the numbers they’ll read in the coming months will be less optimistic and much more useful.
Sources used: https://arxiv.org/abs/2605.15777, https://arxiv.org/abs/2605.15871, https://arxiv.org/abs/2605.15207