Agents in crisis: drift, attacks and the problem of trusting AI

Today’s most important AI news

Today’s research converges on an uncomfortable but necessary message: AI agents already in production behave in ways nobody designed and that few organizations are monitoring. It’s not a day of spectacular launches, but of warning signs that technical and product teams should take seriously before scaling their systems.

Agents in production forget the rules — and nobody notices

Nautilus Compass, a paper published today on arXiv, documents a problem any developer who has worked with long-running agents will have sensed: models “drift” during extended sessions. They forget constraints the user specified at the start, repeat mistakes that were already corrected, and in some cases “confabulate” prior agreements that never happened. The authors call it persona drift, and it’s especially hard to detect because it happens gradually, not all at once.

What makes this work relevant for teams in production is the approach it proposes: a black-box solution, meaning it doesn’t require access to the model’s weights. This matters because most organizations work with closed APIs — Claude, GPT-4 — and don’t have the option of inspecting the system’s internal state. Nautilus Compass acts as a memory and detection layer that runs on top of the model, without depending on its internal architecture.

The practical implication is direct: if your team has coding, support or automation agents that operate in long sessions, you can’t assume behavior on turn 1 holds on turn 50. You need external verification mechanisms, and this paper offers a concrete architecture for building them.

Swarms of cheap agents can bypass the defenses of the safest models

A paper with a deceptively technical title — “Position: AI Security Policy Should Target Systems, Not Models” — contains one of the most provocative claims of the week: coordinated attacks by multiple lightweight agents can achieve, at effectively zero cost, what individual attacks can’t against frontier models. The researchers present swarm-attack, an open-source framework where several small LLM agents coordinate through shared memory, parallel exploration and evolutionary optimization to find safety bypasses.

The paper’s central argument goes beyond the attack itself: AI safety policies that focus on restricting specific models are aiming at the wrong target. When dangerous capabilities emerge from coordination between cheap, open agents, regulating the most powerful model on the market doesn’t solve the problem. It’s the system, not the component, that is the real risk vector.

For product managers and security teams, this changes the mental model. The question is no longer just “what can this model do?” but “what can a network of mediocre models working together do?”. Any organization building multi-agent systems — and today there are many — should consider this vector in its threat modeling.

Hallucinations in RAG are more frequent than benchmarks suggest — and humans aren’t the gold standard either

A study comparing the original annotations of two classic hallucination-detection benchmarks (QAGS-C and SummEval) against predictions from Gemini 2.5 Flash and GPT-5 Mini reaches a conclusion that’s uncomfortable in both directions: modern models detect hallucinations the original human annotators missed, but current benchmarks systematically underestimate LLMs’ real performance because they were built with inconsistent human criteria.

The concrete problem is that the humans who annotated those datasets made their own mistakes — they missed cases of genuine hallucination, and marked as wrong fragments that were correct. When a 2025 LLM is evaluated against those “gold” annotations, its score suffers from the evaluator’s errors, not its own failures. The authors propose a hybrid scheme: LLM-first with human adjudication only in cases where the model expresses low confidence.

For teams building RAG pipelines — which by now are the majority of enterprise AI projects — there are two simultaneous readings: the good news is that their systems probably hallucinate less than internal evals indicate. The bad news is that they can’t blindly trust those evals to make deployment decisions either. Reviewing the evaluation methodology is as urgent as improving the model.

In a nutshell

The pattern emerging today isn’t about new capabilities but about the fragility of systems that already exist. Agents drift, swarms attack from flanks nobody is watching, and the metrics we use to measure quality are built on sand. The industry has spent two years accelerating the deployment of agents in production, but the trust infrastructure — monitoring, evaluation, systemic defense — is running several steps behind. AI’s next big problem won’t be announced with a DeepMind paper; it will appear silently in the logs of a system nobody has checked in weeks.

---

Sources used: Nautilus Compass – Black-box Persona Drift Detection (https://arxiv.org/abs/2605.09863), AI Security Policy Should Target Systems, Not Models (https://arxiv.org/abs/2605.09504), Do Benchmarks Underestimate LLM Performance? (https://arxiv.org/abs/2605.08462)