When A.I. Assistants Can’t Tell Whom to Trust

Security researchers are drawing a sharper picture of what can go wrong when artificial intelligence systems are allowed to act like software assistants — reading code, following setup instructions and making decisions about what to trust.

In a series of recent findings, one team showed how a coding agent could be turned into a vehicle for malware from a seemingly harmless GitHub repository. Another experiment found that thousands of attempts to trick an A.I. assistant into leaking a secret failed. And a new academic paper argues that one of the field’s most stubborn vulnerabilities — prompt injection — may persist because large language models do not reliably distinguish trusted instructions from hostile text.

Taken together, the developments suggest a field in transition: some defenses are getting better, especially against simpler forms of prompt manipulation, even as researchers uncover more realistic ways agentic systems can be compromised in practice.

A Clean Repository, a Hidden Payload

The most concrete warning came from Mozilla’s 0DIN security team, which described a repo-based attack path involving Claude Code, Anthropic’s coding assistant.

The demonstration began with a GitHub project that appeared benign. The malicious code was not plainly embedded in the repository, which meant ordinary scans — and the A.I. agent itself — had little obvious reason to flag it. Instead, the harmful payload was retrieved only at runtime, through a DNS TXT record, after the agent followed routine setup steps.

That indirection is what made the attack notable. The repository could look clean. The setup behavior could appear normal. But once the assistant executed the sequence, the machine could be handed over through a reverse shell, effectively giving an attacker control.

The episode underscores a central problem in A.I. agent security: trust is often fragmented. One system inspects the repository, another executes commands, another resolves network requests. No single layer necessarily sees the entire attack chain.

That matters especially for coding agents, which are being marketed as productivity tools that can clone repositories, install dependencies, debug environments and run tests with limited supervision. The more autonomy those systems are given, security researchers say, the more they can inherit the same risks as a human developer — while adding new ones because they may act too quickly and too credulously.

The Limits of “Just Don’t Obey”

At the same time, separate research suggests that the underlying prompt-injection problem is not merely a matter of better warning labels or stricter phrasing.

A paper presented in connection with ICML 2026, “Prompt Injection as Role Confusion,” argues that many failures arise because models do not robustly understand which text is privileged and which is untrusted. In modern chat systems, instructions are often separated into roles such as system, user and assistant. But the researchers found that models can infer those boundaries from style as much as from the formal tags themselves.

In other words, a model may not consistently recognize, in a hard security sense, who is speaking. Instead, it may respond to text that merely looks like internal instructions.

The authors found that small stylistic changes could drastically affect whether an attack worked. “Destyling” malicious text — rewriting it so that it no longer resembled the model’s internal formatting or reasoning style — sharply reduced attack success in their experiments. To human readers, the content was largely the same. To the model, it was not.

That finding points to a deeper concern for the industry. If role separation is not a hard boundary but a learned, probabilistic behavior, then prompt injection may remain difficult to eliminate outright. Defenses may improve, but still operate more like an arms race than a solved security architecture.

Evidence of Progress — and of Remaining Risk

Yet the latest reports do not point in only one direction.

In a public challenge run by Fernando Irarrázaval, more than 2,000 participants made roughly 6,000 attempts to trick an OpenClaw-based A.I. assistant into revealing a planted secret through email-based prompt injection. The assistant, powered by Claude Opus 4.6 and protected by explicit anti-injection rules, did not yield the secret.

The result does not prove such systems are secure. A public contest can miss more patient or sophisticated attackers, and a test built around secret exfiltration does not capture every way an agent can cause harm. But it does suggest that frontier models and layered prompting defenses may be improving against some straightforward attacks.

That aligns with claims from model providers. OpenAI’s latest system card says the company evaluates prompt-injection resilience for connector-based tools and reports near-perfect scores on an internal benchmark, while still describing prompt injection as an unresolved, industrywide problem that requires monitoring, user confirmation and multiple defensive layers.

That caveat is important. A model that resists being talked into revealing a hidden string is not necessarily safe to let install software, modify files or execute commands. The difference between reading a malicious email and running a compromised setup routine is the difference between a persuasive failure and an operational one.

Why This Moment Matters

For years, warnings about A.I. agents have often sounded speculative: what if a chatbot with tools could be manipulated into doing something dangerous? The recent reporting makes the risks less abstract.

The Mozilla demonstration offers a concrete path from ordinary development workflow to full machine compromise. The role-confusion research offers a plausible explanation for why prompt injection keeps resurfacing, even as models get better at refusing obvious attacks. And the failed public hacking challenge shows that progress is real, but uneven.

What emerges is a more mature view of A.I. security. The question is no longer simply whether models can be tricked by cleverly worded prompts. It is whether systems built on top of those models can safely navigate real environments full of hidden dependencies, ambiguous authority and adversarial inputs.

That question is growing more urgent as coding agents move from novelty to infrastructure. Companies are increasingly asking them to inspect repositories, propose fixes, run scripts and interact with external services. Those are precisely the settings in which subtle trust failures matter most.

Researchers say the likely answer will not come from any single fix. Better model training may reduce susceptibility to prompt injection. Harder approval gates may help prevent dangerous tool use. Greater transparency around what an agent is about to run — and where code or instructions are coming from — could make hidden attack chains easier to detect.

But the recent findings suggest that the industry is still some distance from a world in which an A.I. assistant can safely be treated like a fully trustworthy operator. For now, the lesson appears simpler: even when the repository looks clean, the prompt looks harmless and the model looks more robust than before, the risk may only be hiding one layer deeper.

Sources

Further reading and reporting used to add context:

AI News