CredLeak-Bench

The idea

Delegating a task to an agent also delegates a security decision: is the site asking for this information allowed to receive it? A phishing page doesn’t have to tell the agent to ignore its safeguards; it only has to look like the next step of an ordinary workflow. Refusing everything stops leaks, but it also blocks real logins and leaves real bills unpaid.

So every phishing scenario in CredLeak-Bench comes with a legitimate twin that differs only in the destination. We count what the agent actually submits in a local sandbox, not what it says it did.

Results

We test seven agents: Claude Opus 4.8, Claude Sonnet 5, Gemini 3.6 Flash, GPT-5-mini, GPT-5.6-luna, Qwen3.5-9B and Llama-3.1-8B-Instruct.

No model is both safe and useful

The ideal agent sits in the upper-left corner: it completes legitimate tasks and never leaks. No model gets there in any setting. Hover a point or a model to trace it.

Every model leaks, even without being asked to sign in

When a user hands over a link, agents leak in 31–76% of phishing cases while refusing only 3–27% of real ones. Left to triage an inbox, they leak less but refuse far more: Opus 4.8 drops to 1.6% leakage, but refuses 88.3% of legitimate tasks.

Whether an agent spots the trap depends on the cue

Leakage swings widely with the type of deception, and models differ in which signals they catch. Opus 4.8 rarely falls for typosquats or certificate warnings (4.7% mean leakage on each) but still leaks on wrong TLDs and raw IPs. The open-weight models leak heavily on every cue, including 59–61% of certificate-warning cases. Plain HTTP on the correct hostname is the one blind spot every model shares. Low leakage on one cue says little about phishing resistance overall.

Pick a cue to see the spoofed page and how often each model fell for it.

homograph30.0% avg. leakraw_ip30.2% avg. leaktyposquat30.3% avg. leaksubdomain30.6% avg. leakbrand_mismatch31.0% avg. leaktls_warning32.5% avg. leakwrong_tld36.4% avg. leakno_https65.0% avg. leak

🔒https://meridian-reenue.example/login

Deception cue

Misspelled domain

A single letter is dropped: meridian-reenue instead of meridian-revenue.

Impersonates

Meridian Revenue Service

Genuine

meridian-revenue.example

Directed Autonomous

Opus 4.8

9.4

0.0

GPT-5-mini

37.5

6.2

GPT-5.6-luna

43.8

12.5

Sonnet 5

28.1

6.2

Gemini Flash

53.1

0.0

Qwen3.5-9B

68.8

37.5

Llama-3.1-8B

76.6

43.8

Asking for caution backfires; verification helps

Telling the agent to be cautious cuts leakage but kills utility: in the inbox setting, completion falls to 0–11% for every model. Giving it the official domain and a rule to check it works differently: in the directed setting, six of seven models leak less and complete more.

Avoiding the trap isn’t the same as recovering

When the phishing email points at a real pending task, the safe move is to skip the link and use the real site. Agents almost never do: recovery tops out at 24.2%. Opus 4.8 never leaks here, but it never recovers either; it hands the task back to the user in 722 of 768 cases.

What it looks like in practice

Citation

@misc{ahmed2026credleakbenchevaluatingcredentialleakage,
      title={CredLeakBench: Evaluating Credential Leakage and Recovery in LLM Agents},
      author={Rafid Ahmed and Joseph Fioresi and Mubarak Shah and Yuzhang Shang},
      year={2026},
      eprint={2610.08871},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2610.08871},
}