Redwood Research

[Also posted on the METR blog]

Automatically finding monitor failures by editing real agent transcripts

Empirical work on the Distillation Double Bind.

Asked which decision theory is right, frontier models name FDT/UDT. But add a sentence saying you're an academic philosopher, and they often switch to CDT.

A model of why capabilities research usually increases risk despite enabling safety—and when it can reduce risk

When monitor evasion looks like legitimate learning to your continual learning system, it’s hard to have one without the other

We should have a strong presumption that latent reasoning architectures would make oversight far more difficult.

Unlike other models we tested, GPT-6 Astra performs modestly better on general benchmarks with filler tokens, and significantly better on serial depth-heavy tasks.

Simple prompt optimizations can improve model capability to control their reasoning

"Serial depth between text bottlenecks" as a proxy for latent reasoning abilities.

Architectures that incorporate opaque recurrence or allow agents to communicate using latents could rapidly make it much harder to monitor chains of thought. We propose that AI companies regularly report verified information about opaque serial depth, share monitorability evidence, and publish a policy on architectures that could degrade monitorability.

We recently published the report from our brief independent investigation into this incident.

Unsanctioned coordination, like we saw in the Hugging Face incident, could enable future AIs to take over

Anthropic concluded in the April Mythos Preview alignment risk update that the model “does not possess any unknown propensities that would increase alignment risk.” The report argues that if Mythos Preview were coherently misaligned[1][2], it likely would have been detected by the assessment (following Anthropic, I will call this “reliability of the assessment”

When a misaligned AI can only output tiny amounts of information, it may find sabotage very difficult

We need more details

And what the incident can’t tell us about alignment

What are the broader lessons from this incident?

Yes, but less than had they been schemers.

We recently ran a strategy fellowship through Astra. As part of this, we ran a reading group for our fellows on some of the topics that we think are important for thinking about AI futurism (key dynamics in AI development, existential risk from AI, and approaches to mitigating risk). This post contains the reading list we used.