RSIAgent’s own Table 1 credits most of its headline benchmark gain to the actor-verifier harness itself, before its autonomous exploration step ever runs. Pairing GLM-5.3 as actor with Kimi-K3 as verifier already adds about twice as many OSWorld partial-credit points, and more than ten times as many Agents’ Last Exam (ALE) points, as the exploration step adds afterward. On ALE, that harness-only score already exceeds the paper’s cited number for GPT-6 Astra; on OSWorld, it falls short until exploration runs and closes the gap.
The harness adds most of the score before exploration ever runs
RSIAgent is training-free: three agent roles — a curriculum agent, an actor and a verifier — explore a new software environment, then freeze what they learned into a reusable memory with no weight updates involved. The paper’s reference harness pairs GLM-5.3 as actor with Kimi-K3 as verifier and curriculum agent, and Table 1 isolates what each ingredient contributes. Moving from a bare Kimi-K3 to that GLM-5.3+Kimi-K3 harness, with exploration and memory switched off, adds 13.67 points on OSWorld’s partial-credit metric, 12.15 on ALE’s partial-credit metric and 8.95 on ALE’s binary metric. Turning exploration and memory on afterward adds 7.01 more OSWorld partial-credit points, 1.07 more on ALE partial credit and 1.50 more on ALE binary: smaller gains than the harness alone already produced, and on ALE binary about a sixth of it.
On ALE, the harness alone already beats GPT-6 Astra
The harness-only ALE partial-credit score, 83.75, exceeds the 82.26 the paper cites for GPT-6 Astra, so the open-weight pair’s lead on ALE mostly exists before any exploration runs. OSWorld is different: the harness alone scores 71.97, still below GPT-6 Astra’s cited 72.60, and only the full pipeline’s 78.98 crosses that line. Exploration is optional for the ALE result and necessary for the OSWorld one.
The comparison itself has a limit the paper states directly: every rival score, GPT-6 Astra’s included, comes from that system’s own leaderboard entry or technical report under what the paper calls an unmatched evaluation protocol, not a re-run by RSIAgent’s authors. And on ALE’s stricter binary metric, which awards credit only for a fully solved task, RSIAgent’s 50.75 still trails GPT-6 Astra’s 52.24; the paper itself says its partial-credit advantage “does not extend to every reported metric.”
Half the benchmark tasks kept their old score, and some of the rest were hand-corrected
The 84.82 ALE headline score rests on a narrower re-evaluation than the average suggests: only 19 of ALE’s 67 tasks received a new score after exploration, and the other 48 simply carried their pre-RSI baseline into the same average. OSWorld’s re-run rate was higher: 41 of its 82 tasks, exactly half, got a new score after exploration. Of the 19 new ALE scores, the paper describes some as local corrected grades, others as qualified by “public-label transfer” — a term that appears nowhere else in the 49-page paper and is never defined — and one as a variant that skipped the broad-exploration stage entirely. Separately, the paper discloses that one task’s baseline and RSI scores were both revised after the fact, without saying whether other tasks in either benchmark underwent similar correction.
A four-task test shows the exploration step still has an effect
A four-task ablation shows the exploration mechanism has a measurable effect on its own: the full two-stage exploration scores 74.54% on average, ahead of 65.52% for a broad-only version and 56.50% for a deep-only version, and deep-only exploration alone can score below the no-exploration baseline on two of the four tasks. That leaves the paper’s central claim, that open-weight models can out-score frontier closed ones through recursive self-improvement, resting mostly on the harness that runs before exploration starts: on ALE, the open-weight pair clears GPT-6 Astra’s number mostly before self-improvement begins running. Code and a project page are public under an Apache-2.0 license, with no independent lab or outside benchmark check of the numbers yet.
Further Reading
- RSIAgent: Autonomous Exploration for Recursive Self-improvement in New Environments — the paper, with Table 1’s harness-versus-exploration breakdown and the appendices detailing task coverage and score corrections.
- AetherLabsAI/RSIAgent — the public code release, Apache-2.0, pushed as recently as 2026-09-15.
- RSIAgent project page — the paper’s companion site.