Leaderboard
Every learner after the stream, frozen, on the two tests. Across is the share of its reset's spend it saves on fresh instances of the tasks it met, which is memorization, and up the share it saves on 100 tasks it never met, which is generalization, so the top right is better on both. A result is a learning method on a published student model and, if it has one, a teacher, used unchanged, so each panel is one student, and a method is compared only with others on the same student and teacher. Each learner is read against the same system reset, so a stronger model earns nothing for being stronger, and the absolute view under the plane shows what each started from, since a large share of a small spend is a small gain.
| Learner | Teacher | ||||||
|---|---|---|---|---|---|---|---|
| DAgger, 2 usersimitates the teacher, 2 users sharing one model, each with puzzles of their own, Prime Agent | GLM-5.3 Flash | 0.55 0.46 to 0.64 | 0.40 0.34 to 0.47 | 82% → 37% | 86% → 51% | 18% → 62% | 0.47 0.38 to 0.56 |
| DAgger, 4 usersimitates the teacher, 4 users sharing one model, each with puzzles of their own, Prime Agent | GLM-5.3 Flash | 0.51 0.45 to 0.57 | 0.39 0.33 to 0.46 | 78% → 38% | 86% → 52% | 18% → 63% | 0.53 0.46 to 0.60 |
| DAggerimitates the teacher, Prime Agent | GLM-5.3 Flash | 0.57 0.48 to 0.67 | 0.29 0.23 to 0.35 | 84% → 36% | 85% → 60% | 19% → 49% | 0.43 0.33 to 0.54 |
| told the pageuntrained, told the teacher's page of playing rules, Prime Agent | none | 0.16 0.08 to 0.26 | 0.09 0.05 to 0.14 | 84% → 70% | 85% → 77% | 19% → 28% | – |
| program librarythe program library | none | 0.65 0.46 to 0.86 | 0.03 −0.02 to 0.09 | 65% → 22% | 76% → 73% | 29% → 33% | 0.43 0.25 to 0.64 |
| H-Tthe teacher writes the notes, Prime Agent | GLM-5.3 Flash | 0.06 −0.01 to 0.14 | 0.03 −0.02 to 0.07 | 84% → 79% | 87% → 85% | 17% → 19% | 0.05 −0.02 to 0.14 |
| H-Aasked to save anything that could help, Prime Agent | none | 0.02 −0.02 to 0.07 | −0.00 −0.05 to 0.04 | 84% → 82% | 87% → 87% | 17% → 17% | −0.05 −0.11 to 0.01 |
| Hthe harness as shipped, Prime Agent | none | −0.01 −0.04 to 0.03 | −0.01 −0.05 to 0.03 | 84% → 85% | 87% → 88% | 17% → 16% | −0.02 −0.09 to 0.04 |
| demo-cache floorshows the examples it paid for again | none | 0.37 0.26 to 0.51 | −0.01 −0.05 to 0.03 | 67% → 42% | 73% → 74% | 32% → 33% | 0.31 0.19 to 0.45 |
| H-Egiven our guidance, Prime Agent | none | 0.03 −0.08 to 0.13 | −0.01 −0.05 to 0.03 | 74% → 72% | 80% → 81% | 25% → 23% | 0.05 −0.06 to 0.16 |
| H-T-Ethe teacher writes them, given our guidance, Prime Agent | GLM-5.3 Flash | 0.01 −0.05 to 0.07 | −0.02 −0.07 to 0.03 | 84% → 83% | 87% → 89% | 17% → 15% | 0.07 −0.03 to 0.16 |
| H-E-T-Ethe teacher writes them, both given our guidance, Prime Agent | GLM-5.3 Flash | 0.34 0.26 to 0.42 | −0.05 −0.10 to −0.00 | 74% → 49% | 80% → 83% | 25% → 21% | 0.29 0.19 to 0.40 |
| H0the harness, wiped before every instance, Prime Agent | none | 0.00 −0.05 to 0.07 | 0.03 −0.03 to 0.08 | 84% → 84% | 89% → 86% | 15% → 18% | 0 |
| PL0the program library, wiped | none | 0.03 −0.07 to 0.12 | 0.01 −0.06 to 0.07 | 66% → 64% | 76% → 76% | 28% → 29% | 0 |
| H-E0given our guidance, wiped before every instance, Prime Agent | none | −0.06 −0.14 to 0.01 | 0.00 −0.05 to 0.06 | 72% → 76% | 80% → 80% | 25% → 25% | 0 |
| stateless floorthe model alone, keeping nothing | none | 0.06 −0.03 to 0.15 | −0.02 −0.10 to 0.05 | 69% → 64% | 72% → 74% | 34% → 31% | 0 |
Every learner on this student, sorted by the held-out test until another column is chosen. A share in gray cannot be told apart from zero once its interval is widened for the number of learners on the panel. The rows in light type keep nothing between instances, and each reads half its plays against the other half: how far chance moves a point. Every number is from the paper's own pipeline; Figures has its tables.
Submitted results
Learners from outside unify go here, each placed the same way: the share of its reset's spend it saves on the seen test and on the held-out test, with the student model it plays with and the teacher, if any. A run on the public stream and its two frozen tests, sent by pull request with their records and the learner reset on each, is marked reported. A row is verified once we have rerun its code ourselves on Continual-ARC's private held-out set: season s1, 50 new ARC-style rules that nobody outside unify has seen, each a generator of fresh instances, 25 streamed as arc1 streams its tasks and 25 held out, played at a seed we do not disclose.
The bare model plays the same test instances, and a † beside a share marks a learner whose reset used more of the budget than the bare model did, since a learner that played worse whenever its memory was empty would score for it. A verified row is a result; a reported one is its submitter's claim, since the public tasks, their answers and the stream's solvers could be brought along. Verified rows come first, then the reported, each by its held-out test. Run yours says how to submit a learner and what it may use.
None yet.
Submit yoursFurther introspection: the stream
The plane reads each learner after the stream. These views read the stream itself, for GPT-6 Luna's runs, which cover three harnesses at three reasoning efforts: each learner's share learned while it played, against what its model calls cost per instance, then its budget used. They help explain a place on the plane rather than rank anyone.
Share learned on the stream, against spend
Each point is one learner at one effort on the stream: its share learned there against what its model calls cost per instance, joined from the least effort to the most. Higher and further left is better. A learner that keeps nothing between instances, the stateless floor or a wiped harness, is its own reset and sits at 0 by construction, so this chart leaves it out.
The cost is what it takes to serve an instance. A learner that trains its weights will show what its training cost, in dollars at the rental price of its GPUs, in a column of its own beside it, never added into this axis.
Hover a point for its numbers, and click it to replay the run. A hollow point is a harness with everything it wrote deleted before every instance, in the chart of the budget used below. Click a learner under either chart to hide or show it in both, or Option-click it to show it alone. The intervals switch above draws every point's interval.
Every learner at medium effort, on the stream
| Share learned | Stream | ||||
|---|---|---|---|---|---|
| # | Learner | ||||
| 1 | Program library | +0.60 ±0.13 | 11.5 ±4.0 | 94.5 | $0.005 |
| – | Demo-cache floor | +0.24 ±0.08 | 27.4 ±5.0 | 83.5 | $0.010 |
| – | Stateless floor | 0 | 35.1 ±4.5 | 84.0 | $0.009 |
| – | Prime Agentmemory wiped | 0 | 25.7 ±3.2 | 92.0 | $0.009 |
| 2 | Hermes Agentas shipped | 0.00 ±0.06 | 28.1 ±3.7 | 90.5 | $0.026 |
| – | Hermes Agentmemory wiped | 0 | 28.0 ±4.1 | 91.0 | $0.069 |
| – | OpenClawmemory wiped | 0 | 27.8 ±3.8 | 90.0 | $0.028 |
| 3 | Prime Agentas shipped | −0.04 ±0.05 | 26.6 ±3.5 | 92.5 | $0.010 |
| 4 | OpenClawas shipped | −0.15 ±0.10 | 31.5 ±4.3 | 87.0 | $0.028 |
L is the share learned on the stream: the share of the learnable budget a learner saves over itself reset, the same model in the same harness wiped before every instance, 0 for a learner that keeps nothing and 1 for perfect memory, so a stronger model scores no higher for being stronger. The demo-cache floor is read against the stateless floor, a harness against itself wiped and the program library against its programs wiped. A share in grey cannot be told apart from 0, since its 95 percent interval takes in 0. The floors, the model alone, take no rank, and neither does a learner that keeps nothing, the zero of its own scale. U is the budget used and P the share solved within the cap, both in percent, over the stream's 200 instances. Small type is the standard error over tasks, as in the paper's tables. Click a column to sort by it, and a number to replay its run. After its stream each workstreams run went on to some or all of the 100 held-out tasks while it could still learn, which the two frozen tests replace, so nothing from those tasks is here.
Budget used on the stream, against spend
The same learners on the paper's raw measure: the share of the feedback budget each used on the stream, against what its model calls cost per instance, joined from the least effort to the most. Lower and further left is better.
The budget used rewards the model and the harness as much as the learning, since a stronger model buys fewer demonstrations whether or not it learns. The share learned reads it against the learner itself reset to take that out; this chart shows what it starts from. Its whiskers are one standard error either side.
Harnesses asked to store, or told to keep a program library
These rows change what a harness is told, so the paper reports them apart, in its Appendix F. Asked to store, a harness hears after every verdict that it should save anything that could help with the task's next instances. Told to keep a program library, it carries out the program library's method with its own tools, keeping one tested program per task and running it first when the task returns; wiped, it does the same with everything deleted before every instance. The last row is the program library with nothing kept.
| Share learned | Stream | ||||
|---|---|---|---|---|---|
| # | Learner | ||||
| 1 | Prime Agenttold to keep a program library | +0.47 ±0.09 | 13.0 | 97.5 | $0.011 |
| 2 | OpenClawtold to keep a program library | +0.47 ±0.07 | 14.7 | 95.5 | $0.067 |
| 3 | Hermes Agenttold to keep a program library | +0.44 ±0.10 | 16.9 | 92.5 | $0.061 |
| 4 | Prime Agentasked to store | +0.32 ±0.08 | 18.4 | 95.0 | $0.010 |
| 5 | Hermes Agentasked to store | +0.28 ±0.07 | 21.0 | 90.5 | $0.039 |
| 6 | OpenClawasked to store | +0.03 ±0.08 | 27.2 | 90.0 | $0.043 |
| – | Prime Agenttold to keep a program library, memory wiped | 0 | 22.6 | 95.5 | $0.013 |
| – | Hermes Agenttold to keep a program library, memory wiped | 0 | 28.0 | 88.0 | $0.100 |
| – | OpenClawtold to keep a program library, memory wiped | 0 | 25.5 | 92.0 | $0.12 |
| – | Program library, nothing kept | 0 | 24.6 | 92.5 | $0.008 |
The paper reads these rows against each other, instance by instance, so their budgets carry no error bar here. Each one's share learned is read against itself reset, as every other row's is: a harness asked to store against the same harness wiped, one told to keep a program library against itself told the same and wiped.