The paper’s figures and tables
The Continual-ARC paper leads with Figure 1, every learner on the open models on its two frozen tests, one of the tasks it met and one of 100 it never met, read here beside the same learners in absolute terms. Most of the rest of its main text follows, and after it, from its appendices, the views for further introspection: the benchmark’s details, what the harnesses kept, and GPT-6 Luna’s 228 runs, on which no harness left to itself saves a measurable share of what it leaves to learn and a deliberately simple program library saves 0.49 to 0.62 of its own. Each element carries the paper’s number. Point at a chart to read it, switch effort where the paper has panels, turn learners off and on in the legend, and open the run behind any number in the replay.
Two kinds of learning
An agent can get better with experience in two ways. It can remember the rule of a task it met before, which the paper calls memorization, or it can get better at tasks it has never met, which it calls generalization. After the stream the learner is frozen and plays a test of each, and Figure 1 puts every learner on both.
Every learner on the two frozen tests
The two tests in absolute terms
The benchmark
A learner meets a long stream of ARC instances. The same tasks recur after dormant gaps, each time with a fresh input, while new tasks keep arriving, and every instance is charged for the feedback it uses. Then come the two tests, each a single round of 100 instances played from where the stream left the learner.
One instance
The stream
Learners
The paper asks where learning should go: into the notes an agent harness keeps, or into the weights of the model that drives it. Either can be taught by the model itself or by a stronger one, so every learner is one student, Qwen3.5-9B, placed by those two choices, with GLM-5.3 Flash as the teacher.
Table 3, in the benchmark’s details below, places each row, and the post describes each.
Results
Table 1 gives each row that keeps something at one user on the open models, on the stream and on the two tests, and the table here adds the resets the rows are read against.
Every row on Qwen3.5-9B and GLM-5.3 Flash
| row | Qwen3.5-9B | GLM-5.3 Flash | ||||
|---|---|---|---|---|---|---|
| L | seen | held-out | L | seen | held-out | |
| notes in the harness | ||||||
| H | −0.02 −0.09 to +0.04 | −0.01 −0.04 to +0.03 | −0.01 −0.05 to +0.03 | +0.12 −0.16 to +0.36 | +0.12 −0.03 to +0.27 | +0.00 −0.11 to +0.10 |
| H-E | +0.05 −0.06 to +0.16 | +0.03 −0.08 to +0.13 | −0.01 −0.05 to +0.03 | +0.62 +0.49 to +0.75 | +0.74 +0.64 to +0.84 | +0.07 −0.02 to +0.16 |
| notes the teacher writes | ||||||
| H-T | +0.05 −0.02 to +0.14 | +0.06 −0.01 to +0.14 | +0.03 −0.02 to +0.07 | – | – | – |
| H-T-E | +0.07 −0.03 to +0.16 | +0.01 −0.05 to +0.07 | −0.02 −0.07 to +0.03 | – | – | – |
| H-E-T-E | +0.29 +0.19 to +0.40 | +0.34 +0.26 to +0.42 | −0.05 −0.10 to −0.00 | – | – | – |
| weights | ||||||
| DAgger | +0.43 +0.33 to +0.54 | +0.57 +0.48 to +0.67 | +0.29 +0.23 to +0.35 | – | – | – |
| baselines | ||||||
| program library | +0.43 +0.25 to +0.64 | +0.65 +0.46 to +0.86 | +0.03 −0.02 to +0.09 | +0.68 +0.49 to +0.87 | +0.81 +0.59 to +1.00 | +0.05 −0.06 to +0.14 |
| stateless floor | – | +0.06 −0.03 to +0.15 | −0.02 −0.10 to +0.05 | – | −0.06 −0.16 to +0.04 | +0.03 −0.10 to +0.14 |
| demo-cache floor | +0.31 +0.19 to +0.45 | +0.37 +0.26 to +0.51 | −0.01 −0.05 to +0.03 | +0.41 +0.29 to +0.53 | +0.44 +0.34 to +0.57 | −0.02 −0.11 to +0.06 |
| H0 | – | +0.00 −0.05 to +0.07 | +0.03 −0.03 to +0.08 | – | −0.05 −0.23 to +0.11 | +0.13 −0.01 to +0.26 |
| H-E0 | – | −0.06 −0.14 to +0.01 | +0.00 −0.05 to +0.06 | – | −0.06 −0.24 to +0.07 | −0.01 −0.15 to +0.12 |
| PL0 | – | +0.03 −0.07 to +0.12 | +0.01 −0.06 to +0.07 | – | +0.00 −0.16 to +0.13 | −0.05 −0.22 to +0.09 |
Further introspection
Everything above reads a learner after the stream, on the two tests. What follows is from the paper’s appendices: the benchmark’s details, how the harnesses ran and what each kept, and the stream itself on GPT-6 Luna, whose runs cover three harnesses at three reasoning efforts, with the share learned while each played. These views help explain a place on the plane rather than rank anyone.
Benchmark details
Appendix A of the paper: the words, the comparison, the models, the configurations and the tasks.
The terms
The comparison on Qwen3.5-9B
The models
An input grid does not say which task it belongs to
feca6190B 7fe24cdd91413438B 007bbfb7c3e719e8B ac0a08a4The ARC-AGI-1 track's configuration
The tasks the track serves
456873bc60a26a3e7318201294133066ce8d95cc3d31c5b362ab2642e760a62e929ab4e9759f3fd39f27f097e41c6fd39bebae7af8be4b642037f2c7df8cc3779110e3c509c534e7c48954c125094a639a4bb2263a301edc0934a4d89def23fe2546ccf6e345f17bd5c634a20d87d2a694414823cf133acc1e97544eaa18de87817e6c092a5f82179c1e755f770cc55f2970060767636eace0fb7511e9bb695417b80ad24c1777185a5a2103281123b4e66aafb885b81ff1bf89d739e633a9e5f21745ecda2b0fe394be5b805289ad53a57f2f04103eff5b03560426b7fb29bca3f840883f23242bac605cbb72a961c973c3b0d855783887f9a67cb54acc710793b4f4b3c1990cce626c0bcc73ccf9c24852f2fab7f8a4d8992798f63194b014a04b26028dae5dfc47996f1185fa5666d931c21c67b4a34d6ea4a07e4aab40072c737e39423a55dc5af49b42903d1b4aca8de6eaac2e8ecfbbb1b8b60692e18c5ffb2104506d28a554db823b8597cfd70b17323b6ad5bdfd4e469f39e5c44e8f00dbd492136b0064319f2597ac3e2b04Running the harnesses
Appendix D: each harness runs as a black box in a sandbox, with a fresh conversation for every instance and nothing carried over but what it wrote itself.
How the adapter drives a harness
What the harnesses kept
Appendix E reads everything each harness stored at the end of the stream in its three runs as shipped, one at each effort. Left to decide, none stored what would answer a returning task. The post takes the harnesses one at a time.
What each learner is built to keep
What memory changes in how a harness spends
| pairs bought | wrong answers | solved (%) | budget used (points) | ||||
|---|---|---|---|---|---|---|---|
| harness | effort | first | returns | returns | returns | first | returns |
| Prime Agent | low | +0.64 | −0.04 | −0.25 | +2.9 | +3.5 ±9.1 | −2.0 ±2.2 |
| medium | +0.12 | +0.07 | +0.04 | +0.6 | +2.3 ±4.3 | +0.7 ±1.3 | |
| high | +0.32 | +0.09 | −0.02 | +1.1 | +4.8 ±4.1 | +0.4 ±1.4 | |
| Hermes Agent | low | +0.36 | +0.72 | −0.12 | +2.3 | +2.3 ±7.2 | +1.8 ±2.7 |
| medium | −0.44 | +0.02 | −0.05 | −0.6 | −2.8 ±4.8 | +0.5 ±1.5 | |
| high | +0.16 | +0.29 | +0.02 | +0.6 | +2.8 ±3.8 | +1.4 ±1.7 | |
| OpenClaw | low | +0.20 | +0.57 | −0.26 | +2.3 | −3.3 ±8.7 | −1.1 ±2.7 |
| medium | +0.48 | −0.01 | +0.39 | −4.0 | +3.0 ±5.2 | +3.8 ±2.4 | |
| high | +0.52 | −0.08 | −0.03 | −0.6 | +4.3 ±3.6 | −0.8 ±1.3 | |
What each harness stored, and what it saved
Results on GPT-6 Luna
Appendix I: to check whether what the harnesses do on the open models holds on a stronger one, GPT-6 Luna at each reasoning effort drives the two floors, the program library, and each harness as shipped and with everything it wrote wiped before every instance. Every learner meets the same instances in the same order, and each is scored by the share learned: read against itself reset, the same system wiped before every instance, the share of the budget left to learn that it saves. The figures after it are the site’s own, drawn from the runs’ records beside that table: the share learned by effort, and the stream read by the number of times a task had been met before.
After its stream a run went on to some or all of the 100 held-out tasks while it could still learn, which the frozen tests replace, so nothing from those tasks is drawn here.
Share learned on GPT-6 Luna
| GPT-6 Luna | ARC-AGI-1 | ||
|---|---|---|---|
| effort | low | medium | high |
| learnable budget, model alone (points) | 50.8 | 32.5 | 26.5 |
| Stateless floor | 0 | 0 | 0 |
| Demo-cache floor | +0.19 ±0.05 | +0.24 ±0.08 | +0.28 ±0.08 |
| Hermes Agent, memory wiped | 0 | 0 | 0 |
| Hermes Agent, as shipped | −0.05 ±0.06 | 0.00 ±0.06 | −0.08 ±0.08 |
| OpenClaw, memory wiped | 0 | 0 | 0 |
| OpenClaw, as shipped | +0.04 ±0.06 | −0.15 ±0.10 | +0.01 ±0.06 |
| Prime Agent, memory wiped | 0 | 0 | 0 |
| Prime Agent, as shipped | +0.03 ±0.05 | −0.04 ±0.05 | −0.04 ±0.06 |
| Program library | +0.49 ±0.11 | +0.60 ±0.13 | +0.62 ±0.15 |
Share learned by reasoning effort
| learner | share learned | 95% interval | saved over its reset | |
|---|---|---|---|---|
| Demo-cache floor | +0.28 | +0.11 to +0.44 | +7.3 pts | Replay |
| Program library | +0.62 | +0.37 to +0.93 | +11.3 pts | Replay |
| Prime Agent, as shipped | −0.04 | −0.16 to +0.07 | −0.9 pts | Replay |
| Hermes Agent, as shipped | −0.08 | −0.23 to +0.08 | −1.6 pts | Replay |
| OpenClaw, as shipped | +0.01 | −0.10 to +0.13 | +0.2 pts | Replay |
Budget used by exposure
| learner | budget used | 16th to 84th percentile | instances | |
|---|---|---|---|---|
| Stateless floor | 53.0 | 44.2 to 60.8 | 25 | Replay |
| Demo-cache floor | 42.8 | 33.8 to 51.2 | 25 | Replay |
| Program library | 20.8 | 12.8 to 28.7 | 25 | Replay |
| Prime Agent, as shipped | 50.0 | 42.2 to 58.0 | 25 | Replay |
| Prime Agent, memory wiped | 50.0 | 43.0 to 58.2 | 25 | Replay |
| Hermes Agent, as shipped | 43.8 | 36.2 to 50.7 | 25 | Replay |
| Hermes Agent, memory wiped | 46.2 | 38.2 to 54.0 | 25 | Replay |
| OpenClaw, as shipped | 47.8 | 38.8 to 55.8 | 25 | Replay |
| OpenClaw, memory wiped | 50.7 | 42.2 to 58.8 | 25 | Replay |
Budget saved against the demo-cache floor, by exposure
| learner | saved | 16th to 84th percentile | |
|---|---|---|---|
| Stateless floor | −10.2 | −14.0 to −5.8 | Replay |
| Program library | +22.0 | +14.5 to +28.8 | Replay |
| Prime Agent, as shipped | −7.2 | −13.5 to −1.0 | Replay |
| Prime Agent, memory wiped | −7.2 | −15.8 to −0.5 | Replay |
| Hermes Agent, as shipped | −1.0 | −7.5 to +5.5 | Replay |
| Hermes Agent, memory wiped | −3.5 | −11.5 to +3.5 | Replay |
| OpenClaw, as shipped | −5.0 | −11.5 to +1.5 | Replay |
| OpenClaw, memory wiped | −8.0 | −13.8 to −2.8 | Replay |
Solved by exposure
| learner | solved | 16th to 84th percentile | instances | |
|---|---|---|---|---|
| Stateless floor | 60.0 | 52.0 to 72.0 | 25 | Replay |
| Demo-cache floor | 68.0 | 60.0 to 76.0 | 25 | Replay |
| Program library | 80.0 | 72.0 to 88.0 | 25 | Replay |
| Prime Agent, as shipped | 68.0 | 56.0 to 76.0 | 25 | Replay |
| Prime Agent, memory wiped | 72.0 | 64.0 to 80.0 | 25 | Replay |
| Hermes Agent, as shipped | 72.0 | 64.0 to 80.0 | 25 | Replay |
| Hermes Agent, memory wiped | 72.0 | 64.0 to 84.0 | 25 | Replay |
| OpenClaw, as shipped | 68.0 | 60.0 to 76.0 | 25 | Replay |
| OpenClaw, memory wiped | 64.0 | 52.0 to 76.0 | 25 | Replay |
The second and third tracks
Appendix K: ARC-AGI-2 and ARC-AGI-3 under the first track’s protocol, each checked end to end with four learners on Kimi K3.1 Pro. The Overview says what each track serves and quotes the paper’s Table 21, how often three models solve each one at first sight.
Four learners on each new track
| stream | frozen tests | ||||
|---|---|---|---|---|---|
| track | row | solved | L | seen | held-out |
| ARC-AGI-2 | stateless floor | 61.0% | 0 | −0.01 −0.13 to +0.10 | −0.01 −0.05 to +0.03 |
| demo-cache floor | 71.0% | +0.35 +0.22 to +0.48 | +0.37 +0.26 to +0.48 | +0.01 −0.02 to +0.04 | |
| Prime Agent, as shipped | 64.5% | 0.00 −0.09 to +0.08 | +0.01 −0.06 to +0.09 | −0.02 −0.04 to +0.01 | |
| Prime Agent, wiped | 65.0% | 0 | −0.04 −0.21 to +0.09 | +0.01 −0.03 to +0.06 | |
| ARC-AGI-3 | stateless floor | 25.8% | 0 | −0.06 −0.12 to 0.00 | +0.01 −0.05 to +0.06 |
| demo-cache floor | 32.5% | +0.11 +0.01 to +0.23 | +0.13 +0.04 to +0.24 | −0.02 −0.10 to +0.04 | |
| Prime Agent, as shipped | 60.0% | +0.04 −0.07 to +0.14 | +0.03 −0.05 to +0.09 | −0.04 −0.11 to +0.02 | |
| Prime Agent, wiped | 52.5% | 0 | +0.01 −0.04 to +0.06 | +0.06 −0.03 to +0.17 | |
arc2 and arc3 at seed 0. Solved is the share of the stream’s instances solved. L is the share learned on the stream, and the seen and held-out columns the share of the reset’s spend on each frozen test that the row no longer pays, read as in Table 1: the demo-cache floor against the stateless floor and Prime Agent against itself wiped, from four plays of each test by the row and four by its reset. A row that is its own reset learns 0. Higher is better for all three, and small type is the 95 percent interval over task resamples.In the paper and not drawn here: the open models’ share solved and budget used by visit over the stream and on each test (Figure 4); every row on Qwen3.5-9B that keeps notes in the harness and every row on GLM-5.3 Flash, row H-A among them (Tables 11 and 12); how often DAgger’s phases changed its adapter (Figure 7); DAgger at more users (Figure 8, Table 17); every row in absolute terms as a table (Table 16), drawn above as a chart beside Figure 1; the budget a solved instance used (Table 15), whose curves the blog post draws; GPT-6 Luna’s share solved and budget used by visit at each effort, and its budget on the instances it solves (Figures 9 to 12); GPT-6 Luna’s frozen tests (Table 19), which the leaderboard draws; the ablations (Table 13); the stream process written out (Algorithm 1); where the code keeps every other text a learner is shown (Table 6); the software behind the harness rows (Table 7); the comparison with related benchmarks (Table 14); the first-sight solve rates on the private held-out season (Table 20); and those on the second and third tracks (Table 21), which the Overview quotes.