Run Continual-ARC on your own learner.

Step 01

Install the benchmark

You need uv and Python 3.10 or newer. The last command clones Re-ARC and ARC-AGI at pinned commits into ~/.cache/continual-arc unless CONTINUAL_ARC_DATA says otherwise.

git clone https://github.com/unifyai/continual-arc
cd continual-arc
uv venv && uv pip install -e ".[dev]"
uv run continual-arc data fetch

Step 02

A smoke run

arc1-smoke is a short stream, 128 instances of 16 tasks, and the learner here gives up on every instance, so it needs no model and no key and takes about a second. The runner writes the run directory: records.jsonl, one line per instance, with summary.json, schedule.json, config.json and meta.json.

uv run continual-arc schedule make --config arc1-smoke --seed 0 --out smoke.json
uv run continual-arc run --learner continual_arc.testing:GiveUpLearner \
  --schedule smoke.json --out runs/smoke
uv run continual-arc report runs/smoke

The report's score is 100 minus the budget used, so the leaderboard's U is 100 minus the score, or 100 times the mean cost over 16; its solve rate is P.

Step 03

Write a learner

A learner is any object with these methods, and whatever it keeps between instances is the method under test: notes, programs, a cache, or the weights of a model it trains. The runner calls reset wherever the schedule wipes the learner, which the reset reference does before every instance. What it keeps changes only in end_round, once a round is over, so every user of a round, and every instance of a frozen test, meets the state the round began with. This one keeps the demonstrations it pays for on each task, holding a round's new ones apart by user until the round ends, and asks solve for an answer; put your method there and it runs as it stands.

The learner sees only the instance: the task under a name drawn for the run, such as task-3f2a9c1e, the input, and its own attempts and requests against their caps. It is told nothing about the schedule, neither the phase nor how often the task has come before, and the run's records keep the real ids. The real id would be a lookup key into the public tasks and their solvers.

from continual_arc.env import DemosFeedback, GiveUp, LearnerObservation, RequestDemos, Submit
from continual_arc.learner import BaseLearner


def solve(pairs, grid):
    """Your method: an answer from the task's pairs so far, or None to pay for one more."""
    return None


class MyLearner(BaseLearner):
    def __init__(self):
        self.pairs = {}                  # what it keeps between instances, by task id
        self.bought = {}                 # this round's pairs, by user and task id

    def reset(self):                     # the reset reference wipes it before each instance
        self.pairs.clear()
        self.bought.clear()

    def act(self, obs: LearnerObservation):  # obs.task_id, obs.input, obs.user, obs.attempts_used, obs.demo_cap, ...
        pairs = self.pairs.get(obs.task_id, []) + self.bought.get((obs.user, obs.task_id), [])
        guess = solve(pairs, obs.input)
        if guess is not None:
            return Submit(guess)         # wrong costs 1; right ends the instance
        if obs.demo_requests_used < obs.demo_cap:
            return RequestDemos()        # one more pair of the task, costs 1
        return GiveUp()                  # ends the instance, charged as failed

    def observe(self, obs, action, feedback):
        if isinstance(feedback, DemosFeedback):
            self.bought.setdefault((obs.user, obs.task_id), []).extend(feedback.pairs)

    def end_round(self):                 # what it keeps changes only once a round is over
        for (_, task_id), pairs in self.bought.items():
            self.pairs.setdefault(task_id, []).extend(pairs)
        self.bought.clear()

    def resources(self):                 # cumulative counters, logged and never scored
        return {"pairs_kept": float(sum(map(len, self.pairs.values())))}

Run it from the folder that holds my_pkg/; PYTHONPATH=. lets the runner import it, or install your package into the same environment.

PYTHONPATH=. uv run continual-arc run --learner my_pkg.learners:MyLearner \
  --schedule smoke.json --out runs/my-smoke

Step 04

Or drive a harness

The baselines repository runs the paper's learners and drives agent harnesses through the benchmark's protocol. arc-baselines run starts a local proxy for the run and points OPENAI_BASE_URL, OPENROUTER_BASE_URL and ARC_LLM_BASE_URL at it, so every model call a harness makes is counted whatever the harness does, and the real key stays in the proxy. Each harness runs in a sandbox, bubblewrap on Linux and Seatbelt on macOS, that can write only its own state and reach the network only through the proxy, and it sees each task under an alias drawn for the run. The baselines need Python 3.12 or 3.13.

git clone https://github.com/unifyai/continual-arc
cd continual-arc/baselines
uv sync
uv run continual-arc data fetch
uv run arc-baselines run --learner continual_arc.testing:GiveUpLearner \
  --config arc1-smoke --seed 0

Adapters for Prime Agent, Hermes Agent and OpenClaw are in systems/, and scripts/launch-cell.sh launches any cell the paper reports, with the harness installed at the version of the paper's Table 7. This is OpenClaw as shipped on GPT-6 Luna at medium effort, playing the stream alone, as every cell does, so that its frozen tests can follow it as yours do below:

export OPENROUTER_API_KEY=...
bash scripts/launch-cell.sh acp workstreams 0 arc1 luna6-medium openclaw

For a harness of your own, subclass SystemLearner, which speaks the protocol, handles re-prompts, restarts and wipes, and leaves you five methods.

from continual_arc_baselines.systems.base import SystemLearner


class MyHarnessLearner(SystemLearner):
    """Everything the harness keeps between instances lives under self.state_dir."""

    name = "my-harness"

    def start(self):            # launch it, its model client on the proxy (self.environ)
        ...

    def stop(self):
        ...

    def ask(self, message, *, reprompt=False):  # one message in, its final reply out
        ...

    def new_session(self):      # a fresh conversation; what it keeps stays
        ...

    def wipe_state(self):       # delete everything it kept (called while stopped)
        ...

"end": "finish" ends an instance as the paper's harness rows did: the harness hears the verdict and nothing else, keeps whatever it chooses, and says when it is done. --llm-defaults sets the reasoning effort on any model call that leaves it unset, and --no-epilogue plays the stream alone, so that the frozen tests start from where it ends.

PYTHONPATH=. uv run arc-baselines run --learner my_pkg.harness:MyHarnessLearner \
  --learner-kwargs '{"end": "finish"}' --llm-defaults '{"reasoning_effort": "medium"}' \
  --config arc1 --seed 0 --no-epilogue --out runs/arc1/my-harness-ws0

A harness keeps one conversation, so it plays each frozen test one instance at a time. Before every instance arc-baselines test stops it and puts its state directory back as the stream left it, at the same path, so every path the harness wrote into its own files still names its own state. It builds the harness as the stream's launch did, so run it from the same folder, with the same key. The two tests play one after the other, each about as long as half the stream, and your harness's reset references, the same instances played with arc-baselines run, which wipes it before each, can play beside them. So can its reset on the stream, which wipes it before every instance of the same stream, from the schedule Step 05 makes.

uv run continual-arc schedule make --config arc1 --seed 0 --test held-out --width 1 --out arc1-held-out0-1.json
uv run continual-arc schedule make --config arc1 --seed 0 --test seen --width 1 --out arc1-seen0-1.json
PYTHONPATH=. uv run arc-baselines test runs/arc1/my-harness-ws0 --schedule arc1-held-out0-1.json \
  --out runs/arc1/my-harness-held-out
PYTHONPATH=. uv run arc-baselines test runs/arc1/my-harness-ws0 --schedule arc1-seen0-1.json \
  --out runs/arc1/my-harness-seen
PYTHONPATH=. uv run arc-baselines run --learner my_pkg.harness:MyHarnessLearner \
  --learner-kwargs '{"end": "finish"}' --llm-defaults '{"reasoning_effort": "medium"}' \
  --schedule arc1-held-out0-1.json --out runs/arc1/my-harness-reset-held-out
PYTHONPATH=. uv run arc-baselines run --learner my_pkg.harness:MyHarnessLearner \
  --learner-kwargs '{"end": "finish"}' --llm-defaults '{"reasoning_effort": "medium"}' \
  --schedule arc1-seen0-1.json --out runs/arc1/my-harness-reset-seen
# the stream with the harness wiped before every instance: arc1-ws0-reset.json from Step 05
PYTHONPATH=. uv run arc-baselines run --learner my_pkg.harness:MyHarnessLearner \
  --learner-kwargs '{"end": "finish"}' --llm-defaults '{"reasoning_effort": "medium"}' \
  --schedule arc1-ws0-reset.json --out runs/arc1/my-harness-ws0-reset

Each test's share saved and the share learned come from these runs, as Step 05 shows, with my-harness in place of my-learner.

Step 05

Run the stream and its tests

The track is arc1's stream at seed 0: 200 instances of 25 training tasks from Re-ARC's generators, and then two frozen tests. The held-out test serves one instance of each of 100 ARC-AGI-1 evaluation tasks the stream never served, and the seen test 100 fresh instances of the stream's 25 tasks, four of each. Each test is one round in which every instance is a user of its own, starting from where the stream left your learner, so nothing learnt on one test instance reaches another. The leaderboard compares runs at seed 0.

uv run continual-arc schedule make --config arc1 --seed 0 --no-epilogue \
  --out arc1-ws0.json
uv run continual-arc schedule make --config arc1 --seed 0 --no-epilogue --reset-each \
  --out arc1-ws0-reset.json
uv run continual-arc schedule make --config arc1 --seed 0 --test held-out --out arc1-held-out0.json
uv run continual-arc schedule make --config arc1 --seed 0 --test seen --out arc1-seen0.json

Play the stream, then give each test its own copy of your learner as the stream left it, in a run directory that holds the stream's alias-salt, so that it meets the stream's tasks under the names it learnt them by. A learner that holds a model client or a process saves what it keeps instead, and loads it into a new instance for each test, and a harness plays its tests with arc-baselines test, as Step 04 shows.

import copy
import shutil
from pathlib import Path

from continual_arc.runner import run
from continual_arc.schedule import Schedule

from my_pkg.learners import MyLearner

learner = MyLearner()
run(learner, Schedule.load("arc1-ws0.json"), out_dir="runs/arc1/my-learner-ws0")
for test in ("held-out", "seen"):
    out = Path(f"runs/arc1/my-learner-{test}")
    out.mkdir(parents=True)
    shutil.copy("runs/arc1/my-learner-ws0/alias-salt", out)  # the names the stream gave its tasks
    run(copy.deepcopy(learner), Schedule.load(f"arc1-{test}0.json"), out_dir=out)

Each test is read as the share of its reset's spend your learner no longer pays, against itself reset before each of the same instances: the same model, in the same harness if it has one, which plays the test one instance at a time, --width 1, on its own. transfer prints the share with its 95 percent interval, T, the points of the budget saved, and the budget used and share solved on each side, the absolute view. learned prints the share learned over the stream, your run against your learner wiped before every instance of the same stream, the schedule --reset-each made: the share of the budget left to learn that it saves, from 0 for a learner that keeps nothing to 1 for perfect memory.

PYTHONPATH=. uv run python play.py
uv run continual-arc schedule make --config arc1 --seed 0 --test held-out --width 1 --out arc1-held-out0-1.json
uv run continual-arc schedule make --config arc1 --seed 0 --test seen --width 1 --out arc1-seen0-1.json
PYTHONPATH=. uv run continual-arc run --learner my_pkg.learners:MyLearner \
  --schedule arc1-ws0-reset.json --out runs/arc1/my-learner-ws0-reset
PYTHONPATH=. uv run continual-arc run --learner my_pkg.learners:MyLearner \
  --schedule arc1-held-out0-1.json --out runs/arc1/my-learner-reset-held-out
PYTHONPATH=. uv run continual-arc run --learner my_pkg.learners:MyLearner \
  --schedule arc1-seen0-1.json --out runs/arc1/my-learner-reset-seen
uv run continual-arc transfer runs/arc1/my-learner-held-out runs/arc1/my-learner-reset-held-out
uv run continual-arc transfer --test seen runs/arc1/my-learner-seen runs/arc1/my-learner-reset-seen
uv run continual-arc learned runs/arc1/my-learner-ws0 runs/arc1/my-learner-ws0-reset

The bare model is no reference, but a result shows it beside the shares: the stateless floor of the baselines, the model alone with nothing kept, played on the two tests one instance at a time. A learner that played worse whenever its memory was empty would score for it against its own reset, so a result whose reset uses more of the budget than the bare model on a test is flagged there on the leaderboard. Unless the leaderboard already has the bare model's tests for your model and effort, run them from the baselines:

export OPENROUTER_API_KEY=...
uv run continual-arc schedule make --config arc1 --seed 0 --test held-out --width 1 --out arc1-held-out0-1.json
uv run continual-arc schedule make --config arc1 --seed 0 --test seen --width 1 --out arc1-seen0-1.json
MODEL='{"model": "your/model", "max_tokens": 128000, "end": "finish"}'
EFFORT='{"reasoning": {"effort": "medium"}}'
uv run arc-baselines run --learner continual_arc_baselines.systems.bare:BareModelLearner \
  --learner-kwargs "$MODEL" --llm-defaults "$EFFORT" --schedule arc1-held-out0-1.json --out runs/arc1/bare-held-out
uv run arc-baselines run --learner continual_arc_baselines.systems.bare:BareModelLearner \
  --learner-kwargs "$MODEL" --llm-defaults "$EFFORT" --schedule arc1-seen0-1.json --out runs/arc1/bare-seen

The paper's harness runs played the stream, 200 instances, in 4 to 45 hours and 1 to 45 US dollars of model calls on GPT-6 Luna, as the tracking proxy counted them. The two tests add 200 instances to your learner's run, its reset on the stream 200 more, and its reset on the tests 200 more.

Step 06

Check it and submit

Fork unifyai/continual-arc and copy every run directory the runner wrote to results/arc1/, each under a name of its own: your stream, your two tests, your learner reset on the stream and on each test, and the bare model's tests unless the leaderboard already has them. Then add a submission.json to your stream's directory, beside its five files, naming your reset's stream as reset and each test's three runs, run, reset and bare, under tests.

{
  "name": "my-learner",
  "learner": "my-learner 1.0, with the pairs it buys kept between instances",
  "student": "openai/gpt-6-luna",
  "effort": "medium",
  "teacher": null,
  "code": "https://github.com/you/my-learner",
  "contact": "you@example.com",
  "reset": "results/arc1/my-learner-ws0-reset",
  "tests": {
    "held-out": {"run": "results/arc1/my-learner-held-out", "reset": "results/arc1/my-learner-reset-held-out",
                 "bare": "results/arc1/bare-held-out"},
    "seen": {"run": "results/arc1/my-learner-seen", "reset": "results/arc1/my-learner-reset-seen",
             "bare": "results/arc1/bare-seen"}
  },
  "notes": "anything a reader needs to know to read the number"
}

scripts/check_result.py rebuilds every run's schedule from its configuration, test and seed, and checks that its records cover every slot and that its summary is the one the records give. It checks that your tests were played under your stream's task names, that your reset is a run of your learner and that the bare runs are the bare model's. Then it prints the numbers the leaderboard will show, both tests' shares with the budgets beside them, with a flag on any test where your reset plays below the bare model. It calls no model and no network.

mkdir -p results/arc1
cp -r runs/arc1/my-learner-* runs/arc1/bare-* results/arc1/
# write results/arc1/my-learner-ws0/submission.json, then
uv run python scripts/check_result.py results/arc1/my-learner-ws0

Then open a pull request. Your result goes on the leaderboard marked reported, its two shares placing it on the plane beside the others on its models. Questions go to the repository's issues.

Step 07

Have it verified

A verified result is one we have rerun ourselves on Continual-ARC's private held-out set. It is season s1: 50 new ARC-style rules that nobody outside unify has seen, each a generator of fresh instances, and each checked against every public ARC task. 25 of them are streamed as arc1 streams its tasks, and 25 are held out for the frozen held-out test after it. Claude Opus 5.5 at its highest reasoning effort solves every one of them, nearly every time, and GPT-6 Luna fares on them at every effort as it does on the public stream.

Once your run is in, open an issue with the Verify my learner form. It asks for:

• The system. Its name and a contact.
• The code. Its public repository and the commit to run, and the learner's import path: a Python class that implements the benchmark's learner protocol, such as your SystemLearner adapter, installable with pip from the repository and built with no arguments, so a harness's settings, such as end, are its class's defaults. A harness depends on continual-arc-baselines @ git+https://github.com/unifyai/continual-arc@v0.6.0#subdirectory=baselines alone, which brings the benchmark with it.
• The models. The published student model it plays with on OpenRouter and its reasoning effort, the teacher model if it has one, and any other model it calls.
• A key. An OpenRouter key made for this run, capped at its budget and encrypted with our public key. Paste what this prints into the form:

curl -sO https://raw.githubusercontent.com/unifyai/continual-arc/main/verify/unify-verify.pem
OPENROUTER_KEY=...
printf %s "$OPENROUTER_KEY" | openssl pkeyutl -encrypt -pubin -inkey unify-verify.pem \
  -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 | base64

We accept it on the issue within about 15 minutes. Then we rerun your learner on the held-out season at a fresh seed we do not disclose, in the same sandbox as the paper's runs, with every model call going through the tracking proxy on your key. After the stream, your learner plays the season's two frozen tests from where the stream left it: fresh instances of the 25 rules it met, and one instance of each held-out rule. A harness on SystemLearner plays them one at a time, as arc-baselines test does: before each one we stop it and put its state directory back as the stream left it. Any other learner plays them in a single round, so it must serve a round of users as the learner protocol asks, keeping each user's work apart and changing what it keeps only once the round ends. Your learner reset plays the season's stream wiped before every instance, for the share learned, and the same test instances one at a time, wiped before each, for each test's share, and the bare model it plays with plays the test instances too, a reset that plays below it flagged on that test. A run takes hours, as on the public stream.

The result goes on the issue and on the leaderboard, marked verified: both tests' shares with their 95 percent intervals and the budgets beside them, the share learned on the held-out stream, and what the model calls cost. Its intervals are wider than the paper's rows, which average four plays a side where a verified run plays each side once, and its rules are the private season's, so its point is read beside the public stream's, never as the same measurement. Then we delete the key. Delete it at OpenRouter too.

You pay only for the model calls, on your own key, and we pay for the machine. The paper's harness runs on GPT-6 Luna cost 1 to 45 US dollars on the public streams, and the same harnesses wiped before every instance 2 to 37, so a cap of about 100 dollars suits a model of Luna's class, and a stronger model needs more. Keep OpenRouter's Private Input & Output Logging off on the account: with it on, the account's logs would hold our held-out tasks, so a run with it on is void.

For now we cannot verify a learner that trains its weights, or a system whose code cannot be shared.

The rules

What a learner may use

The stream is public, and the answers to its tasks are too: Re-ARC ships a solver for every ARC-AGI-1 training task, and the evaluation tasks' outputs are published. A learner that brought them along would use none of the budget without learning anything. So a result sent by pull request shows as reported, a claim, and it becomes a result only once we have rerun its code on the held-out set, whose rules are not public.

• Nothing about ARC brought in. No ARC, Re-ARC, ARC-GEN or other task data, answers or solvers inside the learner, its prompts or its starting memory, and no rebuilding of the schedule. What a model learned in pretraining is its own prior, which your learner reset on the same model already carries.
• A published student and teacher, used unchanged. The model your learner plays with, and the one that teaches it if there is one, are published checkpoints used as published and named exactly: an OpenRouter model id, or open weights as their Hugging Face repository at a commit, org/name@commit. A result is compared only with others on the same student and teacher, so a model changed before the stream is another student and stands alone: one made to start badly would save a large share of its reset's spend just by recovering, and its absolute budget would show it ending where the published model starts.
• Every model named. Every model the learner calls is named in the submission and reached through the baselines' tracking proxy. Its shares are read against your learner reset, which calls the same models, and the bare model shown beside them is the model it plays with, the strongest if it plays with several. A model that only teaches it, such as a stronger model labelling its games for training, is part of its training: those calls count in its training compute, shown beside the score.
• Learning only from what it was shown. A learner learns from its observations and feedback. The run's records, with their real task ids and exposures, are the benchmark's, not the learner's.
• Anything else is allowed: memory, programs, tools, a cache, training the weights. Compute is logged and shown beside the score, never in it, and a learner that trains shows its training compute apart from what it costs to serve an instance.