A red pen is fast, a pass/fail grade outgrows the teacher: RLVR, OPD and MOPD on a toy model빨간 펜(OPD)은 빠르고, O/X 채점(RLVR)은 선생을 넘어요

A toy experiment: CPU only, about 35 minutes, three seeds per run. The full code is rlsim.py, a single 364-line file that needs only PyTorch and NumPy.

At a glance

There are six experiments. Experiments 1–5 use the same small student model, and every method writes the same 256 answers per training step. Numbers are accuracy on 2,000 held-out evaluation problems (drawn separately from training problems, though an overlap is possible; see known gaps). Experiment 6 scales up to million-sized numbers and all four operations on a GPU.

Why I ran this: the shape of LLM post-training

Open models released in 2025–2026 mostly follow the pipeline below. This note shrinks the blue part, RL and distillation (OPD, MOPD), until it runs on a laptop, and runs it piece by piece.

Figure 1. A typical modern pipeline. The blue steps are what this experiment runs.

Xiaomi’s MiMo-V2-Flash, for example, describes three stages: SFT, per-subject RL to build teachers, then multi-teacher OPD. DeepSeek-V4 reportedly replaced V3.2’s mixed multi-subject RL stage entirely with OPD (I took that from a blog analysis and did not check the original). Experiment 4 compares exactly these two approaches.

The setup: problems, grader, models

Problems: three-digit addition and subtraction, ones digit first

The model sees a problem like 699+684= and writes the answer one character at a time: four digits starting from the ones place, then $ (end of answer). 1383 becomes 3831$.

Ones-first is the order of long addition by hand. You need the ones digit to know the carry, and the carry to write the tens digit. So these five characters are a tiny reasoning trace: a mistake early on shakes the digits after it.

The vocabulary is 14 characters: 0–9, +, -, = and $. Subtraction always takes the smaller number from the larger, so answers are never negative.

The verifier: one point only if all five characters match

This grader is the “V” in RLVR (reinforcement learning with verifiable rewards). There is no partial credit: writing 1183 for 1383 scores 0, even though three of four digits are right.

# rlsim.py: the verifier
def correct(seq, answers):
    return (seq[:, PROMPT_LEN:] == answers).all(1).float()   # 1 if every character matches, else 0

Models: one small student, three larger teachers

Both are GPT-style decoder transformers shrunk down. The teacher is about 8× the student.

The three teachers were trained with SFT on true answers (details). Each learned only its own subject, so each scores close to 0% on the other one. That matters in experiment 4.

How the teachers learned: SFT on the true answers

The teachers use neither RL nor distillation. This toy problem computes the true answer (a + b, a − b) for free, so the teachers simply copy correct answers. In code that is load_or_train_teachers() calling sft_until() (walkthrough 12).

# name: (subjects to learn, target accuracy, seed)
specs = {"add": (["+"], 0.995, 1), "sub": (["-"], 0.995, 2), "add-weak": (["+"], 0.70, 3)}

# repeat until the target is reached
prompts, answers, _ = mixed(BATCH, ops, rng)   # 256 random problems of its own subject + true answers
step_sft(model, opt, prompts, answers)         # copy the answer character by character (cross-entropy)

1. Model: a randomly initialized 4-layer, width-128 TinyGPT, 798,606 parameters (about 8× the student).
2. Training: each step draws 256 fresh problems of its subject and trains it to reproduce the five answer characters with teacher forcing. Adam, learning rate 0.001.
3. Stopping: every few steps it is scored on 1,000 validation problems (seed 999) and stops once it beats the target. These are separate from the evaluation problems used for the charts (seed 12345).
4. Saving: stored as results/teachers/*.pt and only loaded afterwards. Teachers never train during the experiments.

• The weak teacher was taught less on purpose. Same recipe as the addition teacher, stopped as soon as it crossed 70%. Its mistakes are scattered across problems, which is why the student first overtakes it in experiment 3.
• Outside its own subject it scores 0%. The addition teacher never saw a - during training. That character’s embedding never received a gradient and is still its random initial value. Given a subtraction problem it answers 548 − 286 with 834: it adds.
• The student’s 34% starting point uses the same function, with the target lowered to 30%.

This is the same function (step_sft) as the experiments’ “SFT distillation”. Only the target answer differs.

The student’s starting point: about 34%

A real LLM starts RL after pre-training and SFT. So the student was also given a little SFT on true answers, stopping once it passed 30%. That took 380–1,115 steps depending on the seed, and it starts at 34% on average on the evaluation problems. Experiment 2 is the exception: it starts from an untrained 0% student.

Kept equal for a fair comparison

The four methods on one problem

One problem shows what each method teaches: 699 + 684 = 1383. The 34% student (seed 0) writes 1183: only the hundreds digit is wrong.

① SFT distillation: copy the model answer

The teacher solves the problem first and writes a model answer. The student copies it one character at a time, and while doing so it sees the teacher’s prefix, not its own. The student never solves anything itself.

The loss grows as the probability the student gives the teacher’s character shrinks (−log probability, i.e. cross-entropy). Copying the teacher’s 1383:

In the hundreds place the student gives the correct 3 only 10.6%, the largest loss. Training pushes the student toward “3 goes here”.

# step_sft in rlsim.py, condensed
targets = teacher_tokens(prompts, subjects, teachers)      # the teacher's model answer
seq = torch.cat([prompts, targets], 1)
loss = F.cross_entropy(answer_logits(model, seq), targets)  # −log p per character

Line-by-line notes and tensor shapes: walkthrough 7.

② RLVR (GRPO): solve the same problem 8 times, get only pass/fail

The student solves the same problem 8 times itself. The verifier returns a single pass/fail per answer. Each answer is then compared with the average of the eight: answers better than average get their characters’ probabilities raised, worse ones lowered. That is GRPO (RL that compares answers to the same problem against each other).

How much to raise or lower is the advantage: (that answer’s score − mean of the 8) ÷ standard deviation of the 8.

Case A: 699 + 684 attempted 8 times, all wrong.

All eight score 0, so the mean is 0 and every difference is 0. Every advantage is 0, so nothing is learned from this problem, not even which digit was wrong.

Case B: 227 + 189 (answer 416), right 2 times out of 8.

Where +1.62 and −0.54 come from. They measure how much better than the average of its group of eight each answer did.

The sign comes from ③ and the size from ⑤. Dividing by 7 rather than 8 in ④ is PyTorch’s std() default; dividing by 8 gives a std of 0.433 and +1.73 / −0.58.

So the two right answers are pushed up with weight +1.62 and the six wrong ones down with −0.54. But the wrong 516 got its last two digits, “16”, right, and the whole answer still gets −0.54, dragging the right digits down too. A single pass/fail can’t tell characters apart. Summed over all eight answers, though, the picture changes a little (walkthrough 8).

Flip the marks yourself below and watch the learning signal change with the number of right answers.

By number right. Scores are only 0 or 1, so the number right out of 8 determines everything. Same formula as the calculator.

• The rare outcome moves most. If only one of eight is right, it is pushed up by +2.47; if seven are right, the one wrong answer is pushed down by −2.47.
• The advantages always sum to 0. With two right: +1.62 × 2 − 0.54 × 6 = 0. Up and down balance, so characters that all eight answers share cancel out (walkthrough 8).
• All right or all wrong gives 0. Every difference in ③ is 0; there is nothing to compare. So RLVR learns nothing from problems that are too hard or too easy.
• How wrong doesn’t matter. 516 (one digit off) and 906 (two digits off) both get −0.54, because scores are only 0/1. A grader with partial credit would give each answer a different value.

What the number actually does. The loss is −advantage × log p(answer). Minimizing it raises the probability of that answer with strength 1.62 and lowers others with strength 0.54. It is then scaled by the learning rate and averaged over 256 answers × 5 characters, so it is a relative weight between answers, not “the probability moves by 1.62”.

Dividing by the std in ⑤ is debated. A very hard problem with one right answer, or a very easy one with seven, has a small std, so dividing concentrates large values there. Dr. GRPO drops the division and uses the raw differences from ③ (+0.75 / −0.25).

# step_rlvr in rlsim.py, condensed
seq = write(model, prompts.repeat_interleave(8, 0))          # solve each problem 8 times
reward = correct(seq, answers).view(-1, 8)                   # pass/fail
advantage = (reward - reward.mean(1, keepdim=True)) / (reward.std(1, keepdim=True) + 1e-6)
logp = log_prob_of_written_tokens(model, seq)                # log p of the characters written
loss = -(advantage.view(-1, 1) * logp).mean()                # push good answers up, bad ones down

Line-by-line notes, these eight answers' real log-probabilities, and where the signal survives once the eight are summed: walkthrough 8.

③ OPD: a red pen on the student’s own answer

In OPD (on-policy distillation) the student solves the problem itself. The teacher then reads the student’s answer from the start and, at each character, gives its own probabilities for “what I would write after this prefix”. The student learns to make its probabilities match. The teacher is the larger model SFT-trained on true answers (how the teachers learned).

Below is the teacher’s red pen on the student’s 1183. Each row shows the student’s and the teacher’s top probabilities at that position.

The loss is reverse KL: for every character the student puts probability on, how much does the teacher put there? The more the student bets on characters the teacher almost never writes, the larger the loss. As a formula: Σ p_student(c) × log(p_student(c) ÷ p_teacher(c)).

• Hundreds (9.48): the student bets 36% on 1, the teacher almost 0% (and 99% on 3). The wrong digit is pinpointed.
• Tens (1.34): written correctly as 8, but still penalized: the student also bet 22% on 7, which the teacher gives 0%. OPD fixes right but unsure, which RLVR cannot.
• Other places (0.01): student and teacher agree; almost no loss.

“On-policy” means the student learns from answers it actually wrote. SFT distillation only ever sees the teacher’s prefix; OPD also teaches “from here, continue like this” after the student’s own mistakes. The cost is one extra teacher pass per answer.

# step_opd in rlsim.py, condensed
seq = write(model, prompts)                                   # the student solves it (once)
s = answer_logits(model, seq).log_softmax(-1)                 # student probabilities per character
t = answer_logits(teacher, seq).log_softmax(-1)               # the teacher reading the same answer
loss = (s.exp() * (s - t)).sum(-1).mean()                     # reverse KL over all 14 characters

The gradient, and where probability moves in the hundreds place: walkthrough 9.

④ MOPD: each subject gets its own teacher’s red pen

MOPD (multi-teacher OPD) is OPD that routes each problem by subject tag to that subject’s teacher: addition to the addition teacher, subtraction to the subtraction teacher.

The two-subject starting student writes 252 for 548 − 286 (answer 262). The same answer under each teacher’s red pen:

The subtraction teacher fixes only the wrong tens digit. The addition teacher rewrites every digit: on its own it would write 834, i.e. 548 + 286, ignoring a minus sign it has never seen. Hand it subtraction and it teaches the student to add (experiment 4).

# rlsim.py: MOPD is OPD with a per-subject teacher map
step_opd(model, opt, prompts, subjects, teachers={"+": add_teacher, "-": sub_teacher})

The training environment: what each method sees, writes and gets back

All methods play the same “game”: read a problem, write five characters, done. Two things differ: who writes the answer and what comes back.

The game, in RL terms

Per-method environment

• Why SFT is slowest: a teacher 8× the student’s size writes an answer every step, which takes one pass per character (5). OPD’s teacher only reads an answer the student already wrote, producing all five positions in one pass. In real LLMs too, the OPD teacher’s cost is reading (prefill), not generation, which is relatively cheap.
• SFT and OPD never use the true answers. mixed() returns them, and both methods throw them away. They only need a teacher, which is why a wrong teacher passes its mistakes on (experiment 3). RLVR is the reverse: no teacher, just the truth (the grader).
• Only RLVR solves the same problem several times. One pass/fail says nothing about “good”; it has to be compared with other answers to the same problem. So the same 256 answers cover only 32 problems.

What every method shares

• Starting weights: each seed builds one starting student, and every method gets its own copy.deepcopy(start). All start from exactly the same weights.
• Optimizer: a fresh Adam per method, learning rate 0.0005, gradients clipped to total norm 1 (clip_grad_norm_).
• Problem order: each method rebuilds its RNG from the same seed. SFT and OPD draw 256 problems per step, so they see the same problems in the same order. RLVR draws 32 per step, so its order differs.
• Evaluation: every 20 steps, 2,000 fixed problems per subject (seed 12345), greedy decoding. Evaluation never affects training.

What changed per experiment

Hardware

CPU only on an Apple M4 Max laptop (no GPU). Python 3.11, PyTorch 2.10, NumPy 2.3, with torch.set_num_threads(4). Building the three teachers takes under a minute; the four experiments about 35 minutes. No external data and no pre-trained weights: the code generates the problems on the spot.

Experiment 1: a student starting at 34%

Question: from the same starting point, which method learns fastest?
Setup: the 34% student, addition only, 1,000 steps. SFT distillation and OPD learn from the addition teacher (99.8%).

Figure 2. Addition accuracy. Hover a line to see that step's values.

Why: the information per step differs. RLVR writes 256 answers and learns only “how many of the 8 were right” for each of 32 problems. OPD gets a full 14-way probability distribution for each of the five characters of every answer.

On top of that, RLVR ran out of problems worth learning from. At the start 61% of problems carried a signal; by step 1,000 only 24%. Early on, hard problems with 8 wrong answers carry no signal; later, easy problems with 8 right answers don’t either (figure 3, right).

Here SFT distillation was slightly faster than OPD. The answers are only five characters, so there is little “continue from your own mistake” to learn. More on that in what carries over to real LLMs.

Experiment 2: a student that knows nothing

Question: can a blank student learn with each method?
Setup: a randomly initialized student (0%), addition only, 1,000 steps.

Figure 3. Left: accuracy of the 0% student. Right: share of problems where RLVR's 8 answers were not all the same (i.e. carried a signal).

Why: a blank student gets all five characters right by chance with probability about (1/14)⁵, roughly 1 in 540,000. Over 1,000 steps × 256 answers = 256,000 answers, the expected number of right answers is 0.5. Every group of 8 is all wrong, the signal stays 0, and the student never changes.

OPD tells the student “this digit goes here” at every character even when the answer is wrong, so it learns from scratch.

Why OPD suddenly jumps around step 400–500

Accuracy sits near 0% for 400 steps and then shoots up around step 500. I re-ran seed 0 (the accuracy curve reproduces exactly) and measured accuracy per digit.

2,000 evaluation problems, greedy. KL is the average loss over the preceding 20 steps.

• Easy positions come first. The end marker $ hits 100% within 100 steps, and the thousands digit (always 0 or 1) quickly reaches the 70s. Neither needs much arithmetic.
• Positions that need arithmetic sit at 10% for a while. Ones, tens and hundreds hover around 10% until step 300, guessing among ten digits. KL barely moves from 4.3–4.6. The model seems not to have found which input positions to add yet.
• Addition unlocks in long-addition order. The ones digit goes 30% → 90% between steps 300 and 400; then between 400 and 500 the tens go 26% → 90% and the hundreds 10% → 81%. The ones digit (no incoming carry) comes first, the digits that need a carry follow. Subtraction goes in a different order (below).
• Exact match needs all five. At step 400 the ones digit is at 90% but exact match is 2.3%, because the other digits are still guessing. Exact match is close to the product of the per-digit accuracies (step 500: product 71%, actual 73%).

Two things overlap. One: the model takes a while to find the digit rule and then learns it fast; per-digit accuracy really does jump. Two: a 5-for-5 grade makes digit-by-digit progress look like “flat, then a leap”. Feeding the correct prefix (teacher forcing) gives almost the same per-digit accuracy, so this is not early mistakes propagating. Experiment 1 starts at 34% and already knows part of the rule, so it has no plateau.

Subtraction? The order follows how hard each digit’s arithmetic is

Subtraction does not go ones-first, and it doesn’t go in reading order either. All six runs (OPD and true-answer SFT, three seeds each) behaved like this:

• The hundreds digit passes 50% first, at steps 200–220, and then stalls in the 60s–70s for a while.
• The ones digit is the first to finish (90%), at steps 420–940, followed by the hundreds. The tens digit is last; under OPD it never reached 90% within 1,000 steps.
• Subtraction is harder than addition. After 1,000 OPD steps, exact match is 72.3–83.1% for subtraction versus 93.5–98.4% for addition.

Mean of three seeds. "(2/3)" means only two of the three seeds got there; the mean uses those seeds only.

Splitting up the arithmetic each digit needs shows why. Below, a “wrap” is the step that goes past 10 or below 0 and comes back around (7 + 5 → 2, 3 − 8 → 5).

Subtraction here always takes the smaller number from the larger, so subtracting the hundreds digits gives 0–9 directly, with no wrap. Ignoring the borrow and just writing “hundreds of a − hundreds of b” is already right more than half the time. The stall in the 60s–70s looks like that stage. Getting the borrow right needs the lower digits, so the hundreds only pass 90% after the ones digit is learned. The tens digit needs both a wrap and a borrow, so it’s last.

Addition’s ones → tens → hundreds order also looks less like “long-addition order” and more like the order of easiest arithmetic, which happens to coincide. The model learns the easy computation first, not the writing order.

What if the answer is written biggest digit first? It learns far worse

So does writing order not matter? I switched the answer to reading order (thousands digit first) and compared it with the current ones-first format. There is no teacher for that format, so this uses true-answer SFT from a 0% student for 2,000 steps (three seeds, digit_order.py).

Figure 3-3. True-answer SFT after 2,000 steps, per-digit and exact-match accuracy (mean of three seeds). The thousands digit is 100% in every format and is left out.

• Ones first: after 2,000 steps addition is at 100% and subtraction at 99.5–100%.
• Thousands first: addition 8–23%, subtraction 3–33%. The thousands and hundreds digits come quickly (90% reached at steps 0–100 for thousands, 420–920 for hundreds). The tens and ones digits reached 90% within 2,000 steps in only one of six seeds each.

Writing ones first turns the low digits the model has already written into a scratchpad for the next digit. If a’s ones digit is 7 and the model wrote 2 as the answer’s ones digit, it can see there was a carry (7 + 5 = 12). Writing thousands first means computing every carry in its head in one go. The hundreds digit stalling at 93–97% in that format also looks like a carry guessed from the tens digit alone. For addition, the ones digit only decides the carry when the tens digits sum to exactly 9, which is about 4.5% of problems, so guessing gets you to about 95%. This matches the finding that small transformers learn addition much better when the answer is written reversed, ones digit first (Lee et al., 2023).

What I could not explain is why, in the thousands-first format, the ones digit (the easiest arithmetic) is learned last.

Experiment 3: when the teacher is weak

Question: how far does a student get with a teacher that makes mistakes?
Setup: the 34% student, addition only, 3,000 steps. SFT distillation and OPD learn from the weak addition teacher (73%). RLVR uses only the grader.

Figure 4. Addition accuracy. The dashed line is the weak teacher's skill (73%).

At first the student beat the teacher. Taught by a 73% teacher, the student reached 84%. The teacher’s mistakes are scattered across problems, so the student picks up the shared rule before the noise.

With more training it came back down. As the student matches the teacher’s probabilities ever more closely, it copies the teacher’s mistakes too, and falls to 79%. Distillation means “become the teacher”, so the teacher is the ceiling.

RLVR is slow but has no ceiling. Its reference is the truth, not a teacher. It catches up with distillation around step 1,000 and crosses 90% at step 1,660.

Experiment 4: two subjects

Question: how do you make one student good at two subjects?
Setup: a student with a little of both (addition 34%, subtraction 30%), 1,000 steps, four methods.

• Mix-RL: each step mixes 16 addition and 16 subtraction problems, 8 attempts each, pass/fail.
• Cascade-RL: 500 steps of RL on addition, then 500 on subtraction.
• MOPD: each step, the addition teacher red-pens 128 addition problems and the subtraction teacher 128 subtraction problems.
• Addition teacher only: the same OPD, but the addition teacher grades both subjects.

Figure 5. Accuracy on both subjects. At the vertical dashed line (step 500) Cascade-RL switches from addition to subtraction.

MOPD: the right teacher for each subject

Both subjects pass 97% by step 1,000, reaching 90% at step 480 (addition) and 540 (subtraction). Each subject is corrected character by character by a teacher who knows it, so the subjects don’t interfere.

Addition teacher only: you can’t teach what you don’t know

Addition hits 99%, but subtraction falls from 30% to 3% within 20 steps and then to 0%. As shown above, the addition teacher writes 834 for 548 − 286. The student faithfully follows that red pen and adds on subtraction problems too. This is why MOPD keeps a separate teacher per subject.

Mix-RL: mixing makes the subjects shake each other

In the first 80 steps addition dropped from 34% to 18% and subtraction from 30% to 21%. Reinforcing right answers in one subject shook the other through shared parameters (the see-saw effect). Each subject also gets only 16 problems per step, half the signal. It then climbs slowly to 45% by step 1,000.

Cascade-RL: learn one, forget the other

• Steps 0–500 (addition only): addition rose 34% → 67%, but subtraction fell 30% → 3.5% without ever being trained: catastrophic forgetting.
• Step 500 (switch to subtraction): subtraction is already almost gone, so most groups of 8 are all wrong. The share of problems with a signal fell from 58.5% in the addition phase to 4.9%.
• Steps 500–600: addition then collapsed too, 67% → 6.5%. Rare subtraction signals kept moving the parameters and seem to have shaken addition as well; I didn’t isolate the exact cause.

Experiment 5: getting RLVR started on a 0% student

Question: in experiment 2, RLVR never received a single signal from the 0% student. What has to change for it to start learning?
Setup: the same random 0% student, addition only, 256 answers per step, 3 seeds. Five methods for 1,000 steps; the three that started learning were run to 3,000.

def step_rlvr_variant(model, opt, prompts, answers, method: str) -> dict:
    """The same GRPO step as rlsim.step_rlvr, plus a switch for the variants."""
    prompts, answers = prompts.repeat_interleave(s.GROUP, 0), answers.repeat_interleave(s.GROUP, 0)
    seq = s.write(model, prompts)
    if method == "inject":  # replace the first answer of each group with the true answer
        seq[:: s.GROUP, s.PROMPT_LEN :] = answers[:: s.GROUP]
    hit = (seq[:, s.PROMPT_LEN :] == answers).float()  # (256, 5): right at each position?
    exact = hit.all(1).float()
    logp = s.answer_logits(model, seq).log_softmax(-1).gather(-1, seq[:, s.PROMPT_LEN :, None]).squeeze(-1)
    if method == "token":  # per position: mean and std over the same position of the 8 answers
        reward = hit.view(-1, s.GROUP, s.ANSWER_LEN)
        advantage = (reward - reward.mean(1, keepdim=True)) / (reward.std(1, keepdim=True) + 1e-6)
        loss = -(advantage.reshape(-1, s.ANSWER_LEN) * logp).mean()
        signal = (reward.std(1) > 0).float().mean().item()
    else:
        reward = (hit.mean(1) if method == "partial" else exact).view(-1, s.GROUP)
        advantage = (reward - reward.mean(1, keepdim=True)) / (reward.std(1, keepdim=True) + 1e-6)
        loss = -(advantage.view(-1, 1) * logp).mean()
        signal = (reward.std(1) > 0).float().mean().item()
    s.update(opt, model, loss)
    own = exact.view(-1, s.GROUP)[:, 1:] if method == "inject" else exact.view(-1, s.GROUP)  # student-written answers only
    return {"signal": signal, "own_exact": own.mean().item()}

exp5.py (lines 50–71) adds a switch to rlsim.py's step_rlvr; s is rlsim, left unchanged.

After 1,000 steps

Per-digit accuracy is the mean of three seeds. "Unchanged" means the gradient stayed 0 and the weights never moved from their initial values.

To 3,000 steps: the three that started

Figure 6. Addition accuracy of the 0% student. Hover a line to see that step's values.

Why

• Easy problems first: the weights never moved. Even 0–9 problems still have five-character answers (7 + 5 → 2100$). A student that doesn’t know the format never gets all five right, so every group scores 0, advantages are 0 and so is the gradient. Per-digit accuracy after 1,000 steps is identical to the start. Easy-first only helps a student that is already sometimes right.
• Partial credit (per answer): only the easy positions. The end marker $ reached 100%, and the thousands digit stopped at exactly 50.2% in all three seeds: 50.2% of evaluation sums are ≥ 1,000, so it appears to always write 1. Ones, tens and hundreds stay at 10%, pure guessing. The score is one number for the whole answer, so +0.2 for a lucky ones digit is spread equally over all five characters and drowned out by the noise of other positions being right or wrong by chance. It can’t tell which position earned it.
• Per-digit grading: learns per position, but slowly. An answer with the right ones digit raises only the ones character. Unlike partial credit, per-digit accuracy really does rise (ones, tens, hundreds at 82 · 69 · 49% by step 3,000). But it learns by trial and error, only when it happens to write the right digit (10%), so exact match is just 24% at step 3,000. It is a tiny version of a process reward.
• Injected answer: works, but about 2× slower than SFT. By step 3,000 two seeds are at 99–100% (90% at steps 1,480 and 1,800). One seed is at 9.7%; that seed was also the slowest under true-answer SFT. At first all 7 student answers are wrong and each gets −0.35. Uniformly lowering answers sampled from the student’s own distribution points nowhere on average (the expected gradient of the log-probability of a sampled answer is 0). So what is actually learned is the 32 injected answers at +2.47: an eighth of SFT’s data (where all 256 are correct), plus noise. Once the student starts getting answers right itself, it becomes real RLVR; by the end, 99% of the answers it wrote were right.
• True-answer SFT: the fastest. At step 1,000 it ranges from 17% to 100% by seed; by step 3,000 all three are at 100%. It reached 90% at steps 680–1,140, similar to OPD in experiment 2 (520–840).

Experiment 6: million-scale arithmetic, four teachers

Question: experiments 1–5 use three-digit addition and subtraction, which is very simple, and the right answer comes for free. If the problems grow to million-scale arithmetic with all four operations and one teacher per operation, does MOPD clearly beat the other methods?
Setup: 61 jobs on a dev GPU box (two H100 NVL), about 3 hours 40 minutes. Each student run is 10,000 steps, three seeds. The code is a single file, big/bigsim.py (not published here).

Making “the answer is free” less free

• True-answer SFT baseline: the result when the correct worked answer can be handed over directly. Other methods have to beat this to mean anything.
• RL-polished teachers: only 10,000 steps of true-answer SFT, then 20,000 more steps of RLVR. Closer to how real teachers are made.
• A systematically wrong teacher: an addition teacher that always drops the carry into the thousands digit. Not random errors: the same mistake every time.
• A grader that’s sometimes wrong: during RL, each pass/fail mark is flipped with 10% probability.

The teachers are nearly perfect

A 10.7M-parameter teacher gets 6-digit × 6-digit multiplication right 99.8% of the time with no worked steps. That looks like the ones-first format plus an effectively unlimited supply of problems. The teacher that learned all four operations (4× the training) is as good as the specialists: 100% on +, − and ×, 97.3% on ÷. For the RL-polished teachers, 20,000 RL steps did not raise accuracy (multiplication 99.9 → 99.6%, division 91.8 → 90.5%).

The starting student is 1,000 steps of true-answer SFT. It is already at 99.9–100% on addition and subtraction, and starts at 28–32% on multiplication and 23–32% on division.

Results: the distillation methods are about the same, and RL barely learns

Figure 6-1. The student's multiplication and division accuracy after 10,000 steps (mean of three seeds).

• All four distillation methods match true-answer SFT. True-answer SFT 82.5%, SFT distillation 82.1%, MOPD 81.2%, single-teacher OPD 81.1%. With a near-perfect teacher, copying the teacher and copying the truth are the same thing. MOPD did not come out ahead.
• Why MOPD didn’t win: here even the teacher that learned all four operations is as good as the specialists, so a teacher per subject buys nothing. MOPD won experiment 4 because the addition teacher knew nothing about subtraction. Answers are also short (13 positions), so OPD’s strength, learning to continue after one’s own mistakes, has little room to show.
• The ceiling is the student. The teacher is at 99.8% on multiplication; the student sits near 58% whatever the method, stuck on large multiplications (table below). The distillation curves were still rising 0.9–2.3 points over the last 2,000 steps, so 10,000 steps is not converged.
• Mix-RL barely moves. Multiplication 30 → 32%, division 27 → 30%. Only about 3% of problems have some but not all of the 8 answers right, because a point needs all 13 positions correct and the student rarely gets large × or ÷ right.
• Cascade-RL forgets the earlier subjects. Running + → − → × → ÷ for 2,500 steps each dropped addition 100 → 69.8% and subtraction 100 → 76.1%. Multiplication and division stay near the start.
• MOPD → RL ends below MOPD alone (78.0 vs 81.2%). After 5,000 MOPD steps, 5,000 RL steps added 1.0–1.4 points; the same 5,000 steps spent on more MOPD added 4.0–4.9.

Realism conditions

Figure 6-2. Baseline condition versus the bug teacher (addition accuracy) and the 10%-wrong grader (overall accuracy). Mean of three seeds.

• RL-polished teachers: the student ends up the same. The teacher’s division is 90.5%, below the true-answer SFT teacher’s 97.1%, but the student’s division is around 70%, so the teacher difference only exists above the student’s ceiling.
• Bug teacher: distillation copies the mistake. The starting student was at 99.9–100% on addition, and both MOPD and SFT distillation brought it down to 80.4%, exactly the bug teacher’s score. The teacher’s mistake overwrote something the student already knew.
• Bug teacher + RL: only one seed of three recovered. For seed 1, addition was back to 99.8% within 1,000 RL steps and ended at 100%; seeds 0 and 2 stayed at 80.4% to the end. Once the student writes the buggy answer confidently, all 8 answers are identically wrong and there is no signal. It only gets fixed when an answer happens to get the carry right. Same reason as experiments 2 and 5.
• A sometimes-wrong grader: Cascade-RL collapsed. During the addition and subtraction stages (already ~100%), “problems with something to learn” showed up as 57%. With a clean grader it was 0.7–2.1%, so nearly all of it was fake signal. In the multiplication stage, one mostly-wrong answer flipped to a pass gets pushed hard with +2.47, and around steps 5,250–6,000 even addition collapsed below 50%. Mix-RL didn’t collapse, but multiplication and division ended below the starting point. MOPD → RL mostly held because of what it had already learned from OPD.

RL learning rate: at 3e-4 the model falls apart

At first I ran RL with the same learning rate as OPD, 3e-4. A multiplication teacher that was at 99.9% after SFT dropped into the 80s within 1,000 RL steps. Only a few percent of problems have anything to learn, so the RL gradient is small and noisy, and Adam normalizes the step size per parameter, which turns that noise into steps of size 3e-4 as well. OPD’s signal is dense (every character), so the same learning rate was fine there.

Figure 6-3. Accuracy after RL with learning rate 3e-4 versus 3e-5. Teachers: after 20,000 RL steps that followed 10,000 SFT steps (SFT ended at 99.9% for multiplication and 91.8% for division). Students: seed 0.

The last row is from the big/rl_lr_test.py log (seed 0 starting student, 1,200 steps).

Lowering just the learning rate to 3e-5 stopped the collapse. Every RL run in the results above was re-run at 3e-5. It’s the same reason real LLM RL uses learning rates around 1e-6, far smaller than SFT.

What carries over to real LLM training, and what doesn’t

This is a toy for seeing the concepts. Take the direction, not the numbers.

Code walkthrough: rlsim.py, line by line

rlsim.py is a single 364-line file using only PyTorch and NumPy. The snippets below are cut straight from it, and the captions give the file’s line numbers. Tensor shapes and values were extracted by running the same code.

0. Map of the file

Figure 7. Call order. The methods split inside the blue box.

1. Characters and problems: VOCAB, problems(), mixed()

VOCAB = list("0123456789+-=$")  # "$" = end of answer
STOI = {c: i for i, c in enumerate(VOCAB)}
PROMPT_LEN, ANSWER_LEN = 8, 5  # after "123+045=": four digits ones-first, then "$"
SEQ_LEN = PROMPT_LEN + ANSWER_LEN
BATCH = 256  # answers the student writes per step; identical for every method
GROUP = 8  # RLVR: answers per problem (32 problems x 8)
LR = 5e-4
LOG_EVERY = 20
WARM_TARGET = 0.30  # student starting point: 30% accuracy (mimics pre-training + a little SFT)

rlsim.py, lines 30–38

def problems(n: int, op: str, rng: np.random.Generator) -> tuple[torch.Tensor, torch.Tensor]:
    """n problems (prompts) and answers as token ids. Subtraction always subtracts the smaller number from the larger."""
    a, b = rng.integers(0, 1000, n), rng.integers(0, 1000, n)
    if op == "-":
        a, b = np.maximum(a, b), np.minimum(a, b)
    c = a + b if op == "+" else a - b
    prompts = [[STOI[ch] for ch in f"{x:03d}{op}{y:03d}="] for x, y in zip(a, b)]
    answers = [[STOI[ch] for ch in f"{z:04d}"[::-1] + "$"] for z in c]
    return torch.tensor(prompts), torch.tensor(answers)


def mixed(n: int, ops: list[str], rng: np.random.Generator) -> tuple[torch.Tensor, torch.Tensor, list[str]]:
    """n problems split evenly across subjects. The third value is each row's subject."""
    per = n // len(ops)
    parts = [problems(per, op, rng) for op in ops]
    subjects = [op for op in ops for _ in range(per)]
    return torch.cat([p for p, _ in parts]), torch.cat([a for _, a in parts]), subjects

rlsim.py, lines 45–61

• VOCAB has 14 characters and STOI maps character → id: 0–9 are 0–9, + 10, - 11, = 12, $ 13.
• f"{x:03d}{op}{y:03d}=" pads to three digits: 45 + 7 becomes 045+007=, so a problem is always 8 characters.
• f"{z:04d}"[::-1] + "$" pads to four digits and reverses, so an answer is always 5 characters.
• Subtraction puts the larger number first with np.maximum / np.minimum.
• mixed() splits n across subjects, addition in the first half and subtraction in the second. The third value, subjects, is each row’s subject tag; MOPD uses it to pick a teacher.

699+684=  →  [6, 9, 9, 10, 6, 8, 4, 12]   one row of prompts (8)
1383      →  reverse "1383" → "3831" + "$"
          →  [3, 8, 3, 1, 13]              one row of answers (5)

2. The verifier: correct()

def correct(seq: torch.Tensor, answers: torch.Tensor) -> torch.Tensor:
    """Verifier: 1 if the written answer matches the reference character for character, else 0."""
    return (seq[:, PROMPT_LEN:] == answers).all(1).float()

rlsim.py, lines 64–66

seq                   (256, 13)  8 prompt positions + 5 answer positions
seq[:, PROMPT_LEN:]   (256, 5)   the 5 answer positions
== answers            (256, 5)   True / False per position
.all(1)               (256,)     are all five True?
.float()              (256,)     1.0 or 0.0

The end marker $ is compared too: four correct digits followed by anything other than $ score 0.

3. The model: TinyGPT

class TinyGPT(nn.Module):
    def __init__(self, layers: int, width: int, heads: int = 4):
        super().__init__()
        self.tok = nn.Embedding(len(VOCAB), width)
        self.pos = nn.Embedding(SEQ_LEN, width)
        block = nn.TransformerEncoderLayer(
            width, heads, 4 * width, dropout=0.0, activation="gelu", batch_first=True, norm_first=True
        )
        self.blocks = nn.TransformerEncoder(block, layers, enable_nested_tensor=False)
        self.norm = nn.LayerNorm(width)
        self.head = nn.Linear(width, len(VOCAB))
        self.register_buffer("mask", nn.Transformer.generate_square_subsequent_mask(SEQ_LEN), persistent=False)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        t = x.shape[1]
        h = self.tok(x) + self.pos.weight[:t]
        h = self.blocks(h, mask=self.mask[:t, :t], is_causal=True)
        return self.head(self.norm(h))


def student() -> TinyGPT:
    return TinyGPT(layers=2, width=64)


def teacher() -> TinyGPT:
    return TinyGPT(layers=4, width=128)

rlsim.py, lines 72–97

• self.tok maps a character id to a 64-dim vector (token embedding); self.pos is a learned vector for each of positions 0–12 (position embedding). Their sum is the input.
• nn.TransformerEncoderLayer with a causal mask (generate_square_subsequent_mask) lets each position see only itself and earlier positions. Despite the “Encoder” name, this makes it a GPT-style decoder-only model.
• norm_first=True places LayerNorm before attention and the MLP (pre-LN, the standard since GPT-2). Dropout is 0.
• self.head outputs 14 scores (logits) per position: shape (batch, positions, 14).
• Student and teacher are the same class at different sizes. Sharing the vocabulary lets the teacher’s probabilities be compared with the student’s position by position and character by character, which is what makes OPD possible.

4. Picking out the answer predictions: answer_logits()

def answer_logits(model: TinyGPT, seq: torch.Tensor) -> torch.Tensor:
    """The prediction made just before each of the five answer characters (batch, 5, vocab)."""
    return model(seq)[:, PROMPT_LEN - 1 : -1]

rlsim.py, lines 100–102

A GPT’s output at position i predicts the character at position i+1. So the prediction for the first answer character (position 8) lives in the output at position 7 (=). The one-position shift is the key.

  0  1  2  3  4  5  6  7  8  9 10 11 12   ← position
  6  9  9  +  6  8  4  =  3  8  3  1  $   ← input character
  9  9  +  6  8  4  =  3  8  3  1  $  -   ← what this position's output predicts
                       ^  ^  ^  ^  ^      ← [:, PROMPT_LEN-1 : -1] = positions 7–11

model(seq) is (batch, 13, 14), and slicing [:, 7:12] gives (batch, 5, 14). Position 12 (after $) has nothing to predict and is dropped. SFT, RLVR and OPD all get “the student’s probabilities at the five answer positions” from this one function.

5. Writing an answer: write()

@torch.no_grad()
def write(model: TinyGPT, prompts: torch.Tensor, greedy: bool = False) -> torch.Tensor:
    """The model writes the answer one character at a time. greedy=False samples from its probabilities (temperature 1)."""
    seq = prompts
    for _ in range(ANSWER_LEN):
        logits = model(seq)[:, -1]
        nxt = logits.argmax(-1) if greedy else torch.multinomial(logits.softmax(-1), 1).squeeze(1)
        seq = torch.cat([seq, nxt[:, None]], 1)
    return seq


def accuracy(model: TinyGPT, test: tuple[torch.Tensor, torch.Tensor]) -> float:
    prompts, answers = test
    return correct(write(model, prompts, greedy=True), answers).mean().item()

rlsim.py, lines 105–118

• @torch.no_grad(): no gradients are recorded while writing. The probabilities needed for training are computed by reading the finished answer again with answer_logits(). Real RL frameworks do the same: an inference engine (vLLM, etc.) generates rollouts and the trainer re-reads them to compute probabilities.
• Five iterations: each feeds the whole seq so far and uses only the last position’s logits ([:, -1]). There is no KV cache, so the prefix is recomputed every time, which is fine at 13 positions.
• greedy=True → argmax: always the top character. Used for evaluation and for teacher answers.
• greedy=False → torch.multinomial(softmax): sample by probability (temperature 1). Used for student answers in RLVR and OPD. A character the student believes at 30% appears 30% of the time, which is why eight attempts at the same problem disagree.

6. The update: update()

def update(opt: torch.optim.Optimizer, model: TinyGPT, loss: torch.Tensor) -> None:
    opt.zero_grad()
    loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    opt.step()

rlsim.py, lines 124–128

All three methods move parameters with these four lines; only the loss differs. If the total gradient norm exceeds 1 it is scaled down to 1, so no single step jumps too far.

7. One step of SFT distillation: step_sft(), teacher_tokens()

def step_sft(model: TinyGPT, opt, prompts: torch.Tensor, targets: torch.Tensor) -> dict:
    """Copy the given answer character by character (cross-entropy)."""
    seq = torch.cat([prompts, targets], 1)
    loss = F.cross_entropy(answer_logits(model, seq).reshape(-1, len(VOCAB)), targets.reshape(-1))
    update(opt, model, loss)
    return {}


def teacher_tokens(prompts: torch.Tensor, subjects: list[str], teachers: dict[str, TinyGPT]) -> torch.Tensor:
    """For each subject, the model answer written by that subject's teacher (greedy)."""
    targets = torch.empty(len(prompts), ANSWER_LEN, dtype=torch.long)
    for op, model in teachers.items():
        rows = torch.tensor([s == op for s in subjects])
        if rows.any():
            targets[rows] = write(model, prompts[rows], greedy=True)[:, PROMPT_LEN:]
    return targets

rlsim.py, lines 131–146

• Line 143, rows: a mask that is True for rows of this teacher’s subject. Only those rows are written by this teacher (greedy). With two teachers the rows are split and written in two passes.
• Line 133, torch.cat: the teacher’s answer is appended to the problem, and the student predicts each next character while reading it. The prefix is always the teacher’s (teacher forcing), so the student never actually writes; it never gets to learn how to continue after its own mistake.
• Line 134, F.cross_entropy: the mean of −log p(teacher’s character) over 256 × 5 = 1,280 characters.
• No true answers: line 215 of post_train() passes teacher_tokens(...) instead of answers.

8. One step of RLVR: step_rlvr()

def step_rlvr(model: TinyGPT, opt, prompts: torch.Tensor, answers: torch.Tensor) -> dict:
    """GRPO: write 8 answers per problem; raise the probability of the characters in correct answers, lower it in wrong ones.
    If all 8 are right or all 8 are wrong there is nothing to compare, so that problem teaches nothing."""
    prompts, answers = prompts.repeat_interleave(GROUP, 0), answers.repeat_interleave(GROUP, 0)
    seq = write(model, prompts)
    reward = correct(seq, answers).view(-1, GROUP)
    advantage = (reward - reward.mean(1, keepdim=True)) / (reward.std(1, keepdim=True) + 1e-6)
    logp = answer_logits(model, seq).log_softmax(-1).gather(-1, seq[:, PROMPT_LEN:, None]).squeeze(-1)
    update(opt, model, -(advantage.view(-1, 1) * logp).mean())
    return {"signal": (reward.std(1) > 0).float().mean().item(), "reward": reward.mean().item()}

rlsim.py, lines 149–158

Line 152, repeat_interleave: eight copies of the same problem side by side. Easy to confuse with repeat:

problems           [A, B, C]
repeat_interleave  [A, A, B, B, C, C]   ← this one
repeat             [A, B, C, A, B, C]

Line 154’s .view(-1, 8) groups 8 consecutive rows as one problem, so this layout is what makes “one row = eight attempts at one problem”. With repeat, answers to different problems would be averaged together: a bug that fails silently, without an error.

Line 155, advantage: mean and std per problem. mean(1) and std(1) run across the eight attempts (columns). PyTorch’s std divides by n − 1 by default. With 2 of 8 right, as in 227 + 189, the mean is 0.25 and the std 0.463, so right answers get +1.62 and wrong ones −0.54; dividing by n would give a std of 0.433 and +1.73 / −0.58. If all eight are equal the std is 0; 1e-6 avoids dividing by zero, and the numerator is 0 too, so the advantage is exactly 0.

Line 156, logp: pick out the log-probability of each written character. log_softmax(-1) gives 14 log-probabilities per position, (256, 5, 14), and gather keeps only the character actually written: (256, 5).

Line 157, the loss. Multiplying the advantage (256, 1) by logp (256, 5) applies one answer’s advantage equally to its five characters. The minus sign is there because the optimizer reduces the loss: reducing it means increasing advantage × log p, so characters in answers with positive advantage gain probability and those with negative advantage lose it.

That is the policy gradient. The log-probability of a whole answer is the sum of its five characters’ log-probabilities. The gradient of the expected score is the average of score × ∇log p, and subtracting the group’s mean score (a baseline) leaves its expectation unchanged while reducing noise.

Real values: the eight answers to 227 + 189

A log-probability of −0.00 is a character the student is already almost 100% sure of. The gradient of log p is proportional to (1 − p), so such positions barely move however large the advantage. What moves is the hundreds place, where the student hesitated.

Where the signal survives once the eight are summed. Earlier I said the wrong 516 drags its right digits down too. For one answer in isolation, that’s true. But the loss sums all eight, and answers that wrote the same character after the same prefix add their advantages, because the same prefix means the same student probability.

• Ones: all eight wrote 6. +1.62 × 2 + (−0.54) × 6 = 0: exact cancellation. Advantages always sum to 0, so a character everyone wrote gets no signal.
• Tens: seven wrote 1 and only 906 wrote 0. 1 gets +0.54 and 0 gets −0.54.
• Hundreds (the seven with prefix “61”): 4 gets +3.24 and each of the other five digits −0.54. The signal concentrates where the answers diverge.
• Thousands and end marker: prefixes differ between answers so nothing cancels, but the student is already almost 100% sure, so they barely move.

So even a single pass/fail per answer, compared across several answers, reveals to some extent which position was the problem. But when all eight are wrong, as for 699 + 684, there is nothing to compare and this is 0 too.

Line 158, signal: the share of problems whose 8 scores have a nonzero std (i.e. were not all the same). That is the “problems with signal” in figure 3. It is only logged, never used for training.

9. One step of OPD and MOPD: step_opd()

def step_opd(model: TinyGPT, opt, prompts: torch.Tensor, subjects: list[str], teachers: dict[str, TinyGPT]) -> dict:
    """OPD: the student writes; keeping the student's own prefix, shrink the gap between the teacher's and the student's
    distribution at every character (reverse KL over the whole vocabulary). A different teacher per subject makes it MOPD."""
    seq = write(model, prompts)
    s = answer_logits(model, seq).log_softmax(-1)
    t = torch.empty_like(s)
    with torch.no_grad():
        for op, model_t in teachers.items():
            rows = torch.tensor([x == op for x in subjects])
            if rows.any():
                t[rows] = answer_logits(model_t, seq[rows]).log_softmax(-1)
    kl = (s.exp() * (s - t)).sum(-1)
    update(opt, model, kl.mean())
    return {"signal": 1.0, "kl": kl.mean().item()}

rlsim.py, lines 161–174

• Line 164: the student samples one answer per problem; no gradient recorded.
• Line 165: the student’s log-probabilities on re-reading its own answer. This is where the gradient flows.
• Lines 166–171: the teacher’s log-probabilities when reading the student’s answer. It only reads, so all five positions come out in one pass. Routing rows to a different teacher per subject with the rows mask makes it MOPD.
• Line 172: reverse KL = Σ over 14 characters of p_student × (log p_student − log p_teacher). s.exp() is the student’s probability and s - t the log-ratio. This is the exact sum over all 14 characters, not a one-sample estimate.
• Lines 173–174: the mean over 256 × 5 positions is the loss. Every position gets a signal, so signal is always 1.

OPD, MOPD and experiment 4’s control are the same function with a different teachers map:

OPD   (exp. 1, 2)   {"+": add}
OPD   (exp. 3)      {"+": add_weak}
MOPD  (exp. 4)      {"+": add, "-": sub}
control (exp. 4)    {"+": add, "-": add}   # subtraction problems read by the addition teacher too

The shape of the gradient. Differentiating by hand, the gradient of one position’s reverse KL with respect to its logit z_c is below. It matched autograd to within 1.2 × 10⁻⁷.

∂KL/∂z_c = p_student(c) × ( log p_student(c) − log p_teacher(c) − KL )

Gradient descent moves against the gradient, so in logit space characters whose log-ratio exceeds that position’s KL go down, and those below it go up. SFT’s gradient is p_student(c) − [c is the teacher’s character]: it raises only the teacher’s character and lowers the rest in proportion to the student’s probability.

Real values: the hundreds place of 699 + 684 (prefix 3, 8). Nudging the logits a tiny step along the gradient, here is the share of probability leaving (−) and arriving at (+) each character. The four non-digit characters are at 0% and omitted.

• SFT: 91% of the incoming probability goes to the correct 3. The outflow is proportional to the student’s probability: 1 (69%), 0 (18%), 2 (13%). The other 9% goes in small amounts to 4–9, which had little probability. Their logits go down too, but 1, 0 and 2 go down much more, so in probability they rise a little. Probability is shared among characters, so it is always relative.
• OPD: the outflow comes entirely from 1, the character the student actually wrote; the teacher gives it almost 0, so its log-ratio (13.75) is the largest. On the incoming side 3 gets the most (47%), but 2 rises too (34%). 0’s logit goes down, but 1 goes down so much more that 0’s probability barely changes (+2.5%).

Why does 2, which the teacher gives only 0.004%, go up? Its log-ratio (8.53) is below this position’s KL (9.48). OPD is moving probability off 1, the biggest problem right now, and 2, which already has a lot, takes a big share. Once 1 shrinks and the KL drops, 2 flips to going down. Below is a thought experiment moving only this position’s 14 logits directly, step size 0.5.

OPD clears 1 out first (36% → 19%) while 2 briefly grows (20% → 25%); from the second step 2 shrinks too and 3 rises. SFT raises 3 steadily from the start. The two losses have different gradient scales, so compare only the direction, not the speed per step. Real training also moves parameters with Adam, so logits don’t move independently like this.

10. The training loop: post_train()

def post_train(method: str, model: TinyGPT, plan: list[tuple[int, list[str]]], teachers: dict, tests: dict, seed: int) -> dict:
    """plan = [(steps, subjects), ...]; two stages make a cascade (finish one subject, then the next)."""
    torch.manual_seed(seed)
    rng = np.random.default_rng(seed)
    opt = torch.optim.Adam(model.parameters(), lr=LR)
    curve, signals, step, started = [], [], 0, time.time()
    schedule = [ops for steps, ops in plan for _ in range(steps)]
    for step in range(len(schedule) + 1):
        if step % LOG_EVERY == 0:
            point = {"step": step, **{op: accuracy(model, tests[op]) for op in tests}}
            point["signal"] = float(np.mean(signals)) if signals else None
            curve.append(point)
            signals = []
        if step == len(schedule):
            break
        ops = schedule[step]
        if method == "rlvr":
            prompts, answers, _ = mixed(BATCH // GROUP, ops, rng)
            stats = step_rlvr(model, opt, prompts, answers)
        else:
            prompts, answers, subjects = mixed(BATCH, ops, rng)
            if method == "sft":
                stats = step_sft(model, opt, prompts, teacher_tokens(prompts, subjects, teachers))
            else:
                stats = step_opd(model, opt, prompts, subjects, teachers)
        if "signal" in stats:
            signals.append(stats["signal"])
    return {"curve": curve, "seconds": round(time.time() - started, 1)}

rlsim.py, lines 193–220

• Line 199: expands plan into a per-step subject list. [(1000, ["+", "-"])] is 1,000 entries of ["+", "-"]; Cascade’s [(500, ["+"]), (500, ["-"])] is 500 of ["+"] followed by 500 of ["-"]. This one line is the whole difference between Mix-RL and Cascade-RL.
• Lines 201–205: every 20 steps (including step 0 and the last), measure greedy accuracy per subject and record the mean signal collected since. Each is one point on a chart: 51 points for 1,000 steps.
• Lines 209–211: RLVR draws BATCH // GROUP = 32 problems (16 per subject with two subjects).
• Lines 213–217: the others draw 256. SFT builds teacher answers for step_sft; everything else goes to step_opd. OPD and MOPD share the line.

11. Experiment definitions: experiment()

    if name == "multi":
        runs = {
            "mix-rlvr": ("rlvr", [(steps, ["+", "-"])], {}),
            "cascade-rlvr": ("rlvr", [(steps // 2, ["+"]), (steps // 2, ["-"])], {}),
            "mopd": ("opd", [(steps, ["+", "-"])], {"+": add, "-": sub}),
            "opd-add-teacher-only": ("opd", [(steps, ["+", "-"])], {"+": add, "-": add}),
        }
    else:
        runs = {
            "single": {
                "sft": ("sft", [(steps, ["+"])], {"+": add}),
                "rlvr": ("rlvr", [(steps, ["+"])], {}),
                "opd": ("opd", [(steps, ["+"])], {"+": add}),
            },
            "cold": {
                "rlvr": ("rlvr", [(steps, ["+"])], {}),
                "opd": ("opd", [(steps, ["+"])], {"+": add}),
            },
            "weak": {
                "rlvr": ("rlvr", [(steps, ["+"])], {}),
                "opd-weak": ("opd", [(steps, ["+"])], {"+": weak}),
                "sft-weak": ("sft", [(steps, ["+"])], {"+": weak}),
            },
        }[name]

rlsim.py, lines 303–326

A run is three things: (method, plan, teacher map). This table is the entire experimental design.

    for seed in range(seeds):
        if name == "cold":
            torch.manual_seed(seed)
            start, info = student(), {"steps": 0, "+": accuracy(student(), tests["+"])}
        else:
            start, info = warm_student(ops, seed)
        result["warm"].append(info)
        if name == "single" and seed == 0:
            result["example"] = example_card(start, add)
        for run, (method, plan, run_teachers) in runs.items():
            out = post_train(method, copy.deepcopy(start), plan, run_teachers, tests, seed)
            result["runs"].setdefault(run, []).append(out)

rlsim.py, lines 331–342

Each seed builds one starting student (a random model for experiment 2), and every method gets its own copy.deepcopy(start) for post_train(). Methods never share weights.

12. Teachers and the starting student: sft_until(), load_or_train_teachers(), warm_student()

def sft_until(model: TinyGPT, ops: list[str], target: float, rng, val: dict, check_every: int, max_steps: int) -> dict:
    """SFT on the true answers until accuracy reaches target (used to build teachers and the student's starting point)."""
    opt = torch.optim.Adam(model.parameters(), lr=1e-3)
    for step in range(max_steps + 1):
        if step % check_every == 0:
            accs = {op: accuracy(model, val[op]) for op in ops}
            if np.mean(list(accs.values())) >= target or step == max_steps:
                return {"steps": step, **accs}
        prompts, answers, _ = mixed(BATCH, ops, rng)
        step_sft(model, opt, prompts, answers)
    raise AssertionError

rlsim.py, lines 180–190

def load_or_train_teachers() -> dict[str, TinyGPT]:
    """An addition specialist, a subtraction specialist (each learns only its own subject), and an under-trained addition teacher (~70%)."""
    folder = OUT / "teachers"
    folder.mkdir(parents=True, exist_ok=True)
    specs = {"add": (["+"], 0.995, 1), "sub": (["-"], 0.995, 2), "add-weak": (["+"], 0.70, 3)}
    teachers, report = {}, {}
    for name, (ops, target, seed) in specs.items():
        torch.manual_seed(seed)
        model = teacher()
        path = folder / f"{name}.pt"
        if path.exists():
            model.load_state_dict(torch.load(path))
        else:
            started = time.time()
            info = sft_until(model, ops, target, np.random.default_rng(seed), val_sets(), 25 if target < 0.9 else 100, 30000)
            torch.save(model.state_dict(), path)
            print(f"teacher {name}: {info} in {time.time() - started:.0f}s", flush=True)
        teachers[name] = model
        report[name] = {op: accuracy(model, t) for op, t in test_sets().items()}
    (folder / "accuracy.json").write_text(json.dumps(report, indent=1))
    print("teacher test accuracy:", report, flush=True)
    return teachers


def warm_student(ops: list[str], seed: int) -> tuple[TinyGPT, dict]:
    torch.manual_seed(seed)
    model = student()
    info = sft_until(model, ops, WARM_TARGET, np.random.default_rng(1000 + seed), val_sets(), 5, 5000)
    return model, info

rlsim.py, lines 236–264

• sft_until(): SFT on true answers (learning rate 0.001). Every check_every steps it is scored on 1,000 validation problems (seed 999) and stops at the target. The stopping set and the charts’ evaluation set (seed 12345) are different.
• The three teachers: addition only to 99.5% (seed 1), subtraction only to 99.5% (seed 2), and the weak one: addition only to 70% (seed 3), checked every 25 steps, which first passed 70% at step 275. It scores 73% on the evaluation set.
• Saving: teachers are saved as results/teachers/*.pt and loaded instead of retrained on later runs.
• warm_student(): target 30%, checked every 5 steps. Weights are initialized with the experiment seed and problems drawn with 1000 + seed. Experiments 1 and 3 call the same function with the same seeds, so their starting students are identical.

13. How these lines change in real LLM training code

14. Known gaps

• Evaluation problems may have appeared in training. There are 1,000 × 1,000 = 1 million addition problems. SFT and OPD draw 256,000 at random over 1,000 steps, so a given evaluation problem appears at least once with probability 22.6% (3.1% for RLVR’s 32,000). A 100k-parameter model can’t memorize a million problems, so accuracy still measures whether the rule was learned, but strictly these aren’t “unseen” problems. Filtering evaluation problems out in problems() would fix it.
• Experiment 2’s recorded starting accuracy: line 334 calls student() twice, so the recorded 0% comes from a second, different random model rather than the actual starting model. Both are random models at 0%, so the results are unaffected.
• Subjects without a teacher: step_opd() would compute with uninitialized rows (torch.empty_like) for any subject missing from teachers. Every subject had a teacher here, so it never happened. A fix would stop with an error when a subject is missing.
• No safeguards: no KL(π ‖ π_ref) constraint, no replay of earlier subjects, no learning-rate schedule. This is why the cascade collapse may be exaggerated.

Glossary

Run it yourself

Python 3, PyTorch and NumPy are enough; it takes about 35 minutes on a CPU. Download rlsim.py.

# train the three teachers → experiments 1–4 (1,000 steps, 3 seeds) → experiment 3 again at 3,000 steps
python3 rlsim.py all --steps 1000
python3 rlsim.py weak --steps 3000

# quick check (1 seed, 100 steps, under a minute)
python3 rlsim.py single --seeds 1 --steps 100

rlsim.py reproduces experiments 1–4, and results land in results/*.json. Experiment 5, the per-digit traces and the charts came from small helper scripts (exp5.py, cold_digits.py, digit_order.py, trace.py, summary.py, charts.py) that build on rlsim.py and aren’t published here. The core of experiment 5 is the step_rlvr_variant function shown above. Experiment 6 is a separate, larger script (big/bigsim.py, run with big/orchestrate.py; about 3 h 40 min on two H100s), also not published. Note that the CLI’s default is --steps 500; the experiments here used 1,000.

A three-digit addition/subtraction toy, built to see the direction of real LLM post-training with your own eyes. The numbers hold only for this setup.

장난감 실험이에요. CPU 만으로 약 35분, 실행마다 시드 3개예요. 전체 코드는 PyTorch 와 NumPy 만 쓰는 364줄짜리 파일 하나, rlsim.py 예요.

한눈에 보기

실험은 여섯 개예요. 실험 1~5 는 모두 같은 작은 학생 모델로, 한 스텝에 답안 256장을 쓰는 같은 조건에서 비교했어요. 숫자는 평가용으로 따로 고정한 문제 2,000개에서 잰 정답률이에요(학습 문제와 따로 뽑지만 우연히 겹칠 수는 있어요. 허술한 점). 실험 6 은 GPU 에서 백만 단위 사칙연산으로 키운 거예요.

왜 이 실험을 했나 — 요즘 LLM 학습 순서

2025~2026년 공개 모델들의 학습 순서는 대체로 아래와 같아요. 이 글은 그중 파란 칸, 곧 RL 과 증류(OPD·MOPD) 를 아주 작게 만들어서 하나씩 돌려 본 거예요.

그림 1. 요즘 학습 순서. 파란 칸이 이 실험에서 돌려 본 부분이에요.

예를 들어 Xiaomi MiMo-V2-Flash 는 “SFT → 과목별 RL 로 선생 만들기 → 다중 선생 OPD” 세 단계라고 밝혔어요. DeepSeek-V4 는 V3.2 에서 쓰던 “여러 과목을 섞은 RL” 단계를 OPD 로 통째로 바꿨다고 해요(블로그 분석 기준이라 원문은 직접 확인하지 않았어요). 실험 4 가 바로 이 두 방식을 비교해요.

실험 무대 — 문제, 채점기, 모델

문제: 세 자리 덧셈·뺄셈, 답은 1의 자리부터

모델은 699+684= 같은 문제를 받고 답을 한 글자씩 써요. 답은 1의 자리부터 네 자리를 쓰고, 끝에 $(끝 표시)를 붙여요. 1383 이면 3831$ 이에요.

1의 자리부터 쓰는 건 손으로 세로셈할 때 순서예요. 1의 자리를 먼저 써야 받아올림을 알고, 그걸 알아야 10의 자리를 쓸 수 있어요. 그래서 이 다섯 글자가 아주 작은 추론 과정이에요. 앞 글자를 틀리면 뒤 글자도 흔들려요.

모델이 쓸 수 있는 글자는 0~9, +, -, =, $ 이렇게 14개예요. 뺄셈은 항상 큰 수에서 작은 수를 빼서 답이 음수가 아니에요.

채점기(검증기): 다섯 글자가 정답과 전부 같으면 1점

RLVR 의 “V” 가 이 채점기예요(Verifiable reward, 기계로 확인되는 보상). 부분 점수는 없어요. 1383 을 1183 이라고 쓰면, 네 자리 중 세 자리가 맞아도 0점이에요.

# rlsim.py — 채점기
def correct(seq, answers):
    return (seq[:, PROMPT_LEN:] == answers).all(1).float()   # 다 같으면 1, 아니면 0

모델: 작은 학생 한 명, 큰 선생 세 명

둘 다 GPT 와 같은 구조(트랜스포머)를 아주 작게 줄인 거예요. 선생이 학생보다 약 8배 커요.

선생 세 명은 정답으로 SFT 해서 만들었어요(자세히). 각자 자기 과목만 배웠기 때문에 다른 과목은 거의 0% 예요. 이게 실험 4 에서 중요해져요.

선생은 이렇게 배웠어요 — 진짜 정답으로 SFT

선생은 RL 도 증류도 쓰지 않았어요. 이 장난감 문제는 코드가 정답(a + b, a − b)을 공짜로 계산해 주니까, 정답 답안을 따라 쓰게만 하면 됐어요. 코드로는 load_or_train_teachers() 가 sft_until() 을 부르는 게 전부예요 (코드 해설 12).

# 이름: (배우는 과목, 목표 정답률, 시드)
specs = {"add": (["+"], 0.995, 1), "sub": (["-"], 0.995, 2), "add-weak": (["+"], 0.70, 3)}

# 목표에 닿을 때까지 반복
prompts, answers, _ = mixed(BATCH, ops, rng)   # 자기 과목 무작위 256문제 + 진짜 정답
step_sft(model, opt, prompts, answers)         # 정답을 글자마다 따라 쓰기 (cross-entropy)

1. 모델: 4층 · 폭 128 TinyGPT 를 무작위로 초기화해요. 798,606 파라미터로 학생의 약 8배예요.
2. 학습: 한 스텝에 자기 과목 문제 256개를 새로 뽑고, 정답 다섯 글자를 teacher forcing 으로 따라 쓰게 해요. Adam, 학습률 0.001 이에요.
3. 멈추기: 일정 스텝마다 검증용 1,000문제(시드 999)로 정답률을 재서 목표를 넘으면 멈춰요. 그래프용 평가 문제(시드 12345)와는 따로예요.
4. 저장: results/teachers/*.pt 로 저장하고, 실험에서는 불러다 쓰기만 해요. 실험 중에 선생은 전혀 학습하지 않아요.

• 약한 선생은 일부러 덜 가르쳤어요. 방법은 덧셈 선생과 같고, 70% 를 넘자마자 멈춘 것만 달라요. 그래서 실수가 문제마다 들쭉날쭉해요. 실험 3 에서 학생이 처음엔 이 선생을 넘은 이유예요.
• 자기 과목 말고는 0% 예요. 덧셈 선생은 학습 중에 - 글자를 한 번도 못 봤어요. 그 글자의 임베딩은 기울기를 한 번도 받지 않아 무작위 초깃값 그대로예요. 뺄셈 문제를 주면 548 − 286 을 834 로, 곧 더하기로 풀어요.
• 학생의 출발점(34%)도 같은 함수로 만들었어요. 목표만 30% 로 낮췄어요.

실험의 “SFT 증류” 와 같은 함수(step_sft)예요. 넘기는 답만 달라요.

학생의 출발점: 정답률 약 34%

실제 LLM 은 RL 을 시작할 때 이미 사전학습과 SFT 를 마친 상태예요. 그래서 학생도 정답으로 조금 SFT 해서 정답률이 30% 를 넘으면 멈췄어요. 시드마다 380~1,115스텝이 걸렸고, 평가 문제 기준으로 평균 34% 에서 출발해요. 실험 2 만 예외로, 아무것도 안 배운 0% 학생에서 출발해요.

공정하게 비교하려고 맞춘 것

학습법 네 가지를 한 문제로 하나씩

같은 문제 하나로 네 방법이 무엇을 가르치는지 봐요. 문제는 699 + 684, 정답은 1383 이에요. 34% 출발점 학생(시드 0)은 이 문제를 1183 이라고 써요. 100의 자리 하나만 틀렸어요.

① SFT 증류 — 모범답안 베껴 쓰기

선생이 먼저 문제를 풀어 모범답안을 써요. 학생은 그 답안을 한 글자씩 따라 써요. 이때 학생은 선생이 쓴 앞부분을 보면서 다음 글자를 맞혀요. 학생이 직접 푸는 단계가 없어요.

벌점은 “선생이 쓴 글자에 학생이 준 확률” 이 낮을수록 커요(−log 확률, cross-entropy). 아래는 선생 답안 1383 을 따라 쓸 때의 벌점이에요.

100의 자리에서 학생은 정답 3 에 10.6% 만 걸어서 벌점이 가장 커요. 학습은 이 벌점을 줄이는 쪽, 곧 “이 자리에선 3” 쪽으로 학생을 밀어요.

# rlsim.py 의 step_sft 를 줄여 옮긴 것
targets = teacher_tokens(prompts, subjects, teachers)      # 선생이 쓴 모범답안
seq = torch.cat([prompts, targets], 1)
loss = F.cross_entropy(answer_logits(model, seq), targets)  # 글자마다 −log 확률

줄 단위 해설과 텐서 모양은 코드 해설 7에 있어요.

② RLVR (GRPO) — 같은 문제를 8번 풀고 O/X 만 받기

학생이 같은 문제를 8번 직접 풀어요. 채점기는 답안마다 O/X 하나만 줘요. 그리고 8장의 평균과 비교해요. 평균보다 잘한 답안은 그 글자들의 확률을 올리고, 평균보다 못한 답안은 내려요. 이게 GRPO 예요(같은 문제의 답안끼리 비교하는 RL).

얼마나 올리고 내릴지는 advantage(이점)로 정해요. advantage = (그 답안 점수 − 8장 평균) ÷ 8장 표준편차예요.

예 A — 699 + 684 를 8번 풀었더니 전부 틀렸어요.

8장이 전부 0점이라 평균도 0, 차이도 0 이에요. advantage 가 전부 0 이라서 이 문제에서는 아무것도 배우지 못해요. 어느 자리가 틀렸는지도 몰라요.

예 B — 227 + 189 (정답 416) 는 8번 중 2번 맞혔어요.

+1.62 와 −0.54 는 이렇게 나와요. 같은 문제 8장 안에서 그 답안이 평균보다 얼마나 잘했는지를 재요.

부호(+/−)는 ③ 에서, 크기는 ⑤ 에서 정해져요. ④ 에서 8 이 아니라 7 로 나누는 건 PyTorch std() 의 기본값이에요. 8 로 나누면 표준편차가 0.433 이라 +1.73 / −0.58 이 돼요.

그래서 맞은 두 장은 +1.62 만큼 밀어 올리고, 틀린 여섯 장은 −0.54 만큼 밀어 내려요. 그런데 틀린 516 도 뒤 두 자리 “16” 은 맞았어요. 그래도 답안 통째로 −0.54 라서 맞은 글자까지 같이 내려가요. O/X 하나로는 글자를 가려 주지 못해요. 다만 8장을 합치면 조금 달라져요(코드 해설 8).

아래에서 직접 O/X 를 바꿔 보세요. 맞은 장수에 따라 학습 신호가 어떻게 바뀌는지 보여요.

맞은 개수별로 정리하면. 점수가 0 또는 1 뿐이라 8장 중 맞은 개수 하나로 숫자가 다 정해져요. 위 계산기와 같은 식이에요.

• 드문 쪽이 크게 움직여요. 1장만 맞히면 그 한 장이 +2.47 로 크게 올라가요. 7장 맞히고 1장 틀리면 그 한 장이 −2.47 로 크게 내려가요.
• 합은 항상 0 이에요. 2장 맞힘이면 +1.62 × 2 − 0.54 × 6 = 0 이에요. 올리는 양과 내리는 양이 같아서, 8장이 모두 똑같이 쓴 글자는 상쇄돼요 (코드 해설 8).
• 8장이 다 맞거나 다 틀리면 0 이에요. ③ 의 차이가 전부 0 이라 비교할 대상이 없어요. 그래서 RLVR 은 너무 어려운 문제에서도, 너무 쉬운 문제에서도 배우지 못해요.
• 틀린 정도는 반영되지 않아요. 516(한 자리 틀림)과 906(두 자리 틀림)이 똑같이 −0.54 예요. 점수가 0/1 뿐이라서예요. 부분 점수를 주는 채점기였다면 답안마다 값이 달라져요.

이 숫자가 실제로 하는 일. 손실은 −advantage × log 확률(그 답안) 이에요. 이걸 줄이는 방향으로 학습하니까 +1.62 는 그 답안이 나올 확률을 1.62 의 세기로 올리고, −0.54 는 0.54 의 세기로 내려요. 여기에 학습률이 곱해지고 256장 × 5글자로 평균을 내기 때문에 확률이 1.62 만큼 바뀐다는 뜻은 아니에요. 답안끼리의 상대적인 세기예요.

⑤ 의 표준편차 나누기는 논란이 있어요. 1장만 맞힌 아주 어려운 문제나 7장 맞힌 아주 쉬운 문제는 표준편차가 작아서, 나누면 큰 값이 몰려요. Dr. GRPO 는 이 나누기를 빼고 ③ 의 차이(+0.75 / −0.25)를 그대로 써요.

# rlsim.py 의 step_rlvr 를 줄여 옮긴 것
seq = write(model, prompts.repeat_interleave(8, 0))          # 문제마다 8번 풀기
reward = correct(seq, answers).view(-1, 8)                   # O/X
advantage = (reward - reward.mean(1, keepdim=True)) / (reward.std(1, keepdim=True) + 1e-6)
logp = log_prob_of_written_tokens(model, seq)                # 쓴 글자들의 log 확률
loss = -(advantage.view(-1, 1) * logp).mean()                # 잘한 답은 올리고, 못한 답은 내리고

줄 단위 해설, 이 8장의 실제 log 확률, 그리고 8장을 합치면 신호가 어느 자리에 남는지는 코드 해설 8에 있어요.

③ OPD — 내가 푼 답안에 선생의 빨간 펜

OPD(On-Policy Distillation)는 학생이 직접 풀어요. 그다음 선생이 학생 답안을 처음부터 읽어 가면서, 글자마다 “이 앞부분 다음에 나라면 뭘 쓸지” 확률을 줘요. 학생은 자기 확률이 선생 확률과 비슷해지도록 배워요. 선생은 진짜 정답으로 미리 SFT 해 둔 더 큰 모델이에요(선생은 이렇게 배웠어요).

아래는 학생이 쓴 1183 에 선생이 매긴 빨간 펜이에요. 각 칸은 그 자리에서 학생과 선생이 어떤 글자에 확률을 얼마나 걸었는지예요(상위 3개).

벌점은 reverse KL 이에요. 쉽게 말하면 “학생이 확률을 건 글자마다, 선생은 거기에 얼마나 걸었나” 를 봐요. 학생이 많이 걸었는데 선생은 거의 안 건 글자가 있을수록 벌점이 커요. 식으로는 Σ 학생확률(c) × log(학생확률(c) ÷ 선생확률(c)) 예요.

• 100의 자리 (벌점 9.48): 학생은 1 에 36% 를 걸었는데 선생은 1 에 거의 0% 예요. 선생은 3 에 99% 를 걸어요. 틀린 자리가 정확히 짚여요.
• 10의 자리 (벌점 1.34): 8 로 맞게 썼는데도 벌점이 있어요. 학생이 7 에도 22% 를 걸었는데 선생은 7 에 0% 라서요. 맞았지만 자신 없는 것까지 고쳐 줘요. RLVR 은 이걸 못 해요.
• 나머지 자리 (0.01): 학생과 선생이 거의 같아서 벌점이 없어요.

“On-policy” 는 학생이 자기가 실제로 쓴 답안으로 배운다는 뜻이에요. SFT 증류는 선생이 쓴 앞부분만 보지만, OPD 는 학생이 실수한 앞부분 뒤에서도 “여기서부터는 이렇게” 를 배워요. 대신 답안마다 선생 모델을 한 번 더 돌려야 해요.

# rlsim.py 의 step_opd 를 줄여 옮긴 것
seq = write(model, prompts)                                   # 학생이 직접 풀기 (한 번)
s = answer_logits(model, seq).log_softmax(-1)                 # 글자마다 학생 확률
t = answer_logits(teacher, seq).log_softmax(-1)               # 같은 답안을 선생이 읽은 확률
loss = (s.exp() * (s - t)).sum(-1).mean()                     # reverse KL, 글자 14개 전부

기울기 식과 100의 자리에서 확률이 어디로 옮겨 가는지는 코드 해설 9에 있어요.

④ MOPD — 과목마다 그 과목 선생의 빨간 펜

MOPD(Multi-teacher OPD)는 OPD 와 똑같은데, 문제마다 과목표를 보고 그 과목 선생에게 보내요. 덧셈 문제는 덧셈 선생이, 뺄셈 문제는 뺄셈 선생이 빨간 펜을 잡아요.

뺄셈 문제 548 − 286(정답 262)를 두 과목 출발점 학생이 252 이라고 썼어요. 같은 답안에 두 선생이 각각 빨간 펜을 들면 이렇게 달라요.

뺄셈 선생은 틀린 10의 자리만 6 으로 고쳐 줘요. 그런데 덧셈 선생에게 이 답안을 맡기면 모든 자리를 다르게 고쳐요. 덧셈 선생이 이 문제를 혼자 풀면 834 을 써요. 548 + 286 이에요. 빼기 기호를 본 적이 없어서 무시하고 더해요. 이런 선생에게 뺄셈을 맡기면 학생에게 “더하라” 고 가르치게 돼요(실험 4).

# rlsim.py — MOPD 는 OPD 에 선생 목록만 과목별로 넘겨요
step_opd(model, opt, prompts, subjects, teachers={"+": add_teacher, "-": sub_teacher})

세 방법의 학습 환경 — 무엇을 보고, 누가 쓰고, 무엇을 돌려받나

세 방법은 같은 “게임” 위에서 돌아요. 문제를 보고 글자를 하나씩 다섯 개 쓰면 한 판이 끝나요. 방법마다 다른 건 답안을 누가 쓰는지와 무엇을 돌려받는지 두 가지예요.

게임의 규칙 — RL 용어로

방법별 환경

• SFT 가 가장 느린 이유: 학생보다 8배 큰 선생이 매 스텝 답을 써요. 쓰기는 글자 수만큼(5번) 모델을 돌려야 해요. OPD 의 선생은 학생이 이미 쓴 답안을 읽기만 해서 한 번에 다섯 칸이 다 나와요. 실제 LLM 에서도 OPD 의 선생 비용은 생성이 아니라 읽기(prefill)라서 상대적으로 싸요.
• SFT 와 OPD 는 정답을 쓰지 않아요. mixed() 가 정답도 함께 돌려주지만 두 방법은 버려요. 선생만 있으면 되고, 그래서 실험 3 처럼 선생이 틀리면 학생도 그대로 배워요. RLVR 은 반대로 선생 없이 정답(채점기)만 있으면 돼요.
• RLVR 만 같은 문제를 여러 번 풀어요. 한 장의 O/X 로는 “잘했나” 를 알 수 없고, 같은 문제의 다른 답안과 비교해야 해서예요. 그래서 같은 256장이라도 문제 수는 32개로 줄어요.

모든 방법이 똑같이 쓰는 것

• 출발 가중치: 시드마다 출발점 학생을 한 번 만들고, 방법마다 copy.deepcopy(start) 로 복사해서 써요. 세 방법이 정확히 같은 가중치에서 출발해요.
• 옵티마이저: 방법마다 새 Adam, 학습률 0.0005. 기울기 전체 길이가 1 을 넘으면 1 로 잘라요 (clip_grad_norm_).
• 문제 순서: 방법마다 같은 시드로 난수 생성기를 새로 만들어요. SFT 와 OPD 는 한 스텝에 256문제씩 뽑아서 똑같은 문제를 같은 순서로 봐요. RLVR 은 32문제씩 뽑아서 문제 순서가 달라요.
• 평가: 20스텝마다, 시드 12345 로 고정한 과목별 2,000문제를 greedy 로 풀게 해서 정답률을 재요. 평가는 학습에 영향을 주지 않아요.

실험마다 바뀐 것

실행 환경

Apple M4 Max 노트북의 CPU 만 썼어요(GPU 없음). Python 3.11, PyTorch 2.10, NumPy 2.3 이고, torch.set_num_threads(4) 로 스레드 4개만 써요. 선생 셋을 만드는 데 1분 안쪽, 실험 넷에 약 35분 걸렸어요. 외부 데이터나 사전학습 가중치는 하나도 안 써요. 문제는 코드가 그 자리에서 만들어요.

실험 1 — 34% 에서 출발한 학생

알고 싶은 것: 출발점이 같을 때 어느 방법이 빨리 배우나요?
설정: 34% 학생, 덧셈만, 1,000스텝. SFT 증류와 OPD 는 덧셈 선생(99.8%)에게 배워요.

그림 2. 덧셈 정답률. 선 위에 마우스를 올리면 그 스텝의 값이 보여요.

왜 이렇게 됐나: 한 스텝에 받는 정보량이 달라요. RLVR 은 256장을 쓰고도 받는 건 “32문제 각각에서 몇 장이 맞았나” 뿐이에요. OPD 는 답안 한 장에서 다섯 글자 각각의 확률 분포(14가지)를 받아요.

게다가 RLVR 은 배울 거리가 있는 문제가 갈수록 줄었어요. 처음에는 61% 였는데 1,000스텝 무렵엔 24% 예요. 처음엔 8번 다 틀리는 어려운 문제가, 나중엔 8번 다 맞히는 쉬운 문제가 신호 0 이에요(그림 3 오른쪽).

이 실험에서 SFT 증류가 OPD 보다 조금 빨랐어요. 답이 다섯 글자뿐이라 “자기 실수에서 이어 쓰기” 를 배울 일이 거의 없어서예요. 이 점은 같은 점, 다른 점에서 다시 다뤄요.

실험 2 — 아무것도 모르는 학생

알고 싶은 것: 아무것도 모르는 학생도 각 방법으로 배울 수 있나요?
설정: 무작위로 초기화한 학생(0%), 덧셈만, 1,000스텝.

그림 3. 왼쪽: 0% 학생의 정답률. 오른쪽: RLVR 에서 8장이 다 같지 않아 신호가 있던 문제의 비율.

왜 이렇게 됐나: 아무것도 모르는 학생이 다섯 글자를 우연히 다 맞힐 확률은 대략 (1/14)⁵, 약 54만분의 1 이에요. 1,000스텝 동안 256,000장을 써도 맞을 거라 기대되는 답안이 0.5장이에요. 8장이 매번 다 틀리니 신호가 계속 0 이고, 학생은 그대로예요.

OPD 는 틀린 답안에도 글자마다 “여기엔 이 숫자” 를 알려 줘서 바닥부터 배워요.

OPD 는 왜 400~500스텝에 갑자기 오르나

처음 400스텝은 거의 0% 였다가 500스텝 무렵 갑자기 올라요. 시드 0 을 똑같이 다시 돌리면서(정답률 곡선이 원래와 똑같이 재현돼요) 답의 자리별 정답률을 따로 재 봤어요.

평가 문제 2,000개, greedy. KL 은 그 직전 20스텝의 평균 손실이에요.

• 쉬운 자리부터 배워요. 끝 표시 $ 는 100스텝 만에 100% 예요. 1000의 자리(0 아니면 1)도 금방 70% 대에 와요. 계산이 거의 필요 없는 자리예요.
• 계산이 필요한 자리는 한동안 10% 예요. 1 · 10 · 100의 자리는 300스텝까지 10% 안팎이에요. 숫자 10개 중 아무거나 찍는 수준이에요. KL 도 4.3~4.6 에서 거의 안 내려가요. 문제의 어느 칸끼리 더해야 하는지를 아직 못 찾은 상태로 보여요.
• 덧셈은 세로셈 순서대로 풀려요. 1의 자리가 300 → 400스텝에 30% → 90%, 이어서 400 → 500스텝에 10의 자리가 26% → 90%, 100의 자리가 10% → 81% 로 올라요. 받아올림이 없는 1의 자리가 먼저고, 앞 자리 받아올림이 필요한 자리가 뒤따라요. 뺄셈은 순서가 달라요(아래).
• 전체 정답은 다섯 자리가 다 맞아야 해요. 400스텝에 1의 자리는 90% 인데 전체 정답은 2.3% 예요. 나머지 자리가 아직 찍기 수준이라서요. 전체 정답률은 자리별 정답률을 곱한 값과 거의 같아요(500스텝: 곱 71%, 실제 73%).

정리하면 두 가지가 겹쳐요. 하나는 자리 규칙 자체를 한동안 못 찾다가 찾으면 빠르게 배우는 거예요. 자리별 정답률도 실제로 뛰어요. 다른 하나는 다섯 글자가 다 맞아야 1점인 채점이에요. 자리마다 차례로 배우는 과정이 전체 정답률로는 “가만히 있다가 한 번에 뛰는” 모양으로 보이게 해요. 앞 칸에 정답을 넣어 줘도(teacher forcing) 자리별 정답률이 거의 같아서, 앞 칸 실수가 뒤로 번진 탓은 아니에요. 34% 에서 출발한 실험 1 은 이미 규칙을 어느 정도 알고 시작해서 이런 정체 구간이 없어요.

뺄셈은? — 순서를 정하는 건 “계산이 얼마나 복잡한가”

뺄셈은 1의 자리부터가 아니에요. 그렇다고 가로(읽는) 순서도 아니에요. OPD 와 정답 SFT 의 시드 6개 모두 이렇게 됐어요.

• 100의 자리가 가장 먼저 반을 넘어요. 200~220스텝에 50% 를 넘고, 그 뒤 60~70% 대에서 한동안 멈춰요.
• 완성(90%)은 1의 자리가 먼저예요. 1의 자리가 420~940스텝에 90% 를 넘고, 100의 자리가 그 뒤예요. 10의 자리가 가장 늦어요. OPD 는 1,000스텝 안에 10의 자리가 90% 를 못 넘었어요.
• 뺄셈이 덧셈보다 어려워요. OPD 1,000스텝 전체 정답이 뺄셈 72.3~83.1%, 덧셈 93.5~98.4% 예요.

시드 3개 평균이에요. (2/3) 은 시드 셋 중 둘만 닿았다는 뜻이고, 평균은 닿은 시드로만 냈어요.

자리마다 필요한 계산을 나눠 보면 이유가 보여요. 아래에서 “한 바퀴” 는 10 을 넘거나 0 아래로 내려가서 한 바퀴 돌아오는 계산이에요(7 + 5 → 2, 3 − 8 → 5).

이 실험의 뺄셈은 늘 큰 수에서 작은 수를 빼서, 100의 자리끼리 빼면 0~9 가 바로 나와요. 한 바퀴 돌 일이 없어요. 그래서 받아내림을 무시하고 “a 의 100의 자리 − b 의 100의 자리” 만 해도 절반 넘게 맞아요. 60~70% 에 멈춘 게 이 단계로 보여요. 받아내림까지 맞히려면 아래 자리를 봐야 해서, 1의 자리를 배운 뒤에야 90% 를 넘어요. 10의 자리는 한 바퀴와 받아내림이 둘 다 필요해서 가장 늦어요.

덧셈의 1 → 10 → 100 순서도 세로셈 순서라서라기보다, 그 순서가 계산이 쉬운 순서와 같아서로 보여요. 모델은 쓰는 순서가 아니라 쉬운 계산부터 배워요.

답을 1000의 자리부터 쓰면? — 훨씬 못 배워요

그럼 쓰는 순서는 상관없을까요? 답을 읽는 순서(1000의 자리부터)로 쓰게 바꿔서, 1의 자리부터 쓰는 지금 형식과 비교했어요. 이 형식에 맞는 선생이 없어서 정답 SFT 로, 0% 학생에서 2,000스텝 돌렸어요(시드 3개, digit_order.py).

그림 3-3. 정답 SFT 2,000스텝 뒤 자리별·전체 정답률(시드 3개 평균). 1000의 자리는 모든 형식에서 100% 라 뺐어요.

• 1의 자리부터 쓰면 2,000스텝에 덧셈 100%, 뺄셈 99.5~100% 예요.
• 1000의 자리부터 쓰면 덧셈 8~23%, 뺄셈 3~33% 예요. 1000·100의 자리는 금방 배워요(90% 도달: 1000의 자리 0~100스텝, 100의 자리 420~920스텝). 10의 자리와 1의 자리는 시드 6개 중 하나씩만 2,000스텝 안에 90% 에 닿았어요.

1의 자리부터 쓰면 모델이 앞에서 직접 쓴 낮은 자리가 다음 자리를 계산하는 메모가 돼요. 예를 들어 a 의 1의 자리가 7 인데 답의 1의 자리에 2 를 썼다면 받아올림이 있었다는 걸 바로 알 수 있어요(7 + 5 = 12). 1000의 자리부터 쓰면 이 메모 없이 받아올림을 머릿속에서 한 번에 계산해야 해요. 이 형식에서 100의 자리가 93~97% 에서 더 오르지 않은 것도 받아올림을 10의 자리만 보고 어림한 결과로 보여요. 덧셈이라면 10의 자리끼리 합이 딱 9 일 때만 1의 자리가 받아올림을 결정하는데, 그런 문제가 약 4.5% 라 어림만으로 95% 안팎까지 가요. 작은 트랜스포머에게 덧셈 답을 거꾸로(1의 자리부터) 쓰게 하면 훨씬 잘 배운다는 연구 결과(Lee 등, 2023)와 같은 방향이에요.

다만 1000의 자리부터 쓸 때 계산이 가장 쉬운 1의 자리가 가장 늦게 배워지는 이유는 확인하지 못했어요.

실험 3 — 선생이 약하면

알고 싶은 것: 선생이 틀리는 선생이면 학생은 어디까지 가나요?
설정: 34% 학생, 덧셈만, 3,000스텝. SFT 증류와 OPD 는 약한 덧셈 선생(73%)에게 배워요. RLVR 은 선생 없이 채점기만 써요.

그림 4. 덧셈 정답률. 점선은 약한 선생의 실력(73%)이에요.

처음엔 학생이 선생을 넘었어요. 73% 선생에게 배운 학생이 84% 까지 갔어요. 선생의 실수가 들쭉날쭉해서예요. 문제마다 다른 자리에서 틀리니까, 학생은 실수보다 공통된 규칙을 먼저 배워요.

오래 배우면 다시 내려왔어요. 학생이 선생의 확률을 점점 정확히 닮으면서 선생의 실수까지 닮아 79% 로 내려와요. 증류는 결국 “선생을 닮아라” 라서 선생이 천장이에요.

RLVR 은 느리지만 천장이 없어요. 기준이 선생이 아니라 진짜 정답이라서요. 1,000스텝 무렵 증류를 따라잡고, 1,660스텝에 90% 를 넘었어요.

실험 4 — 과목이 둘이면

알고 싶은 것: 학생 한 명이 두 과목을 다 잘하게 하려면 어떻게 하나요?
설정: 두 과목을 조금씩 배운 학생(덧셈 34%, 뺄셈 30%), 1,000스텝, 네 가지 방법.

• Mix-RL — 한 스텝에 덧셈 16문제와 뺄셈 16문제를 섞어 각 8번씩 풀고 O/X.
• Cascade-RL — 덧셈만 500스텝 RL 한 다음, 뺄셈만 500스텝 RL.
• MOPD — 한 스텝에 덧셈 128문제는 덧셈 선생이, 뺄셈 128문제는 뺄셈 선생이 빨간 펜.
• 덧셈 선생 혼자 — 같은 OPD 인데 두 과목 다 덧셈 선생이 빨간 펜.

그림 5. 두 과목 정답률. 세로 점선(500스텝)에서 Cascade-RL 이 덧셈에서 뺄셈으로 넘어가요.

MOPD — 과목마다 맞는 선생

두 과목 모두 1,000스텝에 97% 를 넘었어요. 90% 에 닿은 건 덧셈 480스텝, 뺄셈 540스텝이에요. 과목마다 그 과목을 아는 선생이 글자마다 고쳐 주니까 서로 방해하지 않아요.

덧셈 선생 혼자 — 모르는 과목은 가르칠 수 없어요

덧셈은 99% 인데 뺄셈은 처음 20스텝 만에 30% 에서 3% 로 떨어지고 0% 가 됐어요. 앞에서 본 것처럼 덧셈 선생은 548 − 286 을 834 라고 써요. 학생은 그 빨간 펜을 충실히 따라서 뺄셈 문제도 더했어요. MOPD 가 과목마다 선생을 따로 두는 이유예요.

Mix-RL — 섞으면 서로 흔들어요

처음 80스텝 동안 덧셈은 34% 에서 18% 로, 뺄셈은 30% 에서 21% 로 오히려 떨어졌어요. 한 과목의 맞은 답을 강화한 게 같은 파라미터를 쓰는 다른 과목을 흔든 거예요(시소 효과). 과목당 문제도 한 스텝에 16개로 절반이라 신호도 적어요. 그 뒤 천천히 올라 1,000스텝에 45% 였어요.

Cascade-RL — 하나를 배우면 다른 하나를 잊어요

• 0~500스텝 (덧셈만): 덧셈은 34% → 67% 로 올랐지만, 뺄셈은 30% → 3.5% 로 떨어졌어요. 뺄셈은 한 번도 안 했는데 잊은 거예요(파괴적 망각).
• 500스텝 (뺄셈으로 전환): 이미 뺄셈을 거의 못 해서 8번 다 틀리는 문제가 대부분이에요. 배울 거리가 있는 문제가 덧셈 단계의 58.5% 에서 4.9% 로 줄었어요.
• 500~600스텝: 덧셈도 67% → 6.5% 로 무너졌어요. 드물게 오는 뺄셈 신호로 파라미터가 계속 움직이면서 덧셈까지 흔든 것으로 보여요. 정확한 원인은 따로 확인하지 않았어요.

실험 5 — 0% 학생에게 RLVR 을 시작시키려면

알고 싶은 것: 실험 2 에서 RLVR 은 0% 학생에게서 신호를 하나도 못 받았어요. 무엇을 바꾸면 배우기 시작하나요?
설정: 실험 2 와 같은 무작위 학생(0%), 덧셈만, 스텝마다 답안 256장, 시드 3개예요. 다섯 방법을 1,000스텝으로 비교하고, 배우기 시작한 셋은 3,000스텝까지 더 돌렸어요.

def step_rlvr_variant(model, opt, prompts, answers, method: str) -> dict:
    """rlsim.step_rlvr 와 같은 GRPO 한 스텝 + 변형 스위치."""
    prompts, answers = prompts.repeat_interleave(s.GROUP, 0), answers.repeat_interleave(s.GROUP, 0)
    seq = s.write(model, prompts)
    if method == "inject":  # 문제마다 첫 장을 정답으로 바꿔 넣는다
        seq[:: s.GROUP, s.PROMPT_LEN :] = answers[:: s.GROUP]
    hit = (seq[:, s.PROMPT_LEN :] == answers).float()  # (256, 5) 칸마다 맞았나
    exact = hit.all(1).float()
    logp = s.answer_logits(model, seq).log_softmax(-1).gather(-1, seq[:, s.PROMPT_LEN :, None]).squeeze(-1)
    if method == "token":  # 칸마다 따로: 같은 문제 8장의 같은 칸끼리 평균·표준편차
        reward = hit.view(-1, s.GROUP, s.ANSWER_LEN)
        advantage = (reward - reward.mean(1, keepdim=True)) / (reward.std(1, keepdim=True) + 1e-6)
        loss = -(advantage.reshape(-1, s.ANSWER_LEN) * logp).mean()
        signal = (reward.std(1) > 0).float().mean().item()
    else:
        reward = (hit.mean(1) if method == "partial" else exact).view(-1, s.GROUP)
        advantage = (reward - reward.mean(1, keepdim=True)) / (reward.std(1, keepdim=True) + 1e-6)
        loss = -(advantage.view(-1, 1) * logp).mean()
        signal = (reward.std(1) > 0).float().mean().item()
    s.update(opt, model, loss)
    own = exact.view(-1, s.GROUP)[:, 1:] if method == "inject" else exact.view(-1, s.GROUP)  # 학생이 직접 쓴 답안만
    return {"signal": signal, "own_exact": own.mean().item()}

exp5.py (50~71줄) — rlsim.py 의 step_rlvr 에 스위치만 붙인 거예요. rlsim.py 는 그대로 두고 함수만 가져다 써요.

1,000스텝 결과

자리별 정답률은 세 시드 평균이에요. "처음 그대로" 는 기울기가 계속 0 이라 가중치가 출발점에서 한 번도 안 바뀐 경우예요.

3,000스텝까지 — 배우기 시작한 셋

그림 6. 0% 학생의 덧셈 정답률. 선 위에 마우스를 올리면 그 스텝의 값이 보여요.

왜 이렇게 됐나

• 쉬운 문제부터 — 가중치가 한 번도 안 움직였어요. 0~9 끼리의 쉬운 문제도 답은 다섯 글자예요(7 + 5 → 2100$). 형식부터 모르는 0% 학생은 한 장도 다 맞히지 못하고, 8장이 다 0점이면 advantage 가 0 이라 기울기도 0 이에요. 1,000스텝 뒤 자리별 정답률이 처음과 똑같아요. 쉬운 문제부터 가르치는 건 “가끔은 맞히는” 학생에게만 통해요.
• 부분 점수(답안 하나에) — 쉬운 칸만 배웠어요. 끝 표시 $ 는 100% 가 됐고, 1000의 자리는 세 시드 모두 정확히 50.2% 에 멈췄어요. 평가 문제 중 합이 1,000 이상인 비율이 50.2% 라서, 항상 1 을 쓰는 것으로 보여요. 1·10·100의 자리는 10%, 곧 찍기 수준이에요. 점수가 답안 전체에 숫자 하나라서, 1의 자리를 우연히 맞힌 +0.2 가 다섯 글자 모두에 똑같이 나눠져요. 다른 칸이 우연히 맞고 틀린 잡음에 묻혀서 어느 칸 덕분인지 가려지지 않아요.
• 칸마다 채점 — 자리별로는 배우지만 느려요. 1의 자리를 맞힌 답안은 1의 자리 글자만 올라가요. 그래서 부분 점수와 달리 자리별 정답률이 실제로 올라요(3,000스텝에 1·10·100의 자리 82 · 69 · 49%). 하지만 맞는 숫자를 우연히(10%) 써 봐야 배우는 시행착오라, 전체 정답은 3,000스텝에도 24% 예요. 실제 LLM 의 과정 보상(process reward)을 아주 작게 만든 셈이에요.
• 정답 끼워 넣기 — 되지만 SFT 보다 2배쯤 느려요. 3,000스텝이면 두 시드는 99~100% 예요(90% 도달 1,480 · 1,800스텝). 한 시드는 9.7% 인데, 이 시드는 정답 SFT 에서도 가장 느렸어요. 처음엔 학생이 쓴 7장이 전부 틀려서 똑같이 −0.35 씩 내려가요. 학생이 확률대로 쓴 답들을 고르게 내리는 건 평균적으로 아무 방향도 아니에요(확률대로 뽑은 답의 log 확률 기울기는 평균이 0). 그래서 실제로 배우는 건 +2.47 을 받은 정답 32장뿐이고, 256장이 모두 정답인 SFT 에 비하면 데이터가 1/8 이고 잡음이 섞여 있어요. 학생이 스스로 맞히기 시작하면 그때부터 진짜 RLVR 이 돼요. 마지막엔 학생이 직접 쓴 답의 99% 가 정답이었어요.
• 정답 SFT — 가장 빨라요. 1,000스텝에서는 시드에 따라 17~100% 였고, 3,000스텝이면 세 시드 모두 100% 예요. 90% 에 닿은 건 680~1,140스텝으로, 실험 2 의 OPD(520~840스텝)와 비슷해요.

실험 6 — 백만 단위 사칙연산, 선생 4명

알고 싶은 것: 실험 1~5 는 세 자리 덧셈·뺄셈이라 너무 단순하고 정답이 공짜라는 한계가 있었어요. 문제를 백만 단위 사칙연산으로 키우고 연산별 선생 4명을 두면 MOPD 가 다른 방법보다 확실히 앞설까요?
설정: dev GPU(H100 NVL 2장)에서 작업 61개를 약 3시간 40분 돌렸어요. 학생 실행마다 10,000스텝, 시드 3개예요. 코드는 big/bigsim.py 하나예요(여기에는 싣지 않았어요).