GLYPH
Post-training a 4B Rust coding agent with verifiable rewards
- Built: verified traces → full SFT → three RL rewards → pass@8.
- Found: no RL arm reliably beat SFT; the protocol was already saturated.
- Next: compare base → SFT → RL on independently authored Rust tasks.
1. What I built
GLYPH is a verifiers environment. It owns the Rust tasks, tool loop, and rewards;
PRIME-RL orchestrates rollouts and policy updates. Each rollout edits a disposable Cargo
crate and receives reward from real compiler, test, or stdout results.
ASSISTANT CALL read_file {…} TOOL RESULT src/lib.rs ASSISTANT CALL apply_patch {…} ASSISTANT CALL cargo_test {…} TOOL RESULT 4 passed ASSISTANT FINAL: fixed and tested
The full conversation—not just the patch—is the SFT training unit. The same ChatML renderer and tool runtime power data generation, RL, evaluation, and the TUI.
2. GLYPH in action
In the TUI, the trained agent solved the independently authored score_summary
crate. It fixed score filtering and capping, passed four tests, then emitted FINAL.
The task appears in neither training nor evaluation.
One selected success, not a benchmark. The crate is new, but its filtering and aggregation concepts resemble training archetypes. Harder exploratory crates failed within their tool budgets and were not a controlled evaluation. This shows transfer to one new crate—not broad Rust capability. crate and tests · reproduction
3. How I made the SFT data
A generator wrote each task as structured JSON: a buggy crate, intended edits, and expected
tool outcomes. A deterministic materializer built the crate and ran every step through Cargo.
I kept a case only when planned failures failed and the final fix passed. rustc,
not the generator, produced the compiler output.
Full fine-tuning on those verified traces produced SFT_HALF_A_V8. It solved
95 of 150 held-out crates at valid@8 and learned the tool protocol almost
perfectly.
Holdout limit: training and evaluation came from the same generator and task families. The holdout tests new instances of known bug types—not new Rust problem classes. Next time: reserve whole bug families and add human-written tasks.
4. The RL experiment
One SFT control. Three RL reward designs. One training run per RL arm.
valid@8 counts a crate when at least one of eight sampled rollouts reaches Cargo success and ends with one clean FINAL.
Sparse
Clean verifier success dominates; failed attempts receive coarse penalties.
Dense
Adds credit for compiling and for the fraction of tests passed.
Compiler-aware
Grades failed code by the furthest rustc phase reached.
| Model | valid@8 (retained) |
|---|---|
| SFT | 95 |
| Sparse RLVR | 98 / 96 / 98 |
| Dense RLVR | 102 |
| Compiler-aware | 95 / 96 / 94 |
No retained-run comparison reached p<0.05 · methodology →
Prompt-paired and family-grouped tests gave dense versus SFT p≈0.12 and p≈0.15. Aggregate-only SFT 97/100 and dense 102/99 repetitions remain in the chart but not the claims because their rollouts were not saved. Repetitions were not independently seeded.
5. One verifier gap
One retained RLVR trace passed every test after reversing the written TLS precedence.
Before (spec-correct): tls = direct.or_else(profile).or(defaults) After (model's patch): tls = profile.or_else(direct).or(defaults) Tests: 3/3 pass. Spec: violated. Verifier: blind.
Full trace →
Full system, user, assistant, and tool transcript.
6. What the evidence says
SFT saturated the protocol
Sparse RLVR produced correct syntax on 150/150 greedy prompts versus 149/150 for SFT. The models still shared 74 greedy failures and 48 retained pass@8 failures. The bottleneck was task execution, not formatting.
Sparse reward could not separate the hard tail
At step 0, 64 of 96 rollouts belonged to tied groups and were filtered. Some compiled or passed partial tests, but sparse reward barely distinguished that progress.
I still cannot isolate the cause
No matched base-model evaluation means I cannot separate model capacity, narrow synthetic-data coverage, or post-training regression. That is the next experiment.
Verify these claims →
reproducible statistics · three-run sensitivity analysis · SFT greedy eval · sparse greedy eval · pass@8 metadata
7. Next: isolate the bottleneck
Run base → SFT → RL on independently authored Rust tasks. Score Cargo/task success separately from GLYPH trace formatting.
| Result | Evidence for |
|---|---|
| Base fails; SFT improves | Post-training added capability |
| Base passes; SFT fails | Post-training caused regression |
| All fail | Test targeted data or a stronger base |
| SFT improves only same-family tasks | Specialization, not broad Rust transfer |
Result: a working, auditable post-training stack—but no established RL gain. The next experiment now has a clear target.