I RL-trained an agent whose job is to write RL training jobs for smaller models, and open-sourced the whole thing.<p>The setup is two nested RL loops:<p>- Outer loop: the trainer agent (Qwen3.6-35B-A3B, LoRA) is handed a task spec ("teach a small model to do X"). It works in a sandboxed workspace with file tools and writes a complete prime-rl training job: a verifiers environment + rubric, a dataset, and a hyperparameter config. Submitting triggers a validation probe with capped retries.
- Inner loop: each validated job is dispatched to a warm pool of up to 16 Runpod GPU pods, where prime-rl GRPO-trains a small Qwen (0.6B or 1.7B). The checkpoint is scored pre/post on a hidden eval the agent never sees.
- The inner model's improvement flows back up as the outer loop's reward (plus a validation-efficiency term and a small train-speed tie-breaker). The outer loop is tinker-cookbook's importance-sampling GRPO, run async off-policy so one slow episode doesn't stall a batch.<p>Results, over 54 outer-loop steps (~1,750 real GPU training jobs):<p>- Episode reward went from ~0.0 to a ~0.63 peak.
- Learning came in two distinct rungs: first "stop failing validation and dying on GPUs", then "make better models". GRPO took the steepest gradient first — the entire early gain was process reliability, and only once that saturated did the hidden-eval scores of the trained models start climbing.
- It transferred to a held-out task family that the agent never trained on: mean reward 0.399 untrained → 0.545 at step 34, easing to 0.49 by step 54 (n=10 per arm, so noisy — a rise then a plateau/dip).
- The agent learned to stop picking the weaker 0.6B base model (1.7B share of its jobs: 42% → 95%) and started actually using the hyperparameter surface (21% → ~78% of episodes).<p>Cost: the headline arc was ~$1.3k all-in (~$810 Runpod, ~$465 Tinker). Each inner training job cost ~$0.13–0.30 — a benchmark matrix over GPU × base-model picked cheap pairs (mostly A40s in practice, since the cost-winner was rarely in stock).<p>Two honesty notes: the outer loop runs through Tinker's managed API rather than local GPUs — the inner loop is all open-source stack on rented pods. And ~$1.3k is the headline arc, not the project; the pilots and blind alleys that got me there cost a few hundred more, and every one of them is written up in the retros in the repo, including the failures.<p>I did this because I think agents that improve other AI systems are going to be a big part of the next few years, and I wanted to know what it actually takes to get the reward moving. Turns out: way more debugging of the process than the policy, and it's all more accessible than it looks.<p>Happy to answer questions about the reward design, the GPU orchestration, or the things that didn't work.