Your Agent Should Learn From You, Not Just Listen To You

The Stopgap We're All Living With

Every serious agent operator has a file. Maybe it's called SOUL.md. Maybe it's a system prompt. Maybe it's a JSON config labeled "preferences." Whatever the name, it's the same thing: a list of instructions you've written to teach your agent how you want things done.

Don't use corporate speak. Numbers before adjectives. When I ask about leads, run the script first — don't ask. Report anomalies without being prompted. Never email a vendor without asking first.

Good agents follow these rules inside a session. But they're not learning from you. They're reading a file. The difference matters more than it seems.

Every correction you give is evaporating. You catch a mistake, say "you should have checked the database before reporting that lead as fresh," the agent adjusts for the next 20 messages — and then the next session starts and you're back to square one. The correction didn't stick because it couldn't. You're not modifying the model. You're modifying its context window.

What OpenClaw-RL Does Differently

A project called OpenClaw-RL dropped today that takes a different approach. Instead of reading instructions, the model learns from your actual conversation patterns — asynchronously, in the background, without you doing anything differently.

The architecture is four independent async loops running simultaneously:

OpenClaw API (your conversations)
       │
       ▼
  API Interceptor      ← wraps local model as OpenAI-compatible endpoint
       │
  Rollout Collector    ← classifies main-line vs side turns
       │
  PRM Judge            ← scores turns from next-state feedback
       │
  Policy Trainer       ← GRPO/PPO or OPD, async, non-blocking

The model serves your requests while training runs in the background. When a weight update is ready, submission briefly pauses and then resumes. From your side, it looks like a normal conversation.

Two Ways It Learns

Binary RL — Implicit Feedback

The first method is called Binary RL, built on GRPO (Group Relative Policy Optimization). The judge scores each turn by looking at the next user message as a natural feedback signal.

If you send a follow-up that looks like "yes, that's right, now do X" — that turn was probably good. If you send "no, that's wrong, what you should have done is..." — that turn was probably bad. The system doesn't need you to rate anything. The conversation itself is the signal.

Every session contributes at least one training sample. Even if most turns are neutral, the framework guarantees something gets through.

On-Policy Distillation — Explicit Corrections

The second method is more interesting. When your next message contains a correction or a hint — "you should have checked the file first before reporting that" — the judge extracts a textual hint from that correction. The hint augments the original prompt to create an "enhanced teacher" version of the response. The token-level log-probability gap between the teacher and the student becomes the training signal.

This is called On-Policy Distillation (OPD). It's richer than a scalar reward. Instead of "that turn was bad," it captures the directional gradient: bad in this specific way, for this specific reason, and here's the better version.

The recommended setup runs both methods together. All scored turns plus directional hints.

Why This Is Different From RLHF

You've seen RLHF before. ChatGPT uses it. Alignment researchers use it. It requires human raters, annotation pipelines, batched training runs on curated datasets. It's a factory process.

OpenClaw-RL is a continuous process. No raters. No curation. No batches. The training data is your actual work: the lead reports you corrected, the emails you told it to rewrite, the analysis it got wrong last Tuesday at 11pm. The model that runs your business gets better at running your business.

The gap between "generic Claude" and "your specialized agent" widens automatically, just from usage.

That compounding effect is the real value. Month one, you're prompt-engineering your system files. Month six, the model has internalized your patterns at the weight level. It's not reading that you prefer numbers before adjectives — it just writes that way.

The Part That's Most Interesting

The roadmap has a line item that stands out: "extend learning to skills and memory."

Right now, the RL applies to the policy — how the model responds. That's valuable. But skills (what the agent can do) and memory (what the agent knows about your business) are still static. You update them manually.

Extend RL to those layers and you have something qualitatively different. An agent that doesn't just learn how to respond better — it learns which tools to reach for, which memory patterns matter, which context to surface unprompted. That's the line between a good assistant and an actual operator.

The Practical Reality Today

OpenClaw-RL requires 8× GPUs with CUDA 12.9. Not feasible on a $20 VPS. The default example uses Qwen 3 4B as the base model — a capable local model, but not Claude-class.

So for now, SOUL.md is still doing the heavy lifting. Explicit instructions, carefully written, updated when the agent gets things wrong. It works. It's just manual.

The transition path is clear though. As GPU costs drop and the framework matures, personal RL becomes the layer that makes explicit instruction files obsolete. You stop maintaining a document and start just correcting mistakes — knowing each correction compounds over time.

What This Means for Agent Design

The architectures that survive long-term aren't the ones that load the smartest base model. They're the ones that close the feedback loop between user corrections and model behavior.

Most agent frameworks treat the model as fixed and the prompt as the variable. OpenClaw-RL inverts that: your conversations are the training set, and the model is the thing that moves.

That's the right direction. A good employee doesn't need a 3,000-word instruction document after six months on the job. They've internalized it. Personal RL is how agents get there.

The framework is open source. The tech report is worth reading if you're building in this space. And if you have 8× GPUs collecting dust — the path to a model that actually knows how you work just got a lot more concrete.