Productionizing RLMs as Enterprise Agent Orchestrators

I’ve been circling this problem for a while.

It’s followed me across jobs: leading generative AI R&D at Booz Allen, where the focus was agentic AI for mission transformation, and consulting with multiple Fortune 500s at AWS Professional Services, building out their enterprise agent platforms. The work kept having the same basic shape as what people are now rediscovering through Recursive Language Models (RLMs) and dynamic subagents. These environments already had distributed AI and automation systems. The hard part was finding them, understanding their capabilities, composing them safely, and adapting when their interfaces changed.

The answer was always some version of just-in-time composition. You don’t want one permanent graph with every registered agent sitting under a supervisor. You want to retrieve the agents relevant to the current task and construct the multi-agent system that workflow needs, right then. (LangGraph-style graphs were a natural fit: wrap an existing agent as a node, coordinate it under a supervisor for that one workflow, tear it down after.)

In practice that meant registries, usually knowledge graphs in something like Neo4j or AWS Neptune: agents, tools, data sources, owners, permissions, interfaces, dependencies. The registries I built were never just catalogs; that was core to the design. They carried history: success statistics on agents, which shapes of context packet worked with which systems. When it was time to compose a workflow, that history was what the system reasoned over. The same decision is still the right one when the orchestrator becomes an RLM.

What we didn’t have was topological control. We could select which agents became children of a supervisor graph. We couldn’t let the system write the workflow itself.

RLMs open up the topology

That’s what RLMs change. By RLM I mean the pattern from Recursive Language Models by Alex L. Zhang, Tim Kraska, and Omar Khattab: the model treats the prompt as part of an external environment (usually a Python REPL), programmatically inspects and transforms its context, and recursively calls itself or other models over the relevant pieces. The key idea is model-written code controlling context.

Models are already good at writing programs over systems they saw during training. smolagents and Anthropic’s programmatic tool calling both bank on exactly that (I covered both in my end-of-year post). The next jump is code that orchestrates other models and agents. Recent RL experiments on RLMs make the point directly: small models RL-fine-tuned as native RLMs learned task-specific orchestration behavior that prompting or supervised fine-tuning alone couldn’t elicit.

The lineage here is long. HuggingGPT, AutoGen, LLMCompiler, TaskWeaver, and plenty of others pointed the same direction: let a model compose models, tools, and agents through a programmatic substrate. RLMs made it newly salient by putting recursive context control at the center, and the pattern is now in the harnesses people actually use. Claude Code calls these code-driven procedures dynamic workflows. LangChain’s Deep Agents call them dynamic subagents: instead of issuing one subagent task at a time through tool calls, the main agent writes a short script that dispatches subagents through a code interpreter. Their follow-up, How to Use RLMs in Deep Agents, adds the nuance that matters most here: those recursive calls aren’t plain LM calls — they’re agents with their own tools and state.

Inside an enterprise, that nuance is the whole point, and it goes further than the current implementations. Deep Agents’ dynamic subagents, as shipped today, run in-process: the orchestration script executes in an embedded interpreter inside the agent’s own runtime, and each task() call invokes a subagent built in that same process. That’s not what a large enterprise or government environment looks like. The systems worth composing are distributed: separately deployed, separately owned, sitting behind their own auth on somebody else’s infrastructure. A recursive call might be a cyber triage agent with SIEM access, a procurement agent wired into contract systems, or a legal review agent with citation constraints. The top-level RLM is writing a context-routing program over a distributed system of organizational capabilities: what to retrieve, what to redact, which system to call, what output format to ask for, how to transform the result for the next call, when to stop.

That’s just-in-time agent composition with the topology finally opened up. Which brings the old problem back, sharper than before.

The model never learned your systems

The systems that matter inside a company or government environment are internal, permissioned, idiosyncratic, and invisible during both pretraining and post-training. The model was never trained on your procurement workflow, your incident ontology, your data-access rules, or the exact shape of the agents in your registry.

A model can obviously write code. The question is how you get it to write useful context-routing programs over systems it never learned.

My answer: the same backbone we were building back then, upgraded for the new topology. A registry that says what exists and owns each system’s contract (an OpenAPI-style spec or agent-card-style description for every agent in the catalog, so an interface change is a registry update). An orchestration interface that gives the model structured primitives for composing against those contracts. And telemetry from every rollout, folded into the same data system, so the next program can reason over what actually happened. The telemetry is the memory.

Existing standards make that contract layer less speculative. Agent2Agent (A2A) is aimed at interoperability between independent, potentially opaque agent systems, and Agent Cards give agents machine-readable descriptions of their capabilities, skills, security posture, and coordination surface. I’d treat those as a natural starting point for the registry entry. But the argument here starts where the card ends: production systems also need to remember which agents were composed, what context packets and transformations crossed the edges, what outcome was achieved, and what should change next time.

I’d argue this backbone matters more now than it did in our hand-built era. Back then the registry mostly fed retrieval and selection. Now the same data layer can feed trajectory mining, credit assignment, prompt and program optimization, RL, even generating the training environments. I’ll come back to that at the end.

You can only optimize the interchange

So what do you actually optimize? The clean case is the DSPy / compound AI systems one: define a single LM program and optimize its modules end to end against a metric. That works when you own the whole program.

Enterprise orchestration is messier for a different reason: ownership and distribution. The legal review agent, the procurement workflow, the cyber triage system — different teams, different prompts, different evals, different release processes. Each gets optimized by its owner. The orchestration layer doesn’t get to fuse them into one program and tune it end to end.

What it can optimize is the interchange: which system gets called, in what order, under which workflow pattern, with what context packet, and how one system’s output gets transformed into the next one’s input.

If these were simple functions with fixed argument types, that would be easy. But compound AI systems are sensitive to context shape. The contract in the registry gets you syntactic correctness — the call won’t reject your payload. It doesn’t tell you which of several valid shapes actually gets a good result out of the system: the same information can work or fail depending on whether it shows up as a typed table, a prose brief, or source-linked excerpts. This was exactly the packet-statistics problem from our old registries.

Workflow patterns are learned the same way. The dynamic-subagents post names the recurring shapes (fan-out and synthesize, classify and act, adversarial verification, loop until done), and these are really test-time compute patterns: spend more calls and more structure to buy coverage, comparison, or verification. But there’s no universal answer for which shape a task needs. Legal review probably wants adversarial verification; procurement might need a fan-out across policy, budget, and contract agents before synthesis. That’s a learned property of the systems being composed, not a framework setting.

The model writes the orchestration program. The platform’s job is knowing which programs actually work. And that takes evidence.

Don’t just give it a REPL

The RLM should absolutely still write code — that’s the point. But in production it shouldn’t compose internal systems through arbitrary free-form Python. It needs a small set of orchestration primitives to write against: registered nodes, typed handoff edges, policy checks, result objects. The nodes are the compound systems in the registry. The edges carry the context packet passed to the next node, plus the transformation code that produced it.

This isn’t about developer tidiness. It’s how you get consistent telemetry. If every node call, edge, retry, and result flows through the same interface, every RLM-written program produces comparable traces. You can see that the procurement-to-legal edge works when it hands off a typed clause table and fails when it hands off a prose summary. Without the interface you still have logs, but you can’t reliably reconstruct the graph, inspect what each edge passed, or compare this workflow to prior rollouts.

So the production version of an RLM is a REPL plus the primitives that make its programs observable.

The telemetry is the memory

A useful trace here is a full program trace: the task, the context it retrieved, the systems called, the topology, the typed edges, the outcome. LangSmith-style observability already makes agent trajectories inspectable; dynamic subagents add a more specific need: visibility into the edges. What got passed between agents, how it was produced, whether it was useful. The edge matters as much as the node.

Walk through one workflow. An RLM is asked to produce an action memo for a procurement decision. It retrieves policy context, shapes a procurement packet, calls the procurement agent, notices a legal-risk clause, extracts it with source pointers, calls legal review, translates the legal constraints back into procurement terms, and drafts a memo for a human approver. If the memo is approved, that’s a validated program topology. If it fails, the trace should say where: legal review came too late, or the source evidence got summarized away, or legal needed a typed table and got prose.

What you want out of a failure like that is a reusable lesson — something like:

When passing procurement output to legal review, preserve source IDs, confidence markers, and the normalized clause schema. Don’t compress the finding into prose. Use this typed handoff template.

You can call that lesson procedural memory: a verbalized thing the next rollout retrieves before it writes a line of code. But notice it’s a derivative. It only exists because the trace existed, in a minable shape, attached to the systems it touched. Extracting lessons is its own problem (trajectory reflection, credit assignment, plenty of approaches), and it’s separable from the claim I actually care about: get the telemetry into the data system, or none of those approaches are even available. Ours lived in the graph as success statistics and packet shapes. Same idea, smaller scale.

One backbone, every improvement method

For any of this to improve, the system has to answer a few questions after every rollout: what program did the model write, what did each edge pass, did it get the result, and what should change next time. The outcome is one label: was the memo approved, did the system of record end up in the right state. The trace is the explanation: topology, edges, retries, policy checks, cost. (OpenInference, OpenTelemetry’s GenAI conventions, and benchmarks like tau-bench are all converging on the same idea: score the trajectory and the state changes rather than the final text alone. More on this in an upcoming evals post.) And sometimes the workflow succeeds while the orchestration is still bad (right answer, bought with an underspecified packet that cost three clarification round trips), which is why I like credit assignment at lower levels of the trajectory over final-outcome-only rewards — for RL, and just as much for prompt and program optimization. This turn-level reward work (co-authored by Will Brown, built on his verifiers library) shows the shape: extend GRPO to multi-turn with per-turn rewards on things like whether each turn emitted a well-formed tool call and whether retrieval actually surfaced the right information.

Once the telemetry exists, the menu is long. Trajectory mining, which is exactly the loop LangChain Labs is chasing (get traces, enrich them, improve from them, repeat). Hardening repeated lessons into rules: if procurement-to-legal handoffs fail without a normalized table, enforce the schema in the interface. Context optimization of the orchestrator itself. DSPy now ships an experimental dspy.RLM module that takes custom tools callable from inside its REPL, so you can hand it retrieval over the registry — and since it’s built from ordinary DSPy predictors, you can in principle optimize it like any other DSPy program, bootstrapping few-shot demonstrations of how to query the registry well and tuning the instructions for how it uses what comes back. Then GEPA-style program optimization against labeled trajectories, which I think mints something for free: GEPA evolves candidate programs along a Pareto frontier, so the winners are natural preference data over the original RLM-written programs. Not that RL has to wait for preference pairs; with verifiable rewards you do the credit assignment and go straight to policy optimization. (Letta’s framing fits here too: agents improve by restructuring memory rather than waiting on weight updates.)

The part I’m most bullish on: the registry makes RL environments generatable. It defines the action space (which systems can be called, what contracts they expose, what policy gates apply), trace history supplies realistic tasks and failure modes, and validated outcomes supply the reward. Instead of hand-building an environment per workflow, you sample tasks from prior rollouts, simulate the domain agents, and train the model to write better programs over your actual catalog. When the catalog changes, the environment changes with it. That one deserves its own post.

Somebody’s going to build this

There’s a real product in this post. The registry system plus the telemetry layer, sold with forward-deployed engineers who populate the registry with your existing compound AI systems — and, once the traces accumulate, post-train an RLM to orchestrate them.

A couple of players are unusually well positioned. LangChain has most of the components already: Deep Agents as the RLM abstraction, LangSmith for the observability, LangGraph as the workflow primitive, and LangChain Labs already doing continual learning by trace mining. Prime Intellect comes at it from the training side: hosted RL post-training as a service, Will Brown’s verifiers library and the Environments Hub, and an experimental RLM environment pointed at exactly this kind of training. There’s probably room for a new player too. But I wouldn’t be surprised if one of these two just ships a registry-and-procedural-memory product with the service attached.

---

Not every organization should do all of this at once. The first move is the same regardless: put the contracts in the registry and start collecting the telemetry. The rest of the menu opens from there.

---

Cite this post

If you found this useful, feel free to cite it:

Hitchcock, W. (2026). Productionizing RLMs as Enterprise Agent Orchestrators. williamhitchcockai.com. https://williamhitchcockai.com/blog/procedural-memory-for-enterprise-rlms

@article{hitchcock2026rlms,
  title={Productionizing RLMs as Enterprise Agent Orchestrators},
  author={Hitchcock, William},
  year={2026},
  month={July},
  url={https://williamhitchcockai.com/blog/procedural-memory-for-enterprise-rlms}
}