Just Let Agents Write Code

AI agents are everywhere, and everyone is trying to figure out how to make them work best. Ask any engineer building AI products and they’ll tell you their favorite combination of model, harness, router, eval system, and workflow system. Ask them again a week later, and they’ll have a new answer.

But one pattern that keeps popping up over and over and over again is the concept of “Code Mode”. Originally named by Cloudflare last year, code mode is where agents write code to access tools as functions instead of explicit “tool calls” in the LLM output.

At Redo, we believe this “code mode” is a powerful unlock for the agents we’re building, and have created a homegrown sandbox that’s both fast and secure, and works seamlessly with the rest of our systems.

But first, a quick primer.

What is code mode???

In case you aren’t familiar with the concept, here’s how it works.

Normally, to let an AI agent work in your system, you’d expose a series of tools giving it access to various CRUD operations. For example, here we are exposing tools for searching through products and updating their prices.

import { generateText, tool } from 'ai';
import { z } from 'zod';

const result = await generateText({
  tools: {
    searchProducts: tool({
      inputSchema: z.object({
        query: z.string(),
      }),
      execute: async ({ query }) => {
        // ... find the data and return it
      },
    }),
    updateProductPrice: tool({
      inputSchema: z.object({
        id: z.string(),
        newPrice: z.number(),
      }),
      execute: async ({ id, newPrice }) => {
        // ... save the updated price for the given product id
      },
    }),
  },
  prompt: "Please increase the prices of all red shoes by 10%."
});

For the agent to update product prices, it first has to call the searchProducts tool (which would return all the relevant info about any products that match the query), and then, for any products that need updating, it calls the updateProductPrice tool with the product IDs and the new prices.

For an agent with a couple of tools and a simple task, this can work well! It’s still wild to me how good the LLM is at regurgitating exact IDs for future tool calls; the models have gotten so good that they rarely make mistakes here.

But there are some other problems with this approach:

To update the product price, we had to make three round trips to the LLM, more depending on how many times it wanted to call the updateProductPrice tool.
A large amount of unnecessary information was added to the context, i.e. all of the product data.

Code mode solves these problems by having the agent write code that interacts with the tools deterministically, giving it more control over what data it gets and what it does with it. Tool calls can be chained together, with conditional statements, loops, and functions, providing all the flexible flow control we love about code.

This technique requires exposing only two actual tools to the LLM:

search-tools accepts a query string and enables the LLM to discover which functions are available in the code. The tool response is a list of the matching functions, their parameters, and a description of how to use each one, similar to how a “normal” tool would be sent.
execute-code accepts a chunk of code to be executed in the sandbox and returns the results. The response in this case is the text written to stdout / stderr during code execution.

This not only enables a very large number of tools to be exposed to the LLM, but also lets the agent take many chained actions much much faster and cheaper, due to not filling the context window.

This seems simple, until you realize you have to execute the untrusted code the agent wrote, connect the tool implementations, validate tool arguments, capture stdout and stderr, enforce permissions, allow human-in-the-loop, and about a dozen more things… all without the security team hunting you down.

Remote Code Execution as a Service

At first, we tried several code sandbox tools that basically ran a new container for each request. Linux containers are great for sandboxes, and cgroups let you control pretty much everything: CPU limits, memory limits, network access, etc. Plus, you can run any language you want by just installing it in the container.

In order for the tool calls from the agent’s code to run our implementations, we created a small library that the LLM-generated code could import, giving it tools that, when called, make HTTP requests to our internal services. This worked fairly well, but adding new tools was painful because it required modifying the container image each time, not to mention a secondary auth path, which added complexity around which tools an agent is allowed to run.

Those issues were tolerable, but there was a bigger problem: speed. These services came with high startup times, leading to several seconds of waiting before running code for a few milliseconds. More recently, we’ve seen services like Google Cloud Run sandboxes and AWS Lambda MicroVMs, which seem to have better cold start times, but they don’t give us a great way to interact with the sandboxed code at runtime.

While we felt we could overcome many of these shortcomings with optimizations, we recognized we were fighting an uphill battle. So we went back to the drawing board. How could we achieve similar isolation while decreasing startup times and providing better runtime integration with the agent?

After getting talked off the ledge of writing a whole new programming language to solve this, I went back to an old Cloudflare article, where they talked about how they solved a similar problem in their Workers platform with V8 isolates.

V8 is the JavaScript engine that powers Google Chrome, and for every browser tab you open, it creates a separate “isolate”, providing isolation without the overhead of starting a whole new process. Unlike traditional virtualization or containerization methods, which require significant resources for each environment, V8 isolates enable Cloudflare to run thousands of different sandboxed code snippets on the same machine, and start them up extremely quickly (single digit milliseconds).

For us, this meant we could run hundreds of instances of agent-authored code concurrently with almost no latency, without fear of that code wreaking havoc on our machines. V8 provides built-in per-isolate memory limits, and with some tunable limits on the CPU time and wall time of each agent’s execution, we were just missing one thing: our own custom runtime to bridge the gap between the agent’s code and our own.

Shoutout to Deno for the inspiration.

RedoJS

We started with a Rust-based HTTP server running Rusty-V8 that accepted execute-code tool calls and executed the generated JavaScript. V8 itself comes with very few of the utilities you’d expect, so we initialize the common global objects an agent would reach for, like console, setTimeout, and URL. We also added a small shim for intercepting tool calls.

The HTTP server accepts WebSocket connections, which enables us to send messages back and forth between the harness and the RedoJS server. What kinds of messages? Well, the lifecycle of a single RedoJS request looks something like this:

All the tools are available to the agent under a global namespace, so when the agent writes Redo.tools.toolname("abc", 123, true), RedoJS sends a message back over the WebSocket and returns a promise that resolves when the tool finishes on the harness side. Because the messages are handled on the same WebSocket connection that started the original request, all the context necessary to run the tool is already available in memory. No additional auth, no agent context to load, and you can even manipulate the agent state directly from the tool calls.

And because they are just promises in the isolate, we’re even able to execute many of them at the same time and await them in one big Promise.all.

For one final example, let’s say you want to find all the products you ordered more than 1,000 units of in the last 90 days that had the new-product tag, and order them all again. Unless you wrote a very specific tool to handle this scenario, an agent would struggle to put all the pieces together accurately without burning millions of tokens.

Or, it could just write code like this:

// 90 days ago
const cutoff = new Date();
cutoff.setDate(cutoff.getDate() - 90);

// Grab every purchase order placed in that window
const orders = await Redo.tools.searchPurchaseOrders({ placedAfter: cutoff });

// Tally how many of each product we ordered across all of them
const quantities = new Map();
for (const order of orders) {
  for (const { productId, quantity } of order.lineItems) {
    quantities.set(productId, (quantities.get(productId) ?? 0) + quantity);
  }
}

// The products we ordered more than 1,000 units of
const candidateIds = [...quantities]
  .filter(([, qty]) => qty > 1000)
  .map(([id]) => id);

// One tool call per product — but they all fire concurrently in the
// isolate and resolve together, instead of a round trip each.
const products = await Promise.all(
  candidateIds.map((id) => Redo.tools.getProduct(id)),
);

// Keep only the ones tagged "new-product"
const reorder = products.filter((p) => p.tags.includes("new-product"));

// Roll them all into a single new purchase order
const po = await Redo.tools.createPurchaseOrder({
  lineItems: reorder.map((p) => ({
    productId: p.id,
    quantity: quantities.get(p.id),
  })),
});

console.log(`Created PO ${po.id} with ${reorder.length} products`);

TL;DR

By giving agents a fast and secure sandbox to write and execute code in, we’re building higher-quality, more token-efficient experiences in our product. V8 isolates are a great way to provide that environment because they are fast and put you in control of which parts of the outside world are exposed in that sandbox.

Agents are great at writing code, just let them do it.