• Intro
• What we’re doing
• Why we can do this
• Where to start
• Preparing the engine
  • Authentication
  • Environment
• Going full throttle
• Results
• Conclusions

Intro

Several weeks ago, the first sign of Bun being rewritten from Zig to Rust was spotted. Starting as just an experiment before showing curious results, we then finally got the highly awaited blog post. In the past, rewriting a codebase was a sign of either a bad design decision made in the beginning, or, a non-productive nerd snipe. Today, the implied cost of engineering hours towards a rewrite is more affordable in tokens.

What we’re doing

Since rewriting a codebase should be doable now, how do we go about a large autonomous refactor? To show by example, we will be rewriting jq in Odin with ChatGPT 5.6 Luna.

Why we can do this

jq has a language agnostic test suite which matches the same template as Bun.

In non-technical terms, this is like defining a word in the dictionary without using the word in that sentence. Regardless of the specific language used, the purpose of the verifier (the test suite in the case of Bun or jq) is to give a strict thumbs up or down for some given code being “correct”.

Note: It doesn’t matter whether or not the codebase was generated or artisanally written by a person, the purpose of the verification is the “good” or “not good”. If you’re on a team with a robust CI/CD, then seeing all green on a pull request is fair grounds for an “LGTM” and approval. Whether you view that as being a good or bad thing, the level of confidence in the result is what matters.

Where to start

Like how you could ask an LLM to prepare a local coding environment (fetching files, installing dependencies), you can ask the LLM to plan a larger codebase refactor before proceeding. Below is my summary of some LLM output on things about Odin that would be useful to know in advance of rewriting a C project.

• Outlining the package graph such there are no import cycles
• No native coroutines or async behaviors
• No closures that can access outside-scope values

Preparing the engine

Since we’re going to have a flexible number of agents working in parallel, we will also prompt the initial setup to have multiple files that prevent merge conflicts.

Authentication

For this project, I used Vers VMs which basically give you git operations but for computers. First, create an authenticated Codex snapshot that we can restore for ephemeral agents. Authenticating a VM with Codex is as simple as going through the Device Code flow. After you’ve gotten the computer logged in to Codex, make a commit of that VM, and you’ll now be able to restore as many new VMs from this snapshot with your OpenAI account already signed in!

Aside from the security of delegating work to separate sandboxes or the space saved on my local laptop, Codex seems to have a limit of four parallel worktree agents on a given machine. Orchestrating these cloud computers was what allowed me to work on this in the timeframe I did.

Environment

After this first setup, we need some simple plumbing such as GitHub PAT and preparing the Codex instance to be cloned when handling new prompts. After some basic branch protections, every PR would be reviewed by a short-lived agent (leaning on the point from the Bun blog post where an agent owning the PR would want for their changes to be merged whereas a blank review agent tends to be more unbiased)

Author note: GitHub being down at points did cause nuisances and I did migrate some of the review/evaluation to the cloud VMs. The end result of 522 passing tests is still the same.

Going full throttle

“Now rewrite jq into Odin, make no mistakes”

Vibe coding is magical when you can tell a computer to make an app and it does. But, it’s much more useful when you give it the right direction, environment, orchestration, or even input (what are you telling the computer to do?). By lining up each of these in the setup, I’m then able to enter that prompt verbatim and know it should produce a meaningful result.

Results

After a bit more than a week since writing that last sentence in this blog post, I finally got a rewrite of jq into Odin based on their test suite!

https://github.com/yevbar/jq-odin

I reminded ChatGPT at times to get “back on track” or use the cloud VMs instead of the count-limited local worktrees but I didn’t insert myself anywhere relating to the implementation of jq in Odin. As soon as ChatGPT completed the Goal (their version of a loop with Claude), it hid the banner showing the exact amount of time spent but it’s reported to be close to seven days and nine hours.

For the cases strictly covered by the test suite, it gets the job done, however, there’s one caveat with this being a drop-in replacement for jq off the shelf. It strictly covers as much is expected by the test suite, which is only 522 test cases. While this may seem like a lot, test driven development tends to play well with language oriented projects (while I was working on LSD, I had a slew of tests across the stack since there’s a broad surface area for a parser or interpreter).

For functionalities not covered by the test suite but only implemented in the CLI, there’s not a high certainty those capabilities would work in the Odin rewrite as it is. But, knowing agents can recursively improve, it’d likely be a matter of implementation or orchestration to work on this such that the Odin-based jq would eventually have more feature parity.

Conclusions

For gojq and jaq, other implementations of jq in different languages, there was weeks of time in between the start of those projects and their first working releases. While those projects seem to be focused on targeting the jq codebase instead of just the test suite gate, the results here further highlight that agentic coding can dramatically speed up work.

The partial result from the existing test suite alone may contribute to jq’s difficulty on ProgramBench and why there are thousands of “generated behavorial tests”. If you’re wondering whether or not agents can write or rewrite real things, we’re there now. Whether or not agents can complete real things is a different question. It’s less what the team of agents is made of and more a question of what the team of agents is doing, you could have hundreds of “auto-researchers” but it means nothing if they’re as unproductive as a tech company trying out a “flat organization structure”.

Hack the planet!

To compensate for joining a multi-month competition halfway through, I decided to timebox to a couple days and try approaching a generalized bot. Knowing Magic and understanding strategies to the game is what enabled me to hone in on stronger heuristic bots with the previous project. Given there are likely to be people who have that level of familiarity but for Pokemon, I know I’m faced with two options:

• Study the game and the allowed cards to find a strong meta
• Learn just enough to make a functional bot and see if I can iterate on that

While an LLM could do the first and I have seen that done before, I didn’t trust my intuition for evaluating its output so I opted for the latter. I shipped a few bots that were able to play and win games but nothing I vibed broke the threshold for actually performing well against the field. Based on the result of the Legends of Code and Magic annual competitions, most people there focused on heuristic bots that leaned on specific strategies or they applied neural networks to certain mechanics of the game. My suspicion is you could analyze the submissions in the Pokemon competition and find something similar.

So, right away, the Pokemon game deserves a kudos for not being immediately LLM’able. Perhaps if the AGI narrative holds, then benchmarks would be less measures of intelligence but more akin to post-quantum cryptography. Several years ago, quantum computing was more hype but now the US Government takes it seriously. What post-quantum cryptography is about is technically just algorithms but they can also be understood as techniques for dealing with that strength of technology. Likewise, if AI on its own gets good at real world problems, we may benefit from some “post-AI riddles” that keeps AI in check like how gravity keeps us grounded.

As for takeaways, I think there are two I’d like to walk away with at least:

• Friendshaped doesn’t mean friend: Data can meet a format but the model may not be representative of its behavior
• It’s better to be early and underdressed to a money making party than late: I can’t find the original tweet but it went along that idea.

However, a take I heard mid-conversation from someone who heard it from someone else stuck with me. What we’re seeing in the tech industry (especially with regard to programming roles thanks to coding agents) is similar to what happened to formerly Communist countries when they opened their borders after the collapse of the Soviet Union. It’s not a clean direct metaphor so, to deliver the point, let’s imagine we’re someone who works on some assembly line in a factory for a home-grown product.

The product you work on isn’t the greatest quality but it’s precious, it’s marked with the direction and instructions of the nation you’re producing value for. Your friends working in non-State-backed markets seem silly to you because they must cope in some way when their trade isn’t performing well. You’re smart for contributing to something the very nation you’re a part of says is worth something. Then, one day, that security and dependency disappears, democracy is trendy, and your neighbors can purchase from abroad rather than locally produced. While you protest consumer choice, you can’t help but notice your industry go belly up practically overnight.

Suppose, one day, North Korea stops being an authoritarian state and everyone who used to only buy from local manufacturers can buy from a neighboring Asian country. Assuming we’re past the “calibrating purchasing power with the rest of the world” phase, the majority of people would be expected to buy from abroad if it’s better or cheaper. Call it free market capitalism or competition, but the change in consumer behavior would be clear. Every SWE sitting with guilt or uncertainty over their future career is experiencing something similar to factory workers in a formerly shut-in nation that’s now opened up its borders.

If your immediate reaction is similar to when tech workers ask for a union, pity for techies is not the point of this post. Regardless of the average SWE salary to global poverty baselines, coding agents being able to replace technical talent is revealing of underlying problems similar to how COVID didn’t worsen institutions but made their problems clearer.

Back in the day, technical talent was as non-fungible as brand identities. If you were doing a venture-backed startup, you’d accumulate engineers to bring to life the vision in your pitch deck. It was a required dependency since, otherwise, who’d make the app? Nowadays, you can vibe the entire MVP with nothing more than a prompt or markdown file if you care that much. People would establish their careers on being specifically familiar with a certain programming language or framework (remember when bootcamps were shipping React or Angular developers?) but now that knowledge is only relevant insofar as it’s not immediately obvious to the models.

Whenever you were faced with a problem in the past, there was an actual tradeoff between finding something off the shelf or making it yourself (the classic “build vs buy” argument). Nowadays, people are able to make their own in-house solutions as soon as something off the shelf gives even the slightest inconvenience. While specific needs or “enterprise” use cases may require the existence of focused companies on certain solutions (ie Salesforce or Datadog), open source work or lower token costs help in the direction of people being able to build their own solutions. There was a time when it seemed crazy to leave bespoke hand-crafted books for the printing press. There was a time when it seemed crazy to leave fine cultery for plastic utensils. There was a time when it seemed crazy to leave intentionally made things for mass produced items. There was a time when it seemed crazy to leave physical media for the internet. The same will happen to software and how it gets engineered.

Other theories of the AI bubble generally describe how it will behave economically today and around the peak of the bubble, but they don’t always describe how the world will change afterwards. With the industrialization of software, seeing how consumers responded to the industrialization of mass produced goods, it paints a narrative for how generative UIs shift from being toys to real applications. An ordinary consumer looking for an item will go to Walmart or Amazon first before they look for a small local business that hand-crafts that said item. A UI that provides to users just what they need informationally makes a lot more sense than an app that was engineered for a world of capturing attention.

While apps are technically playing the same game as the “attention economy”, the goal is slightly different. Before, you wanted consumers or users to be viewing your displays to get the value you provided. Now, you want consumers or users to be getting the value you provide, regardless of the display[s] you may provide out of the box. MCP turns your buttons and forms into text questions and answers. Generative UIs turn your pages into partially renderable views for users to get no more than they’re interested in. You don’t care if a person pays you after messaging a chatbot or after clicking through a series of forms so long as your service is ultimately provided.

Some things do not change, of course. An app that’s trendy or popular will play the same today as it did in the past with word of mouth. Programs do ultimately depend on the value they’re providing and not just their aesthetic value. But, funnily enough, the vibe coders got it right with being able to view software as nothing more than an incidental problem for an AI to deal with, not people. People making toys don’t care about the exact chemicals used with the plastics, they just want a plastic toy in the end. Likewise, people solving problems with software will critique much less whether an app was developed in some language or another. I think people can care or not care what language an app was developed in. We may find a world where people care less about technical implementation details or a world where people actually care about programming languages like the environmental impact of certain culinary categories.

• The gist
• What makes Magic particularly hard?
  • How can information be incomplete?
  • A technique for incomplete information
  • Turing complete whatnow?
  • Distinguishing between the engine and the bot
• How other people have “hacked” Magic?
  • Official Products
  • Java-based
  • MageZero
  • Deck building bot
  • Simplified Engines
    • Legends of Code and Magic
    • open-mtg
• What I did
  • Frustrations which led to a new engine
  • How I vibed the engine
  • Datalog
  • Transpiling English to Datalog
  • Simple API
  • Bot
  • Vision interaction
• Results
• What I did with my account
• GitHub

The gist

Months ago, I participated in a hackathon where the prompt was to make a programming language and then make a game in that language. There I built a few DSLs specifically for card games. Since then, I’ve been curious as to what it would look like or take to have a Stockfish for Magic and went through a tumultuous journey to put together the pieces I have. Hope you enjoy :)

What makes Magic particularly hard?

Magic the Gathering is one of the most popular collectible card games and literally the only reason the company that makes Monopoly is still alive. Due to being an incomplete information game as well as one whose mechanics are Turing complete, it is one of the hardest games in the world.

How can information be incomplete?

Unlike games like chess or Go, Magic operates with incomplete information like poker or mafia. By “incomplete information”, there is knowledge relevant to the game not publicly available to all players. The simplest example of this would have to be rock-paper-scissors; knowing what an opponent is going to select makes all the difference from the game being “solveable” versus a disguised Monty Hall problem.

In the case of the Monty Hall problem, the “solution” can be determined by broadening out beyond what’s at face value. Let’s say you selected the first door and then learned behind the third door is a goat.

This leaves us with two options: either the first door, or the second door.

On first impression, this would lead one to think there’s a 50% chance of being right whether or not you swich. But, two options does not mean two equal possibilities. You have a 2/3 chance of choosing a door with a goat in the beginning, and then a 100% chance of winning when you switch (since you’d always be getting the car in that scenario).

A technique for incomplete information

Similar to the Monty Hall problem, with ReBeL, the expansion from hidden to public knowledge is the key technique. In the ReBeL paper, the authors suggest a version of rock-paper-scissors with a twist: whenever you win using scissors, you get two points and, whenever you lose with scissors, you lose two points.

Should we only consider what’s immediately in front of us (a player about to play a shape you don’t know), then there’s no reason to deliberate and you may as well see what goes.

If we consider the possible responses from the opponent player, then we can work out an optimal policy for how often to make certain plays. This is thanks to being able to weigh the different expected points or rewards from these different outcomes.

With poker, you have ranges, the different probabilities of winning with different hands in different seats. The important detail about ranges is there is a finite number of probabilities to determine, whether or not you map exact permutations (ie a king of hearts with jack of spades) versus general hands (ie a seven-deuce offsuit). Unlike rock-paper-scissors or poker, there does not exist a finite ceiling to the amount of total information that could be contained in a game of Magic. This is thanks to it being Turing complete and there being an infinite amount of information that could emerge in a game.

Turing complete whatnow?

Back in 2019, it was shown Magic itself could run a computer like a Redstone computer in Minecraft. By “computer”, it was the simplest version of a computer (the Turing machine). How it ties into the complexity of a game has to do with the Halting problem which gives a limitation to what can be done with computers. For instance, with physics, you can’t run into a wall and expect to pass through. (Source to below animation)

Unlike a game like chess or poker where players make one decision per turn (moving one piece in chess, a call or fold in poker), you can have a sequence of interactions in a single turn in Magic. Coupling this together with the large number of cards published, you can end up with infinite combos that may or not ruin the friend group you’re playing with. Unfortunately, infinite storage does not exist so we can lean on a claim made by the original authors where the game itself may not be computably decidable but it may be transition computable.

They leave this as something they believe in more than they can formally prove given the 20,000+ corpus of cards at the time. However, I think it can be safely assumed given the following:

• It’s possible to establish the static rules of the game for non-infinite situations
• It’s possible to define how cards and effects apply to each other for non-infinite situations
• There are rules for players to assign finite numbers to infinite loops

While it does not rigorously cover all cases, the amount of gameplay around the world and work done by employees at Wizards of the Coast would leave me to think we’ve exhausted most of the obvious ones. Therefore, we should be able to have some software that handles moving from one state in the game to the next and not fret over the technical un-computability of it all.

As a final note on the Turing completeness, while Powerpoint and other games such as Minecraft do share the ability to have a computer run inside its environment, Magic is the only game that is both Turing complete and requires more than a single player by design. With this, it’s the closest thing to a capture-the-flag for people who choose fantasy over sci-fi.

How other people have “hacked” Magic

Magic the Gathering has been around for over 30 years and, in that time, plenty of softwares have been built around and for the game. Below are some of those different products and projects:

Official Products

Arena is the latest app distributed by Wizards of the Coast and it covers all the great stuff from cards to mechanics, and even Universes Beyond! While their first attempt at programming the game was infamously riddled with bugs, Arena stands strong at millions of downloads plus players. In the game, they have a bot who’s available to play against named “Sparky”. While not the strongest in performance, Sparky does act as their effective benchmark for a codified Magic player.

Relevant to one of the other projects and what I ended up building is Arena has a setting to write game events to a log file in real time. What this means is, rather than have to OCR the entire screen to get data relevant to game, you could parse predictable strings (this is also how Untapped.gg is able to “replay” historical games).

Java-based

For a reason that’s not entirely clear to me, a lot of card game development in and outside of research uses Java. The two big contendors here in open source are Forge and Mage. Both consist of engines to handle the game as well as UIs for local playing experiences. When I was attempting some early experiments, I found that its headless mode was not behaving in an actually “headless” mode without UI methods being discoverable in the stack trace. And so, running the engine in headless mode was like running a browser in headless mode, it may be running some less stuff but it kinda doesn’t work without the rendering parts.

Both Forge and Mage having this UI overhead (and Forge eating up my memory faster than an old person’s Alzheimer’s), ultimately contributed to my decision later on to develop a new engine. However, they do have extensive (or exhaustive depending on your views) heuristics already programmed so these two can also act as benchmarks for playing against.

MageZero

MageZero is, sadly, not a shipped bot or even an collection of bots. Instead, it’s a toolkit (remember when people wouldn’t shut up about LangGraph?) for training your own deck-specific RL agents. The purpose behind this project is simple: different decks of cards affect the game as much as the game itself does. When you train a chess or Go bot, you’re always starting with the same pieces on the same board with the same rules. In Magic, there are cards that make people draw extra cards at the start of their turn or prevent players from casting any spells so the cards involved in a player matchup matter a great deal.

While the implication of game-changing by cards is striking, it’s built on top of Mage which means it’s dependent on its Java-based engine; ergo another not-so-preferred target for me.

Deck building bot

I found this through a comment on Hacker News where a draft player built a bot to both choose cards during drafting as well as consolidate his final deck. While this did yield a comparable win rate to when he’s not using algorithms for deckbuilding, he was the one playing those games, not the bot.

Additionally, it’s unclear whether the sets he trained against were the same as the ones in the drafts he played. Even in the case that it’s true, it still presents an interesting result to be able to map his preferences onto a neural network based on his historical gameplay (one may have expected a more intricate model to be needed for a game like Magic).

Simplified Engines

Legends of Code and Magic

For a couple years in a row, there was an online competition based on Legends of Code and Magic, a card game similar to Magic but designed so that bot matches would be fair. By having a simpler game to work with, this meant more people with less compute would be able to participate as well. Over the years, various techniques from heuristics to neural networks have all been employed but either battling over marginal improvements or scoping each improvement into modular strategies that are plug-and-play.

open-mtg

The open-mtg project might be the closest thing to what I was looking for in the first place but it falls short in a few ways.

Firstly, the last commit was seven years ago and the game has changed quite a bit since then. Secondly, it operates on a limited subset of the entire rules of the game which may or may not be misleading with respect to observed results (maybe there exists a ceiling for the number of rules that suffice in some given architecture in a non-obvious way).

What I did

Frustrations which led to a new engine

After reading papers and trying experiments, the one I’m most disappointed didn’t work was making a DSL for game strategies (imagine COBOL for describing patterns or plays) and then letting NEAT improve the underlying syntactic graph of a program. Maybe I didn’t add enough richness to the DSL or the NEAT implementation wasn’t granular enough but a lot of failures and crashes pointed to one common frustration: the Java-based engine was more than I needed and computationally expensive.

While Forge’s DSL for cards is neat, it exposes an inherent issue with keeping the engine up to date. In Magic, there are both rules and cards which may alter the game; all of these being relevant across an engine’s stack. Let’s take a look at what happens when a new set of cards is released with changes to the rules:

New work needing to be done across the stack isn’t too terrible since there is a finite number of cards out there and only a finite number of new sets or rule changes happening each year. However, the intertwining of the pieces inside the engine does mean a “vibe refactor” like what Bun did going from Zig to Rust becomes a lot trickier. In fact, a “vibe refactor” for either of the Java engines would be more non-trivial since, unlike Bun, the test suites are written in the same language as the project.

However, building a new engine with a focus on performance would help in a number of ways. First, it’d help wrestle with the question of “if we looked far enough ahead, could we generally find wins?”. Second, it’d only use the compute necessary for games so training models on more data (synthetic or self-played) becomes more realistic.

How I vibed the engine

So, if not a refactor, I need some way to vibe code a new engine (there are tens of thousands of cards out there and I am but one mere mortal). Each time I was reviewing a possible solution for verification to check agent output, the pattern I was looking for was: take the latest rules (which are available on the Wizards of the Coast website as TXT or DOCX), loop over each rule and convert it to some spec, append to our verification tool (building up a test suite rule by rule).

At one point, I was looking into preparing another swarm with formal verification like TLA+ to verify the behavior in the same way that integration test suites are used for verifying “vibe refactors”. The one I had the most hope for was dafny since it officially outputs to Python. However, it wasn’t clear if the language would be able to represent enough logic to handle the rules relating to Magic. If involved, the purpose of the verification piece is to be the checkbox that matters with respect to completion, otherwise, it may as well be AI psychosis to think anything succeeded.

But then, another realization sunk in: however I’m interpreting the text in the rules to be converted into a formal verification language is in of itself going to be easily argued or prone to faults. Then, I remembered that rules are privy to change for interpretation, an example being when a player named a card that was not the one on the board but understood to be implicitly (“Borborygmos Incident”). Arguments over the US Constitution aren’t about the content itself so much as they’re about the interpretation of said content. Given this as well as the general complexity in Magic, it seemed like the answer could be simpler:

Instead of architecting a new engine with its own complexity and costs as the game expands, boiling down the game into a means of interpreting the English texts from the rules and cards makes it more adaptable to changes to the game (which must be somewhat legalese given millions play the game and some compete for millions). Maintaining this “English interpreter” is less a problem with a codebase and more reviewing the interpretation of grammatical patterns in texts.

Which makes the choice of output language more significant. If they were more functionally complete, I’d have gone for the DSLs I made at the hackathon months ago, which were solely for this type of problem. There is, fortunately, a solution that not only follows what’s been used historically with games academically but also offers a C++ build in the end (making an integration with Python similar to how PyTorch uses Python to drive a Torch C++ backend).

Datalog

In the world of academic work relating to games, there is a tool used by the name of Game Description Language (or GDL for short) which is a variant of Datalog. While it was originally developed in the pursuit of general game playing, its base language is suited rather well for what we’re looking for. Unlike its parent language, Prolog (which was used to write the first version of Erlang!), Datalog is not Turing complete. You never have to worry about “what if this goes off to infinity?”. This is thanks to the underlying bottom-up architecture where it builds up truths from known facts in contrast to Prolog which will start with a query and then inquire into its truth-iness based on established rules.

Following the end of the Turing complete section where I describe Magic as being “transition computable” but not “game computable”, there cannot exist a deterministic program that accepts a Magic game in its entirety and strictly computes whether or not the game ends. But, there can be a program that would accept a game and tell you what happens next. This is what we would look for in an engine that describes the state of the game in order to provide to a bot which drives decisions according to wherever the engine is currently at.

Another variant of Datalog, Souffle, is the one I’m using and it also happens to have been used to find vulnerabilities in the Java JDK plus used by Amazon to verify their VPN connections. To remedy changes to larger game states (ie late in the game or a large number of creatures are in play), I had Claude translate the work from a 2021 paper and branch to a fork of Souffle. This is so our engine can handle editing the state versus refreshing from scratch and being able to discern which case is more optimal. This is useful for when a single creature kill does nothing versus when a single creature kill requires untangling a bunch of effects.

Transpiling English to Datalog

There is an old fashioned way of looking at sentences called the Reed-Kellog sentence diagram where you deconstruct a sentence into a tree where the branching describes structure and modifiers.

Programmatically, there are tools that provide “dependency parsing” where you take in some text and get back a graph like the ones shown above. The one I used is spaCy and, once you have a graph containing relations among words or tokens, then you have an abstract syntax tree!

Why this is powerful is programming languages from Rust to Odin all under the hood start by taking the source code (which is really just text), shuffling it around a graph structure, then finally producing your output. As an example in our application, let’s consider the following rule:

“If a creature has toughness 0 or less, it’s put into its owner’s graveyard.”

Right away, we know this is a conditional with the “If” at the beginning. We can prepare the conclusion “it’s put into its owner’s graveyard” as the effect from satisfying what is entailed in the condition clause. Here the only real value that matters is the “creature” and it having the “toughness” that is, in this case, “0 or less”, which maps to <= 0. One comment about the below diagram is the “O”s down in the final Souffle are uppercase o’s and not zeros (which have zero fills), this is the convention for referring to “object” variables in Souffle.

Souffle being a logic oriented programming language also makes aspects of encoding the game rather neat, such as expressing some rules as assertions rather than a chain of conditions and function calls.

As a caveat, I tried to keep it faithful to this architecture, albeit some subagents wrote in some straight regex replacements as the “intrepreting”. As funky as that choice was, I did some refactoring to backtrack and tidy up the English-to-Souffle pipeline. Ideally, regex would be optionally used structurally (ie replacing URLs with strings to get picked up as a single word by a parser) before being fed into the more grammatically aware English-to-Souffle pipeline more utilizing spaCy.

Simple API

One of the first things I had wanted when I started looking into a “Stockfish for Magic” was there being something similar to python-chess where you could install as easily as:

$ pip install chess

Then play like so:

# Import the library
import chess

# Create a board (you need one to play a game of chess)
board = chess.Board()

# First move e4
board.push_san("e4")

mtg and a couple other good names were already taken so I went for python-mtg for simplicity sake. Now you can install as simply as:

$ pip install python-mtg

Then play like so:

# Import the library
from mtg import Game, mountain

# Create a game (you need players and cards to play a game of magic)
game = Game.new(
  [mountain] * 40, # Every player has a basic deck
  starting_hand=lambda: [mountain], # Every player has at least a Mountain in their starting hand
)

# Play a land
game.play(mountain)

Bot

For making the bot, I set up a Player class that would make it easier to define and read programmatic Magic players. Rather than approach defining bots as an implementation problem, this helps heuristics be more written like how you would instruct someone to play your deck (ie when a friend borrows your cards to play a game). It also enables the below bot where all it does is play lands, cast spells, swing with everything, and never blocks.

from mtg import Player, PriorityOption as Do

class BlindAggroPlayer(Player):
  def choose_move(self, game):
    return game.prioritize(
      Do.LANDS,
      Do.SPELLS,
      Do.ATTACKS,
      Do.SKIP
    )

Or, to have more calculated preferences for different decision paths (ie if you were to MCTS):

from mtg import Mover, Player, PriorityOption as Do

class HeuristicPlayer(Player):
  def choose_move(self, game) -> Move | None:
    self.bind(game)  # refresh self.creatures, self.opponent, self.life

    return game.prioritize(
      Do.LANDS.prefer(self.land_choice),

      Do.RESOLVE_TRIGGER.prefer(self.resolve_choice, floor=0.0),
      Do.SPELLS.prefer(self.develop_choice, floor=0.0),
      Do.ABILITIES.prefer(self.develop_choice, floor=0.0),

      Do.ATTACKS.prefer(self.attack_choice),
      Do.BLOCKS.prefer(self.block_choice),
      Do.SKIP,
    )

Or, my personal favorite, being able to search for a winning path and taking it (verified against Forge with turn-1 kills). While the test of being able to perform a Thassa’s Oracle win is forced with a certain starting hand, solving the problem of finishing a game is often the last problem in minmaxing before optimizing or pruning tree search.

Vision interaction

It’s easy to assume that wrangling an entire screen with OCR would be difficult and that’s generally correct. However, we can take advantage of something from web scraping. If we took a look at the home page for Hacker News and ask how to get the titles of the top posts.

Extracting the text itself from this screen would be a challenge but if we note that each of the elements of interest match the CSS selector span.titleline then we can go from that string to the strings we’re interested in.

By going bottom-up we’re able to go from the substance to the content of interest. In the case of Arena where there are screens consisting of a lot of text.

The thing we’re then more interested in are the general shapes appearing on the screen (ie orange or blue buttons) rather than trying to find the button that corresponds to a “Play” or “Pass” action. As such, automation here can be the effect of going from content to substance.

I use Moondream for this and it improves the problem from impossible to somewhat dependable similar to spaCy. A necessary disclaimer is both of these tools under the hood are stochastic models and therefore should not be treated as formalizations but applicable tools.

Results

After getting a minimal working setup to connect the “blind aggressive” bot defined at the top of the bot section to some simulated interactions, I was able to let it drive my seat autonomously and beat Arena’s Sparky following those minimal heuristics (play lands, cast spells, swing with everything, never block).

Following that, I began working on extending the simulated interactions further for a heuristic bot that can target cards (so blocking can be translated from the engine to Arena) and got it to the point where it drove its own seat autonomously and beat Sparky three times in a row. The cards chosen for that white life-gain deck were intentional so it had an easier time with targeting; call it convenience or laziness but it got the job done.

I then began iterating on a more improved heuristic player to drive a deck I had successfully played with against others online (mono-red with Bayo, Irritable Instructor as the commander). On more than one occassion, it would drive the seat through 10+ moves and hit a snag with a new interaction with Arena that was not yet handled (ie warp casting), I’d then take over the seat only to win using actions already validated with the bot (ie it could cast spells targeting creatures owned by the opponent or myself).

While the cards in my red deck were both competitive as well as annoying (some opponents may resign out of hinderance rather than surrender), the end of the third game in the video shows the bot ignoring casting any spells and goes straight into combat since its lookahead identified a path to win the game. This validates the player framework being able to both develop a board following established heuristics, and finish a game to completion when a winning sequence of moves is identified.

Had I modified the player evaluation to always load in the heuristics before making a move, then it could have had a similar developer experience to live coding except for Magic (where I could edit the file and the player’s strategy updates in real time). But, after a couple cycles of setting up a game, letting the bot take over and drive my seat, and hitting more snags with missing interactions, it felt less like I was validating the tech as much as I was developing a cheat software. Since that’s not my interest here, I decided to let my work reach a halting point to tidy up the code and put together this write-up.

The latest inthearena work may be buggy or not even working and I’d think that’s possibly for the best to be that way. Sharing nonetheless for the sake of sharing work.

What I did with my account

Did I feel bad about doing this project with no transparency about being a bot? Yes. I tried setting up a new account solely for doing this but I already had built up a collection of cards in my first account and the onboarding took forever. Having played Magic since middle school, the game has a special place in my heart so I certainly wasn’t the most proud to be ruining a potentially fun experience for other players. At the same time, playing any game online does entail a bit of figuring out how natural or artificial your opponent’s gameplay is.

If you’re reading this, I’ve already submitted a ticket to delete my Wizards account and I have no interest in abusing this software (I also have already spent enough time playing the game, kudos to the folks who made it). By sharing publicly, I hope to share both content with folks who’d be interested in the engine work I did as well as potentially to folks working at Wizards of the Coast so they maybe have an example of what to block with anticheat work.

My hope is that Wizards of the Coast does not remove the detailed logging setting or delay it by some time window (1), but, instead, sees this as a new API or game in of itself. Lichess used to have bot tournaments and there are technically games you can code but these usually tend to be simple games or knockoffs from real ones. Universes Beyond can happen because the IP they borrow (ie Marvel, Doctor Who, Lord of the Rings) don’t have an involvement with card games (2). As such, I think even a separate tournament could be neat (don’t repeat Legends of Code and Magic where it’s just a research project, put up a small cash prize to attract attention).

Thank you for reading and, as always, hack the planet!

(1) A few-minute delay to the detailed log output would prevent it from being usable for automated gameplay but allow it to still be used by services like Untapped.gg

(2) Otherwise, I’d have really wanted to see a One Piece theme

GitHub

https://github.com/yevbar/witchcraft

From the burnout of AI psychosis to the ennui from software engineering becoming irrelevant, it’s easy to be someone working in tech or another impacted skill area (not industry) and feel like we can just scroll through websites engineered to our dopamine while we sit and wait for UBI (or UHI, take your pick). But, this is the same tune of what we said about self driving cars where, a decade ago, we were told people driving would become meaningless in five years.

What if I told you the commodification of software is an inevitable conclusion to a story that’s been building for years? Let’s begin with the opening sentence from geohot’s Sloptember post:

I’m calling it now, the adoption of AI agents into software development will be one of the most costly mistakes in the field’s history.

If you’re skeptical of coding agents, you’d probably agree with the assertion and are waiting for people to stop wasting money on LLMs. If you’re fully surfing the vibes, then you may think he’s “just not using it correctly”. However, let me suggest to you a view that’s not skeptical of coding agents while agreeing with the sentence. The previous event before we commodified software was when we commodified software talent. By this, I don’t mean recruiting but, rather, coding bootcamps.

The sales pitch for coding bootcamps is simple: you spend less time studying than you would on a computer science degree but get the same job in the end. At one point, there was a small frenzy and some would even grant a Bachelor’s degree upon completion. Someone who prided themselves on being one of the nerds who sat in the back of the classroom and studied for years to get ahead of their peers may view this as a saturation of software engineering talent. However, considering software engineers are still people and you’ll have high performers as well as low performers, all it did was add more domestic labor. Coupling this with devshops that just trebuchet engineers at problems (which only saw increased demand through the 2010s as software was eating the world), we were getting closer to a world where more people would be able to code.

If, before, your problem was you couldn’t find a person who knew how to code, now there’s a whole crowd who got the skills you need. However, like why we don’t only hire remote workers for a fraction of the salary, domain expertise can matter as much as high/low performance. Anyone who’s worked on a project with others knows people have strengths and weaknesses; a person who’s awesome at writing SQL queries may be allergic to frontend development whereas the one who’s great with CSS animation transitions may roll their eyes back at the idea of a database. Just throwing money at technical labor without clarity of the goal is as effective as waiting for monkeys on typewriters to hand you Shakespeare.

At the same time, like how all large companies are remote companies, all software practices can be codified and we can assign agents to the responsibilities that comprise an engineering org. Economically, it may be the case that smart people orchestrate systems and spend a fraction of what hiring a team would cost but it may also be the case (as practices are figured out and token prices stop being subsidized) that we end up spending more in total with “artificial talent” than if we had hired a team of people.

As software products continue to become further industrialized, holding opinions such as “LeCunn is right about LLMs” only gets a part of the picture right. Like Warren Buffet’s remark on how he’s “never met a rich economist”, something similar could be said about there not being a directly impactful theorist (emphasizing directly since Marx is what the communists read but Lenin and Trotsky were the ones who directed impact). It is true software engineering is experiencing a deskilling like frontend development, but, it doesn’t change how the internet promises all of humanity’s knowledge yet plenty of dumb people still exist. Just because AI is capable of doing things doesn’t mean it will, someone will still need to invoke it or prepare a system that does so.

Unfortunately, if industrialization and commodification has taught us something (ie mass printing of books, manufacturing and assembly lines), it’s that people will broadly value convenience or accessibility over something more bespoke (why buy a single knife when you can get the whole cutlery set). We could ruminate in that and say this is what people are actually worried about when it comes to AI replacing software engineers. Although, it would be the same as trying to make money with an investment thesis that the world is going to end; buying gold makes no sense and you’re just hoarding bananas. Instead, this should be seen as a dawn similar to that of the internet when everyone online suddenly got the superpower of reading from and communicating with others across the world.

There are numerous problems in math that get shelved or put up with a bounty because the person putting up the problem does not have the capacity or ability to tackle that problem. In the world of software, there have been numerous ideas that wouldn’t exist because there wasn’t any technical effort put in that direction (ie when your friend says “what if there was an app that could X?”). Now, we can technically make anything and delegate our focus more towards the design of the product. In the words of Meek Mill:

I feel like they can make my claude smarter who can help em do that … or what is the smartest ai program available to the people? Because the things I am learning in a week would take me 5 years to learn.

Meta recently published a benchmark named ProgramBench where the goal is simple: replicate a project without the internet (more details in their blog post). In contrast to rewriting from one language to another, this benchmark for rewriting projects without source code comes at a funny time. Extending the idea that engineers will be replaced in a couple years, software companies will begin to look more like hedge funds where models perform trades instead of people and agents write code instead of people.

Hypothesis

Rather than tackle all the problems, which may require a combination of techniques, we focused on the ones which were related to programming languages such as interpreters or compilers. LLMs are good at coding because code is just text, therefore shouldn’t a PLT toolkit help an agent across the finish line?

The original benchmark prohibits using the internet since an agent could cheat and clone the source code from GitHub. Unfortunately, this also prevents the agent from installing packages; nobody’s expected engineers to write software from scratch since the days of Bell Labs! To preserve the original intent of not letting the agent find an easy solution off the web, this experiment sticks to one language and toolkit.

Approach

We built a toolkit with OCaml, which already has an extensive history of being used for problems related to programming languages. The toolkit was given to mini-swe-agent, the same one used by Meta when publishing the original scores, as well as our harness intended for greenfield development.

Interestingly, the former performed much better than the latter. A note on that is shared after the results table.

Results

Below shows a comparison of the progress made in the original publication using Sonnet 4.6 and mini-swe-agent. veldt is the name of the OCaml PLT toolkit and the zagent captain is what was used to make sterling, our open source OpenAPI-to-SDK generator.

Note: Both veldt and zagent were run with Sonnet 4 (claude-sonnet-4-20250514) rather than 4.6

Note 2: Developing veldt was done with knowledge of the sorts of abstractions that would be useful in these tasks (as shown with jq/lua results being overfit due to there being more methods available for parsing). The validation is in other technically different problems showing benefits like typst.

zagent failures stood out considering its intention for greenfield work. My suspicion is it’s the same problem as internet data not containing reasoning data and only final outputs. Were an engineer to approach these problems from scratch (again, not a new problem), they’d maybe start by deconstructing the requirements and rationales behind design decisions. As an analogy, chimpanzees can memorize numbers but we do not expect them to abstract away addition of numbers; coding agents can implement software but we’re not expecting them to explain the business requirements.

Conclusion

We’re not yet at the point where software engineering from first principles is solved. Although, simply giving coding agents access to relevant tools makes it more productive, especially when they’re in the direction of abstractions and services related to the problem being solved. Otherwise, it’s comparable to making a web app in Assembly instead of Python, or sending SMS messages without Twilio, or using kubernetes in your stack without being able to explain why.

However, this does not mean agents can’t improve your productivity at a small or larger scale. If you’re interested in having your own fleets of coding agents working for you, then we’ve got the platform for you. Go ahead over to our docs to learn more and get started!

We took Hermes Agent, developed by the folks at Nous Research, and brought it to WebAssembly in two different ways, detailed below, and share what our general takeaways from this experiment are.

Should I replace my Hermes with one of these?

Probably not. If doing things with Python in WebAssembly is of interest to you, then continue reading!

Where did Hermes come from?

Having taken the world by storm, OpenClaw is an AI assistant which uses the computer you run it on similarly to a person (so it can do more than just recite Wikipedia articles). Following this, Hermes was written in Python on top of mini-swe-agent (unlike OpenClaw which is written in TypeScript on top of pi).

Both are “general-purpose agents” which are really batteries-included coding agents. The “magic” of their utility comes from the assembling of the “harness” the agent sits inside of when handling users’ prompts; which gives it the capability to do things like click around a browser or send an email.

Why WebAssembly?

I’m admittedly biased when it comes to WebAssembly but, having worked on multi-agent projects before, I was interested in seeing if WebAssembly would give a win with regard to isolation (ie spinning up multiple separate agents in parallel) or granular configurability (ie assembling agents precisely with certain tools for different “modes”).

Additionally, since Hermes is written in Python, I wanted to see if eagerly compiling to WebAssembly would offer closer-to-native performance.

How to WASM Hermes

Pyodide

Run Hermes in pyodide yourself

This app serves an index.html with the full agent running client-side in Pyodide. To run the below command, you may need to install the vers CLI first.

vers run-commit f46a2b21-73fe-4835-ad79-8eccd523fc07 \
  --format json --wait \
  | sed -n 's/.*"vm_id"[: ]*"\([^"]*\)".*/https:\/\/\1.vm.vers.sh/p'

Public Vers VM Commit: f46a2b21-73fe-4835-ad79-8eccd523fc07

How it works with Pyodide

The first approach is by using Pyodide, a Python runtime that’s ported to WebAssembly so Python programs can be interpreted and run in the browser. You can think of this as being similar to the approach that was taken with bringing Postgres to WebAssembly:

Postgres itself doesn’t get run in WebAssembly but instead a Linux emulator in WASM runs a modified version of Postgres so the whole thing can actually work together inside a browser.

Hermes in Pyodide source

You can view and modify the source code here: https://github.com/hdresearch/hermes-pyodide

pywasm

Run Hermes in pywasm yourself

This app serves the hermes_agent.wasm binary. Hit the “Run” button in the UI to execute it live.

vers run-commit f83df1ac-0a53-4ca6-bc26-205584fe65a3 \
  --format json --wait \
  | sed -n 's/.*"vm_id"[: ]*"\([^"]*\)".*/https:\/\/\1.vm.vers.sh/p'

Public Vers VM Commit: f83df1ac-0a53-4ca6-bc26-205584fe65a3

How it works with py2wasm

This second approach works by using py2wasm, a Python-to-WebAssembly compiler and the pywasm split design keeps the security boundary clean:

The host extracts real schemas from the Hermes’ ToolRegistry at startup before sending them to the WASM binary via init protocol. The LLM always sees the same parameter names as the actual handlers (ie path instead of file_path or old_string instead old_text).

Hermes in pywasm source

You can view and modify the soure code yourself here: https://github.com/hdresearch/hermes-pywasm

Benchmarks

Below are benchmarks obtained from running on a M4 macbook. As admitted earlier, this probably won’t meaningfully replace running Hermes on your laptop. However, if porting the harness itself to alternative environments (ie a browser) is of interest to you, then you can see some of the tradeoffs between pyodide and py2wasm.

Takeaways

While the founder of Docker years ago suggested WASM+WASI was the missing sandboxing solution, it’s evidently not a magic bullet considering the missing capabilities from a full-fledged computer or container. If having a full but branchable VM with incredibly fast startup times sounds like what you’re looking for, then go on over to Vers and get started!

We took MemPalace and extended its techniques to close the gap in the LongMemEval recall@5 retrieval benchmark to get a reproducible 100% score using only local compute (no LLM or API calls).

What this is not

• Not a LongMemEval leaderboard score. The full LongMemEval benchmark is end-to-end and involves generating answers plus GPT-4 judging. This experiment is strictly about the same retrieval metric that MemPalace was tackling.
• Not a strong metric. The metric is recall_any@5, the softer variant. recall_all@5 (requiring every gold session in the top 5) would be a harder bar.
• Not an novel algorithm. Iterating on failures from the dataset, the patches made are general NLP patterns. A new benchmark could be put together with different heuristics required but that just continues the cat-and-mouse game of developing “human-comparable intelligence”.

These caveats aren’t intended to steer your attention away but more set the expectation of an interesting result. The central takeaway of grammatical patterns in text being applicable to vector stores still deserves some acknowledgement.

What we did do

We achieved 100% recall@5 retrieval on all 500 LongMemEval questions. The system uses no language model, makes no API calls, and requires no GPU. The MemPalace baseline on the same metric is 96.6%, so the +3.4% improvement represents a real engineering output. Shared in a project dubbed Retaining, it does:

• 500/500 R@5 (100% recall at rank 5)
• 500/500 R@10 (100% recall at rank 10)
• Fully deterministic and reproduced across multiple runs

Context

On April 6th, Ben Sigman shared that Milla Jovovich had fun with coding agents and built a solution for long-term memory named “MemPalace”. For those who are fans of sci-fi movies, you may recognize Jovovich as the one who played the Fifth Element as well as Alice in Resident Evil. The cherry on top is, at the end of the Resident Evil series, Alice is enabled to tackle the antagonist after her childhood memories were uploaded to her; a rather similar message to enabling agents after giving them a “memory palace”.

Originally proclaiming it to score 100% with optional Haiku rerank before backtracking, it’s racked up a good volume of attention and validation so it’s not totally “viral slop”. By both compressing content and making historical context navigatable, it highlights the efficacy of simple NLP techniques when applied with LLMs.

Improving MemPalace

What worked

If you’ve seen structured note taking like the Cornell Note Taking System or backlinks in Obsidian, then you know there’s more to outlining text than just indexing when or where words occur. With spaCy and named entity recognition, we can extend the existing pipeline by including noun phrases or other grammatical relations that give a more detailed picture of the “ontology” representing the content at hand.

Below is a table of newly added techniques and how much they contributed to the recall@5 performance:

Individual: technique alone added to the baseline. Cumulative: technique added on top of prior ones. Deltas overlap and do not sum to total.

The top three contributors are all simple re-ranking heuristics. The logic engine contributes modestly and actually causes the most regressions. The finding from this experiment: enrichening NLP extraction in a retrieval pipeline can produce more than improving the logic engine that queries them. (damn you bitter lesson!)

In more detail

1. spaCy-based extraction

Every session gets processed through spaCy’s en_core_web_sm pipeline. We extract entities, noun phrases, relations (subject-verb-object triples), time-related markers, and quoted phrases. This takes ~5 seconds per question’s haystack when run on my Macbook.

2. Pure-Python logic engine

A LogicKB Python class that stores extracted facts as inverted indexes. For each query, it looks up matching objects across all sessions, returning a weighted score per each one. This replaced an earlier Prolog approach with the same idea but much less complexity and no IPC overhead.

3. NER-enriched synthetic documents

For each session, we create an document containing its extracted facts and details. These get indexed alongside the raw session text, giving the embedding model a richer retrieval surface. This is the single biggest contributor to accuracy.

4. Noun-phrase embedding bridge

We embed each session’s extracted objects into a separate ChromaDB collection and query it with the question’s noun phrases. This bridges gaps that neither keywords nor full-document embeddings can cross. “Battery life phone” → “portable power bank” has a close enough embedding distance in the noun phrase space to pick up the right session.

5. Time related bridge

For time-related questions (“What did I buy 10 days ago?”), we first identify all sessions in the date window, then run the noun phrase bridge within that filtered set. This discriminates between 14 sessions that all share the same date by finding the one whose noun phrases are topically closest to the question.

What didn’t work

When people complain about LLMs not being able to answer questions or hallucinating false information, what nobody complains about is the LLMs’ ability to identify the question it needs to answer (we can depend on AI to write code that does a thing rather than depend on it to end-to-end handle a task). In the subject of answering questions or digging through “long context problems”, I first attempted to have the LLM use Prolog for storing and retrieving facts.

However, the semantic fuzziness (ie synonyms or finding similar topics to a query) ended up hurting the overall score more than helping. The approach in MemPalace to depend on a vector store actually showed to be “more correct” in this experiment.

Nevertheless, I do think there may be types of problems where realistic input queries (ignoring cases where people are funny and test jailbreaking support agents) would be usable with a more structured and queryable store of relations between objects. Prolog just may not be a low-hanging fruit solution for long-term memory problems where semantic similarity is something worth indexing.

Running yourself

First, clone the repo and install dependencies.

git clone https://github.com/hdresearch/retaining
cd retaining
python3 -m venv .venv && source .venv/bin/activate
pip install spacy chromadb
python -m spacy download en_core_web_sm

Next, download the dataset for the benchmark.

# Download LongMemEval data (~265MB)
curl -fsSL -o /tmp/longmemeval_s_cleaned.json \
  https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json

Lastly, run the benchmarks.

# Vector-only baseline: 96.6% R@5, ~5 min
python bench_v2.py /tmp/longmemeval_s_cleaned.json --mode vector

# Full hybrid: 100% R@5, ~50 min
python bench_v2.py /tmp/longmemeval_s_cleaned.json --mode hybrid

No API keys. No GPU. Python 3.9+ and ~300MB of disk.

Conclusion

AI famously hit “winters” in the past when some wall prevented computers from becoming sufficiently intelligent. Interestingly, the problem in the past was that “symbolic” approaches to AI would fall short when it came to the last mile of complexity. Similarly, LLM-maximalist approaches also run into a “last mile problem” when it comes to ensuring accuracy of details (ie hallucination).

By incorporating older NLP techniques to tackle the “last mile problems” with modern approaches involving LLMs, there are rather interesting results to be found! Albeit, the implementation used here to game recall@5 is, certainly, by no means a complete solution for knowledge retrieval.

The beauty of the finding is that the problem of “if only someone had sat down long enough to write every NLP grammar rule” now becomes somewhat negligible in a world with coding agents. So, rather than continue to see human text as black boxes, know that a richer pipeline may get the sufficient amount of complexity for some information to be adequately indexed.

• What is this?
• What is this not?
• The background
  • Coding agent
  • Ralph loops
  • RLMs
• zagent terms
  • Code cannon
  • Code pirate
  • Code captain
• Takeaways

What is this?

This is an overview of the principles I used to assemble zagent, a coding harness for getting more progress out of a single “shot”. I’ll be both describing the topics I worked on top of as well as the structure behind what I put together.

If you’re looking for a post to read that gives copy-and-paste’able commands, this isn’t for you. If you’re alright with reading something more explanatory, then continue on!

What is this not?

zagent is not going to replace your Claude code or pi (which I predominately use) but the ideas below should be high level enough you can implement it in your own harnesses or coding agent systems.

The background

I didn’t by any means invent a new model or algorithm, I simply applied together some existing concepts which are accessible yourself. Being transparent, this is my way of building up towards a general purpose version of what Google accomplished with AlphaEvolve.

To break down what the heck is going on with zagent, there are three “primitives” in the area of coding agents that would be useful to know.

Coding agent

From editors like Cursor to headless systems like Devin, there’s a large variety of offerings that all fall under the notion of “coding agents”. Simplifying it to the bare minimum, a coding agent is an AI that can take a prompt from someone and write code to accomplish some goal. However it may be accessed by a user (could be tagging in a Slack workspace, sending a message on Telegram, writing a prompt from a UI, etc), the underlying step from general agents is that it can write and run code.

Sometimes underappreciated in domains other than literally writing software, the power of coding agents is in how much is built on top of code, making them immediately ‘effective’ in the world around us today. It could be a “short lived” agent that only runs to solve a specific problem before exiting or a “long running” agent with a growing memory. Depending on the particular use case you’re looking for, one may be better than another.

In the context of writing software that delivers something, I’ve personally found the philosophy of short lived agents to be better suited.

Ralph loops

Ralph Wiggum loops, named literally after the Simpsons character, is a technique for working with coding agents that looks something like the below pseudo-code:

while not done:
    fire coding agent at task(s)
    repeat until done

For instance, the Claude code plugin would run a while true loop in bash until the LLM outputted a specific string indicating it had actually completed the task rather than said things which sounded nice. In pseudo-code that’d look something like:

while "DONE" not in last_output:
    fire claude code and tell it to say "DONE" when finished
    continue until done

To avoid running out of context (and to keep productive when one goes to sleep), folks would run “ralph loops” since the while true serves as a way to reset the context over and over, letting it run ‘infinitely’. In the case of problems where the task is going through a large bullet point list of items (ie meticulously writing unit tests across a large codebase), it works well since the tokens that filled up the context about prior solved items isn’t relevant to the context needed for solving problems moving forward.

However, in the case of problems where you do lose something by resetting the context (ie a complex integration which requires knowing about all the pieces involved to be useful), then ralph loops can fall short. While still a useful technique, it’s no longer meme’d as a solution for “solving programming” for this reason.

RLMs

An idea popularized from a blog post and then published to arXiv, RLMs broadly solve the problem of “running out of context” but in an importantly different way. Rather than place the “infinite loop” above the LLM (like done in the ralph loop), what if the loop were conceptually brought into the agent loop itself? In RLMs, this is done by letting the agent recursively call itself or other agents before coming back with a final answer.

Explaining how this works with LLMs but with an analogy: suppose you wake up to a text message asking you to research something that you have five minutes to respond to but you haven’t had the chance to even have coffee yet. Lacking the energy to Google around, you text someone else who you think either knows the answer already or wouldn’t mind finding it, they get back with the answer, you forward to the first person, and then all’s done.

A profound utility from this is being able to “stretch” your context window since spawned sub-agents can go through their context windows exploring something rather than the top-level agent you provided the original prompt to. Nowadays, in conjunction with stuff like memory, some of the older problems with arbitrarily large context windows have tools for tackling them.

Where “infinite context” can fall short can be broadly explained by how “completely illuminating a house such there no shadows” makes it uninhabitable. It’s no secret LLMs can be convincing whether to themselves to users falling into AI psychosis. As a result, letting an agent ruminate on some goal or task (even if it’s rational like programming), can lead to adverse results which are seen as unproductive to the person hoping to finish an app or such.

zagent terms

Inspired by my experience with herding coding agents, there are three layers I’ve assembled into zagent that apply the above ideas. Before you ask, yes, the names are inspired by One Piece.

Code cannon

In my prior projects using “code cannons” like rewriting git in zig or developing a modern toolkit between Elixir and WebAssembly, what I was really doing was leveraging Vers VMs as the RLM environments in which sub-agents were working on scoped problems. To differentiate from the ideal of a code factory, this RLM pattern is what I’ve referred to as a “code cannon”.

In the case of rewriting the git CLI, there are several subcommands which can be worked on in parallel (and on different files which can prevent conflicts when merging changes). You can think of this like how, at a hackathon, you may have one person working on the backend, one person working on the frontend, and one person working on the slideshow presentation; each of them can work on their piece of the overall project without stepping on each others’ toes.

Code pirate

Taking a step back and contemplating what I was really doing when “firing code cannons”: I would see what the progress or status of changes were, break down the next wave of changes I wanted to see, provisioning new agents with their respective prompts, and letting it run for a while before coming back to my laptop and repeating.

Enter the “code pirate”, a ralph loop that works from a markdown file firing code cannons until it finishes more substantial progress.

By bridging together the context-resetting of the Ralph loop (the pirate) and the context-mindfulness of the RLMs (the cannons), it establishes a coding harness which is able to accomplish larger diffs like building out sterling (if it’s still private, it’s coming soon!).

When I come back to my computer to review a captain result, it’s less about knitting knots in feature intentions and more about steering the army of coding agents overall. Making sterling with the code pirate was less about firing it over and over at a goal but more setting goal(s), it finishes them through, and then setting new goals to be implemented (like making a peanut butter jelly sandwich).

Code captain

Everything up to this point I can say truthfully has yielded a real result that would have taken more time or effort if I used a different tool. This next “layer” is something I’ve been tinkering with and have not yet found something that feels like I “cracked it”. However, I’m sharing here in case the concepts are of use to someone else facing similar problems.

When tackling projects where it “working” is non-negotiable (ie it meets a test coverage quota, an ambiguity that would lead some agents to giving up early), totally depending on the LLM to come back with a result can be anticlimatic.

To remedy this while tinkering with Lean, I’ve started working on a “code captain” which behaves like a code pirate but, rather than let the agent exit when it’s gone astray, I added a gate which prevents the pirate from exiting until all conditions are met.

If the gate’s not well defined, then the agent can find a way to exit early. If the gate’s redefinable (ie learning about new objectives or constraints over time) or even appendable, then the agent may still find a way to exit early. So, ultimately, software engineering’s a game of scoping objectives well.

Takeaways

Training employees versus hiring interns is like the difference between vertical and horizontal scaling. Likewise, the difference between leveling up a single person versus spinning up agents to fill in certain tasks is like the difference between vertical and horizontal scaling but for responsibilities. The underlying problem with coding harnesses is boiling down the responsibilities of a software engineer into horizontally scalable skills.

It’s already the case in some hedge funds that folks will develop models for executing strategies but aren’t picking up the phone and placing orders themselves. While there are still some firms which rely on old fashioned methods, the analog to software is that there will eventually be categories of products where the code defining these products isn’t governed by people but instead by the systems established by them.

Until the day coding’s finally solved, we shall still have problems to solve. Hack the planet!

We rewrote git in zig and:

• Sped up bun by 100x
• Got 4x faster than git on an arm Macbook
• Compiled to WASM to be 5x smaller with 8.5x more exports
  • Check out this demo to clone a repo in your browser!

Rather than start with the theory behind the “swarming”, we’ll share how to code cannon yourself, describe how our zig rewrite of git went, and then dive into some of our theory behind why this works.

How to code cannon yourself

Install vers CLI

First you’ll need the vers CLI installed.

$ curl -fsSL https://raw.githubusercontent.com/hdresearch/vers-cli/main/install.sh | sh

After you’ve installed it, log in.

$ vers login

Now you have a working vers CLI ready to prepare your swarm infrastructure.

Configure environment variables

With the vers CLI you can define environment variables which get injected to all the VMs you create, making authentication for some CLIs a breeze. Here we’ll walk through the environment variables we included for this project.

First, create a new GitHub repository and then follow the instructions for creating a fine-grained personal access token. You’ll want to create one that has Read and write access to content for the repository you’re going to work on.

We configured it to have access to only the repos we’re interested in code cannoning at for this project. Our rationale being we don’t want one or multiple agents to get creative and start integrating other projects that aren’t relevant. Once you have that API key then set it like so:

$ vers env set GITHUB_API_KEY github_pat_...

Next, from the vers dashboard click on the API Keys tab and create a new API key. After you’ve written it down someplace you won’t lose it, you can set it to your environment variables (so an agent running in a VM would be able to on its own spawn further agents).

$ vers env set VERS_API_KEY abc123...

Finally, since we’ve been driving this using Claude, let’s set an ANTHROPIC_API_KEY so any coding agent running in a VM works out of the box.

$ vers env set ANTHROPIC_API_KEY sk-ant-...

Write your initial plan

We’ve shared the plan.md file we used for the zig rewrite of git, you’re welcome to copy it and tweak for your project. Once you have it written, simply point your coding agent at it.

$ pi "Read plan.md and let me know when I can quit this session"

The prompt specifies that contributing agents should be spun up in VMs so you can close your laptop and know things are still progressing.

Let it start running

Eventually pi or your coding agent will tell you agents are working and you’re good to quit the session. Congrats, you’ve successfully created a code cannon to work on some problem.

Check where it’s at

Regardless of the size of the project, since there may be small features or nits you’d like to include anyways, it’s good to check in after agents began working to verify what it is they’re working on. If you find yourself glossing over the agent descriptions and more crossing your fingers than walking away knowing your progress, you’ve likely depended on the agents too much for your goal.

Repeat running and checking in

We found it useful to check in on the swarm similar to checking in with a team during standup but on an admittedly more frequent basis. Rather than provide a prompt to scale up/down the swarm after certain checkpoints, being more hands-on with steering allowed us to also get a clearer understanding of the scope of this project as well.

How we rewrote git in zig

Anthropic took a stab at rewriting the C compiler and Cursor took a stab at rewriting a web browser. It’s not that hard for you to do the same and here’s how we went about rewriting a big open source project with the help of agents!

Environment

The Vers VMs spawned had the environment variables injected at startup (so running pi with instructions will always work).

• ANTHROPIC_API_KEY: For the LLM powering the coding agent
• VERS_API_KEY: For further orchestration
• GITHUB_API_KEY: A strictly scoped API key for just one repository

The initial plan

Below was literally the one markdown file used for the original agent to spin up a swarm.

The goal is to make a modern version control software like git or jj but written in zig

ALL SYSTEMS AND AGENTS MUST use this github -> https://github.com/hdresearch/ziggit.git

For each of the below goals, create a VM and run code like the following

```bash
while true do
  pi -run "GOAL"
end
```

NOTE - pi is running on the VM itself rather than running on the host machine and then ssh'ing commands. This should be done so we can quit this pi session

So agents are just infinitely running since there is always something to improve in a piece of software. Include pi-vers extension so each infinite loop can provision further VMs or agents.

- first person like jj but does not have a `jj git` subcommand and instead is drop in replaceable with `git` so `ziggit checkout` not `ziggit git checkout`
- feature compatibility with git (copy over test suite from git source)
- can compile to webassembly
- can yield performance improvements to oven-sh/bun codebase by using directly with zig integration instead of libgit2 or git cli

Maybe wait for some progress before starting on replacing bun's usage of the git cli (which they use over libgit2 for performance reasons, our suspicion is that a modern solution in zig could be better). Every VM should have the env vars `VERS_API_KEY`, `ANTHROPIC_API_KEY`, `GITHUB_API_KEY`. Also use the hdresearch/bun fork with changes so a real PR can be created pointing at oven-sh/bun BUT DO NOT MAKE THIS PR YOURSELF. Provide instructions for a person to validate the benchmark results with ziggit usage first

We copied over the plan.md used for firebird and the -run argument is not a real argument, the correct one is -p but the top-level agent figures it out anyways.

The produced agent loop

From the markdown plan, our local pi agent created a golden image for the VMs working on the ziggit codebase to use and configured each agent to have different git commit authors so progress would be identifiable.

Every agent additionally got a /root/prompt.txt file with that agent’s specific prompt. The agent tasked with covering git’s test suite would have that file populated with contents like "You are the CORE agent. Run git's test suite and fix CLI bugs." and the agent tasked with improving certain git index functionality would have that file with contents like "You are the NET-SMART agent. Rewrite idx_writer.zig to be 10x faster.".

Finally, every VM runs the exact same bash loop encompassing the coding agent itself as well as the git cleanups referenced earlier. The below was generated by the top-level pi agent orchestrating these coding processes in VMs for how to define a given agent.

#!/bin/bash
set -a; source /etc/environment 2>/dev/null; set +a
export HOME=/root
export NODE_OPTIONS="--max-old-space-size=256"

cd /root/myproject || exit 1

while true; do
    echo "$(date): === Starting agent run ==="

    # 1. SYNC — save dirty work, pull latest from other agents
    git add -A
    git diff --cached --quiet || git commit -m "auto-save before sync"
    git fetch origin master
    git rebase origin/master || {
        git rebase --abort
        git reset --hard origin/master  # nuclear option on conflicts
    }

    # 2. BUILD — rebuild the project
    zig build  # or whatever your build command is

    # 3. RUN PI — the actual agent work
    pi --no-session -p "$(cat /root/prompt.txt)"

    # 4. PUSH — commit and push whatever pi did
    git add -A
    git diff --cached --quiet || git commit -m "auto-save after pi run"
    for attempt in 1 2 3; do
        git pull --rebase origin master || {
            git rebase --abort
            git reset --hard origin/master
        }
        git push origin master && break
        sleep 5
    done

    sleep 10
done

It executes every loop by saving work from the prior loop run, pulling in latest changes, rebuilding the project, running the pi agent, and then repeating the same git operations at the end with also pushing. The agent prompts themselves also mention to use git operations for auditability but these git failguards around the agent itself help ensure the agent loop doesn’t get stuck along the way.

Meta note

To reiterate a point at the end of a another section, the sub-agents aren’t doing anything differently from if you were to be manually starting new agents with their respective prompts yourself. These shouldn’t be doing anything you can’t directly understand, whenever we found ourselves starting with an initial research goal (ie understanding a point of integration before beginning new loops) and letting the agent in front of me handle the rest, we’ve ended up with a mess to clean up.

Similar to how LLMs can be poor at writing configuration files, we’d guess complex integrations fall under a similar category of “problems LLMs do a lot better with a human around” and, should you be working on one of these tasks, make sure every detail relevant to your intended prompt or plan for an agent to carry out is in the context you hit Enter on.

What it cost

At the end of this crunch which consisted of nearly a week and ~13 billion tokens, we successfully created a rewrite of git in zig. If you were to do that as a human, say writing a new git of your own, and you were to work towards 100% test coverage, you’d be in for a world of pain.

The git CLI test suite consists of 21,329 individual assertions for various git subcommands (that way we can be certain ziggit does suffice as a drop-in replacement for git). If it took a person four minutes to write enough functionality to pass each test (overlooking some tests being more complex than others), that’d amount to 85,316 minutes total, or about two months! And that’s without sleeping or eating included in the number.

While we only got through part of the overall test suite, that’s still the equivalent of a month’s worth of straight developer work (again, without sleep or eating factored in).

The final results

bun improvements

The bun team has already tested using git’s C library and found it to be consistently slower hence resorting to literally executing the git CLI when performing bun install. With ziggit, it becomes possible to see upward of 100x speedups for some git operations.

Tested on an M4 Macbook with 24gb of RAM across multiple runs, it scored an average of 85.4x speedup for findCommit, 7.3x speedup for cloneBare, and a ~10x speedup for the entire workflow comprising of git operations. In a x86_64 Linux VM with 8gb of RAM, it scored an average of 6.3x speedup for findCommit, 34.3x speedup for cloneBare, and a ~30x speedup for the full workflow.

When evaluating the complete bun install improvements, it came out speed-wise to about the same as the existing git usage (due to networking being the big bottleneck time-wise despite more cases being slightly faster with ziggit over multiple benchmarks). Except, it’s done in 100% zig and those internal improvements pile up as projects consist of more git dependencies. All in all, it seems like a sensible upstream contribution.

git drop-in

In addition to covering enough functionality to replace bun’s usage of the git CLI, ziggit covers enough subcommands and arguments to be a viable drop-in replacement for git with numerous performance improvements. While there are codepaths where the two are at 1x performance comparisons, it’s remarkable that a modern rewrite in a modern programming language was able to reach that level and even get up to 10x speedup for some commands!

While git itself has had much more development and optimizations for x86_64 Linux, ziggit’s performance really outshines git when measuring on an arm64 Macbook. On our macbook, it’s across the board more than 4x faster than git in both smaller repositories as well as larger ones.

Of course, ziggit comes with git-lfs support as well and a useful succinct mode meant for agents working in new or existing git projects to save significantly in tokens!

WebAssembly

Currently, there’s a wasm-git project which compiles git’s C library directly to WASM and comes out to 806kb large. ziggit, when compiled to WASM, produces a binary that’s only 148kb big. That’s 5.4x smaller already on its own and then it can get down to just 55kb when compressed, making it more portable and accessible.

Additionally, ziggit’s WebAssembly binary provides 68 named distinct exports (ziggit_init, ziggit_clone_bare, ziggit_diff, ziggit_log, etc) in contrast to wasm-git’s 8 obfuscated exports (X, Y, Z, _, $, aa, ba, ca) which are Emscripten-compiled C bindings. Nonetheless, talk’s cheap so you can go ahead and clone an open source repository in our web demo.

Succinct mode

Inspired by rtk, a CLI proxy which reduces LLM token consumption by 60-90%, ziggit also includes a “succinct mode” that’s enabled by default and dramatically slims down outputs. For example, the below:

$ git commit -m "chore: add another file"
[master b6eeb42] chore: add staged file
1 file changed, 1 insertion(+)

Becomes the below:

$ ziggit commit -m "chore: add another file"
ok master 640fe38 "chore: add another file"

Or compare the below difference between git status and ziggit status:

--- normal ---                              --- succinct ---
On branch master                            * master
                                             + Staged: 1 files
Changes to be committed:                      staged.txt
  (use "git restore --staged ..." ...)       ~ Modified: 1 files
        new file:   staged.txt                 README.md
 Changes not staged for commit:
  (use "git add ..." ...)
  (use "git restore ..." ...)
        modified:   README.md

Succinct mode is turned on by default and can be toggled off by passing --no-succinct like so.

$ ziggit --no-succinct status

Or by setting the GIT_SUCCINCT environment variable.

GIT_SUCCINCT=0 ziggit status

Theory

Now, why does any of this work? Here’s our guess having done a similar thing before when making a modern toolkit to bridge Elixir and WebAssembly.

Agent spawned agents is like being a manager of managers

Having direct reports who work with you is vastly different from working with reports who themselves have reports.

Normally, when you’re working in an organization of people, you need to be mindful of the balance and delegation of tasks; this has to do with everyone’s experiences as well as APMs. When you work with coding agents, you could sit and create a coding agent for every individual task or you could have an agent (which itself has a high APM) be the one doing the orchestration:

But, really, this wasn’t a “hands off the wheel” project where we hit Enter once and left the laptop; although we got sleep in the process. Instead, this was more like doing exactly what we would have done if we had a row of laptops on a table and we’re typing on each one except there’s an agent to do the menial part of setting up subsequent coding agents:

For the early part of the work, we prompted the top-level agent to create certain agents for the initial scaffold (in this case: core git functionality as well as identifying where to place the zig code in Bun’s codebase). Once there was enough groundwork laid out, we directed the top-level agent to spawn different agents we knew could work in parallel (ie one was focusing on WebAssembly capability, one was focusing on the exact git functionalities to rewrite to 100% Zig for Bun).

For scenarios where we figured one agent was not going to fulfill some capability in a reasonable amount of time (mind you, this stuff is eating up billions of tokens so not like it’s absurdly unreasonable in the first place), we’d have multiple agents working in the same part of the codebase except the logic wrapping the agent itself (both in the prompt and in literal shell scripts), we use git to rebase or stash or push changes along the way. This both ensures agents don’t tunnel vision themselves into stuff that’s never pushed and agents can be failure tolerant when one gets a task that was already handled by another agent.

Why we think this works

We’ve successfully applied this approach before when bridging Elixir and WebAssembly and have a guess as to why this works. To explain, let’s talk about making a peanut butter and jelly sandwich.

For context, one of our favorite examples for introducing computer science is the exercise of writing instructions for how to prepare a peanut butter and jelly sandwich. It’s a staple I remember from Harvard’s CS50 and have done enjoyably a number of times when I was teaching others how to code pre-LLMs.

The way it goes is you have all the ingredients and tools you’d use to prepare a PB&J (bread, peanut butter, jelly, plates, so on) as well as something to write on and something to write with (such as blackboard, whiteboard, paper, text editor). You begin by instructing the group to provide (so it can be written down) the instructions for preparing a PB&J while, along the way, you follow instructions extremely literally such that a sandwich never gets made (unless you’re nice about it). The goal isn’t to demoralize your students into thinking they can’t define steps but more to emphasize how “dumb” computers can be and how explicit code needs to be for a program to do what you expect.

If you prompt an LLM to make a PB&J, assuming it has access to whatever’s needed in the real world with robot arms plus all the cool hijinks, you’ll likely end up with something much like how you can prompt a coding agent to make some program and it will likely end up with something. If you want to ensure that every sandwich made uses apricot jam, that’s something to specify in the instructions. If you want to ensure some web app generation always uses a certain component library, that’s something to specify in the instructions as well. LLMs are great because they can do things but whichever details you care about must be specified similar to how a human doing the PB&J exercise would need the orientation of the knife and so on to be specified.

The peanut butter and jelly sandwich example works for standard coding because computers need programs to be precise. The example also works for LLMs coding because agents need prompts to be precise. To tie together how one could see that coding agents have the potential to solve a hefty number of engineering problems, let’s consider two things that we know today LLMs are able to do:

1) Build out an initial MVP or prototype

• While this was an early critique for coding applications of LLMs (since they can’t do “real engineering work”), it’s worth admitting this does knock off legitimate work that’d otherwise take a person time to do. 2) Targeted optimizations that are verified by the LLM
• Google showed this already with AlphaEvolve and, in a more broad way, the Deepseek moment shows this point further. Rather than throwing hands up in defeat and running LLMs over and over like monkeys on typewriters, giving LLMs access to the metrics a human would be trying to steer towards in the first place lets them self-guide till they get the job done.

By being able to both legitimately start a project as well as improve it in the directions desired, putting aside the verbosity needed in the prompt or time needed to process, LLMs and coding agents have the capability of tackling a “real” number of engineering problems. It’s not about replacing humans or finding things humans can’t do at all, it’s about overall coordination in the vein of enriched productivity.

At this point, we have all the fundamental pieces for why this approach is productive: meaningfully organizing and directing coding agents with a “top-level” agent doing the administrative work for you. Being able to work with the top-level agent and improve sub-agent prompts or loops also let a deployed agent not be the end all be all but instead iterative.

What was funny about steering this system of agents is it was reminiscent of seeing demands of engineering teams evolve over time like the startups we’ve been at; when the group needs to focus on a refactor or tasks can be divided in parallel, agents can be redirected towards something or spawned/killed according to the codebase’s demands. The point here being there wasn’t a single organizational structure or scaffold which was the “best”, our orchestration was more dynamic as I went along with the project.

An important note about organizations of these agents we’ll add is Kernighan’s Law.

Everyone knows that debugging is twice as hard as writing a program in the first place. So if you’re as clever as you can be when you write it, how will you ever debug it?

If you point the top-level agent at the task of figuring out the most clever tricks possible, you’ll end up with a mess of agents and a lot of token burn for no good reason.

We don’t yet have a prescriptive solution for this but the rule of thumb we’d state is, at any given point in time, you should be able to see a list of running agents and understand the progress they’re making. If you find yourself in a spot where you wouldn’t know where to begin steering, you’ve likely leaned too much on the agents to do something you were responsible for.

Hack the planet.