July 10, 2026
As part of research for the Magic project, I came across the Pokemon AI Battle Challenge. At the time, it looked like yak shaving to go into it considering I didn’t play their TCG growing up. After wrapping up and shipping the Magic engine, I thought I was perfectly prepared to revisit the Pokemon competition. Like Magic, Pokemon is an incomplete information game. Unlike Magic, Pokemon doesn’t explode to the same degree of complexity and is simpler in a number of appreciable ways.
To compensate for joining a multi-month competition halfway through, I decided to timebox to a couple days and try approaching a generalized bot. Knowing Magic and understanding strategies to the game is what enabled me to hone in on stronger heuristic bots with the previous project. Given there are likely to be people who have that level of familiarity but for Pokemon, I know I’m faced with two options:
- Study the game and the allowed cards to find a strong meta
- Learn just enough to make a functional bot and see if I can iterate on that
While an LLM could do the first and I have seen that done before, I didn’t trust my intuition for evaluating its output so I opted for the latter. I shipped a few bots that were able to play and win games but nothing I vibed broke the threshold for actually performing well against the field. Based on the result of the Legends of Code and Magic annual competitions, most people there focused on heuristic bots that leaned on specific strategies or they applied neural networks to certain mechanics of the game. My suspicion is you could analyze the submissions in the Pokemon competition and find something similar.
So, right away, the Pokemon game deserves a kudos for not being immediately LLM’able. Perhaps if the AGI narrative holds, then benchmarks would be less measures of intelligence but more akin to post-quantum cryptography. Several years ago, quantum computing was more hype but now the US Government takes it seriously. What post-quantum cryptography is about is technically just algorithms but they can also be understood as techniques for dealing with that strength of technology. Likewise, if AI on its own gets good at real world problems, we may benefit from some “post-AI riddles” that keeps AI in check like how gravity keeps us grounded.
As for takeaways, I think there are two I’d like to walk away with at least:
- Friendshaped doesn’t mean friend: Data can meet a format but the model may not be representative of its behavior
- It’s better to be early and underdressed to a money making party than late: I can’t find the original tweet but it went along that idea.