what building my own devin taught me

I built devin.baby because I wanted to understand Devin from the inside. You type a coding task. Something decides whether you even need a machine. If you do, an isolated environment boots, a model drives tools against a real filesystem, and progress streams back to a browser. There's an architecture board on excalidraw if you want the shape of the product before the plumbing.

Building it taught me that the interesting part is not the chat UI. It is every layer underneath: what a hypervisor is, why Firecracker exists, how a "sandbox" becomes desired state in Kubernetes, why restored guests hang on TLS, and why the model should never sit on the same machine as the shell it controls. The deploy side of that story lives in ops. This one follows a single task from the browser down into a microVM and back out again, and stops at each layer long enough to make the next one make sense.

Why isolation is the product

A coding agent that can run shell is a remote code execution API with a nicer prompt. If that shell shares a filesystem with your API keys, your Postgres, or other users' workspaces, you do not have an assistant. You have an incident waiting for a clever prompt.

Containers help, but they share a kernel with the host. A kernel bug, a misconfigured mount, a privileged sidecar: the blast radius is still "this machine". Full virtual machines help more. Each guest has its own kernel. Escape means getting out of the hypervisor, which is a much smaller surface than "every syscall the host kernel exposes to containers".

That is the product requirement hiding under "run the agent in a sandbox". Not a Docker-in-Docker trick. A real boundary between planning and execution, the same split Cognition describes as Brain vs Devbox. The Brain decides. The Devbox runs code. They share a network. They do not share a trust boundary.

Once you accept that, most of the architecture stops being optional. You need hardware virtualization. You need a way to boot guests quickly. You need a control plane that can say "there should be a sandbox for this task" without SSHing into a box. You need a tool path that does not require the Brain to route into the guest's private network. The rest of this post is those needs, one at a time.

Machines inside machines

Before Firecracker, three words get used as if they mean the same thing: virtual machine, hypervisor, and container. They don't.

A virtual machine (VM) is a fake computer. From inside it, you see CPUs, memory, a disk, a network card. There is a full guest operating system: its own kernel, its own process table, its own page tables. As far as software in the guest is concerned, it is on bare metal. As far as the host is concerned, that guest is a process (or a set of processes) consuming host RAM and CPU time.

A hypervisor is the software that creates and runs those fake computers. It sits between the guest and the real hardware. When the guest tries to do something privileged (touch a physical device, change certain CPU registers), the hypervisor intercepts and either emulates the behavior or forwards it safely to real hardware.

There are two classic styles:

• Type 1 (bare metal): the hypervisor is the host. Xen, VMware ESXi, and in practice the Linux kernel with KVM live here. There is no "regular OS" under the hypervisor that you casually log into for other work. The machine's job is to run guests.
• Type 2 (hosted): the hypervisor is a program on top of a normal OS. Old-school VirtualBox and VMware Workstation are the intuition. You open an app, that app boots a VM.

Linux's KVM (Kernel-based Virtual Machine) is the path almost everyone on cloud VMs actually uses. KVM turns the Linux kernel into a hypervisor. The host is still a normal Linux box you can SSH into. Guests are processes that use the KVM API to run guest code at near-native speed. Hardware support for this shows up as /dev/kvm on the host. No /dev/kvm, no hardware-accelerated VMs.

A container is a different idea. Containers share the host kernel. Isolation is namespaces (what you can see) and cgroups (how much you can use). Start time is tiny because you are not booting a kernel. Escape and attack surface are also different, because every container is talking to the same kernel you SSH into.

So when people say "sandbox the agent in Docker", they usually mean container isolation. When people say "sandbox the agent in a microVM", they mean a real guest kernel, just a very small one. That difference is the whole reason Firecracker exists.

Nested virtualization, or why EKS is not enough

Cloud VMs are already guests. Your EC2 instance is a VM on someone else's hypervisor (Nitro, on modern AWS). Firecracker on that instance wants to create another layer of guests. That is nested virtualization: a VM inside a VM.

For nested virt to work, the outer hypervisor has to expose hardware virtualization features to the inner guest, and the inner guest kernel has to see /dev/kvm. Sometimes that works. Often on managed Kubernetes it does not, or it works poorly enough that you will not want it under latency-sensitive restores. EKS worker nodes in particular are a bad place to assume Firecracker will behave. You can run the Brain, the API, the web UI, and the orchestrator there happily. You should not plan to run Firecracker sandboxes as ordinary Pods on those nodes and then act surprised when /dev/kvm is missing or slow.

There is a second, quieter reason to keep sandboxes off the app nodes even when nested virt exists: noisy neighbors and blast radius. A Devbox that compiles Rust and runs Chromium for screenshots is a bad roommate for your API server. Putting execution on a dedicated pool (labeled nodes, or a separate EC2 host entirely) means a runaway guest fills that disk and that memory, not the nodes serving devin.baby to browsers.

That constraint forced the shape of my system: Kubernetes is the control plane. A separate EC2 host with hardware KVM is the execution plane. The cluster decides that a sandbox should exist. The KVM host is where it actually exists. Cognition's Brain vs Devbox split maps onto that physical split almost one-to-one.

I still keep an "in-cluster KVM pool" path in the docs for self-managed clusters that have real /dev/kvm workers. Production on EKS uses the external host path. Same CRDs, same Brain, different place the firecracker daemon runs. The control plane API does not care. That is the point of putting desired state in Sandbox objects instead of in a runbook that knows the hostname.

What Firecracker actually is

Firecracker is a virtual machine monitor (VMM) built by AWS for Lambda and Fargate. A VMM is the userspace program that sets up a VM and talks to KVM: how much memory, which kernel to boot, which block device is the disk, which network interface the guest sees.

QEMU is the famous general-purpose VMM. It can emulate exotic devices, boot Windows, attach USB, play VGA. That flexibility costs memory and attack surface. Firecracker deliberately does almost none of that. It implements a tiny device model: virtio net, virtio block, a serial console, a few balloon bits. No USB. No fancy graphics. The point is to boot thousands of small VMs quickly and keep the trusted computing base small.

People call those VMs microVMs. The "micro" is about the VMM and the device model, not about a new kind of CPU mode. Underneath, it is still KVM. The guest still has a kernel. The isolation story is still "separate kernel", not "shared kernel with namespaces".

Device models, virtio, and why "small" matters

When a guest kernel wants to talk to a disk or a NIC, it does not touch real PCI hardware (unless you do PCI passthrough, which Firecracker is not about). It talks to a virtual device that the VMM pretends exists. The guest driver writes to MMIO or PIO. The VMM wakes up, interprets the request, and does something on the host: read a file that backs the disk, send a packet on a tap device, and so on.

Old VMMs emulate real hardware (an Intel E1000 NIC, an IDE controller). That is compatible with anything, and slow to start, and a large C codebase full of decades of device quirks. Modern guests prefer virtio: a deliberate fake device family designed for virtualization. The guest knows it is virtualized. The protocol is rings of descriptors in shared memory. Less emulation, more "please DMA this buffer".

Firecracker only implements a thin virtio set. That is the product decision. Fewer devices means less code in the trusted path, less memory per VM, faster boot. You will not attach a GPU or a USB webcam to a Firecracker microVM and enjoy yourself. You will boot a kernel, mount a rootfs, get a network interface, and run a supervisor. For a coding agent, that is exactly the machine you want.

What I still have to build around it

Firecracker alone is not a product. Something has to:

1. choose a kernel and root filesystem
2. configure the VM over Firecracker's API (it speaks a Unix socket JSON API, not "just run this binary with flags and forget it")
3. attach networking so the guest can reach the internet (and so something can reach the guest)
4. tear everything down when the task ends
5. do that over and over without filling the disk

In devin, that something is a host daemon I call firecracker (the app), speaking HTTP on :9092. The orchestrator calls it. It never runs on EKS. It runs on the KVM host next to /dev/kvm and /var/lib/devin.

I looked at Kata Containers along the way. Kata runs pods inside lightweight VMs and plugs into containerd. That is a great fit if your unit of work is already a Kubernetes Pod and you want stronger isolation under the same API. My unit of work is a task with a warm snapshot pool, a custom in-guest supervisor, and a control plane that is not "schedule a Pod". Borrowing Kata's "dedicated KVM worker pool" idea was useful. Adopting Kata itself would have meant fighting the Pod lifecycle for a shape that is not a Pod. Firecracker plus my own daemon kept the Devbox contract explicit.

Brain vs Devbox, in practice

Cognition's language maps cleanly onto the boxes in the hero diagram.

The Brain is the control-plane service. It owns durable task state in Postgres, holds OPENAI_API_KEY, decides whether a prompt needs a sandbox at all, picks a stack runtime, and runs the OpenAI tool loop once a Devbox is ready. It never executes user shell commands itself.

The Devbox is the Firecracker microVM. Inside it, a Go supervisor is PID 1 and listens on :8081. /workspace is a tmpfs where the agent reads, writes, builds and commits. The guest can have a desktop stack (Xvfb, x11vnc, noVNC) for screenshots and interactive preview. It does not hold the OpenAI key.

Between them, in the same VPC:

• a scheduler worker on the execution host, which receives jobs from the Brain and drives sandbox lifecycle
• an orchestrator, which turns "I need a sandbox" into Kubernetes objects and host API calls
• the firecracker daemon, which owns snapshots and VMs
• a tool-gateway, which turns gRPC tool calls into runtime HTTP

The last sentence is load-bearing. Firecracker guests sit on a host-local CNI network around 192.168.127.0/24. That netns is reachable from the KVM host. It is not reachable from a random pod on EKS. So tool calls cannot be "Brain opens http://192.168.127.8:8081". They have to bounce through something that is on the KVM host: worker HTTP proxy → tool-gateway gRPC on :9095 → runtime HTTP on :8081. The Brain only ever sees structured tool results. Once that path existed, keeping secrets off the Devbox stopped being a slogan and became the default.

Following one task

Users do not create sandboxes. They create tasks.

POST /api/v1/tasks
{ "prompt": "add a health check to the Go API", "agent": "brain" }

The web UI hits the API server. The server authenticates (better-auth) and forwards to the Brain. From there the path matches the excalidraw board:

1. Intent. Brain asks OpenAI whether this needs a microVM. hi / how are you get an in-process reply, emit task.sandbox_skipped, and finish. No worker job, no Firecracker.
2. Runtime. For coding work, Brain picks a stack (nextjs | node | go | rust | python), stores it on the task, emits task.runtime_selected. An explicit runtime on create, or createRepository, still forces a sandbox.
3. Enqueue. Brain POSTs an internal job to EXECUTION_WORKER_URL (the scheduler worker on the EC2 host).
4. Provision. Worker asks the orchestrator for a Sandbox. Controllers create a FirecrackerMachine, pick a host, claim or restore a warm VM.
5. Ready. Worker waits until the in-guest supervisor answers HTTP, then POSTs Brain .../sandbox-ready.
6. Harness. Brain runs the OpenAI tool loop: shell, read_file, write_file, list_dir, git_commit, git_push, browser_open, …
7. Stream. Every phase and tool result is an event. The UI listens on GET /api/v1/tasks/{id}/events.
8. Review. On success the Devbox can stay up for follow-ups (awaiting_review / session wake) instead of being deleted immediately.

Task status is a small state machine:

queued → scheduling → sandbox_starting → runtime_ready → running
       → awaiting_review → completed
                         ↘ failed / cancelled

Greenfield work inserts drafting / draft_ready before a sandbox exists: Brain plans a scaffold, may create a GitHub repo, then boots the Devbox into that repo.

Why the Brain can skip a sandbox

Early builds always booted a Devbox. Saying "thanks" cost a Firecracker restore. That hid a product question: what is the unit of work?

The unit is a task, not a sandbox. The intent step is a small model call with a strict schema. If it fails, Brain falls back to greeting heuristics for obvious small talk, otherwise prompt heuristics for runtime. The UI still shows a completed task either way. Only coding work pays the KVM tax.

That branch also changes how you debug. Failures at "sandbox create" are infrastructure. Failures at "intent" are model or prompt routing. Collapsing both into one "agent failed" event makes every outage look the same.

Desired state for a microVM

If you've read what happens after kubectl apply, you already know the pattern: write desired state, let a loop push reality toward it, put observed state in status. I used that pattern for sandboxes instead of a script that SSHs to the host and hopes.

The orchestrator is a Kubernetes controller-runtime binary with a small internal HTTP API on :9090. The worker does not talk to Firecracker directly. It says "give me a sandbox" and watches status. Three CRDs in group devin.baby do the work:

Custom resource groups need a dot. My first attempt used group devin, so sandboxes.devin. The API server rejects that. They became sandboxes.devin.baby, which meant renaming the group in every object and RBAC rule. Annoying once. Permanent after.

A create looks like this (fields from the CRD in ops):

apiVersion: devin.baby/v1
kind: Sandbox
metadata:
  name: task-a1b2c3
spec:
  taskId: a1b2c3
  runtime: go
  cpu: 2
  memory: 8Gi
  preferredHost: devin-production-fc-01

The chain:

1. SandboxReconciler adds a finalizer, ensures a child FirecrackerMachine, copies vmId / host / runtimeURL into Sandbox status.
2. FirecrackerMachineReconciler runs host selection: honor preferredHost if set, otherwise pick the host with the most ReadyVMs that still has CPU. Then POST /v1/vms on that host.
3. Finalizers on both resources wait until the VM is deleted, so kubectl delete sandbox cannot leave rootfs clones on the EC2 disk.

The production FirecrackerHost in ops is just an address book entry:

apiVersion: devin.baby/v1
kind: FirecrackerHost
metadata:
  name: devin-production-fc-01
  namespace: devin-firecracker
spec:
  address: http://10.0.4.73:9092
  schedulerAddress: http://10.0.4.73:9091
  capacity:
    cpu: 8
    memory: 16Gi

The daemons on that host are installed outside the cluster. Kubernetes only knows how to reach them. Names are part of the protocol: metadata.name has to match what the daemon and preferred-host pinning expect. When staging started sharing production's EC2 box, the staging object had to be named devin-production-fc-01 inside the staging namespace. The YAML has a comment that says the name MUST match, because I forgot once.

Golden snapshots, or how to make microVMs feel fast

A cold Firecracker boot is fine for a demo and too slow for a product. Users feel the gap between clicking Submit and seeing the first tool call. The answer is the same idea AWS uses for Lambda: do the slow work ahead of time, then restore.

A golden snapshot in this stack is:

1. build a Docker image for a stack (go, nextjs, python, …)
2. export it to an ext4 root filesystem
3. boot that rootfs once under Firecracker with a known kernel (vmlinux)
4. capture guest memory and VM state (mem.snap, vm.snap)
5. on demand, restore from that snapshot instead of booting from scratch

/var/lib/devin/snapshots/go/
  rootfs.ext4
  mem.snap
  vm.snap
  meta.json

Restore lands in a few hundred milliseconds. The firecracker daemon also keeps a warm pool: a configurable number of already-restored VMs waiting to be claimed. A create either takes a warm VM or restores on demand. Rootfs clones use cp --reflink=auto so the golden file is never mutated in place.

Every stack image installs the language toolchain and a desktop stack (Xvfb, x11vnc, noVNC) so previews and screenshots work regardless of language. Runtime selection used to be pure prompt sniffing. The Brain now asks OpenAI for a stack enum first, with heuristics as fallback. The Sandbox CR carries the name. The host restores the matching snapshot.

The contract you cannot resize

Firecracker cannot change guest RAM or vCPU count on restore. Snapshot size is a build-time decision (8 GiB / 2 vCPU by default in my setup). The cpu and memory fields on the Sandbox have to match what the snapshot was captured with, or the host refuses the claim. "Flexible" resource fields on the CR were a fiction until the snapshot pipeline and the CR agreed.

That sounds like a limitation. It is also a clarifying constraint. Once snapshot shape is fixed, capacity planning on the host becomes arithmetic: how many warm VMs fit in RAM, how many active clones fit on disk, what happens when both pools are exhausted. The daemon reports that through /v1/status. The orchestrator uses ReadyVMs and capacity when picking a host. Without a fixed shape, "host selection" is hand-waving.

How the guest gets on a network

A restored microVM with no network is a brick that can compile Go and nothing else. Firecracker attaches a virtio NIC backed by a tap device on the host. A tap device is a virtual ethernet interface: packets the guest sends appear on the host as if they came from a NIC, and packets you inject into the tap appear in the guest.

Between the tap and the rest of the world sits CNI (Container Network Interface), the same plugin model Kubernetes uses for Pods. My daemon sets up a point-to-point network named fcnet, with static IPAM so the guest lands on a known address (around 192.168.127.8). tc-redirect-tap bridges the Firecracker tap into that netns. NAT on the host gives the guest egress to the internet for npm, go mod, pip, and model-adjacent HTTPS.

Static IPAM matters for snapshots. If the guest got a DHCP address at golden-boot time, restoring memory brings back that old address and lease state. Pinning the address means the runtime URL the orchestrator writes into status is still correct after restore. From the guest's point of view it has a normal NIC. From the host's point of view each VM is a netns you can break in interesting ways if you forget to clean up iptables chains and orphaned namespaces. The daemon spends a surprising amount of code on that cleanup. Networking bugs look like "agent cannot clone the repo" from the UI.

One more consequence of this design: EKS cannot dial 192.168.127.8. That netns lives on the KVM host. The Brain's tool path has to enter through the worker and tool-gateway on that host. There is no safe shortcut that skips it.

Disk fills up without a stack trace

Every VM clones a rootfs under /var/lib/devin/vms/<id>/. Failed creates leave orphans. The first symptom is not a clean error in the Brain. It is tasks dying at sandbox create while /v1/status on the host quietly reports no free disk.

The daemon now:

• caps concurrent VMs (FIRECRACKER_MAX_ACTIVE_VMS)
• refuses create/warm below a free-disk floor (FIRECRACKER_MIN_FREE_DISK_GIB)
• prunes orphan dirs on start and on a timer
• retries one rootfs clone after prune on ENOSPC

That writeup lives in docs/microvm-guardrails.md. Without it, you will debug the harness for an evening before you df -h the execution host.

Inside the guest: the runtime contract

Every microVM runs a Go supervisor as PID 1 on :8081. PID 1 is special in Linux. It is the first userspace process. If it exits, the guest is done. It is also responsible for adopting orphaned children. Putting the supervisor there means the Devbox has one obvious place that owns lifecycle: mount what the snapshot did not, start the HTTP server, reap tasks, shut down cleanly when the host asks.

On start the supervisor:

1. mounts /proc, /sys, /dev/pts (a restored rootfs is not a full boot sequence with systemd doing this for you)
2. seeds entropy (RNDADDENTROPY)
3. forces a sane resolv.conf
4. mounts /workspace as a tmpfs

The restored root drive is read-only. Writable state has to live on that tmpfs (and in the snapshot's memory image). Agent output, npm install caches, cargo targets: all of it lands under /workspace. Workspace size is itself a product knob. Too small and npm install dies mid-task. Too large and the host runs out of memory for warm VMs. I landed on a large tmpfs default and a prune watcher that cleans build caches when usage climbs, because agents do not notice disk until it is too late.

Entropy deserves its own paragraph. Snapshot restore brings memory back exactly as it was when the golden VM was captured. The kernel's entropy pool comes back cold or wrong. Cryptographic libraries block waiting for randomness. The first HTTPS call hangs in a way that looks like "network is broken" from above. Crediting /dev/random before anything else talks TLS fixed a class of bugs that never showed up in dry-run mode, because dry-run never restores a snapshot and never replays a cold RNG.

DNS is the sibling bug. The host resolves fine. The guest's resolv.conf still points at something that made sense at snapshot build time, maybe a resolver that only existed on the machine that baked the golden image. Forcing a known-good resolv file in the supervisor made "agent cannot reach the package registry" stop being a mystery. Same debugging rule as entropy: when the Brain is healthy and the task stalls after runtime_ready, SSH to the KVM host and talk to :8081 yourself before you touch the prompt.

The supervisor's HTTP surface is intentionally small:

The Brain harness does not call these URLs itself. The tool-gateway exposes a gRPC DevboxTools service (Exec, ReadFile, WriteFile, ListDir, GitClone, GitCommit, GitPush, DesktopScreenshot, BrowserOpen). Each RPC carries runtime_base_url so the gateway stays stateless. The worker proxies Brain tool calls onto that gRPC port on localhost.

That fixed contract is why the control plane can stay dumb about agents. Swap the harness model, change the snapshot, rebuild the supervisor: the worker still only knows "sandbox ready" and "forward this tool".

The harness loop

Once sandbox-ready lands, Brain runs a bounded OpenAI tool loop (about 40 steps for a fresh task, 20 for a follow-up), with context compaction after a few dozen messages. "Bounded" matters. An unbounded agent with shell will eventually find a way to start a foreground next dev, wait forever, and burn a warm VM until you notice.

Tools the model can call:

• shell, read_file, write_file, list_dir
• git_commit, git_push
• browser_open (and related desktop helpers)

Each tool call leaves the Brain, hits the worker's tool proxy, crosses gRPC into the tool-gateway, becomes an HTTP request on the runtime supervisor, and comes back as structured output. The model never sees the guest IP. The UI never sees the gRPC. Progress is mapped into events like Write app/page.tsx, Shell go test ./..., Commit feat(api): add health check so the dashboard feels like watching an IDE, not reading raw completions.

Sessions complicate the pretty diagram. After a successful run the Devbox can stay up for review: screenshots, a preview URL, follow-up prompts that should not pay for another cold (or even warm) boot. The worker tracks active and review sessions, can sleep a VM when idle, and wake it when the user returns. That is another reason sandboxes are CRDs with suspend/wake APIs instead of fire-and-forget processes. A follow-up that says "also add tests" should resume /workspace, not scaffold a new repo.

Never trust the guest, or the README

Anything that crosses into the harness can lie: user prompts, tool stdout, repo files, recalled memory. The harness wraps those in <untrusted>…</untrusted> or TOOL_RESULT blocks and tells the model they are data, not instructions. The system prompt says platform rules always win. Content inside those wrappers is never instructions. Extra guards sit in front of the worker proxy too:

• refuse shell that dumps known secrets (GITHUB_TOKEN, OPENAI_API_KEY, DATABASE_URL, …)
• refuse edits under node_modules / .next / dist / target
• refuse long-lived dev servers that would hang the loop
• filter save_memory facts that look like "ignore previous instructions"
• optional HydraDB memory is recalled inside the same untrusted wrappers

The injection that surprised me was indirect. Greenfield scaffolding used to write the user prompt into README.md. The agent later read_file'd its own README. Jailbreaks stopped needing the chat box. Scaffolds now use a safe template. Requirements stay in the session prompt only. Next.js metadata got the same treatment: a generic description, not the raw prompt echoed into the tree.

This is defense in depth, not a proof. Models can still be socially engineered across a large repo. The job is to close the common paths and keep secrets off the Devbox when you can. A coding agent with shell is always one clever file away from trying something stupid. Fail closed.

Keeping the whole thing alive

A Brain that cannot roll forward safely is not a Brain. Image tags, Vault paths, the Firecracker host address, staging's callback URL: all of that is product state. It lives in ops. Argo CD is the only writer to the cluster. CI builds images, tags them with the git SHA, and commits those tags into ops so git log is the deploy history.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: devin
  namespace: argocd
spec:
  source:
    repoURL: https://github.com/rshdhere/ops.git
    path: production/devin/overlays/external
  destination:
    namespace: devin-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Staging and production share a Kustomize base. Each environment is a small overlay: namespace, hostnames, Vault paths, image tags, the FirecrackerHost address, and staging's internal NLB so the shared EC2 host can call the right orchestrator. Secrets never land in git; Vault + the External Secrets Operator copy them into the cluster.

For Devin it comes down to three rules: pin every image to a SHA, let Argo CD self-heal, and treat the Firecracker host as registered state in git, not as something kubectl edit fixes forever.

Things that only show up after you ship

Architecture diagrams are calm. Production is not. A few evenings, in the order they hurt, because this is the part building your own Devin actually teaches you.

Staging used production's orchestrator. The execution host calls back into the cluster when it needs to create a sandbox or report status. For a while staging and production shared more than a Firecracker box. They shared a callback URL. Creating a sandbox on staging returned HTTP 500 because production's orchestrator had never heard of that task id. The UI blamed the agent. The logs blamed nothing obvious until someone compared environment variables on the host. Staging now gets its own internal Network Load Balancer Service in the overlay. The host config is environment-specific on purpose. Shared hardware does not mean shared control plane addresses.

latest lies across environments. Same class of bug, different layer. Staging's orchestrator was pinned to a git SHA. Production still pulled latest. The API shape matched. The bug fixed on staging was not in production's binary. Creates worked in one environment and 500'd in the other. Immutable tags in ops are how you know which binary failed. CI committing those tags is not ceremony. It is the difference between "git revert" and "rebuild and pray".

Snapshot entropy and guest DNS. Covered above, repeated here because they waste so much time. The control plane is green. Argo CD is Synced. The Brain emitted sandbox_starting and then the task stalled. From the laptop it looks like the model hung. Inside the guest, the first TLS handshake never finished, or DNS never resolved. Debug the guest before you debug the prompt. Dry-run mode will not save you. Dry-run never restores a snapshot.

Guest filesystem corruption. Occasionally an ext4 guest went read-only mid-task ("Structure needs cleaning"). Package installs failed. Writes failed. The harness looked broken. The job pipeline now detects that shape of error, deletes the sandbox, and retries once with forceSandboxRecreate. MicroVMs are real machines. Real machines corrupt disks sometimes. Treating them as cattle means having a retry that actually provisions new cattle.

Shared clusters need clear ownership. Staging and production Devin run in one Kubernetes cluster to save money. CRDs and some objects are cluster-scoped. If both Argo CD Applications try to manage the same CRD, each sync fights the other: one applies wave 0, the other prunes or overwrites. Production owns the cluster-scoped objects. Staging only creates namespaced ones inside devin-staging. Staging ClusterRoles are prefixed with staging- for the same reason. Cheap clusters are fine. Ambiguous ownership is not.

The host name is load-bearing. When staging started sharing production's EC2 box, the FirecrackerHost object in the staging namespace still had to be named devin-production-fc-01 because that string is what the daemon advertises and what preferred-host pinning matches on. Rename it to something prettier and host selection fails in a way that looks like capacity exhaustion. The YAML now has a comment that yells. Comments are not documentation of intent. Sometimes they are scar tissue.

Putting it together

Counting the layers one task walks through:

None of these layers knows the full story. The Brain does not know how Firecracker restores memory. Firecracker does not know what a Conventional Commit is. Argo CD does not know what a task is. They share contracts: HTTP, CRDs, image tags, events.

The product idea stayed small on purpose. Decide if you need a machine. Boot an isolated one. Let a model drive tools against it. Stream the result. Keep the box for review or throw it away. What building my own Devin taught me is that every hard week was about keeping that loop honest when the guest is lying, the disk is full, staging is talking to production, or the "hypervisor" you thought you had was just a container with good marketing.

Further reading

• rshdhere/devin
• rshdhere/ops
• Architecture board (excalidraw)
• Cognition: Brain vs Devbox
• Firecracker
• Linux KVM
• what happens after kubectl apply
• MicroVM guardrails in the repo