Key takeaways
Harbor can now run every eval task in Docker Sandboxes
Docker Sandboxes are MicroVM environments to run agents like Claude Code, Codex, Cursor, and more safely
With Sandboxes, define custom network policies for fair, reproducible evals
Run your evals locally or in the cloud: starting for free on Docker Sandboxes on local surfaces and scaling to Cloud Sandboxes
To evaluate an agent, you give it a real task, let it work, and check the result. Along the way, the agent installs packages, edits files, and runs commands. That’s untrusted, AI-written code, and a benchmark runs it thousands of times. And agents don’t always play fair. An agent can sometimes earn a passing score on an eval by gaming the check instead of doing the work, for example by tampering with tests or overwriting the file that records its score. This is called reward hacking, and it’s a growing concern across the eval community. So running agent evals needs two things:
An eval framework that runs each task, lets the agent work, checks the result, and collects the scores and logs. An environment for every task that’s clean and isolated, so evaluating agents is safe and consistent. Harbor gives you the first. It’s an open-source eval framework. Give it a benchmark like Terminal-Bench, and it simplifies the process of running evals. Now, Docker Sandboxes gives you the second: a safe way to run those eval tasks, on your laptop or in the cloud. Here’s what this means for you
Strong isolation by default. Docker Sandboxes were designed from the ground up for deeper isolation. This means you can run every task in a microVM, not a shared container. A microVM has its own kernel, so whatever the agent does stays inside its sandbox. It can’t break out to reach your machine, other tasks, or anything else outside its environment. Fair, reproducible evals…
