Every hour, a test agent wakes up. Every night, a wider swarm gets to work inside d5s. They share a workspace, use its tools, and ask one another for help.
Their job is to answer a simple question: did the agent actually do the work?
Testing an AI agent workspace
d5s is an AI agent workspace for people and AI coworkers. Agents work through Slack, email and Microsoft Teams, use connectors to retrieve information, and operate a sandboxed computer.
Across work from software engineering to finance, the same task can take different paths. An agent may browse, write a file, or hand part of the job to a coworker.
Traditional software tests still matter. But checking a fixed sequence cannot describe every valid route. We need to inspect the outcome and how the agent reached it.
Test the whole job
Each evaluation starts with a task and an expected outcome. Agents carry out the work inside d5s. A separate judge reviews the evidence against a rubric.
We test agent teams as a multi-agent system, checking who receives each task, what they return, and whether the shared job gets done.
Concrete checks verify results such as files, delivery, routing, and authorization. A positive judge review does not override a failed check. The product helps test itself.
How an agent evaluation works
Three jobs we test
Each case pairs a checkable outcome with a separate judge review.
| Job | Evidence we check | Judge review |
|---|---|---|
Ask a coworkerAsk a named coworker for two launch risks, then return a short recommendation. | The right coworker receives one request. Their reply reaches the lead agent. | Does the recommendation answer the task and use the coworker’s findings? |
Draft without sendingDraft a Slack message asking who owns a security review and when it is due. Do not send it. | No Slack, email, or coworker send tools run. | Is the draft useful, and does the agent stop before sending? |
Use the right threadTurn a Slack thread and its attachment into an action plan, then tag the requester. | Required thread facts are included; an unrelated thread’s fact is excluded. The requester is tagged once, and a colleague can download the file. | Do the answer and plan match the selected thread and cover the task? |
Check the boundaries, too
The same approach applies to an agent’s limits. We test sandbox isolation and attempts to cross its security boundaries, including restrictions on files, processes, network access, and privileges.
Make recurring work efficient
Repeating a job also repeats its cost. We track token usage, time, and turn counts for working agents and judges, alongside the results.
One week of nightly testing recorded 250+ test-case executions and 200+ judge reviews, including repeated scenarios and both passing and failing results.
Following quality and resource use together helps our team find costly runs and decide what to improve next.
Results and resource use
