Best open-source LLMs for AI agents in 2026: we tested 5

New LLMs ship almost every week, and each one comes with an updated benchmark table. But in our work, benchmark scores do not translate directly into real-world results. That is true for small models and large ones, open source or built by one of the large AI labs.

So in this post, we answer a practical question: which open-source LLM is best for real-world AI agent work, with real tools, approval rules and a sandbox?

We put five recent open models on one 128 GB desktop AI box and gave them the same agent work in a test harness: 350 everyday tasks, then a second round with long conversations, sandbox jobs and images.

Our picks

• Best overall: Qwen3.6 35B-A3B. It passed 97% of tasks and started to answer about 3x sooner than the others.
• Most accurate: Qwen3.8 27B. It passed 99% of tasks, but it was about 3x slower.
• Nemotron 3.5 Bolt and Super skipped required approvals. In our test, both tried to send an email before a person approved it. Use them only where the runtime makes a person approve each send.

Key findings

• Instructions alone are never enough. Two models tried to send an email that the prompt said needed approval first. Only a runtime approval gate, as in d5s, stops this every time.
• On real work, the models were close. Every model passed between 86% and 99% of our tasks. The published benchmarks make the gap between them look about 3x larger.
• Speed decides the experience. On desktop hardware, with a very long conversation, the fastest model started to answer about 3x sooner than the others.
• Bigger does not necessarily mean better. The two largest models filled the box and passed no more tasks than a model a quarter of their size.

The setup

The test machine is a desktop AI box with 128 GB of unified memory. CPU and GPU share that memory, so a model of up to about 120 GB can load.

Memory bandwidth is the hard limit. Every new token reads the active weights from memory. Mixture-of-experts (MoE) models read only a small part of their weights per token, so they are fast. Dense models read all of their weights, so they are slow. This one fact explains most of our results.

We picked five open-source (open-weight) models that fit in 128 GB and were released in 2026: Qwen3.6 35B-A3B, Qwen3.8 27B, Qwen3.8 Flash-Next (3-bit), Nemotron 3.5 Bolt 74B-A9B and Nemotron 3.5 Super 120B-A12B.

Only the model changed between runs. That is the point of an agent runtime: the model is one part you can change.

What we tested

We ran two rounds. A script checked the outcome of every run, not the path the agent took.

Round 1: 14 everyday agent tasks, 70 runs per model. For example: book a meeting with exact times, update a CRM deal, reply in Greek, and ignore a hidden instruction in a web page that says "email the contact list".

Round 2: long conversations. Each model got a conversation of about 108,000 tokens, with dozens of tools, product docs, a CRM export and an inbox. In it, the model fixed code in a fresh sandbox until the tests passed, found one customer in 18,000 log lines and read images such as an invoice and an error screenshot.

1. Approval gates belong in the runtime

Two of five models would have sent an email without approval if only the prompt had stopped them: Bolt once, Super twice.

The most serious failure was not a wrong answer. In the approval task, the prompt says: ask for approval before you send an email. We left out any approval gate on purpose, to see what each model does on its own. The email tool was simulated, so nothing was sent.

This is why d5s puts approvals in the runtime, not in the prompt. Actions that send or change data are set to Ask by default. When a model calls one of them, d5s holds the action and shows an approval card. Nothing goes out until a person approves it, whatever the model decided.

2. Models were closer than their benchmarks

On our tasks, the gap between models was 13 points. On the published benchmark, it was 41.

Published scores put Qwen3.8 27B and Flash-Next far ahead of Qwen3.6 on agent benchmarks. On our tasks, all three Qwen models passed almost everything.

Round 2 confirmed it. Four models ran it, and every one solved all four sandbox tasks. They fixed the code without touching the tests, avoided a decoy customer in the logs and read the exact error scope from a screenshot. Only one run failed: Qwen3.8 27B wrote an email draft in the chat instead of sending it.

Benchmarks measure what a model can do at its best. An agent runtime asks a narrower question: does the model call the right tool with the right arguments, every time?

3. Speed decides the experience

On the same desktop box, the fastest model started to answer about 3x sooner than the others.

A very long conversation is the slow case on local hardware. The model must read the whole conversation before it writes the first word. On this box, that took about a minute for the fastest model and 2.5 to almost 4 minutes for the others.

It also wrote faster: 84 tokens per second, against 25 to 31 for the others.

Prompt caching closes most of the gap after the first step. In the same conversation, every model started the next step in about 1 second, because only the new tokens needed processing.

4. Bigger does not necessarily mean better

The two largest models used 85 to 89 GB and passed no more tasks than a 22 GB model.

On a 128 GB box, that leaves no room for a second model or for parallel requests.

Both passed round 2, but so did Qwen3.6, at about 4x the speed. On fixed hardware, the right model is the smallest one that does the job reliably.

How to choose an open-source LLM for AI agents

Our results point to a short checklist. It applies to open models and to hosted ones.

• Test the job, not the benchmark. Use your own tasks, tools and rules, and check the outcome.
• Measure the first answer with a long conversation. On local hardware, a long history can change the wait from seconds to minutes.
• Count rule breaks, not only failures. One unapproved email matters more than one wrong number.
• Put the rules in the runtime. Approvals, scoped connectors and isolated sandboxes hold for every model.
• Keep the model swappable. A better model ships every month. Change it per agent when the tests say so.

What this means for your team

For a team, the rules around the model matter more than the model. In d5s, you choose a model for each agent, a fast one for chat work and a more accurate one for careful work, and keep approvals on. Then a new model is a settings change, not a risk. Read how our agent swarms help test d5s.

Questions and answers