The most capable models in 2026 still live in the cloud, but you no longer need them for agentic work. A new generation of small language models — 2B to 9B parameters — now ships with native function calling, structured output, and tool-use training, and several of them run on an ordinary laptop. Here is what actually matters when picking one, and the models worth downloading today.
What makes an LLM "agentic"?
An agent does more than answer prompts: it plans, calls tools, reads files, checks results, and recovers when its first plan fails. When evaluating small models for this, benchmark hunters look at five things: function calling accuracy, long-context handling, structured output reliability, instruction following, and recovery behaviour after a failed tool call. The two benchmarks that matter most are the Berkeley Function-Calling Leaderboard (BFCL) for tool calls and TAU2-Bench for multi-step tool use in realistic workflows.
Why run small models on-device at all?
Three reasons dominate. First, privacy: prompts and documents never leave the machine, which matters for client code, company files, and offline research. Second, cost: open models are free to download and there is no per-token bill — you pay in hardware and electricity only. Third, resilience: a local agent keeps working on a train, during an outage, or anywhere the connection is flaky. The trade-off is real too: local small models are weaker at hard reasoning, competitive maths, and broad world knowledge, and they generate text more slowly than datacenter GPUs.
The shortlist: 9 models worth your disk space
Qwen3.5-9B — the best tool caller under 10B
If you test one model, make it this one. Qwen reports 66.1 on BFCL-V4 and 79.1 on TAU2-Bench, the strongest function-calling scores in its family, and its model card documents the tool parser to use with it. On Ollama it runs at about 6.6GB, which makes it the new default recommendation for machines with 16GB of RAM. Apache 2.0 licensing keeps commercial use simple.
Qwen3.5-4B — the 8GB sweet spot
On machines with only 8GB of shared system RAM, you have roughly 4–5GB available for the model. Qwen3.5-4B fits in about 3.4GB while carrying the same attention architecture as the 9B, so it loses very little capability per parameter. Do not drop to the 2B or 0.8B variants unless memory genuinely forces it — capability falls off a cliff below 4B.
Phi-4-mini — the 3.8B reasoning pick
Microsoft's Phi-4-mini remains the best math-and-reasoning model at its size, trained on a synthetic-data recipe that punches above its parameter count. It supports function calling, offers a 128K context window, and is MIT licensed. The catch: it is text-only, so pair it with another model if your agent needs image input.
Ministral 3 8B — vision plus tools
Mistral's Ministral 3 8B combines image input with native function calling, which makes it one of the few small models that can look at a screenshot or document image and then act on it. Ideal for UI-automation agents and document-processing pipelines that need both eyes and hands.
Granite 4.2 8B — IBM's text-only agent model
IBM lists tool calling and agentic workflows among Granite 4.2 8B's primary uses. It is text-only and its KV cache is heavy relative to peers, so it suits short-context RAG and tool-use tasks rather than giant document windows. Worth testing if your agent loop is API calls over short context.
Gemma 4 E4B — the efficiency trick
Google's Gemma 4 E-series uses Per-Layer Embeddings, so E4B's 8B total parameters behave like about 4.5B at inference time. The quantization-aware QAT build needs roughly 6.1GB, accepts audio as well as text and images, and carries one of the lightest KV caches measured — a 32K context costs under half a gigabyte of memory. Excellent for long-session agents on modest hardware.
Nemotron 3 Nano 4B — NVIDIA's agentic pick
NVIDIA's Nemotron 3 Nano is a hybrid Mamba-Transformer distilled from a larger Nano model, purpose-built for agentic work with configurable reasoning traces and a 256K context window. It runs in about 2.8GB. One caution: it ships under the NVIDIA Open Model License rather than Apache 2.0, so read the terms before commercial deployment.
Llama 3.2 3B — the safe fallback
Meta's Llama 3.2 3B is a long-established choice where only about 3B fits. It will not top any leaderboard in 2026, but its tooling support, quantization options, and documentation are the most mature of any model this small — which means fewer surprises in production.
gpt-oss-20b — if you have 16GB
OpenAI's open-weight release remains the best local reasoning pick when you can spare around 16GB. Not a small model in the strict sense, but a natural upgrade path once your agentic workflows outgrow the sub-10B tier. It slots into the same Ollama and LM Studio pipelines below.
Also worth a mention: Salesforce's xLAM-2-8B, which reportedly scored above GPT-4o and Claude 3.5 on the Berkeley Function-Calling Leaderboard — further proof that tool calling, the core primitive agents exercise in production, no longer requires a datacenter model.
How to run them locally
Two tools cover almost everyone. Ollama is the developer favourite: install it, then ollama pull qwen3.5 in a terminal and you are chatting within minutes. It serves a local API on port 11434, so agents and automations can talk to it through an OpenAI-compatible endpoint at http://localhost:11434/v1 — the trailing /v1 is required. Ollama officially added tool calling support in 2026; look for the tools tag on the Ollama model library, because chat-only models cannot complete agent tasks and will stall on file operations.
LM Studio is the friendlier option for non-technical users: a desktop app where you browse, download, and chat with models in a ChatGPT-style interface, with sliders for settings and a built-in hint about which models fit your machine. Its tool-calling support is still experimental, so Ollama remains the safer choice for actual agent loops. For serving many users at once, vLLM and LocalAI provide full OpenAI-compatible function calling including parallel tool invocation.
Format note: Ollama and LM Studio both run the GGUF format (built on llama.cpp); vLLM-style servers typically use GPTQ, AWQ, or Safetensors builds. Download the right one for your engine.
Picking the right one for your task
- General agent workflows on 16GB RAM: Qwen3.5-9B first, Granite 4.2 8B second.
- Laptops with 8GB RAM: Qwen3.5-4B, or Gemma 4 E2B's QAT build at 4.3GB.
- Reasoning-heavy single tasks: Phi-4-mini (3.8B) or gpt-oss-20b if memory allows.
- Agents that need to see images: Ministral 3 8B or Gemma 4 E4B.
- Longest context per gigabyte: Nemotron 3 Nano 4B (256K) or Gemma 4 E4B.
Whichever you choose, do not trust vendor benchmark tables alone — every vendor runs its own harness. Download two candidates, give them the same tool-calling task, and keep whichever one follows instructions more reliably.
Where small models still fall short
Keep expectations honest. Qwen's own tables show large models far ahead on LiveCodeBench, and Google's reporting concedes that frontier models still lead on hard coding, competition mathematics, and broad world knowledge. A small local model makes an excellent file-organising, data-extracting, form-filling, and drafting agent. It makes a poor research mathematician. The winning pattern in 2026 is routing: run the narrow, repeated, private tasks locally, and escalate the hard ones to a frontier API.