Your Mac can look at photos, read documents, and describe what's on screen — no cloud API, no subscription, no data leaving your machine. Vision-language models (multimodal models) now run locally on Apple Silicon thanks to quantisation and Apple's unified memory design. This guide shows three working routes: Ollama, LM Studio, and Apple's MLX framework, with the exact models that fit each Mac.
Why Apple Silicon is unusually good for local vision models
A vision-language model has two parts: a vision encoder that turns pixels into tokens, and a language model that reasons over them. Running both needs more memory than a text-only model of the same size — the encoder adds overhead, and a single high-resolution photo can inject thousands of tokens into the context.
This is where Apple Silicon earns its reputation. On a Mac, CPU and GPU share one pool of unified memory, so the model doesn't get copied between system RAM and VRAM like on a PC with a discrete graphics card. An M-series Mac with 16 GB of memory can hold an 11-billion-parameter vision model comfortably; the same model on a PC would need a 12 GB VRAM graphics card just for the weights. The Metal GPU backend also gives respectable token speeds without any tuning.
The practical rule of thumb for quantised (Q4) models: 8 GB of Mac memory runs 7B vision models, 16 GB runs 11B–13B, and 24 GB or more opens up 30B-class models. Quantisation cuts the file size by roughly 75% with a quality loss you will rarely notice in vision tasks.
Match the model to your Mac
Before installing anything, decide what your hardware can hold. The list below uses Q4-quantised weights, which are the standard for local inference.
| Mac memory | Fits comfortably | Good first models |
|---|---|---|
| 8 GB (M1/M2 Air) | 3B–7B vision models | Phi-3.5-vision, LLaVA-1.5-7B (4-bit) |
| 16 GB (most M1–M4) | 7B–13B vision models | Llama 3.2 Vision 11B, Qwen2-VL 7B |
| 24–36 GB | 13B–27B vision models | Larger Qwen2-VL, Gemma 3 vision variants |
| 64 GB+ (Mac Studio) | 30B+ vision models | Heavyweight open multimodal checkpoints |
A 7B vision model at 4-bit quantisation needs roughly 4–6 GB of disk space and about 8 GB of RAM during inference. The Llama 3.2 Vision 11B checkpoint needs around 8 GB on disk. Keep at least 10 GB of disk free for model downloads and the runner app.
Two constraints to respect: an image-heavy prompt eats context fast, so don't set absurdly long context windows on a base Mac; and don't run two models at once — one vision model can saturate the memory bandwidth of a fanless MacBook Air on its own.
Option 1: Ollama — the fastest route
Ollama is the quickest way from zero to a local vision chat on a Mac. Its minimum requirement is 8 GB of RAM, and the Metal backend is enabled by default on Apple Silicon.
1. Download Ollama from ollama.com and move it to your Applications folder, or install from Terminal with curl -fsSL https://ollama.com/install.sh | sh.
2. Pull a vision model. The simplest pick is Meta's Llama 3.2 Vision (11B), packaged for Ollama:
ollama pull llama3.2-vision3. Start it with ollama run llama3.2-vision. The daemon serves an API at localhost:11434 the moment it's installed.
Ollama's Terminal chat is text-focused, so for image input use a front end. The common setup is Open WebUI running in Docker, pointed at Ollama — you get a browser chat window with image upload. Alternatively, send images through the API: POST to /api/generate with an images array containing the base64-encoded photo. Ollama auto-offloads to Metal on Apple Silicon, and the 11B vision model runs at a genuinely usable pace on a 16 GB Mac.
If you only have 8 GB, skip the 11B model and pull a lighter multimodal checkpoint instead — a 3B–7B vision quantisation keeps the Mac responsive and still reads documents and photos accurately.
Option 2: LM Studio — the easiest graphical route
If you'd rather point and click, LM Studio gives you a proper desktop chat UI with an image-attachment button and an MLX backend built specifically for Apple Silicon.
1. Download LM Studio for macOS (Apple Silicon build) from lmstudio.ai and install it.
2. Use the search bar inside the app to find a vision model: llama-3.2-vision, qwen2-vl, or llava-1.5-7b. LM Studio lists each checkpoint's file size and minimum memory, so pick one under your Mac's comfortable limit.
3. Download the model, load it into the chat tab, and click the paperclip icon to attach a photo. Type your question — "Transcribe this receipt and list the total" — and the model reads the image locally.
Two LM Studio features are worth knowing. First, the MLX backend option in settings is tuned for M-series chips and noticeably faster than the default CPU/Metal path for supported models. Second, flipping on the built-in server (localhost:1234) gives you an OpenAI-compatible endpoint, so any app that talks to the OpenAI API can use your local vision model with zero code changes. LM Studio's GUI adds roughly 500 MB of overhead versus Ollama's ~100 MB — irrelevant on a 16 GB Mac, but worth noting on an 8 GB Air.
Option 3: MLX and mlx-vlm — the Python power route
MLX is Apple's native machine-learning framework, and it only runs on Apple Silicon. It is the fastest path on this list because it is designed around unified memory — no data shuffling between CPU and GPU memory. If you're comfortable in Terminal, the mlx-vlm package runs vision models in one command:
pip install mlx-vlm
python -m mlx_vlm.generate \
--model mlx-community/llava-1.5-7b-4bit \
--image path/to/image.jpg \
--prompt "What does this document say?"The mlx-community organisation on Hugging Face hosts pre-converted, MLX-optimised checkpoints — download once, run forever, no account needed. A 7B model at 4-bit uses roughly 4–6 GB of disk and 8 GB of RAM during inference, comfortable on most M1/M2/M3 Macs.
This route is the right choice for batch work: extract tables from a folder of screenshots, generate alt text for a hundred product photos, or OCR a stack of receipts. Write a small Python loop, pay nothing per image, and your data never touches a server. The marginal cost of processing one image or a hundred thousand is identical: zero.
Option 4: Phi-3.5 Vision for smaller or older Macs
If your Mac has 8 GB of RAM, Microsoft's Phi-3.5-vision is the sweet spot: a genuinely capable multimodal model in a small package, with a minimum requirement of 8 GB when quantised. The phi-3-vision-mlx package wraps it for Apple Silicon:
pip install phi-3-vision-mlx
phi3vFor the best speeds, 16 GB or more is recommended, but the model genuinely runs on an 8 GB machine — the right pick for a base MacBook Air or a Mini doing occasional document reading. Like the other options, it handles visual question answering, document extraction, and image captioning entirely offline.
Get good answers from local vision models
Local vision models behave like their cloud cousins, but a few habits pay off:
- Ask for structure. "List the items on this receipt as a table: item, price, quantity" beats "read this." Structured prompts reduce hallucinations in every model on this list.
- Watch the resolution. Very large images inject huge numbers of tokens. Downscale a 4K photo to around 1536 pixels on the long edge before sending it — you keep all the readable detail and halve the token cost.
- Keep context moderate. Vision models fill context with image tokens before they see your question. On a 16 GB Mac, a 4K–8K token context is the practical ceiling for an 11B model; bigger contexts spill to slower memory and tank speed.
- Unlabeled files stay on disk. A genuine advantage of local inference: medical scans, contracts, kids' photos — the stuff you'd never upload to a cloud API — are safe because nothing uploads at all.