Run AI agents on local models with Ollama or LM Studio

You already have a model running on your own machine, in Ollama or LM Studio, and you would like your agents to think with it: no API bill, nothing leaving the house. Bloks ships with no model of its own on purpose, and each agent picks its own engine, so a local model is a first-class choice.

This guide shows how to connect one, which switches matter, and what to honestly expect from a small model.

Two ways in

Bloks reaches local models in two ways:

Ollama, which Bloks finds on its own.
A custom OpenAI-compatible endpoint, for anything that speaks that API: LM Studio, llama.cpp, vLLM, LiteLLM, a self-hosted Qwen, or Ollama itself if you prefer to set it up by hand.

Both show up as engines, and you pick one per agent.

Ollama

Run Ollama with the models you want to use.
Open Bloks. It finds Ollama on this machine and lists whatever models it serves.
If Ollama runs on another machine, set OLLAMA_HOST to its address where Bloks runs, and Bloks looks there instead.

Whether a model served by Ollama can run tools depends on the model, so Bloks makes no promise there and marks it accordingly.

LM Studio, llama.cpp and other OpenAI-compatible servers

Start your model server so it is listening. Bloks wants the server’s /v1 root, the base URL that OpenAI-compatible clients use.
In Bloks, open Settings, Engines and keys, and add a custom endpoint under Your endpoints.
Paste the base URL. Add one or more keys if your server asks for one; leave the key out if it takes none.
Wait for the model list to load. If it cannot be loaded, the endpoint’s row says why in the host’s own words, with Retry, and it is not offered as ready until a list arrives.
If your host wants more than a key (a gateway’s org id, say), add up to ten headers under Add a header. Their values are kept like keys: shown only as set, never sent back to the window, and left out of backups.
Open an agent’s settings and pick the new engine and a model.

On a server or in Docker

On bloks-server, you can list a model server as an engine with environment variables instead, which is what a container wants:

BLOKS_MODEL_URL=http://127.0.0.1:8000/v1
BLOKS_MODEL_NAME=Qwen
BLOKS_MODEL_TOOLS=1

BLOKS_MODEL_KEY holds a key if the server takes one, BLOKS_MODEL_VISION=1 says it sees pictures, and BLOKS_MODEL_EFFORT=1 says it takes a reasoning effort. They are read on each start and never written to config.json. In Docker, host.docker.internal is the host machine from inside the container. Install the Bloks server on a VPS with Docker has the full setup.

The switches that matter

A custom endpoint starts as chat only. Three switches tell Bloks what the model behind it can do. Turn each on only when the model really can.

This model can use tools

With This model can use tools on, the endpoint gets the same tools as the built-in API engines. Turn it on only when the model supports tool calls. If the host refuses them, the turn stops and says so, rather than carrying on without them.

With vLLM, tool calls also need the server started with --enable-auto-tool-choice and the tool call parser for your model (hermes for most Qwen models).

Sees images

Mark an endpoint Sees images and the pictures you attach go to its model, which is what a llama.cpp server with its vision projector needs. An engine that is not set to see images says the image was not sent, so you are never left wondering.

Takes reasoning effort

Turn on Takes reasoning effort for an endpoint whose model takes OpenAI’s reasoning_effort (the o-series, gpt-oss, or a gateway that passes it on). Its agents then get the effort picker with Low, Medium and High, sent with each turn. If the host refuses it, the turn stops and says so.

What a local model can do in Bloks

It helps to know where an endpoint engine sits among the others.

Engines such as Claude Code, Codex and Gemini CLI work in your folders: they run commands and read and edit files on your computer. Engines reached through an API get a smaller set of tools from Bloks itself, and a local endpoint gets them once This model can use tools is on:

asking you a question;
their own memory, read and written with memory_read and memory_write;
app connections;
commands in a sandbox or a cloud computer, when the agent has one.

Bloks badges the difference everywhere it shows an engine. The rest of Bloks works the same: rooms, routines, watchers, undo, rehearsals, approvals and memory you can read.

If you want a CLI engine on your own model, Claude Code reads ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN and ANTHROPIC_MODEL from the environment Bloks runs in, so a gateway that speaks Anthropic’s API in front of your model (LiteLLM, for one) works there.

What to expect from small models

Local models have come a long way, and plenty of everyday work suits them: summaries, drafts, sorting notes, answering from memory. Here is how to get the best from them in Bloks.

Start with tools off. Leave This model can use tools off until you know the model handles tool calls. A model that answers well in chat may still fumble a tool call.
Give heavier work to a heavier engine. Each agent picks its own engine. A local model can run a note-taking agent while a stronger engine leads the room. In a room, the lead speaks last and reviews what came back.
Set a backup engine. In an agent’s settings, a Backup engine picks up a turn when the main engine is out, with the conversation so far. It takes over for limits, overload and outages, not for the work going wrong.
Let Engine scout tell you. Settings, Engines and keys shows how often each engine’s work is kept, learned from what you undo, rewind and discard. Nothing switches on its own.
Use rehearsals for anything risky. Press the flask and the agent works on a copy of its folder, for you to apply or discard.
Count the cost your way. Usage says Not reported for a model nobody has a price for. For a host of your own, you can give your own price per million tokens when you set a spending limit, and the limit counts by that.

One more limit to keep in mind: if you move to managed hosting, the hosted machine cannot reach a local model on your own Mac. Keep Bloks where the model is. For the full engine list, see the docs.

Questions

Does Bloks find Ollama by itself?

Yes. Run Ollama and Bloks finds it on this machine, or wherever OLLAMA_HOST points, and lists whatever models it serves.

Can a local model run tools in Bloks?

It depends on the model, so Bloks makes no promise. For a custom endpoint, turn on This model can use tools only when the model supports tool calls; if the host refuses them, the turn stops and says so.

Can a local model read files in my project folders?

Not directly. Models reached through an endpoint get a smaller set of tools from Bloks itself: asking you questions, their own memory, app connections, and commands in a sandbox or cloud computer when the agent has one.

Does managed hosting work with a model on my Mac?

No. A hosted machine cannot reach a local model running on your own Mac. Run Bloks on the same machine or network as the model instead.