Contents
In brief
The de-swarm project turned a fine-tuned 3B text-to-SQL model into a local API built with FastAPI and Ollama. It runs on a CPU, accepts natural-language questions about SQLite data, and produced textbook-correct SQL for 16 of 22 production-style queries.
The important shift is not merely that the model runs on a laptop. It is that another application can call a constrained HTTP service without granting an LLM unrestricted access to the database.
What happened
The author placed a gateway around Qwen2.5-Coder-3B-Instruct with six endpoints: health and schema discovery, natural language to SQL generation, SQL execution, and a combined query-and-execute route.
The application is deliberately small: configuration, X-API-Key checking, an Ollama client, SQLite schema discovery, a SQL executor, and Pydantic request and response models. There is no agent framework in the serving path—only an HTTP server calling a model server.
Across 22 queries, median latency was 17 seconds, ranging from 9 to 61 seconds. The 3B model handled aggregations and joins across as many as four tables well, but window functions remained a clear limit.
Why it matters
A route that turns a question into SQL and executes it has a larger failure surface than a text generator. A model instruction not to modify data is insufficient: its output must be constrained by the service and the database.
The implementation uses five independent controls:
- Regular expressions reject
DROP,DELETE,UPDATE,INSERT,ALTER, and other forbidden statements. - Only the first statement is retained, preventing patterns such as
SELECT ...; DROP .... - SQLite opens with
mode=ro, so the database itself rejects writes. - A timeout bounds expensive queries.
- A result cap prevents an accidental full-table response.
Six adversarial tests did not result in a security violation. That is not a substitute for access control and review, but it demonstrates a useful rule: model-generated SQL needs enforcement below the model layer.
In practice
For CPU inference, schema context was the dominant performance cost. The initial description of a 16-table, 135-column SaaS schema reached 14,800 characters, and some requests timed out after several minutes.
Rather than remove examples altogether, the project filters them. Columns likely to contain free text—names, emails, URLs, comments, JSON, and similar fields—are excluded by name. The remaining text columns are sampled only when their distinct-value count is low enough to be useful.
This reduced one schema description from 9,800 to 8,200 characters and its sample values from 202 to 96. Queries then completed in 10–32 seconds. On a CPU, trimming irrelevant context is not a minor optimization; it can decide whether the API responds at all.
For a similar service:
- Open data stores in read-only mode and isolate them from write-capable systems.
- Validate generated SQL, bound execution time, and cap returned rows.
- Give the model only relevant tables and compact examples of categorical values.
- Measure latency, timeouts, syntax errors, and unsafe-operation attempts alongside SQL accuracy.
- Document model limits; complex analytics may need a larger model or an additional verifier.
Takeaway
A local LLM becomes operationally useful when it has a clear, safe interface—not simply when it can generate an answer. FastAPI and Ollama can provide that interface without an external inference bill, but the outcome depends on the controls around the model: SQL safeguards, lean context, and observable limits.
A 3B model will not replace an analyst for window functions or intricate business logic. It can still support a constrained reporting workflow when it has a safe data path and its boundaries are explicit.