Teams blow the LLM bill and still cannot say whether last week's prompt change helped. This track is a gateway: one place to route models, cap tokens and money, log prompts (redacted), and run an evaluation harness. It is not data security (roles, on-prem, encryption)—that page stays the policy contour. It is not RAG. It is the control plane under assistants, RAG, and agents.
I implement a small service your apps call instead of scattering vendor SDKs: API keys in one vault, per-team quotas, fallbacks (provider down, rate limit), and traces you can join to a ticket ID. Quality work uses a frozen eval set—golden questions, expected citations, tool outcomes—not a founder clicking the chat once. Write-ups: LLM gateway and FinOps and AI evaluation harness. Broader quality notes: LLM quality testing 2026.
First slice: wrap one production path (internal FAQ or support draft) with budgets and a weekly eval job. Multi-region, multi-vendor beauty contests wait until you have numbers. On-prem models plug into the same interface when policy requires it.
Stack: TypeScript/Node.js gateway, Postgres for usage, optional Redis for rate limits. Python eval jobs if the rest of the ML contour is Python. Related portfolio: cursor-telegram-integration, thinklens-bot—both need cost and failure modes once they leave the demo.
Out of scope: buying GPU clusters “for independence,” or promising a 40% cost cut without a baseline week of logs.
Timeline: a usable gateway + first eval suite is often 3–7 weeks.
What you get
- Single LLM entrypoint, keys, quotas, fallbacks
- Redacted prompt/response logs with retention you approve
- Eval set, scheduled run, and a short report format your lead can read
- Runbook for rotating keys and raising a team cap
