Self-hosted LLM stack
vLLM serving Qwen behind a LiteLLM gateway with per-app keys, rate limits and capacity management, plus an open-source agent platform. External AI spend went down noticeably.
Context
Several teams wanted AI in their automations: classification in support, email drafts in supply chain, lead scoring in B2B sales. Each new use case meant another external API key, another bill and another place where customer data left the company.
The job
- Design and run a vendor-independent inference service as the default for internal automations.
- Give every app its own key, limit and budget, so one runaway workflow cannot starve the others.
- Evaluate options by cost, security and future-proofing together with the executive team, and write the decision down.
Decisions
- vLLM with an open-weight model
- Qwen on vLLM gives OpenAI-compatible endpoints, batching and good throughput on a single GPU server. Teams integrate with the same client libraries they already know.
- LiteLLM as the one front door
- The gateway issues virtual keys per app, enforces requests and tokens per minute, tracks spend and can route a request to an external provider when a task really needs a larger model. Switching models is a config change, not a migration.
- Capacity is a number, not a feeling
- Limits per key are derived from measured throughput of the server, and the remaining headroom is checked before a new key is issued.
- MCP over operations data
- Supply-chain demand and inbound figures are exposed to AI assistants through an MCP server, so analysts ask questions instead of exporting spreadsheets.
Outcome
- A cost-efficient, vendor-independent foundation for AI across the company.
- Noticeably lower spend on external AI services.
- The basis for an AI customer-service chatbot, a self-hosted agent platform and the B2B lead-scoring flow.
Stack: vLLM, Qwen, LiteLLM, Docker, FastAPI, MCP, PocketBase.