Second of five posts on AI gateways. Before: why it exists. After: what it costs in latency and where local models fit.

Summary

  • Download the stack — five files, docker compose up -d.
  • On the application side, the entire change is swapping api.openai.com for the gateway and the model name for a task alias.
  • Langfuse costs four extra services. Start without it.
  • The part of the file that matters most is two lines, and they are in section 3.

1. The six services

Gateway containers: the essential ones on one side, optional observability on the other.

litellm, postgres, langfuse, clickhouse, redis and minio. Only the first two do the gateway's work.

LiteLLM + Postgres is the gateway. Postgres holds virtual keys and accumulated spend — without it LiteLLM runs, but you lose exactly the two things you built the layer for.

Langfuse is observability: request and response traces, cost per trace derived from token usage, prompt versioning, evaluations. It integrates natively with LiteLLM — two callback lines in the YAML.

And the part the documentation doesn't put up front: Langfuse 3.x requires Postgres, ClickHouse, Redis and S3-compatible storage. Four services for observability alone. It is not "one more container".

Start without Langfuse. LiteLLM's own dashboard already shows spend per key, per model and per team, which answers the first question anyone will ask. Add it when the question you need to answer stops fitting there — and at that point you will know exactly what you are paying the operational cost for.

2. Bringing it up

cp .env.example .env    # change EVERY value
docker compose up -d

LiteLLM on localhost:4000, with a dashboard at /ui. Langfuse on localhost:3000.

Five files, not four — init-langfuse-db.sh is mounted by the compose file and without it the Langfuse database never comes up. They are all linked in section 6.