First of five posts on AI gateways. This one is the why. Then come how to build it, what it costs in latency and where local models fit.
Summary
- iFood and Itaú independently built the same thing: a single layer between applications and model providers.
- It solves three problems at once — a key the application never sees, versioned routing, and cost attributable per team.
- None of the three depends on having many providers. With one provider and one application, the layer already pays for itself.
- The rule for when to build it is three questions you can answer in a meeting. They are in section 3.
1. What they built
1.1 iFood — GenPlat
GenPlat is iFood's unified platform for consuming generative models, described by Igor Martinelli, a machine learning engineer at the company, in a technical account of how it was built.
It acts as a proxy, unifying access to more than 150 models behind a single API. It runs on Kubernetes and centralises, before any call leaves: authentication, access control, sensitive-data redaction, governance and auditing.
The list of mechanisms they describe is worth reading, because it is essentially this series' table of contents: an LLM firewall, RPM and TPM control, intelligent fallback, observability — with RAG-as-a-Service and a Databricks integration on top. Not a thin proxy with a key inside; the whole policy layer.
The most concrete case they documented is menu cataloguing. A restaurant submits content through the Partner Portal; GenPlat mediates the call with a specific prompt; the model returns a structured menu ready for review. The application never knows the provider — and per the case published with Google Cloud, the Vertex AI integration lets internal services reach any model, Gemini included, through the same API.
It isn't only product-facing. Developers use GenPlat daily for writing, reviewing and documenting code — more than 60% of the code produced at the company now has AI support.
The internal agent count gives the scale of the problem the layer solves. The target announced in October 2025 was 8,000 agents by March 2026. In March the company reported 9,000 agents and 32% of activities redesigned; by May, 14,000. CEO Diego Barreto framed the goal as every employee having their own agent, and Forbes ran a piece on the "agent factory" showing how it's operated.
Notice what the scale forced: to keep up with that volume, part of the answer was training their own model. That isn't technical vanity — past a certain point, paying a third party per token on every call stops adding up, and an internal model becomes a cost and control decision.
It's the same path we took, at a smaller scale: distilling an SLM to run procedures cut the bill by 36× — measured against a real invoice and metered GPU-hours, not projected. The method, the evaluation protocol and what didn't work are in I swapped a proprietary LLM for a 9B open model.
On top of it runs the LCM (Large Commerce Model), a proprietary model developed with Prosus, which the company says supported 12 million orders in February. That's 3 billion generative-AI interactions in the first two months of the year.
1.2 Itaú — Iara
Iara is Itaú's internal platform, described by CTO Ricardo Guerra in an interview with Exame INSIGHT.
It works as a gateway and orchestrator over multiple LLMs — OpenAI's GPT, Google's Gemini, Anthropic's Claude, and proprietary engines developed with NeoSpace, a startup the bank took a stake in.
The detail that matters most is the routing criterion. Iara picks the engine per interaction, weighing performance, cost and quality. Not failover, not load balancing — a per-call decision across three dimensions.
Around it the bank built a single knowledge base and short- and long-term memory, so conversations keep context without giving up security and privacy requirements. ia.i, the assistant inside the app, was built on that core.
The scale: more than 1,300 AI models in use, 84% of the institution's initiatives using AI in some form, and more than 760 with generative AI embedded directly in a product, service or internal process.
1.3 What the two have in common
Two companies, two sectors, no coordination between them. Both arrived at the same decision: no application talks to a provider directly.
And the responsibilities they put in that layer are near-identical — access, routing across models, handling sensitive data at the boundary, and governance. GenPlat describes it as "centralising authentication, access control, redaction and governance"; Iara, as "choosing the engine on performance, cost and quality over a single base".
When two operations of that size converge on the same design on their own, that isn't a trend. It's the shape the problem has.
2. The problem this layer solves
It isn't one problem. It's three, and that's why the layer pays for itself.
2.1 The application never sees the provider key
Without the layer, every application carries a real key. They multiply through .env files, CI variables, dev machine configs. Nobody knows how many exist, and revoking any one of them means first finding out what breaks.
With the layer, the application gets a virtual key: model scope, spend cap, rate limit, revocable with one call. The real key lives in exactly one place.
The gain becomes concrete the day someone leaves the company: access dies with the virtual key, without anyone touching a production credential.
2.2 Routing becomes versioned configuration
Without the layer, the model name is written inside every application. Switching models — price dropped, quality improved, compliance vetoed a provider — is a pull request in every repository, with a deploy in each one.
With the layer, the application asks for fast or deep. Which model serves each alias is one line of YAML, versioned, in a single file.
It's literally what Iara does when it picks an engine per interaction. The difference between it and the version your company needs is scale, not concept.
2.3 Cost and traces become attributable
Without the layer, the provider's invoice arrives as a number. Nobody can say which team spent what, which route consumes most, or whether Tuesday's spike was a bug in production.
With the layer: spend per team, per application, per route. It's what turns "we're spending a lot on AI" into a line in a report someone can act on.
2.4 All three hold with a single provider
Notice that none of the three depends on provider count. One application, one real key in a .env, the model name written into the code, an invoice with no per-team breakdown — the layer already pays for itself.
The expectation usually runs the other way — "gateways are for companies juggling many providers". But the three problems above are about how the call leaves, not how many places it goes. You probably don't have too many providers, and you still have all three.
3. When to build it
Three questions. You answer all three in a meeting, without consulting anyone — and that is exactly where the signal is.
3.1 How many applications at this company call a model today?
If you don't know the number from memory, that's already the first signal. Above five, the layer pays for itself on routing alone: that's the point where switching models becomes a pull request in every repository instead of one line of configuration.
And the number is almost always higher than remembered, because the fourth and seventh use cases went through no approval at all — they were someone solving a problem on a Friday.
3.2 Who can create a provider key, and who can revoke one?
If anyone can create one without asking, you don't have access control — you have distributed variable cost.
The practical test is the second verb: ask someone to revoke a key right now. If the answer is "I need to find out what breaks first", the key isn't revocable in practice, and a developer leaving the company becomes an incident rather than a task.
3.3 Does last month's invoice break down by team or application?
If it arrives as a single number, nobody can act on it. You can't tell which route consumes most, which team grew, or whether last week's spike was product or a bug.
Cost you can't attribute is cost you can't cut — it becomes an opinion contest in the budget meeting.
And one that decides on its own
A contractual or regulatory requirement about where the data goes. A customer confidentiality clause, a regulated sector, trade secrets. That one isn't a signal, it's a requirement: with it, the layer stops being an optimisation and becomes a condition of operating.
The reverse holds too
If you have three applications and you know exactly which ones, don't build it. The layer costs maintenance — a database, versions, secrets, one more failure point in the production path — and solves no problem you have today. Use the provider key directly and revisit when the number grows.
Two of the three questions answered badly and the layer pays for itself. None of them and it's infrastructure looking for a problem.
4. The step before any decision
Notice that four of the five signals above are numbers you need to have.
That's why step zero isn't choosing a tool — it's finding out what exists.
With the number in hand, the next two posts in the series handle the rest: how to build the layer, with the full Docker Compose, and what it costs in latency, measured.
How we help teams adopt AI
We work with engineering teams putting AI into the real development workflow — not as an experiment, but as an actual part of how the team ships.
That means choosing the right tools for the team's context, configuring them in a way that makes sense for the company's data policy, and making sure developers know how to use them in a way that genuinely increases productivity instead of adding friction.
If you have a development team and you're trying to put AI to work seriously, get in touch.