AI Gateway, API Gateway, Gateway API, and friends: A Map Through the Gateway Confusion

Talk to a random visitor at KubeCon, one of the largest open-source conferences, about gateways, and there is a decent chance one of these terms will come up: API Gateway, Ingress Gateway, Service Mesh Gateway, Gateway API, Envoy Gateway, Agent Gateway, AI Gateway, LLM Gateway.

We won’t judge you if you are confused. The terms sound related, but they describe very different things.

With the current hype around so-called "AI Gateways", the confusion has not exactly decreased. In the open-source ecosystem alone, you can find projects such as Envoy AI Gateway, agentgateway, APISIX AI Gateway, Higress AI Gateway, LiteLLM, Portkey, or AI Gateway functionality in larger platforms such as MLflow. They all have something to do with AI traffic. They do not solve the same problem.

In our next release, we are integrating a new gateway technology. That meant working our way through this thicket of terms, marketing material, and documentation. This post is an attempt to distill what we learned and bring a bit of light into the landscape.

It is written for technical readers who have a basic understanding of Kubernetes, containers, and LLMs, but who do not need to be networking experts or hardcore DevOps nerds. This post does not try to explain every single technology in detail. The goal is to build a mental model: What kinds of gateways are there? At which layer do they operate? Which problems do they solve? And why does it matter not to mix these layers up? Which gateway we chose, and why, may be the subject of a follow-up post.

Disclaimer: We are an AI infrastructure company. Naturally, this post was written with AI assistance. It is, however, by no means one-shot AI slop. We lovingly iterated with the LLMs and reviewed and edited the result several times.

Why "Gateway" Is So Confusing

At its simplest, a gateway is a controlled transition between two areas, for example:

• between the internet and a Kubernetes cluster,
• between services inside a cluster,
• between an application and multiple LLM providers,
• between agents and tools,
• between agents and other agents,
• between a user and a platform,
• or between an internal AI system and external APIs.

That is exactly why the term is so flexible and so confusing. Two teams, or even two developers on the same team, can mean entirely different layers when they say "AI Gateway".

The Classic Gateway World: Ingress, API Gateways, and Service Meshes

Before looking at AI Gateways, it is worth taking a quick look at the classic cloud-native world. (In our community, "cloud native" and "Kubernetes" are almost used synonymously.)

In Kubernetes, one simple distinction helps: a resource describes desired state, a controller reads that resource and translates it into configuration, and a proxy actually sits in the data path. The proxy accepts requests, decides where to forward them based on its configuration, and returns responses. Depending on the proxy, this may include functionality such as TLS termination, load balancing, routing, retries, timeouts, header manipulation, authentication, or observability.

Ingress was long the standard resource for bringing HTTP traffic from outside into a cluster. Ingress itself does not route traffic. Only an Ingress Controller turns it into actual proxy configuration. Typical implementations include NGINX Ingress Controller, Traefik, HAProxy, or Contour.

The more modern Kubernetes Gateway API pursues a similar goal and can, in many scenarios, be understood as a successor to Ingress. But it is not simply a "better Ingress". It is more explicit, more powerful, and more strongly designed around separation of responsibilities between platform, network, and application teams. Here too, the Gateway API is a Kubernetes API, not a data-path component. Implementations such as Envoy Gateway, Kong, Cilium, or Istio have to read these resources and translate them into real proxy configuration.

An API Gateway, for those not inclined to confuse it with the Gateway API, is a functional or product category. It exposes APIs to clients and typically bundles functionality such as authentication, rate limits, quotas, API keys, transformations, or analytics. It can sit in front of Kubernetes, run inside Kubernetes, or be combined with Ingress and Gateway API.

A service mesh such as Istio or Linkerd goes one step further. It handles not only ingress traffic, but also service-to-service communication inside the cluster: mTLS, policies, traffic splitting, observability, and authorization. For this discussion, the key point is simple: Istio is a control plane, not the proxy itself.

A hopefully useful simplification:

• Gateway API is the Kubernetes-standardized API for gateways and routes.
• Envoy is the actual proxy in the data path.
• Envoy Gateway is one example of a controller that works with Gateway API resources and configures Envoy as the underlying proxy.
• Istio is a service mesh control plane; it may also consume Gateway API resources, but it is not central to this article.

The data path is the path real requests and responses take at runtime. The control plane decides and configures. The data plane handles the actual traffic.

What Makes an "AI Gateway" Different?

With classic API traffic, the target is often a known backend service: a request comes in, and the gateway routes it to the right service based on host, path, headers, or other rules. LLM and agent traffic introduces additional questions:

• Which provider or model should be used?
• How should API keys and secrets be managed?
• How should budgets, quotas, and rate limits be enforced?
• How should tokens, costs, and latencies be tracked?
• How do client applications get a unified way to access external providers such as Azure OpenAI, Anthropic, Gemini, or Bedrock, as well as self-hosted models via vLLM, Ollama, or internal endpoints?
• How do you route queries efficiently to local LLM instances, for example in a KV-cache-aware way?
• How should fallbacks, retries, and timeouts be handled?
• How do you provide observability for prompts, completions, and tool calls?
• How do you control access to MCP servers and tools?
• How do you separate tenants, teams, or workspaces?
• How do you prevent agents from accessing internal systems without control?

For our internal discussions, it proved useful to divide AI Gateways along these questions into three categories.

1. LLM Provider Gateways

These gateways sit between an application and LLM providers. They often expose an OpenAI-compatible API, the de facto interface used by many LLM applications, and route requests to different models or providers.

Typical functionality includes API key management, provider abstraction, cost control, fallbacks, rate limits, and logging. One important pattern is the separation between client access and provider access: an application or user does not receive the actual API key for OpenAI, Anthropic, or Bedrock. Instead, it receives a key issued by the gateway. The gateway holds the real provider secrets centrally, decides per request whether the call is allowed, and can track usage, costs, limits, and errors per app, team, or key.

There are many SaaS solutions in this space, for example OpenRouter, Helicone, Keywords AI, or Portkey. There are also candidates that can be self-hosted, such as LiteLLM, Portkey Gateway, or parts of MLflow AI Gateway. agentgateway can also cover LLM provider traffic, but it is broader in scope and appears again below.

The central problem here is:

How do I give applications, developers, and users unified, controlled access to different LLMs?

Note: For all open-source projects, it is worth looking closely at edition and feature boundaries. Source code being available does not automatically mean that all production-relevant operational features such as SSO, fine-grained RBAC, audit, HA setups, or enterprise support are unconditionally included in the free version.

2. Kubernetes-Native AI Traffic Gateways

These gateways come more strongly from the cloud-native and infrastructure world. They build on Kubernetes, Envoy, Gateway API, or related standards.

This is not just about provider abstraction, but about traffic management as a platform capability: Custom Resource Definitions, or CRDs in Kubernetes jargon, Gateway resources, policies, self-hosted inference, routing to model backends, observability, and integration with existing Kubernetes networking models.

An important building block in this category is the Kubernetes Gateway API Inference Extension. It addresses a problem that rarely comes up in classic HTTP routing: for self-hosted inference, it is often not enough to send a request to any pod via round robin. LLM serving needs model-aware routing, endpoint selection based on load and capabilities, priorities, rollouts, and metrics from model servers. Concepts such as InferencePool, Endpoint Picker, prefix-/KV-cache-aware routing, LoRA adapter availability, or GPU utilization become part of the routing decision.

If you are interested in autoscaling and KV-cache-aware routing for self-hosted LLMs, feel free to read our paper on the topic, which we recently presented at VDI Automation. Link to follow once it is published.

Envoy AI Gateway is one example in this category. Built as an extension to Envoy Gateway, it gives applications a single, consistent way to talk to many different models, whether they run at an external provider or on your own infrastructure. It handles surrounding concerns such as authentication, fallbacks, usage limits, and metrics. It also ties back to the routing described above: instead of reinventing model-aware routing, it can build on the Gateway API Inference Extension. A request can be handed to an InferencePool backend where the Endpoint Picker chooses the best model server, while Envoy AI Gateway wraps that smart routing in the outer layer applications actually see.

The central problem here is:

How do I operate AI traffic as a Kubernetes-native infrastructure component?

3. Agent, MCP, and Tool Gateways

Yes, of course this post also has to talk about agentic AI. AI agents combine LLM calls with tool calls, MCP servers, internal APIs, and increasingly communication with other agents. This creates new security and governance questions:

• Which agents are allowed to use which tools?
• How is tool access authenticated and authorized?
• How do you observe agent-to-tool and agent-to-agent communication?
• How do you prevent uncontrolled lateral movement through internal systems?
• How do you enforce policies for MCP servers, tool calls, and agent workflows?
• How do you assign agents their own identity, and how do you connect that identity to access permissions?

By the time you get to the last question, gateway, identity, and policy topics start to overlap. Concepts such as SPIFFE and SPIRE then become relevant. That, however, is its own topic and possibly material for a follow-up post.

The central problem here is:

How do I control communication between agents, tools, services, models, and other agents?

Examples of such Agent, MCP, and Tool Gateways include agentgateway, which addresses agent, MCP, LLM, and service traffic under a shared gateway layer; the Docker MCP Gateway, which centrally manages MCP servers; and Google's Agent Gateway in the Gemini Enterprise Agent Platform.

Why This Distinction Matters

If you mix these categories, you quickly start comparing apples to oranges.

LiteLLM, Envoy AI Gateway, and agentgateway can all appear under the label "AI Gateway". Still, they represent different layers: provider abstraction, Kubernetes-native traffic management, or agent/tool communication. For architecture decisions, the label is only useful if you name the layer you actually mean.

The right question is therefore not:

Which AI Gateway is the best?

But:

Which gateway layer do we actually need?

As a rough grid, this leaves us with:

The grid is not perfect. Many tools move between several categories. It is still useful for sorting the concepts and choosing the right tool for the right job.

What This Means for prokube.ai

At prokube.ai, we build a sovereign, Kubernetes-native platform for AI, agent, and classic MLOps workloads. We are big open-source fans, deliberately build our platform on fully open tools, and contribute back to upstream projects whenever it makes sense. That is why we do not look at AI Gateways in isolation, but as part of a larger platform architecture made up of Kubernetes, Gateway API, Envoy, MCP, observability, identity, policy, and workspace isolation. Which gateway component we are integrating in our upcoming release, and why, is something we will cover in the next post.

Conclusion

"AI Gateway" is not a single clearly defined product type. It is an umbrella term for different layers that all sit somewhere between application, model, agent, tool, platform, and infrastructure.

Anyone talking about AI Gateways today should first clarify:

• Which boundary should be controlled: application to model, agent to tool, agent to agent, service to service, or internet to cluster?
• Do we primarily need LLM provider routing, or a broader agent/MCP/A2A communication layer?
• Should the gateway be Kubernetes-native infrastructure, or an application-adjacent proxy?
• Is this about self-hosted inference and model serving, or external provider APIs?
• What role do identity, policy, tenant isolation, observability, and cost control play?
• Do we need a specialized gateway for one layer, or deliberately one shared gateway layer for several traffic types?

Only once these questions are answered does a tool comparison become useful.

For us, it certainly helped to clean up these terms and be more precise inside the team about what we are actually talking about. We hope it does the same for you. If not, we hope this post at least helps you land a well-placed nerdy correction in the next hallway conversation.