Skip to content

The MCP gateway payload gap — a reproducible deep-dive

A native MCP gateway and an in-process contract layer sit at different points in the same request. This doc measures exactly where one ends and the other begins, using agent-airlock's own head-to-head bench — no vendor claims, no borrowed analogies. If you only want the raw table, read benchmarks/vs_gateway/RESULTS.md; this is the narrative + the honest "where each one is actually enough."

agent-airlock is a type-checker / contract layer for AI tool calls, not a firewall. The word "firewall" now collides with what native MCP gateways do (auth, transport, sandboxing, egress) — and, as the numbers below show, the two are complementary, not competing.

What was measured

One corpus, two layers. The corpus is 12 malformed tool-call payloads + 3 benign controls, defined once in benchmarks/vs_gateway/corpus.py and pushed through both:

  • the native gateway — a live Docker MCP Gateway v2.0.1, running with its defaults, measured by sending each payload as a real MCP tools/call;
  • agent-airlock — the shipped @Airlock decorator, SafePath/SafeURL validators, the in-process guard chain, and a deny-by-default SecurityPolicy, measured live on every run.

Reproduce it

# The head-to-head. Runs airlock live; replays the recorded gateway measurement.
# No Docker daemon required.
python -m benchmarks.vs_gateway            # human-readable table
python -m benchmarks.vs_gateway --json     # machine-readable summary

To re-measure the gateway yourself (needs Docker + the docker mcp plugin):

docker build -t airlock-bench/echo-mcp:latest benchmarks/vs_gateway/gateway_harness
python -m benchmarks.vs_gateway.gateway_harness.regen   # rewrites gateway_measurement.json

The gateway side is a recorded measurement, not a live call at bench time — benchmarks/vs_gateway/gateway_measurement.json carries the full provenance (gateway version, date, method) and the harness regenerates it. The airlock side is never cached; it runs the real code paths every time.

How the gateway measurement works

The harness runs docker mcp gateway run --transport stdio in front of an echo-oracle MCP backend that does no validation of its own — it just echoes back the arguments it received. That isolates the gateway: if the oracle echoes the payload, the gateway forwarded it (PASS); a JSON-RPC error from the gateway before the backend sees the call is a BLOCK. The claim under test is "the gateway forwards the tool-call payload without validating its argument contract" — not "the backend is exploitable," which an echo oracle deliberately says nothing about.

Results (recorded 2026-07-16)

Layer Malformed blocked Benign false-positive
agent-airlock (in-process contract layer) 12 / 12 0 / 3
Docker MCP Gateway v2.0.1 (native transport/identity) 0 / 12 0 / 3

Airlock decision latency: p50 in the tens of microseconds (well under 0.1 ms) — it's a function call, not a network hop.

Per payload class

payload class example argument airlock gateway airlock mechanism
type confusion amount="100" (str for int) block pass @Airlock strict Pydantic, no coercion
value constraint amount=-1 block pass declared amount > 0 on the contract
ghost argument force=True (not in signature) block pass ghost-argument BLOCK mode
path traversal path="../../../../etc/passwd" block pass SafePathValidator
SSRF url="http://169.254.169.254/…" block pass SafeURLValidator
file:// exfil url="file:///etc/shadow" block pass SafeURLValidator
eval/exec RCE code="__import__('os').system(…)" block pass eval-RCE guard
subprocess injection command="/bin/sh", args=["-c", …] block pass subprocess-arg guard
env code-load env={"LD_PRELOAD": "/tmp/evil.so"} block pass env-interpolation guard
secret interpolation url=".../?k=${JWT_SECRET}" block pass env-interpolation guard
codegen breakout """\nimport os; os.system(…)\n""" block pass codegen-delimiter guard
over-privileged select admin_execute when run_query suffices block pass deny-by-default SecurityPolicy
(benign) well-typed transfer amount=100 pass pass —
(benign) clean path path="reports/q3.txt" pass pass —
(benign) public URL url="https://api.example.com/v1" pass pass —

The benign controls matter: the gateway isn't scoring 0/12 because it's broken or misconfigured. It forwards everything — malformed and benign alike — because per-argument contract validation is not the layer it operates at.

Where the native gateway is actually enough

This benchmark is deliberately scoped to the contract layer, and it would be dishonest to read "0/12" as "the gateway does nothing." Its own logs during the run show real controls firing:

- Scanning tool call arguments for secrets...   > No secret found in arguments.
- Scanning tool call response for secrets...     > No secret found in response.
- Running airlock-bench/echo-mcp:latest with [run --rm -i --init
  --security-opt no-new-privileges --cpus 1 --memory 2Gb --pull never ...]

There are whole threat classes where the gateway is the right tool and airlock adds little or nothing:

  • Authentication and transport. Who may connect, with which OAuth token, over what channel. This is the gateway's core job and the 2026-07-28 MCP spec's resource-server mandate lives here. airlock does no connection-level auth — if your risk is "an unauthorized client reached the server," the gateway is the answer.
  • Container isolation and egress. no-new-privileges, CPU/memory caps, --block-network. If a tool server is compromised, a container sandbox contains it far better than airlock's in-process egress control can.
  • Known-secret exfiltration. The gateway's block-secrets scans arguments and responses against its actual secret store — a value-based check. If a real stored API key would leave the boundary, the gateway catches it. (In this corpus, ${JWT_SECRET} is a literal placeholder, not a stored secret value, so it wasn't caught there — but a real secret value would be.)
  • Supply-chain / image provenance. --verify-signatures on the server image. airlock validates tool calls, not the trustworthiness of the server image.
  • Org-wide policy at the edge. Centralized rate limiting, tenant isolation, and audit are cleaner at a network gateway than replicated in every process.

If those are your threats, a native gateway may be sufficient on its own.

Where the in-process contract layer adds value

The gateway forwards the payload the model produced; something has to check that payload at the function boundary. That's the gap the 12/12 measures:

  • Argument types, with no coercion — a JSON "100" is not an int.
  • Hallucinated / ghost arguments — parameters the model invented that the tool never declared.
  • Business-rule value constraints — amount > 0, an enum, a max length.
  • Path / URL argument shapes — traversal, SSRF, file:// — as argument content, not as network destinations.
  • Argument-injection strings that are valid JSON — eval/exec, subprocess args, env interpolation, codegen breakout. To a forwarding proxy these are just strings; only content inspection at the boundary catches them.
  • Least-privilege tool selection — which tool the agent picked for the task, given a deny-by-default allowlist.
  • A self-healing error the model can read and retry against, instead of a stack trace that ends the run.

Honest caveats (read these before quoting the number)

  • 12/12 is a coverage/regression baseline, not an adaptive-attacker score. The corpus is agent-airlock's own fixtures — shapes the guards are built to catch. A novel payload outside a guard's pattern set can pass airlock too. Treat this as "the contract layer catches the contract-layer classes it claims to," not "airlock is robust against a determined attacker." For the adaptive angle, see the AgentDojo result in benchmarks/agentdojo/RESULTS.md.
  • Airlock's value depends on the developer declaring the contract. The amount=-1 block only happens because the airlocked tool declares amount > 0. A plain amount: int signature would pass -1 through airlock too. Same for the over-privileged block: it requires a configured allowlist. The contract layer enforces the contract you write — it does not invent one.
  • The gateway's 0/12 is contract-layer-specific. Point the same harness at a known-secret-value exfiltration or an unsigned server image and the gateway blocks where airlock would not. Different layers, different jobs.
  • One gateway, one version. This is Docker MCP Gateway v2.0.1 with defaults. A different gateway, or one with a custom before:exec interceptor authored to validate arguments, could score differently — the harness is there so you can check yours instead of trusting this table.

The one-line version

Use both. The gateway secures the connection and sandboxes the server; the contract layer type-checks the payload the model actually produced. Neither replaces the other, and this benchmark exists so you can verify that split yourself rather than take it on faith.

Provenance: Docker MCP Gateway image v2.0.1 · docker mcp CLI v0.42.1 · Docker engine 29.4.3 · MCP protocol 2025-06-18 · measured 2026-07-16. Raw data: benchmarks/vs_gateway/RESULTS.md. Shorter summary: vs-native-mcp-gateway.md. Regeneration harness: benchmarks/vs_gateway/gateway_harness/.