Add internal development MCP server for Airflow API as part of Breeze

[shahar1]

Human Summary

To prepare the ground for production-scale user-facing MCP server for Airflow REST API, as proposed by AIP-91 (still in draft), I came to the conclusion that it might be better to start in lower scope with an MCP server for internal development purposes of the open-source (specifically the API parts), bundled with breeze as an opt-in feature.
There's a breeze ADR included as part of this PR (ADR-0018), I'll give tha main parts here:

Why?

It is intended mostly for using AI agents to tackle issues whose resolution requires multiple API calls
Without MCP, you have to rely on API calls using the CLI (hoping that the agent doesn't make typos), and there's some authentication friction (generating JWT tokens). With MCP - the server is deployed as part of breeze and takes care of the authentication parts, and the methods are exposed to the client.
The bigger payoff, though, is that exposing the API as typed, ready-to-call methods lets the agent cheaply inspect the live state of a running Airflow - so on issues that hinge on what's actually happening at runtime, it converges on a fix with far less trial-and-error instead of guessing.

How do you know that it (may) work?

I've empirically tested the resolving 2*** GitHub issues by subagents, where each issue is resolved independently by 2 subagents with the same model (A/B testing):

• One without access to MCP.
• One with MCP access.

Both agents resolved the tested issues successfully, generating equivalent solutions.
In the issue with less requirement for multiple API calls - the MCP arm was no faster (slightly slower, in fact) and consumed more tokens.
However, in the other issue where there were multiple API calls - the MCP subagent took less than 50% of the running time and reached the fix with far fewer iterations (about half the turns and a third of the test runs).

*** Disclaimer: I'm aware that the sample space of 2 issues doesn't say much about statistical significance, but I'll be happy to reevaluate it when it's in used by others! (I'll also try it out on additional issues as long as the PR is in review)

How do we use it?

Deploying it as part of the breeze deployment is as easy as:

• Runnning breeze start-airflow --mcp-server to activate it
• Running an additional command for adding and exposing the MCP server to the AI client (Claude Code/Codex/Copilot/etc.)

Important things

• There are basic guardrail flags for Create/Update/Delete operations (write - enabled by default, delete - disabled by default).
• Tests and docs are included.
• Unit tests are included in the CI, integration tests that need AI API token - currently manual.

I'd like to thank @aritra24 for the brainstorming, discussion, and implementing a prototype.

AI Summary

Click here

Adds an **internal, development-only** [MCP](https://modelcontextprotocol.io/) server (`dev/mcp_server`) that lets any MCP-capable coding agent inspect and debug the Airflow instance running in Breeze through the public REST API — list Dags/runs/task instances, fetch tailed logs with tracebacks, surface import errors, diagnose a failed run in one call, and (opt-in) trigger/clear runs. It is **not** shipped to users and makes **no** airflow-core changes.

This is the "internal Breeze tool" step of AIP-91: a stateless REST proxy (no direct metastore access, read-only by default) so later AIP-91/AIP-101 phases can grow out of it. Full rationale, alternatives, and the pros/cons are captured in a new ADR: dev/breeze/doc/adr/0018-internal-mcp-server-for-breeze-development.md.

Highlights

• Deployed as a Breeze service: breeze start-airflow --mcp-server (host port 28081), run from mounted sources via uvx; also runnable over stdio by any MCP client. Gone when Breeze is off.
• Safety model: read-only by default; writes behind AIRFLOW_MCP_ALLOW_WRITES; DELETE behind a separate, stricter AIRFLOW_MCP_ALLOW_DELETES (off even when writes are on) and reachable only via the generic escape hatch — no dedicated delete tool. Secrets stripped from list responses.
• Curated tool surface (~20 tools) plus two documented escape hatches, rather than one-tool-per-endpoint.
• Testing/CI: unit tests run in CI as their own Dev MCP server tests job; a contract test asserts every endpoint the server calls exists (with its method) in Airflow's committed OpenAPI spec, with a second test guarding that endpoint list against drift; the live-Breeze and real-LLM-client integration tests are opt-in/manual for now.

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code (Opus 4.8)

Generated-by: Claude Code (Opus 4.8) following the guidelines

[jason810496]

I didn't review the whole implementation and the functionality yet, but based on the title and PR description I would like to cc @RoyLee1224 (our GSoC intern working on Agent Eval / Breeze Agent Skills recently!) here.

We had considered MCP for breeze before as well (I need to recall the context and dig into what we had discussed why not MCP but Agent Skills) or perhaps the purpose of the MCP this PR would like to introduce is different from what we want to achieve in Airflow GSoC Breeze Agent Skills -- #62500.

I will take a thorough review soon, and see what is the better next step for both directions.

[shahar1]

I didn't review the whole implementation and the functionality yet, but based on the title and PR description I would like to cc @RoyLee1224 (our GSoC intern working on Agent Eval / Breeze Agent Skills recently!) here.

We had considered MCP for breeze before as well (I need to recall the context and dig into what we had discussed why not MCP but Agent Skills) or perhaps the purpose of the MCP this PR would like to introduce is different from what we want to achieve in Airflow GSoC Breeze Agent Skills.

I will take a thorough review soon, and see what is the better next step for both directions.

Thanks! What was the intended scope of the solution you and Roy worked on?
I don't see agentic skills and MCP as contradictory, but complementary: agentic skills for Breeze help build the contribution workflow correctly, while the MCP operates at runtime inspecting the live instance.

[potiuk]

We had considered MCP for breeze before as well (I need to recall the context and dig into what we had discussed why not MCP but Agent Skills) or perhaps the purpose of the MCP this PR would like to introduce is different from what we want to achieve in Airflow GSoC Breeze Agent Skills -- #62500.

I would echo what @jason810496 wrote. I think MCP server is only really useful where you want to have some real "server side" content that you want to keep behind the server. Breeze does not really have any "data" behind to expose via MCP only "actions" and I would say SKILLs are much, much better (and cheaper) way of accessing breeze functionality - if we think CLI is not enough.

[shahar1]

We had considered MCP for breeze before as well (I need to recall the context and dig into what we had discussed why not MCP but Agent Skills) or perhaps the purpose of the MCP this PR would like to introduce is different from what we want to achieve in Airflow GSoC Breeze Agent Skills -- #62500.

I would echo what @jason810496 wrote. I think MCP server is only really useful where you want to have some real "server side" content that you want to keep behind the server. Breeze does not really have any "data" behind to expose via MCP only "actions" and I would say SKILLs are much, much better (and cheaper) way of accessing breeze functionality - if we think CLI is not enough.

I feel that there's a misunderstanding here (and I renamed the title, as previous one might have been misleading) -
the MCP is for Airflow REST API which is deployed as part of breeze, not for breeze commands.
I'll be happy if you could read through the use cases in the PR's description, as well as the ADR.

Edit: also clarified in the docs + ADR, it's indeed quite important to clarify the scope.

[potiuk]

I feel that there's a misunderstanding here (and I renamed the title, as previous one might have been misleading) - the MCP is for Airflow REST API which is deployed as part of breeze, not for breeze commands. I'll be happy if you could read through the use cases in the PR's description, as well as the ADR.

Ah... Never judge the book by the cover? :)

[potiuk]

I've read in the description that "breeze command line is unreliable so this is to replace it" - and yes - I have not read the PR, just jumped to the conclusion it's for the "breeze CLI" :)

[Dev-iL]

@abhishekbhakat you worked on a related project - any feedback you can share to improve this feature will be most welcome!

[RoyLee1224]

I think the eval harness in #69308 could turn this n=2 A/B into a proper with/without comparison — happy to explore as a follow-up.

[Dev-iL]

I have some reservations about this approach:

1. Either allow deletion normally, or forbid it completely. At the moment it is "hidden" behind an extra tool use, so the only real effect is making it harder (i.e. require more steps), but not impossible, for an agent to run a risky action. What you'll get is agents that are worse at tool calling will be unable to run it, and agents that are good at it won't care.
2. When an MCP available action is "risky", the right place to flag it is in the SKILL accompanying the MCP - it should use wording like "requires explicit user consent before calling" etc. In the absence of such instructions, there's little stopping the agent from sidestepping the MCP and calling the API directly

Additionally:

1. I don't understand why have a "semi Code Mode" implementation - the split seems arbitrary, and a main benefit of the code mode transform (having an unlimited number of mcp endpoints w/o confusing agents) gets diluted.
2. How about having two flavors of the MCP, one with destructive actions enabled and one without? Edit: I noticed that this is what you already did with the env vars later.

[shahar1]

1. It's not an extra tool use - it's a hard rejection at the request layer, enforced as a barrier on every call to the Airflow REST API.
2. I agree that bypassing the MCP is possible by design, and there's nothing to do about it except for scoping the credentials via RBAC - which is out of scope for this PR. However, we should eventually implement the defense mechanism in two layers:
   a. Skill level - prevents bypass attempts via explicit-consent wording in the SKILL (I'd prefer to leave that implementation to a follow-up PR, since this one is already large).
   b. MCP level - prevents accidents from agents that already got past the skill-level guard.
3. During development of the MCP, I tried to aim towards the most common tools in use for internal development of Airflow (@jason810496 - airflowctl currently does not support some useful features such as task instances). I think that the current list is good enough to start with, we could always change it later as part of the evaluation or on the go. If and when we reach the scale that justifies usage of CodeMode - we could consider using it as well (considering it will also become stable at that time).

[Dev-iL]

It's not an extra tool use - it's a hard rejection at the request layer, enforced as a barrier on every call to the Airflow REST API.

By "extra tool use" I was referring to the below (emphasis mine):

plus two escape hatches (airflow_api_get / airflow_api_call) so no endpoint is out of reach

Perhaps I misunderstood what "no endpoint out of reach" means (might be worth rewording?). If the endpoint is discoverable via one of the MCP methods, it practically begs the AI to sidestep the MCP - I can already imagine the following interaction: "hmm apparently there's a DELETE method in the API, let me try to call it" -> "seems like the MCP doesn't allow calling the method, perhaps it's a bug? <looking for workarounds...>" -> "oh, looks like this .env file has the necessary credentials to access the DB" -> "great, now can I start over with a fresh DB"

we should eventually implement the defense mechanism in two layers: a. Skill level ...

I think the Skill layer is an easy win. I urge you to consider making it now.

[jason810496]

Thanks Shahar for brining this up!

I confirmed that the purpose of MCP server for Airflow API and the Airflow GSoC Breeze Agent Skills #62500, the Breeze Agent Skills only focus on overall developer journey and ensure our developer-facing skills, Agents.md won't drift and will be able to verify the setup -> by eval).

the MCP is for Airflow REST API which is deployed as part of breeze, not for breeze commands.
I'll be happy if you could read through the use cases in the PR's description, as well as the ADR.

IIUC, we could achieve the same purpose and functionally by adding a new "rule" on AGENTS.md to route the query with Airflow CTL by breeze exec ... airflowctl or (even just naive docker exec ... airflowctl ...).

MCP for Airflow API server in Breeze is definitely another direction to achieve it but might introduce more maintenance effort.

I think the eval harness in #69308 could turn this n=2 A/B into a proper with/without comparison — happy to explore as a follow-up.

IMHO. How about waiting for the eval framework that @RoyLee1224 working in #69308 to settle down first before directly introducing MCP for Breeze. WDYT?

Additionally, @shahar1 would you mind to provide more context or even concrete setup for I've empirically tested the resolving 2*** GitHub issues by subagents, where each issue is resolved independently by 2 subagents with the same model (A/B testing): this part you mentioned when you have a moment? Thanks.

Then we're able to eval for both directions (without, MCP approach, AGENTS.md + defer to airflowctl approach)

The usage eval framework will be like the following as #69308 (comment) pointed out.

EVAL_REPEAT=3 prek run run-skill-eval --hook-stage manual --all-files
EVAL_FULL=1 prek run run-skill-eval --hook-stage manual --all-files
MODEL=claude-haiku-4-5-20251001 prek run run-skill-eval --hook-stage manual --all-files

From my perspective, If we miss context of some the server-side runtime, we should improve the coverage of Airflow CTL, leverage the existing feature at much as possible instead of introducing new deps in the first place. (IMO, CTL should be able to represent the all the Core API context)

[shahar1]

IIUC, we could achieve the same purpose and functionally by adding a new "rule" on AGENTS.md to route the query with Airflow CTL by breeze exec ... airflowctl or (even just naive docker exec ... airflowctl ...).
Few issues with that:

1. It introduces friction around building the command string, escaping/quoting it correctly, and parsing the output back into something usable - a typed MCP tool sidesteps all three; params go in typed, the response comes back typed.
2. Authentication has to be handled explicitly on the CLI side (login step, token storage, keyring) - the MCP server takes care of that behind the scenes.
3. airflowctl is still missing some commands we rely on (TIs, logs). Worth noting logs aren't in the legacy airflow CLI either - there's no airflow tasks logs command; fetching task logs has only ever been possible through the REST API (or the web UI), which is exactly what this MCP wraps.

MCP for Airflow API server in Breeze is definitely another direction to achieve it but might introduce more maintenance effort.

I don't think it's too much maintenance in practice:

• The surface is already small on purpose, and narrowly scoped to the most common methods.
• Unit tests already run as their own CI job.
• There's a contract test that checks every endpoint we call against Airflow's own OpenAPI spec, so if something renames/changes, CI just fails and tells us exactly where.
• The only new dependency is fastmcp, and everything it pulls in (mcp, httpx, pydantic, etc.) we already ship elsewhere anyway. So it's a bounded cost, not something that grows on us over time.

IMHO. How about waiting for the eval framework that @RoyLee1224 working in #69308 to settle down first before directly introducing MCP for Breeze. WDYT?

I wouldn't like to gate on that, if it's ok - this is an opt-in feature that doesn't touch any other code areas, so I think we can run independently and evaluate/adjust along the way rather than block on it. I still see agentic skills and MCP as complementary here. In fact, we'll even need a dedicated skill for using the MCP, covering the consent/risk wording around write/delete operations discussed with @Dev-iL (separate from the other breeze-related skills you're developing as part of your current efforts). I'd be happy if Roy wants to contribute this part, once the eval framework settles.

Additionally, @shahar1 would you mind to provide more context or even concrete setup for I've empirically tested the resolving 2*** GitHub issues by subagents, where each issue is resolved independently by 2 subagents with the same model (A/B testing): this part you mentioned when you have a moment? Thanks.

From my perspective, If we miss context of some the server-side runtime, we should improve the coverage of Airflow CTL, leverage the existing feature at much as possible instead of introducing new deps in the first place. (IMO, CTL should be able to represent the all the Core API context)

For each issue I ran two headless agents (claude -p), same model (Sonnet), each in its own git worktree off the same main commit, unaware of each other, given the identical task prompt (reproduce, root-cause, fix, and add a regression test). The only difference was MCP access - one arm had the proposed MCP, the other was hard-denied it. I checked ground truth myself: each arm's regression test passes with its fix and fails without it. I evaluted some metrics from a JSON output (wall-clock, turns, tokens, cost) and parsed each transcript (pytest runs, live API/MCP calls).

I used this method on two issues:

• #42790 (Well-specified / low-API) - Both arms produced a correct, tested fix; the MCP arm was no faster (slightly slower) and used more tokens - no benefit where the fix is clear from the issue text.

• #39801 (Hard / API-heavy) - Both arms again reached a correct, tested fix with the same root cause - but the MCP arm did it in ~9 min / 46 turns / 8 test runs vs ~28 min / 86 turns / 24 test runs. The gain (probably) came from ~2 cheap typed calls to observe live task-instance state early, cutting the trial-and-error roughly in half.

From my perspective, If we miss context of some the server-side runtime, we should improve the coverage of Airflow CTL, > leverage the existing feature at much as possible instead of introducing new deps in the first place. (IMO, CTL should be
able to represent the all the Core API context)

Agreed on the direction, but:

1. airflowctl genuinely doesn't cover this yet (as I stated above)
2. "new deps" overstates it - fastmcp is the only new dependency, everything else is already shipped.
3. Everything from my first reply still stands regardless - auth handled server-side, typed calls vs. CLI parsing, etc.

[kaxil]

Interesting choice to using MCP Server for Breeze, and I'd like to understand the rationale a bit more. Like @potiuk mentioned, Server side/Remote MCP is where I see tremendous value of an MCP Server. In fact, I wrote https://kaxil.substack.com/p/mcp-vs-cli-vs-rest when people were calling MCP dead.

I don't see agentic skills and MCP as contradictory, but complementary: agentic skills for Breeze help build the contribution workflow correctly, while the MCP operates at runtime inspecting the live instance.

Skills that use CLIs can inspect live and are categorically better for such things for the reason mentioned in the blog post, specifically piping and chaining the output of various commands -- that makes output token efficient, which is why the suggestion of airflowctl makes complete sense to me.

IMO, we should first invest on ensuring our CLI tools are agent friendly before we go deeper on this route. Specifically asking as this started with Breeze.

airflowctl is still missing some commands we rely on (TIs, logs). Worth noting logs aren't in the legacy airflow CLI either - there's no airflow tasks logs command; fetching task logs has only ever been possible through the REST API (or the web UI), which is exactly what this MCP wraps.

I think then we should help get airflowctl over the line since that is what we are saying is going to be the recommending Prod way to reach to Airflow Prod deployment. That's one thing to consciously decide from maintenance & focus perspective with Helm Chart, airflowctl, API clients, etc

Adding a "Request changes", just to avoid accidental merges and have discussion. I might have a few weeks where I don't have access to laptop and as such if I don't respond, feel free to "dismiss" the request changes if it has been discussed and other committers are aligned.