PyHEP.dev 2026 · 2026‑09‑07

Model Context Protocol for Scientific Software

Building AI interfaces that actually work

Giordon Stark (University of Chicago) · everything shown runs in production on the UChicago ATLAS Analysis Facility

University of Chicago, MANIAC Lab, Enrico Fermi Institute, IRIS-HEP, CERN

Agentic intelligence for research computing

Slides AI-assisted by Claude (Anthropic) · title illustration AI-generated. · PyHEP.dev 2026

What is the Model Context Protocol?

An open protocol connecting AI assistants (LLMs like Claude or Gemini) to the libraries, databases, and services scientists already use.

AI assistantClaude, Gemini CLI, VS Code, …

MCP (JSON-RPC)

⟷

MCP serveryou write this · runs on your laptop or hosted at a facility

Python · subprocess · REST

⟷

Existing library or servicerucio.client, pyAMI, HTCondor, …

tools functions the LLM calls resources data it reads prompts reusable templates

modelcontextprotocol.io · not prompt engineering, not a custom foundation model. ↓ a minimal server

One minute of level-setting, since this is an introduction for scientific software developers. The Model Context Protocol is an open protocol, originally from Anthropic, now with a public spec and SDKs everywhere. The assistant, whatever it is, Claude, Gemini CLI, an editor, speaks MCP, which is JSON-RPC under the hood, to a server you write. That server can run on your laptop or be hosted at a facility, and it talks to whatever you already have however it wants: Python calls into rucio.client or pyAMI, a subprocess, a REST API. Three primitives, and honestly tools are the one that matters in practice: typed functions the model can call, described by a name, a schema, and a docstring. The pitch to this room: this is not prompt engineering and not training a custom model. It's ordinary interface design for an unusual caller. If you can write a Python API, you can write one of these; there's a minimal example below if we want it.

Around an existing Python project · backup

from fastmcp import FastMCP
from rucio.client import Client

mcp = FastMCP("rucio")
client = Client()

@mcp.tool()
def list_dids(scope: str,
              pattern: str = "*") -> str:
    """Search datasets in a scope by wildcard.

    Use a narrow pattern; results are paginated.
    """
    return ", ".join(
        client.list_dids(scope, {"name": pattern}))

mcp.run()

rucio-mcp started from this barebones structure

Everything the LLM sees is right here: name list_dids, schema scope: str, pattern: str = "*", description = the docstring
Wrap rucio.client, not the CLI: structured data in, curated text out
Start on stdio on your laptop; add HTTP when a facility hosts it
fastmcp handles the JSON-RPC; you only write the function

Backup, for the "how do I start" question. This is the barebones structure rucio-mcp started from: FastMCP, the native Rucio client, one decorated function. The decorator turns the signature into the JSON schema and the docstring into the description. Point at the left and the right together: the name, the two typed parameters, and that docstring are literally everything the LLM knows about this tool, which is why the docstring is API surface. Two pieces of advice if you do this tomorrow: wrap the client library, not the command-line tool, because you want structured data on your side of the boundary; and start with stdio on your laptop, where credentials come from your environment like they do for any script. Hosting it for other people is where the rest of this talk begins.

The simple picture, and what I actually run

every MCP tutorial

Agent

→

MCP server

→

Backend

the UChicago AF, in production

User

→

OIDC identityKeycloak

→

AgentMCP client

→

Gateway + brokerauthn · authz · audit

→

Domain MCPRucio, AMI, …

→

Credential custodiansVOMS · krb5 · condor

→

Scientific servicegrid · KDC · schedd

The point

The limiting factor is rarely the LLM. It's reliable, secure access to what scientists already use.

One facility, one summer · all in production

services behind one gateway

124

tools

credential types

~900

users on the UChicago ATLAS AF

The MCP layer

rucio-mcp · datasets, 45 tools
ami-mcp · metadata, 11 tools + a query DSL
af-jupyterlab-mcp · notebooks, 22 tools
af-filesystem-mcp · 4 read-only tools
condor-mcp · batch jobs, 25 tools
atlas-search-mcp-bridge · bridge to the ATLAS OpenSearch MCP hosted (internal) online

The credential layer

af-mcp-platform · gateway + broker
voms-token-service · grid proxies
krb5-token-service · Kerberos
condor-token-service · HTCondor IDTokens
af-credentials · the shared verify/redeem library

Live gateway snapshot, 2026‑09‑05 · mcp-portal.af.uchicago.edu.

six servers later

Lessons learned

interfaces · errors · context · identity · credentials · the facility

An existing API is not an agent interface

bad: the raw API dump

{"logicalDatasetName": "mc23_13p6TeV.601229...",
 "crossSection": "0.729", "crossSection_unit": "nb",
 "genFiltEff": "1.0", "kFactor": "1.0",
 "principalPhysicsGroup": "...", "prodsysStatus":
 "...", ...95 more fields, duplicate rows...}

every field, no units help, and the same for each of 50 datasets

good: markdown-friendly, LLM-friendly

**mc23_13p6TeV.601229** (PhPy8EG ttbar) · VALID
Cross-section: 729 pb (0.729 nb × 1000)
Events: 2M · k-factor: 1.0

---
**Next steps:**
- ami_get_dataset_prov for provenance
- rucio_list_dataset_replicas to locate copies

~15 curated fields, units converted in the open, follow-up tools named

The docstrings keep getting fixed against each tool's actual behavior, to teach the LLM how to use it correctly. The LLM reproduces examples verbatim, so a doc bug is a behavior bug.

Sketch abridged from real ami-mcp output; the curation commit is 901f339 "optimize MCP tools for LLM usefulness".

This is the first lesson chronologically, shown as bad versus good. ami-mcp wraps a twenty-year-old metadata service. The naive wrapper worked and passed its mocked tests, and the output was unusable: a hundred raw fields per dataset, cross-sections in nanobarn with no hint, duplicate rows, multiplied by every dataset in the answer. The fix commit is literally titled "optimize MCP tools for LLM usefulness": pick the fifteen fields an analyst would pick, convert units visibly so the model can't silently mix nb and pb, deduplicate, and end every result with named follow-up tools so the model knows where to go next. And the callout: the docstrings keep getting fixed against the tools' real behavior. AMI is the one server where I teach the model a query language, so the documentation is executable; when a documented example was wrong, the model reproduced the wrong syntax verbatim, and I fixed the docs the way you fix code, against the live service.

Errors are the agent's control flow

bad: what the agent sees by default

Traceback (most recent call last):
  File ".../replicaclient.py", line 158, ...
rucio.common.exception.DataIdentifierNotFound:
  Data identifier 'user.gstark:mydata'
  not found

a dead end: nothing tells the agent what to try next

good: what rucio-mcp returns instead

Error: Data identifier
'user.gstark:mydata' not found

**Recovery:** The DID does not exist in
Rucio. Use `rucio_list_dids` with a
wildcard pattern to search for it.

returned to the agent, not raised into server logs; the next tool is named

Empty results carry the same routing hints. The error categories also feed my monitoring, so I can see which tools confuse the agent.

MCP results have an isError flag so clients can tell failure from success. I didn't set it at first; do both.

The classifier behind the right panel: rucio-mcp tools/_helpers.py (classify_error).

The consumer of an MCP is probabilistic: it course-corrects from what it reads, so an error message is not diagnostics, it's the next instruction. Left panel: the default, a raw exception. Best case the agent guesses; worst case it retries the same call. Right panel: what rucio-mcp returns for the same failure. The error comes back to the agent as a normal result rather than being raised into the server logs, and the recovery line names the exact next tool, so most failures heal themselves inside the loop. Empty results, which are the most common silent failure, get the same treatment. Two housekeeping notes: the same error categories feed my monitoring, so I can literally chart which tools confuse the agent; and MCP has a flag on results, isError, that lets clients and gateways distinguish failure from success. I skipped it at first and my audit log counted failing calls as successes for a while. Return the friendly prose AND set the flag.

Context is a bounded resource

bad: rucio_list_files("mc23..."), unbounded

mc23_13p6TeV:file_000001.root
mc23_13p6TeV:file_000002.root
mc23_13p6TeV:file_000003.root
... 499,997 more lines ...

one call floods the LLM's entire context window, and the server downloaded all 500k names to do it

good: capped, and the output says so

mc23_13p6TeV:file_000001.root
... 49 more ...

---
Showing 50 results (offset=0).
Pass `offset=50` to see more.

a hard server-side limit, announced in the output, with the way to continue spelled out

And cap the server's work too: rucio-mcp once fetched that whole 500k-file container upstream just to serve one 50-row page. Paginate the upstream iterator, not just the output.

The real footer text and iterator fix: rucio-mcp tools/_helpers.py (paginate_iter).

Same bad-versus-good format. Left: what a naive file-listing tool does with a big container: half a million lines into the LLM's context window, which destroys the conversation, and the server fetched all of it from Rucio to boot. Right: what rucio-mcp actually returns. A hard limit of fifty, enforced server side, and the two lines at the bottom matter most: the output says it truncated and says exactly how to get the next page. Truncation the LLM can't see silently distorts what it believes; truncation it can see is just the next tool call. The callout is the second half of the lesson, and I learned it the embarrassing way: for a while the output was perfectly paginated while the server still downloaded the entire container upstream to serve each page. An external contributor caught it. Cap what the backend does, not just what the LLM reads. Same principle at the gateway: it writes its own short instructions instead of forwarding every server's, and only lists the tools you're allowed to call.

Identity is ambient, never an argument

@mcp.tool()
async def create_jupyter_server(
    image: str,
    name: str | None = None,
    cpu_cores: int = 2,
    memory_gb: int = 8,
    ...
) -> str:
    """...The server is always owned by the caller
    (from their verified broker identity) --
    there is no owner argument. ..."""
    claims = await get_broker_claims(ctx, verifier)
    ... create_notebook(owner=claims.unixname, ...)

No tool takes a username. The schema cannot express "read bob's files"
One rule, different ways to implement, e.g.: k8s owner labels · kernel uid/gid · the VOMS nickname in your proxy
Who you are comes from the identity provider (e.g. Keycloak) on every call, not from what a token claims

The principle

Do not expose dangerous parameters; derive them from verified identity.

af-jupyterlab-mcp tools/jupyterlab.py, abridged.

Every agent-facing schema in the fleet has the same absence: there is no owner or username parameter anywhere. The user is whoever the verified token says, full stop. An LLM argument can be hallucinated or injected; a verified claim can't. One rule, implemented differently per backend: Kubernetes owner labels for notebooks, an actual setuid subprocess for the filesystem so the kernel does the enforcement, and the VOMS nickname inside your redeemed grid proxy for Rucio and AMI. And one sharpening: even the token doesn't carry authority. Who you are, your groups and your unix identity, is looked up from the identity provider, Keycloak in my case, on every call, rather than trusted from token contents. That's what makes removing someone from a group take effect in seconds instead of whenever their token expires.

Keep credentials out of the MCP

Lifetime == one call

e.g. a VOMS proxy or Kerberos ticket: fetch it → 0600 temp file → one backend call → delete it. The MCP server keeps nothing.

Exchange, never forward

The user's token never goes downstream (the spec forbids it). Each hop gets a fresh short-lived JWT, valid for one service, expiring in 10 minutes.

One service per secret

e.g. only condor-token-service touches the Condor pool key; the broker is its only allowed caller, and it returns short-lived IDTokens.

bad: a token service deciding permissions

if "atlas-users" in token["groups"]:
    mint_ticket(user)
# a second copy of the access rules,
# trusting whatever the token claims

good: it only verifies who is calling

claims = verify(token, aud="krb5-token-service")
mint_ticket(claims["sub"])
# the gateway already decided permissions;
# this token proves who, from the broker

sketch; written into all three token services' docstrings (krb5 app.py)

Never store secrets in the MCP server. Keep them in a vault (OpenBao, here) behind the broker, and mint short-lived credentials per call.

The architectural heart, and the abstract's promise: authentication without exposing user credentials. Three moves across the cards. First, credential lifetime equals tool-call lifetime: when rucio-mcp needs your grid proxy, it fetches it, writes a mode-0600 temp file, makes exactly one backend call, and deletes it, success or failure. I test the deletion. Second, exchange instead of forwarding: the user's own token never goes downstream, which the MCP spec explicitly forbids. What goes downstream is a fresh JWT the broker mints, valid for exactly one service and expiring in ten minutes. Heavy material like proxy files never passes through the gateway at all; the backend fetches it directly from the broker. The bad-versus-good pair is about where permission decisions live. Bad: the token service checks the token's groups itself. Now the access rules exist in two places, and this copy trusts whatever the token happens to claim, so a stale token keeps its access. Good: the service verifies only that this is a valid broker token addressed to it, and mints for whoever the token says. The permission check already happened once, at the gateway, against live Keycloak groups. That rule is written into all three token services' docstrings so nobody re-grows a second permission layer. And nothing secret lives in an MCP server: it's all in a vault, OpenBao in my case, behind the broker. Deep-dive below if credential plumbing is your thing.

The custodians · deep-dive

voms-token-service

The only pod that sees home dirs and Globus passphrases. Runs voms-proxy-init impersonated; privilege floor found live, five times.

krb5-token-service

Password → kinit → ticket; zero secrets at rest, passwords zeroed on every exit path. Rate limiter protects CERN's lockout counter, not itself.

condor-token-service

Runs condor_token_create next to the pool key, which "must never reach the broker": it can mint any identity.

Why three services, not one

Different privilege floors (root+caps / none / one init container). Merging them forces the union of privileges onto everything. Each is a one-question audit: "did this call come from the broker?"

Orchestrate the hardened native binary; parse artifacts (ccache bytes, PEM), never CLI prose.

Deep-dive for questions. Three services because three root secrets with three different privilege needs. voms needs root plus capabilities to read user keys, and every one of those facts was discovered in production: grid tooling checks file ownership, not just mode. krb5 needs no privileges at all and holds nothing; its rate limiter exists to protect CERN's account-lockout counter, a third party, which is my favorite example of consequence-driven design. condor sits next to the symmetric pool key that can mint tokens for any identity, which is exactly why it must never reach the broker's trust domain. Common thread: all three shell out to the battle-tested native binary and parse artifacts, ccache bytes and PEM, never the human-readable CLI output, which turned out to be locale-dependent.

The facility between your code and the backend

AF gatewayspeaks OIDC / OAuth 2.0 bearer tokens

↓

atlas-search-mcp-bridge (~600 lines)af-credentials writes the ccache (cached in the broker's vault) to a short-lived file per request

↓

curl --negotiatedoes the SPNEGO/Kerberos exchange; zero lines of my Python

↓

CERN's ATLAS OpenSearch MCPKerberos-only; not mine to change

Common gotchas (each one bit me)

Kerberos and CA trust config must be in the container (krb5.conf, your CA bundle)
network policy and DNS have to allow the pod → KDC path
proxies in between can quietly drop headers

The principle

Unit tests with mocks can't see any of this. Test against the real facility services before calling it done.

atlas-search-mcp-bridge · SPNEGO = negotiated Kerberos auth at the HTTP layer.

By "the environment" I mean everything between your code and the backend: containers, trust stores, network policy, DNS, the facility itself. One backend I genuinely could not change: CERN's OpenSearch MCP only accepts SPNEGO, negotiated Kerberos at the Apache layer, while everything on my side speaks OIDC bearer tokens. The answer is the diagram: a six-hundred line bridge that verifies the broker's token, uses the af-credentials library to drop a throwaway Kerberos ccache for that one request, and then lets curl's GSSAPI do the actual SPNEGO exchange. Zero lines of Kerberos in my Python, no policy, no standing secret, and it never parses the MCP traffic passing through it. The right box is the honest part: this service had full test coverage, and production still failed repeatedly, on Kerberos config and CA bundles inside the container, on a network policy silently blocking the KDC, on reverse DNS, and on a dropped header. Never the code. Mocked unit tests cannot see any of that, so budget real time for testing against the live facility services.

many MCPs, one endpoint

From servers to a platform

what breaks at ten MCPs · one tool call, end to end

What breaks at ten MCPs (that one never sees)

Design decisions baked into the AF MCP Platform

Name collisions: two MCPs serve a tool with the same name; fix = a prefix per service (rucio-atlas_*, fs_*)
Each service gets its own fixed OAuth audience, pinned in config, never derived from a name that can change
Listing needs credentials too: some MCPs require auth even for tools/list, so the gateway mints one just to enumerate

New tools are opt-in: when an MCP releases a tool, it stays disabled until an operator maps it at the broker
Authorization is plain RBAC: Keycloak group → permission → tool
A dead backend disappears from tools/list instead of every call failing

Adding the Nth MCP

Each MCP is added via config (Infrastructure-as-Code). Sometimes it needs a token service, sometimes a bridge — never changes to the MCP itself.

af-mcp-platform issues #113, #257, #121 · 289 commits, 77 days, 1,337 tests. ↓ more scars

Everything so far was about one server at a time. At ten, a different class of problem appears, and none of it is in the MCP spec. Quick tour: name collisions, because two backends will eventually serve a tool with the same name and the failure is silent, so every service gets a prefix. OAuth audiences: each service's audience is pinned in config; I learned the hard way not to derive it from a display name, because a rename broke authentication for everything at once. Listing: some MCPs need auth just to enumerate their tools, so the gateway mints a credential for the listing itself. New tools are opt-in: when a backend ships a new tool, it stays disabled until an operator maps it to a permission at the broker. Authorization itself is nothing exotic, plain RBAC from Keycloak groups. And when a backend dies, its tools leave the list rather than erroring one call at a time. The payoff: adding the Nth MCP is configuration, infrastructure-as-code. Sometimes it needs a token service, sometimes a bridge, but you never modify the MCP itself.

Tried X, discovered Y, ended with Z · backup

Tried	Discovered	Ended with
Rust CLI wrapper	parsing text tables is fragile	Python + native client
Returning raw API rows	noise swamped the LLM	curated fields, converted units, next steps
Docstrings written from memory	the LLM reproduced broken examples verbatim	docstrings tested against the live service
Stateful MCP sessions, 2 replicas	cross-replica "Session terminated"	stateless HTTP (now the spec default)
Dynamic Client Registration	you maintain a database of client ids, and a restart orphans them all	CIMD (later adopted by the spec)
Groups inside JWT claims	revocation waits for token expiry	live IdP lookup, a real kill switch

Full catalogue with commit hashes: mcp-design-talk.md §7.2 (about 40 of these).

Backup, and honestly my favorite slide if there's time. Six of the roughly forty tried-discovered-ended-with lessons in the research document. The top three are interface lessons: wrap the client not the CLI, curate what the model reads, and treat docstrings as code that gets tested against the live service. The bottom three are infrastructure lessons, and two of them, statelessness and client-id metadata documents, I adopted under production pressure months before the MCP spec standardized the same choices. Dynamic Client Registration deserves its line: besides the orphaning, it quietly makes you the owner of a database of client ids, which can grow without bound. Early production pain turned out to be a decent predictor of where the protocol was heading.

One tool call, end to end

mcp-portal.af.uchicago.edu · live

AI Assistant

MCP Gateway

Auth Policy Audit Broker

Rucio

HTCondor

Jupyter

Storage

Credentials stay behind the broker

The assistant never holds a raw credential; the user's token is never forwarded.

Authorize every call

Group-based access; denials audited too.

Short-lived, scoped

Minted per service, cached at the broker, reused until expiry; the per-call copy is deleted after use.

mcp-portal.af.uchicago.edu · the portal landing page's own animation (af-mcp-platform). ↓ the same call, step by step

This is the platform in one animated call: the portal landing page's own diagram, reproduced in the slide. Let it run one cycle while talking. Two authorized calls play through: the assistant asks, Auth, Policy, and Audit check green, the Broker mints a credential off to the side, the call lands on Rucio or HTCondor, and the answer comes back. The third cycle is the governance story in eight seconds: "Delete my /data datasets" passes Auth, fails Policy, the gateway buzzes red, and the assistant reports it is not authorized. Three things to say over it: credentials stay behind the broker, so the assistant never holds anything raw and the user's token is never forwarded downstream. Every call is authorized against group-based permissions, and the denials get audited too. And every credential is minted short-lived for exactly one service, used once, and deleted. The step-by-step token walk is the down-arrow deep-dive if someone wants the details.

The same call, step by step · deep-dive

Claude calls rucio-atlas_rucio_list_dataset_replicas with its personal access token

Gateway authenticates; groups and POSIX identity come from the IdP (Keycloak) on every call

Authorized once: this tool requires read_data; denials are audited too

Broker mints a JWT: aud=rucio-mcp-atlas (one service), exp = 10 min; it says who you are, not what you may do

rucio-mcp verifies that JWT, then redeems the VOMS proxy directly from the broker's vault

One Rucio call as the user; proxy deleted; one audit record, traced and metered

The user's token is never forwarded downstream; proxies and tickets never pass through the gateway.

Interfaces that travel

Across LLM providers

Design for the least-capable LLM client: e.g. Claude Desktop had no OAuth discovery, so I added personal access tokens; gemini-cli choked on an OAuth metadata fallback, so I deleted it
Standard OAuth, standard transports, no provider-specific hooks

Across experiments

rucio-mcp already serves ATLAS, CMS, DUNE, ESCAPE: the domain layer is VO-agnostic
Transplant the shape (broker + custodians + config-only services), not the CERN-specific code

Interoperable tool interfaces are likely to matter more than LLM-specific integrations.

↓ how my choices scored against the evolving MCP spec

The abstract's closing claim, made concrete. Across providers: the discipline is designing for the least-capable client. I had a genuinely cleaner design for routing based on OAuth scopes, fully written up, and I shelved it because only one client on the market supports the feature; the write-up is committed next to the code. Same story with the OAuth metadata fallback I deleted after a different client choked on it. Standard auth, standard transports, nothing provider-specific: that's what "works across providers" costs. Across experiments: rucio-mcp is already multi-VO because Rucio is, and the whole facility shape, broker, custodians, config-only services, transplants to any lab. What doesn't transplant is the CERN-specific plumbing, and that's fine: it's all behind one provider interface and per-service config. If you take one sentence home, take the abstract's: interoperable tool interfaces will matter more than model-specific integrations.

Scored against the MCP spec · backup

My choice	The spec (2026-07-28)	Verdict
Stateless HTTP, state as explicit handles	protocol sessions removed	compliant / anticipated it
CIMD instead of Dynamic Client Registration	DCR deprecated for CIMD	compliant / anticipated it
Tools only; no sampling, roots, logging	all three deprecated	compliant / anticipated it
Exchange tokens, audience-validate everywhere	token passthrough forbidden (MUST)	compliant
Markdown-only outputs, no `outputSchema`	structured content recommended	wrong; retrofitting now
No read-only / destructive tool annotations	annotations recommended	wrong; the gateway must keep its own list of which tools write

Spec: modelcontextprotocol.io/specification/2026-07-28 · full comparison in mcp-design-talk.md §6.3.

When is an MCP server appropriate?

Build one when

a library or service scientists already use
assistants need it repeatedly, across clients
you want to control what context the agent gets injected with
you can bound outputs and make errors recoverable

Skip it when

a one-off script would do
the backend's API is still churning under you
the "tool" would hand the LLM a credential

Interoperable tool interfaces are likely to matter more than LLM-specific integrations.

Design the interface for the LLM; design the architecture for the credential. ↓ the checklist

Giordon Stark · kratsg · PyHEP.dev 2026 · 2026‑09‑07 · slides AI-assisted by Claude (Anthropic).

Close on the abstract's promise: when is an MCP server appropriate. Build one when there's an existing library or service with real users, when assistants will need it more than once and from more than one client, and when you can actually bound the outputs and make the errors recoverable, because those are the two properties the model experiences directly. Skip it when a script would do, when the backend has no stable contract to wrap, or when the honest implementation would hand the model a credential; fix that boundary first. Then the two closing lines, slowly: interoperable tool interfaces will matter more than model-specific integrations. And the one I'd put on a sticky note: design the interface for the model, design the architecture for the credential. The checklist below is the hallway version; the full dossier with commit-level citations is mcp-design-talk.md in the slides repo.

The checklist · backup

One domain per server, with written non-goals: datasets, or metadata, or notebooks; not all three
Docstrings are the LLM's manual: write them for the LLM and test the examples against the live service
Bound every output with a hard server-side limit, and say in the output when you truncated
Errors name the next tool to call, and set MCP's isError flag on the result
Structured outputs (outputSchema) from day one, not retrofitted like mine

No username parameters: identity comes from the verified token, never from the LLM
No stored credentials in the server: fetch per call from a vault, use once, delete
Permissions decided at one gateway, not re-implemented per server
Stateless server: any state lives in the backend, addressed by an id the tools pass around
Test against the real facility: the failures live in config, trust stores, and network, not in your logic

The long version, with commit-level citations: mcp-design-talk.md.

188a739 · 2026-09-07