HTCondor Workshop · Lyon · 2026‑10‑01

The Facility Is
the Context

Building an MCP Platform for Scientific Computing

Giordon Stark (University of Chicago) · everything shown runs in production on the UChicago ATLAS Analysis Facility

University of Chicago, MANIAC Lab, Enrico Fermi Institute, IRIS-HEP, CERN

Agentic intelligence for research computing

Slides AI-assisted by Claude (Anthropic) · title illustration AI-generated. · HTCondor Workshop Autumn 2026

The big picture

From prompt to discovery

$ claude mcp add --transport http atlas-af https://mcp.af.uchicago.edu/mcp

what is needed behind this URL?

Claude Code · any model · any harness
› Using the atlas-af mcp, let's find a 1 TeV Higgsino in the Run 2 + Run 3 dataset.

Adapted from “Towards Agentic Analysis”, ATLAS AMG Weekly, 2026-09-04

Where the work is

Swap the model freely; context makes it useful

Modelfrontier API · or self-hosted open weights

HarnessClaude Code · Codex · Copilot

Facility context & toolsdata, storage, batch, identity, fits · via MCP servers

The first two change week to week. That isn't what limits agentic workflows: it's reliable, secure access to the tools, data, and identities users already have.

Same prompt: “Where should my job outputs go?”

Generic model

“Next to the job, in your home area.”

→ big files fill the home quota in minutes

Facility-aware model

“Big outputs to fast local scratch; small summaries in your backed-up home.”

→ same model, more context

Everything here runs in production at the UChicago ATLAS Analysis Facility · mcp-portal.af.uchicago.edu

One slide of framing, then it's all implementation. The way I think about an agent is three blocks. The model, which you can swap: frontier API or open weights you host yourself. The harness, Claude Code, Codex, Copilot, also swappable. And the third block, the facility's context and tools: where the data lives, how you submit jobs, who you are. That third block is the only durable, tractable piece, and it's the one we, the people running the facility, control. Furnishing context is an infrastructure problem, not a prompt problem, which is why I think it belongs in this room. The model and the harness change week to week, and that's fine; they're not what holds us back. A concrete example, same prompt to both. Ask a generic model where your batch job's outputs should go and it'll say next to the job, your home area, and the big files blow your quota in minutes. A model that knows the facility sends big outputs to fast local scratch and keeps small summaries in the backed-up home. The model didn't get smarter; we gave it context. And the flip side: without context it should stay quiet. Ours once confidently recommended tuning for the wrong filesystem, so our rule is ungrounded means silent.

What is the Model Context Protocol?

An open protocol connecting AI assistants (LLMs like Claude or Gemini) to the libraries, databases, and services scientists already use.

AI assistantClaude, Gemini CLI, VS Code, …

MCP (JSON-RPC)

⟷

MCP serveryou write this · runs on your laptop or hosted at a facility

Python · subprocess · REST

⟷

Existing library or servicerucio.client, pyAMI, HTCondor, …

tools functions the LLM calls resources data it reads prompts reusable templates

modelcontextprotocol.io · not prompt engineering, not a custom foundation model

One minute of level-setting. The Model Context Protocol is an open protocol, originally from Anthropic, now with a public spec and SDKs everywhere. The assistant, whatever it is, Claude, Gemini CLI, an editor, speaks MCP to a server someone writes. That server can run on your laptop or be hosted at a facility, and it talks to what we already have: rucio for datasets, AMI for metadata, HTCondor for batch. For this room: it's a standard way to put a facility service in front of any model, the same way you'd put an API in front of a schedd. Tools are the primitive that matters in practice: typed functions the model can call, described by a name, a schema, and a docstring. Not prompt engineering, not a custom model: ordinary interface design for an unusual caller.

Around an existing Python project · backup

from fastmcp import FastMCP
from rucio.client import Client

mcp = FastMCP("rucio")
client = Client()

@mcp.tool()
def list_dids(scope: str,
              pattern: str = "*") -> str:
    """Search datasets in a scope by wildcard.

    Use a narrow pattern; results are paginated.
    """
    return ", ".join(
        client.list_dids(scope, {"name": pattern}))

mcp.run()

rucio-mcp started from this barebones structure

Everything the LLM sees is right here: name list_dids, schema scope: str, pattern: str = "*", description = the docstring
Wrap rucio.client, not the CLI: structured data in, curated text out
Start on stdio on your laptop; add HTTP when a facility hosts it
fastmcp handles the JSON-RPC; you only write the function

Backup, for the "how do I start" question. This is the barebones structure rucio-mcp started from: FastMCP, the native Rucio client, one decorated function. The decorator turns the signature into the JSON schema and the docstring into the description. Point at the left and the right together: the name, the two typed parameters, and that docstring are literally everything the LLM knows about this tool, which is why the docstring is API surface. Two pieces of advice if you do this tomorrow: wrap the client library, not the command-line tool, because you want structured data on your side of the boundary; and start with stdio on your laptop, where credentials come from your environment like they do for any script. Hosting it for other people is where the rest of this talk begins.

Capabilities and knowledge

MCP: arms and hands

The facility lets my agent do things and learn things:

find datasets and metadata
submit and watch batch jobs
read my files, run my notebooks

Skills: domain expertise

The USATLAS marketplace gives it know-how:

af-bnl, af-uchicago: how each facility works
atlas: Rucio, AMI, Open Data, pyhf, cabinetry, HistFitter, ServiceX, …
hep-python-tools: generic HEP Python tooling

“Towards Agentic Analysis”, ATLAS AMG Weekly, 2026-09-04 · github.com/usatlas/marketplace

A facility-native MCP server must be…

Convenient

No extra barriers for analyzers*; self-service access to what the facility already offers.

Secure

Never leak your credentials to the agent; never leak sensitive experiment data**.

Auditable

Telemetry on usage: which tools get used, which don't.

*apart from having your own LLM assistant · **depends on the experiment’s guidance

“Towards Agentic Analysis”, ATLAS AMG Weekly, 2026-09-04

Every MCP tutorial vs. the UChicago AF

every MCP tutorial

Agent

→

MCP server

→

Backend

the UChicago AF, in production

User

→

OIDC identityKeycloak

→

AgentMCP client

→

Gateway + brokerauthn · authz · audit

→

Domain MCPRucio, AMI, …

→

Credential custodiansVOMS · krb5 · condor

→

Scientific servicegrid · KDC · schedd

The MCP layer

rucio-mcp · datasets, 45 tools
ami-mcp · metadata, 11 tools + a query DSL
af-jupyterlab-mcp · notebooks, 22 tools
af-filesystem-mcp · 4 read-only tools
condor-mcp · batch jobs, 25 tools
atlas-search-mcp-bridge · to CERN's ATLAS OpenSearch MCP

The credential layer

af-mcp-platform · gateway + broker
voms-token-service · grid proxies
krb5-token-service · Kerberos
condor-token-service · HTCondor IDTokens
af-credentials · shared verify/redeem library

7 services behind one gateway · 124 tools · 6 credential types · ~900 AF users · live gateway snapshot, 2026‑09‑05 · ↓ architecture overview

Start with the three-box diagram every MCP tutorial draws, because it's true: agent, server, backend, and the SDK makes that a weekend of work. The second row is the same tool call as it actually runs in production. The user authenticated through Keycloak, the agent holds a personal access token, the gateway authenticates and authorizes once, a broker mints a short-lived identity token for exactly one backend, the backend redeems it for a real credential from a custodial service, and only then does the schedd or Rucio hear about it. Every box in that second row exists because something broke without it. Below is what's actually deployed. The MCP layer: six services across data management, metadata, notebooks, filesystems, batch jobs through Brian Bockelman's condor MCP, and a bridge to CERN's OpenSearch MCP that exists only to translate authentication. The credential layer: the platform itself, three token services that are deliberately not MCPs, and the shared library that ties them together. 124 tools, for roughly nine hundred AF users. One design point: when a backend goes down, its tools disappear from the tool list instead of failing at call time. The day I snapshotted this, OpenSearch was down and its fourteen tools were honestly absent.

Overview · the platform boundary

A credential-brokered MCP gateway

One endpoint in front of the facility’s services. The LLM client never holds raw credentials; the broker authenticates the user, mints per-service credentials, and audits every tool call.

Claude, Gemini or any MCP client to the mcp.af.uchicago.edu front door (oauth2-proxy, AF Keycloak OIDC), through a FastMCP aggregator and AF credential broker (FastAPI /v1) doing identity, authZ, credential brokering and audit, fanning out to rucio-mcp, ami-mcp, openmagic, panda-mcp, condor-mcp, gitlab-mcp, jupyter-control and an Nth backend with no code change

Identity, authZ, credential brokering, and audit in one place: the same boundary a facility wants in front of its services, with or without AI.

Illustrative (July 2026); current services on the previous slide · rucio-mcp · ami-mcp · atlasopenmagic · panda-mcp · condor-mcp (Bockelman, CHEP 2026).

The same thing as one architecture picture, if you want the overview. It's illustrative, from July, so the backends along the bottom aren't exactly today's list; the previous slide has the current one. The client, Claude, Gemini, whatever, holds only a bearer token. It hits the front door, mcp.af.uchicago.edu, which is oauth2-proxy in front of the AF's Keycloak. Behind that, the FastMCP aggregator and the credential broker do identity, authorization, credential brokering and audit, and fan out to the MCP servers. The broker is the only thing that ever touches raw x509 or IAM material. Adding the Nth backend is no code change. It's deployed with Flux and Helm. The point for this room: that boundary is the one you'd want in front of your services anyway, AI or not.

The first design choice

One MCP, many, or a gateway?

One server, many sites

one deploy per-VO auth in one process

One server per site

clean isolation N servers to run

Single gateway

one identity token sprawl / choke point

We run the gateway: one identity, one place to authorize and audit. Next: how we handle token sprawl and the choke point.

FastMCP composition = one auth provider per server; that constraint drives a lot of the design.

Once you have more than one MCP server, the first real decision is topology. One server that knows about every experiment is one deploy, but now per-VO auth lives in a single process. One server per site is clean isolation, and N things to run and patch. Or a single gateway in front of many sub-MCPs: users see one endpoint and one identity, and you get one place to authorize and audit every call. We went with the gateway. It has two costs you can see on the card: token sprawl, because every backend needs its own credential, and a choke point, because everything goes through it. The next few slides are how we handle those two. One practical constraint pushed us there: FastMCP composition gives you one auth provider per server, so you can't just mount ten servers with ten different auth stories into one process.

The MCP Platform: one gateway, many MCP servers

mcp-portal.af.uchicago.edu · live

AI Assistant

MCP Gateway

Auth Policy Audit Broker

Rucio

HTCondor

Jupyter

Storage

Credentials stay behind the broker

The assistant never holds a raw credential; the user's token is never forwarded.

Authorize every call

Group-based access; denials audited too.

Short-lived, scoped

Minted per service, cached at the broker, reused until expiry; the per-call copy is deleted after use.

The agent speaks intent ("get this dataset", "run on the cluster"); the facility wires the backend. Infrastructure‑as‑Config: the Nth MCP server is Helm values on Kubernetes, never a change to the server.

mcp-portal.af.uchicago.edu · the portal landing page's own animation (af-mcp-platform).

This is the platform in one animated call: the portal landing page's own diagram, reproduced in the slide. Let it run one cycle while talking. Two authorized calls play through: the assistant asks, Auth, Policy, and Audit check green, the Broker mints a credential off to the side, the call lands on Rucio or HTCondor, and the answer comes back. The third cycle is the governance story in eight seconds: "Delete my /data datasets" passes Auth, fails Policy, the gateway buzzes red, and the assistant reports it is not authorized. Three things to say over it: credentials stay behind the broker, so the assistant never holds anything raw and the user's token is never forwarded downstream. Every call is authorized against group-based permissions, and the denials get audited too. And every credential is minted short-lived for exactly one service, used once, and deleted. And the design idea underneath, which I showed at Nikhef: the agent speaks intent, "get this dataset", "run this on the cluster", and the facility wires that to Rucio, HTCondor, whatever the backend is. And it's infrastructure-as-config: adding the next MCP server is an entry in the Helm values, deployed like anything else on Kubernetes, never a change to the server.

The same call, step by step · deep-dive

Claude calls rucio-atlas_rucio_list_dataset_replicas with its personal access token

Gateway authenticates; groups and POSIX identity come from the IdP (Keycloak) on every call

Authorized once: this tool requires read_data; denials are audited too

Broker mints a JWT: aud=rucio-mcp-atlas (one service), exp = 10 min; it says who you are, not what you may do

rucio-mcp verifies that JWT, then redeems the VOMS proxy directly from the broker's vault

One Rucio call as the user; proxy deleted; one audit record, traced and metered

The user's token is never forwarded downstream; proxies and tickets never pass through the gateway.

Identity is ambient, never an argument

@mcp.tool()
async def create_jupyter_server(
    image: str,
    name: str | None = None,
    cpu_cores: int = 2,
    memory_gb: int = 8,
    ...
) -> str:
    """...The server is always owned by the caller
    (from their verified broker identity) --
    there is no owner argument. ..."""
    claims = await get_broker_claims(ctx, verifier)
    ... create_notebook(owner=claims.unixname, ...)

No tool takes a username. The schema cannot express "read bob's files"
One rule, different ways to implement, e.g.: k8s owner labels · kernel uid/gid · the VOMS nickname in your proxy
Who you are comes from the identity provider (e.g. Keycloak) on every call, not from what a token claims

The principle

Do not expose dangerous parameters; derive them from verified identity.

af-jupyterlab-mcp tools/jupyterlab.py, abridged.

Every agent-facing schema in the fleet has the same absence: there is no owner or username parameter anywhere. The user is whoever the verified token says, full stop. An LLM argument can be hallucinated or injected; a verified claim can't. One rule, implemented differently per backend: Kubernetes owner labels for notebooks, an actual setuid subprocess for the filesystem so the kernel does the enforcement, and the VOMS nickname inside your redeemed grid proxy for Rucio and AMI. And one sharpening: even the token doesn't carry authority. Who you are, your groups and your unix identity, is looked up from the identity provider, Keycloak in my case, on every call, rather than trusted from token contents. That's what makes removing someone from a group take effect in seconds instead of whenever their token expires.

Keep credentials out of the MCP

Lifetime == one call

e.g. a VOMS proxy or Kerberos ticket: fetch it → 0600 temp file → one backend call → delete it. The MCP server keeps nothing.

Exchange, never forward

The user's token never goes downstream (the spec forbids it). Each hop gets a fresh short-lived JWT, valid for one service, expiring in 10 minutes.

One service per secret

e.g. only condor-token-service touches the Condor pool key; the broker is its only allowed caller, and it returns short-lived IDTokens.

bad: a token service deciding permissions

if "atlas-users" in token["groups"]:
    mint_ticket(user)
# a second copy of the access rules,
# trusting whatever the token claims

good: it only verifies who is calling

claims = verify(token, aud="krb5-token-service")
mint_ticket(claims["sub"])
# the gateway already decided permissions;
# this token proves who, from the broker

sketch; written into all three token services' docstrings (krb5 app.py)

Never store secrets in the MCP server. Keep them in a vault (OpenBao, here) behind the broker, and mint short-lived credentials per call.

The architectural heart, and the abstract's promise: authentication without exposing user credentials. Three moves across the cards. First, credential lifetime equals tool-call lifetime: when rucio-mcp needs your grid proxy, it fetches it, writes a mode-0600 temp file, makes exactly one backend call, and deletes it, success or failure. I test the deletion. Second, exchange instead of forwarding: the user's own token never goes downstream, which the MCP spec explicitly forbids. What goes downstream is a fresh JWT the broker mints, valid for exactly one service and expiring in ten minutes. Heavy material like proxy files never passes through the gateway at all; the backend fetches it directly from the broker. The bad-versus-good pair is about where permission decisions live. Bad: the token service checks the token's groups itself. Now the access rules exist in two places, and this copy trusts whatever the token happens to claim, so a stale token keeps its access. Good: the service verifies only that this is a valid broker token addressed to it, and mints for whoever the token says. The permission check already happened once, at the gateway, against live Keycloak groups. That rule is written into all three token services' docstrings so nobody re-grows a second permission layer. And nothing secret lives in an MCP server: it's all in a vault, OpenBao in my case, behind the broker. Deep-dive below on the three custodians, including the HTCondor one.

The custodians · deep-dive

voms-token-service

The only pod that sees home dirs and Globus passphrases. Runs voms-proxy-init impersonated; privilege floor found live, five times.

krb5-token-service

Password → kinit → ticket; zero secrets at rest, passwords zeroed on every exit path. Rate limiter protects CERN's lockout counter, not itself.

condor-token-service

Runs condor_token_create next to the pool key, which "must never reach the broker": it can mint any identity.

Why three services, not one

Different privilege floors (root+caps / none / one init container). Merging them forces the union of privileges onto everything. Each is a one-question audit: "did this call come from the broker?"

Orchestrate the hardened native binary; parse artifacts (ccache bytes, PEM), never CLI prose.

Deep-dive for questions. Three services because three root secrets with three different privilege needs. voms needs root plus capabilities to read user keys, and every one of those facts was discovered in production: grid tooling checks file ownership, not just mode. krb5 needs no privileges at all and holds nothing; its rate limiter exists to protect CERN's account-lockout counter, a third party, which is my favorite example of consequence-driven design. condor sits next to the symmetric pool key that can mint tokens for any identity, which is exactly why it must never reach the broker's trust domain. Common thread: all three shell out to the battle-tested native binary and parse artifacts, ccache bytes and PEM, never the human-readable CLI output, which turned out to be locale-dependent.

Lessons learned · ↓ four more

What breaks at ten MCPs (that one never sees)

Design decisions baked into the AF MCP Platform

Name collisions: two MCPs serve a tool with the same name; fix = a prefix per service (rucio-atlas_*, fs_*)
Each service gets its own fixed OAuth audience, pinned in config, never derived from a name that can change
Listing needs credentials too: some MCPs require auth even for tools/list, so the gateway mints one just to enumerate

New tools are opt-in: when an MCP releases a tool, it stays disabled until an operator maps it at the broker
Authorization is plain RBAC: Keycloak group → permission → tool
A dead backend disappears from tools/list instead of every call failing

Adding the Nth MCP

Each MCP is added via config (Infrastructure‑as‑Config: Helm charts on Kubernetes). Sometimes it needs a token service, sometimes a bridge, never changes to the MCP itself.

af-mcp-platform issues #113, #257, #121 · 289 commits, 77 days, 1,337 tests. ↓ more scars

Everything so far was about one server at a time. At ten, a different class of problem appears, and none of it is in the MCP spec. Quick tour: name collisions, because two backends will eventually serve a tool with the same name and the failure is silent, so every service gets a prefix. OAuth audiences: each service's audience is pinned in config; I learned the hard way not to derive it from a display name, because a rename broke authentication for everything at once. Listing: some MCPs need auth just to enumerate their tools, so the gateway mints a credential for the listing itself. New tools are opt-in: when a backend ships a new tool, it stays disabled until an operator maps it to a permission at the broker. Authorization itself is nothing exotic, plain RBAC from Keycloak groups. And when a backend dies, its tools leave the list rather than erroring one call at a time. The payoff: adding the Nth MCP is configuration, infrastructure-as-code. Sometimes it needs a token service, sometimes a bridge, but you never modify the MCP itself.

Lessons learned · deep-dive

An existing API is not an agent interface

bad: the raw API dump

{"logicalDatasetName": "mc23_13p6TeV.601229...",
 "crossSection": "0.729", "crossSection_unit": "nb",
 "genFiltEff": "1.0", "kFactor": "1.0",
 "principalPhysicsGroup": "...", "prodsysStatus":
 "...", ...95 more fields, duplicate rows...}

every field, no units help, and the same for each of 50 datasets

good: markdown-friendly, LLM-friendly

**mc23_13p6TeV.601229** (PhPy8EG ttbar) · VALID
Cross-section: 729 pb (0.729 nb × 1000)
Events: 2M · k-factor: 1.0

---
**Next steps:**
- ami_get_dataset_prov for provenance
- rucio_list_dataset_replicas to locate copies

~15 curated fields, units converted in the open, follow-up tools named

The docstrings keep getting fixed against each tool's actual behavior, to teach the LLM how to use it correctly. The LLM reproduces examples verbatim, so a doc bug is a behavior bug.

Sketch abridged from real ami-mcp output; the curation commit is 901f339 "optimize MCP tools for LLM usefulness".

This is the first lesson chronologically, shown as bad versus good. ami-mcp wraps a twenty-year-old metadata service. The naive wrapper worked and passed its mocked tests, and the output was unusable: a hundred raw fields per dataset, cross-sections in nanobarn with no hint, duplicate rows, multiplied by every dataset in the answer. The fix commit is literally titled "optimize MCP tools for LLM usefulness": pick the fifteen fields an analyst would pick, convert units visibly so the model can't silently mix nb and pb, deduplicate, and end every result with named follow-up tools so the model knows where to go next. And the callout: the docstrings keep getting fixed against the tools' real behavior. AMI is the one server where I teach the model a query language, so the documentation is executable; when a documented example was wrong, the model reproduced the wrong syntax verbatim, and I fixed the docs the way you fix code, against the live service.

Lessons learned · deep-dive

Errors are the agent's control flow

bad: what the agent sees by default

Traceback (most recent call last):
  File ".../replicaclient.py", line 158, ...
rucio.common.exception.DataIdentifierNotFound:
  Data identifier 'user.gstark:mydata'
  not found

a dead end: nothing tells the agent what to try next

good: what rucio-mcp returns instead

Error: Data identifier
'user.gstark:mydata' not found

**Recovery:** The DID does not exist in
Rucio. Use `rucio_list_dids` with a
wildcard pattern to search for it.

returned to the agent, not raised into server logs; the next tool is named

Empty results carry the same routing hints. The error categories also feed my monitoring, so I can see which tools confuse the agent.

MCP results have an isError flag so clients can tell failure from success. I didn't set it at first; do both.

The classifier behind the right panel: rucio-mcp tools/_helpers.py (classify_error).

The consumer of an MCP is probabilistic: it course-corrects from what it reads, so an error message is not diagnostics, it's the next instruction. Left panel: the default, a raw exception. Best case the agent guesses; worst case it retries the same call. Right panel: what rucio-mcp returns for the same failure. The error comes back to the agent as a normal result rather than being raised into the server logs, and the recovery line names the exact next tool, so most failures heal themselves inside the loop. Empty results, which are the most common silent failure, get the same treatment. Two housekeeping notes: the same error categories feed my monitoring, so I can literally chart which tools confuse the agent; and MCP has a flag on results, isError, that lets clients and gateways distinguish failure from success. I skipped it at first and my audit log counted failing calls as successes for a while. Return the friendly prose AND set the flag.

Lessons learned · deep-dive

Context is a bounded resource

bad: rucio_list_files("mc23..."), unbounded

mc23_13p6TeV:file_000001.root
mc23_13p6TeV:file_000002.root
mc23_13p6TeV:file_000003.root
... 499,997 more lines ...

one call floods the LLM's entire context window, and the server downloaded all 500k names to do it

good: capped, and the output says so

mc23_13p6TeV:file_000001.root
... 49 more ...

---
Showing 50 results (offset=0).
Pass `offset=50` to see more.

a hard server-side limit, announced in the output, with the way to continue spelled out

And cap the server's work too: rucio-mcp once fetched that whole 500k-file container upstream just to serve one 50-row page. Paginate the upstream iterator, not just the output.

The real footer text and iterator fix: rucio-mcp tools/_helpers.py (paginate_iter).

Same bad-versus-good format. Left: what a naive file-listing tool does with a big container: half a million lines into the LLM's context window, which destroys the conversation, and the server fetched all of it from Rucio to boot. Right: what rucio-mcp actually returns. A hard limit of fifty, enforced server side, and the two lines at the bottom matter most: the output says it truncated and says exactly how to get the next page. Truncation the LLM can't see silently distorts what it believes; truncation it can see is just the next tool call. The callout is the second half of the lesson, and I learned it the embarrassing way: for a while the output was perfectly paginated while the server still downloaded the entire container upstream to serve each page. An external contributor caught it. Cap what the backend does, not just what the LLM reads. Same principle at the gateway: it writes its own short instructions instead of forwarding every server's, and only lists the tools you're allowed to call.

Tried X, discovered Y, ended with Z · backup

Tried	Discovered	Ended with
Rust CLI wrapper	parsing text tables is fragile	Python + native client
Returning raw API rows	noise swamped the LLM	curated fields, converted units, next steps
Docstrings written from memory	the LLM reproduced broken examples verbatim	docstrings tested against the live service
Stateful MCP sessions, 2 replicas	cross-replica "Session terminated"	stateless HTTP (now the spec default)
Dynamic Client Registration	you maintain a database of client ids, and a restart orphans them all	CIMD (later adopted by the spec)
Groups inside JWT claims	revocation waits for token expiry	live IdP lookup, a real kill switch

Full catalogue with commit hashes: mcp-design-talk.md §7.2 (about 40 of these).

Backup, and honestly my favorite slide if there's time. Six of the roughly forty tried-discovered-ended-with lessons in the research document. The top three are interface lessons: wrap the client not the CLI, curate what the model reads, and treat docstrings as code that gets tested against the live service. The bottom three are infrastructure lessons, and two of them, statelessness and client-id metadata documents, I adopted under production pressure months before the MCP spec standardized the same choices. Dynamic Client Registration deserves its line: besides the orphaning, it quietly makes you the owner of a database of client ids, which can grow without bound. Early production pain turned out to be a decent predictor of where the protocol was heading.

Also on the facility's plate · deep-dives linked

Other technical considerations

Client identity (CIMD)

The client_id is a URL the server fetches: no registration database, and the facility picks which domains it trusts. Watch for SSRF.

why · mechanics · live trace

Shell access (ssh-oidc)

Swap permanent SSH keys for short-lived federated credentials. An agent with a shell inherits minutes, not a key that lives forever.

overview · PAM vs. SSH cert

Sandboxing

Pods, then bubblewrap off Kubernetes, then eBPF policy, then human-gated privilege. Harnesses can advise, but can't reliably hard-deny.

overview · cgroups vs. eBPF

Deep-dives from the Nikhef colloquium, 2026‑07‑01 · ssh-oidc (KIT / EOSC-synergy) · ebpf.io

Three more things that land on the facility once agents are users, each with a deep-dive I gave at Nikhef, linked here rather than repeated. Client identity: with CIMD the client_id is an HTTPS URL to a metadata document the client hosts, and the server fetches it. No registration endpoint to abuse and no database of client ids to garbage-collect, which is why the platform moved off dynamic registration; the MCP spec has since deprecated DCR in favor of it. The cost is that the server makes outbound fetches, so you block private IPs and cap size and time. Shell access: ssh-oidc from KIT and EOSC-synergy replaces the permanent private key with a short-lived, revocable federated credential. Anything with your ssh-agent socket, including a coding agent, can sign as you forever; with this, the blast radius is minutes. Sandboxing: pods by default on Kubernetes, bubblewrap or similar on bare login nodes, eBPF class policy for fine-grained file and syscall rules, and privilege that's earned and human gated. The honest gap: today a harness can advise but can't reliably hard-deny a command at the user level. Real blocking needs a policy engine on the node.

Same context layer, for operators

AI at every layer of the facility

Analyst-facing

Find data, generate code, submit, recover, fit. Low autonomy, high convenience.

Operations

Cluster health reports, hold-reason triage, drafted user emails, human-approved.

Sysadmin-facing

Config/runbook copilots in the privileged lane; trust earned, actions gated.

Daily HTCondor cluster report posted to Slack by the AI monitor agent

Morning report: 448 held of 14,671 (3.1%)

Drafted held-job email posted to Slack, approved by a human, then sent by the agent

Drafted fix → human types “approve” → sent

Same context layer, different autonomy and blast radius at each level. The facility decides where the human gate sits.

One slide for the operators in the room, because the same idea applies to us, not just to analysts. Think of it as levels of autonomy. Analyst-facing is low autonomy, high convenience. Operations is the middle, and this is running today: a scheduled agent scans HTCondor every morning and posts a Slack report, the state summary, a hold-reason breakdown, top users, jobs held more than seven days, and recommended actions. That morning it was 448 held out of about fourteen and a half thousand. When a user's jobs are stuck, it drafts an email with a specific fix. The real case here was an out-of-memory hold: raise request_memory based on the usage it actually observed against the limit, and write the big outputs to /data. A human replies "approve" in Slack, and only then does it send. Never autonomous. And honestly, there's no HTCondor MCP in this one, just skills around the condor CLI. Sysadmin-facing, the privileged lane, is where trust has to be earned and actions gated. The facility decides where the human gate sits at each level.

live · the UChicago AF

Demo

$ claude mcp add --transport http atlas-af https://mcp.af.uchicago.edu/mcp

One prompt: submit an HTCondor job · start a Jupyter server · whoami · service catalog · AMI · Rucio · ATLAS OpenSearch · watch the job finish

mcp-portal.af.uchicago.edu · the demo prompt

Switch to the terminal and the portal here. Keep it to about five minutes. That one line is the whole client-side setup: add the facility as an MCP server, log in with the facility identity, and the assistant sees only the tools I'm allowed to use. Then one prompt, the gist linked on the slide. It starts the slow things first: submit a ten-second sleep job to HTCondor, and start a small Jupyter server. While those run, the read-only tour: whoami, with my IdP groups and effective permissions; the live service catalog; an AMI cross-section and dataset lookup; where Rucio has the replicas; and a search of the ATLAS OpenSearch graph through the Kerberos bridge. Then close the loop: watch the condor job finish, and query which slot it ran on, the queue wait, runtime and exit code, and check the Jupyter server's status. The point to say out loud: every one of those calls was me, with a credential minted for one service and thrown away, and no token in the agent. If the network fails me, the trace slide and its step-by-step deep-dive are the fallback.

Interfaces that travel

Across LLM providers

Design for the least-capable LLM client: e.g. Claude Desktop had no OAuth discovery, so I added personal access tokens; gemini-cli choked on an OAuth metadata fallback, so I deleted it
Standard OAuth, standard transports, no provider-specific hooks

Across experiments

rucio-mcp already serves ATLAS, CMS, DUNE, ESCAPE: the domain layer is VO-agnostic
Transplant the shape (broker + custodians + config-only services), not the CERN-specific code

Interoperable tool interfaces are likely to matter more than LLM-specific integrations.

↓ how my choices scored against the evolving MCP spec

The abstract's closing claim, made concrete. Across providers: the discipline is designing for the least-capable client. I had a genuinely cleaner design for routing based on OAuth scopes, fully written up, and I shelved it because only one client on the market supports the feature; the write-up is committed next to the code. Same story with the OAuth metadata fallback I deleted after a different client choked on it. Standard auth, standard transports, nothing provider-specific: that's what "works across providers" costs. Across experiments: rucio-mcp is already multi-VO because Rucio is, and the whole facility shape, broker, custodians, config-only services, transplants to any lab. What doesn't transplant is the CERN-specific plumbing, and that's fine: it's all behind one provider interface and per-service config. If you take one sentence home, take the abstract's: interoperable tool interfaces will matter more than model-specific integrations.

Scored against the MCP spec · backup

My choice	MCP 2.0 spec (2026-07-28)	Verdict
Stateless HTTP, state as explicit handles	protocol sessions removed	compliant / anticipated it
CIMD instead of Dynamic Client Registration	DCR deprecated for CIMD	compliant / anticipated it
Tools only; no sampling, roots, logging	all three deprecated	compliant / anticipated it
Exchange tokens, audience-validate everywhere	token passthrough forbidden (MUST)	compliant
`outputSchema` on tools that return data	structured content recommended	compliant; fixed since PyHEP.dev
Read-only / destructive tool annotations	annotations recommended	compliant; fixed since PyHEP.dev

Spec: modelcontextprotocol.io/specification/2026-07-28 · full comparison in mcp-design-talk.md §6.3 (outdated, pre-fixes).

When is an MCP server appropriate?

Build one when

a library or service scientists already use
assistants need it repeatedly, across clients
you want to control what context the agent gets injected with
you can bound outputs and make errors recoverable

Skip it when

a one-off script would do
the backend's API is still churning under you
the "tool" would hand the LLM a credential

From “Model Context Protocol for Scientific Software”, PyHEP.dev 2026 · the full checklist

Open questions for this room

What we don’t know yet

How should facilities expose context?

One MCP per service or a gateway delegating identity? Where does your audit live?

How do we share context between facilities?

Shipped with the facility, or a shared commons? Single-source vs forks; eval/lint for orthogonality; who keeps it true?

How do we evaluate, and secure, an autonomous operator?

Human gates, k8s vs bubblewrap/eBPF isolation, credential brokering so agents never hold raw secrets, a trust ladder with an eval-harness promotion gate.

What makes an environment truly portable?

pixi-native batch as the default? Supply-chain policy for conda-forge?

Who pays for inference, and is that fair?

Hosted frontier vs self-hosted open weights on facility GPUs; equity as policy.

What becomes standardized, across sciences?

A commons spanning facilities, experiments, and sciences: where’s the facility/experiment line?

My bet: the model and harness are the easy part. Context, identity, trust, and equity are the facility’s homework.

Generic LLM → facility-aware collaborator

Swap the model and harness freely. The facility’s context, identity, and credentials are what make it useful and safe, and that layer ports to the next facility.

Still hard: more brokered backends (PanDA next) · hard-deny for agents with a shell · testing against the live services, not mocks

Pointers

maniaclab/af-mcp-platform
condor-token-service · condor-mcp
rucio-mcp · ami-mcp · usatlas/marketplace

Design the interface for the model; design the architecture for the credential.

Giordon Stark · kratsg · HTCondor Workshop Autumn 2026 · ~40 “tried X, found Y” lessons: mcp-design-talk.md · slides AI-assisted by Claude (Anthropic).

a599170 · 2026-10-01

The Facility Isthe Context

From prompt to discovery

Swap the model freely; context makes it useful

Generic model

Facility-aware model

What is the Model Context Protocol?

Around an existing Python project · backup

Capabilities and knowledge

MCP: arms and hands

Skills: domain expertise

A facility-native MCP server must be…

Convenient

Secure

Auditable

Every MCP tutorial vs. the UChicago AF

The MCP layer

The credential layer

A credential-brokered MCP gateway

One MCP, many, or a gateway?

One server, many sites

One server per site

Single gateway

The MCP Platform: one gateway, many MCP servers

Credentials stay behind the broker

Authorize every call

Short-lived, scoped

The same call, step by step · deep-dive

Identity is ambient, never an argument

Keep credentials out of the MCP

Lifetime == one call

Exchange, never forward

One service per secret

The custodians · deep-dive

voms-token-service

krb5-token-service

condor-token-service

What breaks at ten MCPs (that one never sees)

An existing API is not an agent interface

Errors are the agent's control flow

Context is a bounded resource

Tried X, discovered Y, ended with Z · backup

Other technical considerations

Client identity (CIMD)

Shell access (ssh-oidc)

Sandboxing

AI at every layer of the facility

Analyst-facing

Operations

Sysadmin-facing

Demo

Interfaces that travel

Across LLM providers

Across experiments

Scored against the MCP spec · backup

When is an MCP server appropriate?

Build one when

Skip it when

What we don’t know yet

Generic LLM → facility-aware collaborator

The Facility Is
the Context