I wanted a reason to put an AI agent in front of a real Kubernetes cluster and watch what happens when two different people ask it to fix the same thing. Diagnosis is easy to demo. Remediation is where it gets uncomfortable. Once the agent can patch a Service, “who clicked send?” matters a lot more than whether the chat UI has a nice login page.

So I built a small demo on Akamai LKE with kagent 0.9.12 and Keycloak. One broken Service. Two users. Alice can remediate. Bob cannot. The repo is github.com/sprider/kagent-lke-identity-agent.

Why this bothered me

A lot of Kubernetes + AI demos stop at “here’s a summary of your events.” That is useful. It is also mostly harmless if the wrong person runs it.

Giving the model k8s_patch_resource is a different decision. I kept running into setups where SSO proved someone was logged in, then every agent behind that session could call the same privileged tools. Alice and Bob both looked fine in the browser. Only one of them should be able to change the cluster.

I wanted to see that gap on purpose, not discover it later in a worse environment.

What I used from kagent

kagent lets you declare agents in YAML: system message, model config, and a list of tools. I put Keycloak in front of the UI through oauth2-proxy so Alice and Bob are real OIDC users. The Kubernetes tools come from the MCP tool server: get, describe, events, and for one agent, patch.

One thing I got wrong in my head at first: I assumed the agent pod’s ServiceAccount would gate those MCP calls. In 0.9.12 they run as a shared kagent-tools account, which is cluster-admin by default. So this demo does not claim OBO tokens or per-user RBAC are doing the enforcement. What you can actually show today is simpler. Alice’s remediator gets k8s_patch_resource. Bob’s intern remediator does not.

How the demo is shaped

Keycloak login up front, observer for both users, then a fork: Alice’s remediator can patch, Bob’s cannot.

The incident is deliberately dull. In namespace demo, frontend-svc selects app=frontend-BROKEN. The pods are labeled app=frontend. Endpoints drop to zero.

Selector mismatch. Zero endpoints. That is the whole outage.

There are three agents. observer-agent is read-only. remediator-agent is the Alice path and includes patch. remediator-agent-intern is the Bob path and does not.

Walking through as Alice

make ui, open the SSO page, sign in as Alice.

Everything goes through oauth2-proxy first.

Alice gets a normal Keycloak session.

I disabled the chart’s built-in agents so the UI only shows these three.

I ask observer-agent to look at frontend-svc in demo. It finds the bad selector and suggests a patch without applying it.

Observer stays read-only.

Then I switch to remediator-agent and ask it to set the selector to app=frontend. That agent has the patch tool, so it works.

Alice’s remediator actually applies the fix.

Endpoints come back once the selector matches again.

Then as Bob

I reset the failure, log out, and sign in as Bob.

make break

Back to zero endpoints.

Bob also authenticates. That part is not the difference.

I open remediator-agent-intern and send the same fix prompt. No patch tool on this agent. It refuses.

Same ask, different tool list.

Bob did not fail login. He was handed a remediator that cannot call the dangerous tool. That is the part I wanted to be able to show people.

What I’d do differently next time

I would write Bob’s refusal case before polishing Alice’s happy path. The successful patch is easy to get excited about. The refusal is the proof.

I would also stop assuming “OIDC in front of the UI” means the cluster tools are identity-aware. Login tells you who is in the browser. The tool list on the agent tells you what that session can do in this version of kagent. Those are different questions.

And I would read how the MCP tools authenticate before designing RBAC around agent pods. That assumption cost me a round of debugging.

If you want to try it

You will need a Linode token, an OpenAI key, Terraform, kubectl, helm, jq, and openssl. It spins up three g6-standard-4 nodes, so destroy it when you are done.

git clone https://github.com/sprider/kagent-lke-identity-agent.git
cd kagent-lke-identity-agent

export LINODE_TOKEN="your-linode-api-token"
export KEYCLOAK_ADMIN_PASSWORD="choose-a-strong-password"
export OPENAI_API_KEY="your-openai-api-key"

make up

Then:

make ui
make credentials

Defaults are alice / alice123 and bob / bob123. Run Alice through observer and remediator, make break, then try Bob on the intern remediator.

make down

More detail is in the README.

If any of this resonates and you’re wiring agents into clusters too, find me on LinkedIn.