Skip to content

API Frontend Architecture

This page is under active development for v1.5 GA

The API Frontend (AF) is the unified external protocol layer introduced in v1.5. It provides MCP, A2A, and REST access to Kubernaut for operators, AI agents, and the Backstage console.

Overview

The AF sits between external clients and the Kubernaut Engine, handling:

  • Protocol translation — MCP tool calls and A2A tasks are translated into internal Kubernaut API calls
  • Authentication — OIDC/OAuth2 via JWKS validation with JWT claim extraction
  • Authorization — Kubernetes-native SAR-based tool authorization (PR #1222); fail-closed with TTL cache
  • MCP bridge — Dispatches 23 kubernaut_* MCP tools to their backends (K8s API, KA MCP, DataStorage, Prometheus) with per-tool routing. When interactive.enabled: false (#1366), 11 session-dependent tools are hidden, leaving 12 stateless tools for CRD and data operations only (13 with kubernaut_list_alerts if Prometheus is configured).
  • Streaming — Relays Server-Sent Events from KA's SSE endpoint to MCP clients

Agentic Architecture

EXTERNAL CLIENTS MCP (AI Interface) Claude, Copilot, custom agents A2A Agent Agent-to-Agent protocol Backstage Operator console OIDC / OAuth2 DEX, Keycloak API Frontend MCP Streamable HTTP · A2A JSON-RPC · OIDC Auth · RBAC KUBERNAUT ENGINE Gateway Webhook intake + dedup Remediation Orchestrator Owns RR lifecycle — creates child CRDs Kubernaut Agent LLM investigation (Go) Signal Processing Rego classification AI Analysis LLM RCA + selection Workflow Exec. Tekton / Job / Ansible Effectiveness Health scoring + drift Notification Slack / PD / Teams DataStorage PostgreSQL + Valkey AuthWebhook RAR override validation LLM Provider OpenAI / Anthropic / Vertex AI

Session Lifecycle

Session management spans two layers:

  • KA MCP layer (internal/kubernautagent/mcp/) — Owns interactive investigation state via Lease-based single-driver locking. The AF dispatches interactive tools (kubernaut_investigate, kubernaut_message, kubernaut_complete, kubernaut_cancel, kubernaut_status, kubernaut_reconnect, kubernaut_select_workflow, kubernaut_discover_workflows) to KA's MCP server.
  • AF session layer (pkg/apifrontend/session/) — Manages InvestigationSession CRDs with deferred creation: the CRD is not created when the A2A session starts, but only when kubernaut_remediate succeeds and calls MaterializeCRD(). Sessions that never produce a RemediationRequest leave no cluster footprint.

Lease-based distributed locking

The LeaseSessionManager in KA uses Kubernetes coordination/v1 Leases (prefix: kubernaut-interactive-) for single-driver guarantee (BR-INTERACTIVE-002):

  • Acquisition — On action:start, KA creates a Lease in kubernaut-system
  • TTL — Default session TTL is 30 minutes; configurable via InteractiveConfig.SessionTTL
  • Inactivity timeout — Sessions expire after a configurable period of no activity
  • Max sessions — Configurable cap; rejects new sessions when capacity is exhausted (SEC-03)
  • Orphan reclamation — On startup, KA scans for Leases whose holder identity no longer exists and reclaims them
  • Same-user reconnect — If the same user reconnects, the existing Lease is reused

Session takeover (SEC-TAKEOVER-001)

When User B connects to a session owned by User A:

  1. User A's investigation is abandoned (not completed) — prevents identity confusion
  2. The Lease holder is updated to User B
  3. An audit event is emitted recording the takeover
  4. User B starts a fresh investigation in the same session context

Jump-In session upgrade (#1390)

When an operator calls kubernaut_investigate for an RR that already has a running autonomous investigation, the KA upgrades the session to interactive in-place rather than cancelling and recreating it. This preserves the LLM context accumulated during the autonomous RCA phase.

  1. KA sets an interactiveUpgrade atomic flag on the session
  2. The running investigation goroutine sees the flag at its next InteractiveHold check and pauses for operator input
  3. The AA controller detects the IS CRD and sets Interactive=true + SetActivePhase (no cancel)
  4. Session ID and correlation ID are preserved throughout

If the autonomous session has already completed (ErrSessionTerminal), the system falls back to ForceTransitionToUserDriving to start a fresh interactive session.

Disconnect handling (DD-INTERACTIVE-002)

The SessionClosedHandler monitors MCP connection closures via the DelegatingEventStore. On disconnect, it triggers session release and reconstruction.

Graceful shutdown — SessionDrainer

When a KA pod receives SIGTERM (BR-OPS-013):

  1. The SessionDrainer notifies all connected MCP clients that the server is shutting down
  2. In-flight tool executions are given time to complete
  3. All active session Leases are released
  4. Pod terminates only after all sessions are drained

SSE Streaming

The AF streams investigation output to clients via Server-Sent Events:

  • Token-by-token output from LLM responses
  • Keepalive pings at regular intervals to maintain the connection
  • Investigation progress events (phase transitions, tool calls, results)

Integration Points

Target Protocol Purpose
Kubernaut Agent MCP JSON-RPC + HTTP/REST MCP tool proxying, SSE streaming
DataStorage HTTP/REST Workflow catalog, remediation history, audit events
LLM Provider HTTP/REST (via KA) Severity triage with configurable confidence threshold
OIDC Provider OAuth2/OIDC User authentication via JWKS
Kubernetes API SubjectAccessReview SAR-based tool authorization (verb use, group kubernaut.ai, resource tools)

LLM Provider (A2A Agent)

The AF runs its own LLM-backed agent for the A2A handler. The LLM config (agent.llm in the AF config.yaml) mirrors the KA ai.llm schema so operators use one config style across services.

Supported providers: vertex_ai (Claude on Vertex AI — requires vertexProject + vertexLocation), gemini (Gemini API direct — requires apiKeyFile or OAuth2), anthropic (Anthropic API direct — requires apiKeyFile or OAuth2). When provider is empty, the A2A handler returns HTTP 501.

Multi-provider factory — The AF uses a transport chain that resolves the provider at startup, wires TLS (including mTLS client certificates for corporate LLM gateways via tlsCertFile/tlsKeyFile — #1342), and applies an optional circuit breaker around all outbound LLM HTTP calls.

File-based API key — The apiKeyFile field replaces the former LLM_API_KEY environment variable (#1251). Mount the key as a Kubernetes Secret volume; the AF reads it at startup and trims whitespace.

OAuth2 client credentials — For auth-gated LLM gateways, configure oauth2.enabled: true with tokenURL, scopes, and a credentialsDir containing client-id and client-secret files. Token refresh is handled automatically.

InstructionProvider (#1276) — The A2A agent's system prompt is dynamically generated per-request by the InstructionProvider. This replaces the static Instruction string and injects the controller namespace, available tool names, and persona context into the LLM prompt at runtime.

KA bearer token — The kaBearerTokenFile config field provides the AF with a bearer token for authenticating to the KA MCP server (#1287). When set, the AF includes this token in the Authorization header of all KA MCP requests.

Rate limiters (#1392) — Two rate limiters protect the AF:

  • ProviderLimiter — Rate-limits JWKS endpoint fetches. When the limit is hit, cached keys are returned instead of fetching new ones.
  • LLMSemaphore — Bounds concurrent LLM requests. Requests exceeding capacity are rejected with ErrLLMCapacity (HTTP 429).

Re-invocation loop (#1392) — The StreamingExecutor re-invokes the LLM agent when a turn ends with text-only output (no tool calls), up to MaxReinvocations. This handles cases where the LLM produces reasoning text before deciding on a tool call.

JWT ClaimMappings (#1392) — CEL expressions for extracting username and groups from JWT claims (e.g., claims.email, claims.roles). Falls back to hardcoded paths (preferred_username/sub/groups) when expressions are empty, preserving backward compatibility.

See AF LLM Configuration for the full field reference.

A2A Streaming Events

A2A streaming uses TaskStatusUpdateEvent messages classified by metadata.type:

metadata.type Purpose status.message
reasoning LLM inner thoughts / investigation deltas Set (sanitized text)
status Orchestration progress (tool starts, phase transitions) Set
output Final LLM answer Set
investigation Investigation-specific events from KA Set
keepalive Proxy idle-timeout prevention Not set (metadata-only)

Keepalive events are emitted every 5 seconds during long-running KA tool executions. They carry {"type":"keepalive", "dot":"."} in metadata but no status.message, preventing them from polluting the A2A task history. Clients that only render status.message will not see keepalive dots (by design).

A2A integrators should inspect metadata.type to distinguish ephemeral events from history-worthy messages.

Health Checks

Probe Endpoint Port
Liveness GET /healthz 8081 (plain HTTP)
Readiness GET /readyz 8081 (plain HTTP)
Metrics /metrics 9090 (plain HTTP)
API HTTPS 8443