Architecture overview
What runs where, and the path one request takes — from someone's browser, through sign-in and the agent, to a model and back.
Each customer gets their own deployment, built from the same code and infrastructure templates and run inside their own AWS account. One deployment serves one organisationOrganisationTổ chứcYour company's space in the portal. Each deployment runs one organisation, inside your own AWS account.Open in glossary, shares nothing with other customers, and sends every model call through one gateway.
Key rules
- CloudFront is the only public entrance. Everything else sits behind it.
- Every model call goes through LiteLLM. Nothing calls Bedrock directly.
- Identity and policy travel with each request, and the agent can’t widen them.
- One agent host runs every agent. Creating an agent deploys nothing.
- Admins work in the portal only. The LiteLLM and Keycloak consoles are ECV break-glass.
The system at a glance
Hover or focus any box to see what it does.
The browser reaches the portal through CloudFront. The static web app comes from a private bucket; requests under /api go to the portal API, and requests under /idp go to Keycloak. From the portal API, a chat message travels to the agent runtime, which calls models through LiteLLMLiteLLM (model gateway)LiteLLM (cổng mô hình)The gateway every model call passes through. It routes to Bedrock, applies each department's key and budget, and records the cost of every call.Open in glossary, which calls BedrockAmazon BedrockAmazon BedrockAWS's model service, running in your account's region. The portal's models and embeddings come from here, always through LiteLLM.Open in glossary.
Supporting services sit alongside that path:
| Part | What it does |
|---|---|
| LiteLLM | Also the MCP gateway for applications (connected apps) |
| Redis | Portal API sessions, the chat stream buffer, caching |
| Secrets Manager and KMS | Credentials and encryption |
| CloudWatch and S3 | Metrics and alarms |
| S3 | Files, including the web app’s private bucket |
Infrastructure is described in the customer account’s own Terragrunt repository and applied by that account’s owner; the portal’s own repository provisions nothing. A new customer is a deployment that meets the deployment requirements, plus seed configuration — identity provider, branding, model catalog and budgets — with no code change.
Layers
| # | Layer | Does | Built with |
|---|---|---|---|
| 1 | Web app | Chat, workspace and the Admin console | React, Vite, TypeScript |
| 2 | Portal API | Access checks, data, the admin facade, relaying events to the browser | NestJS, as a backend-for-frontend (BFF) |
| 3 | Agent | The reasoning loop, tool orchestration, memory | Strands Agents TypeScript SDK, in the agent runtime on the shared stateless host |
| 4 | Model gateway | Routing, virtual keys, quota, cost attribution | LiteLLM in front of Bedrock |
| 5 | Tool gateway | Application discovery, the authentication boundary, invocation policy | LiteLLM as the MCP gateway, behind a portal adapter |
| 6 | Memory and knowledge | Ingest, hybrid search over memory and knowledge, citations | pgvector in the portal’s Postgres, embeddings through LiteLLM |
| 7 | Identity | Single sign-on, SCIM provisioning, the organisation claim | Keycloak brokering Entra ID; the portal is the SCIM server |
| 8 | Governance | Model access, budgets, audit | LiteLLM policy, with the audit trail in the portal’s Postgres |
| 9 | Infrastructure | Compute, network, encryption, backup | CloudFront, a stateless host, a data host, S3, the account’s secret store |
Customer admins use the portal only. The portal calls the LiteLLM and Keycloak admin APIs on their behalf; the consoles of those two services are kept for ECV break-glass.
The path of one message
-
Sign in
Keycloak signs the person in, brokering your Entra ID single sign-on and MFA. The portal API completes the OIDC code exchange.
-
Hold the session
The portal API keeps the tokens; the browser only ever holds a session cookie. The token carries the account and its realm roles.
-
Check who and what
The portal API reads the person’s seat from the portal database — their department, groups, custom roles and capabilities — fresh on every request.
-
Run the agent
The portal API invokes the agent runtime with the agent’s configuration and the person’s identity and resolved policy. When the agent calls back for memory, knowledge, web or artifacts, each callback carries a shared secret.
-
Call models
Each model call goes to LiteLLM with the department’s virtual keyVirtual keyKhoá ảoA gateway key issued per department, so each model call is limited and billed to the right department. People never handle it.Open in glossary, then to Bedrock. One turn can make many calls.
-
Stream back
Progress and the answer stream back through the portal API as server-sent events, replayed from a Redis Streams buffer so a refreshed page picks up where it left off. See Turn lifecycle.
-
Record
Every model call leaves one audit row, with its cost attributed to the department.
Invariants
| Invariant | Effect |
|---|---|
| Every model call goes through LiteLLM; none reaches Bedrock directly | Quota, cost attribution and model access are enforced in one place |
| The person’s verified identity and resolved policy propagate into the agent, which cannot widen them | Department membership is direct user data in the portal database, read on every request, so a reassignment takes effect on the next one |
What leaves your account
Most traffic stays inside your AWS account. These are the exceptions:
| Traffic | When |
|---|---|
| Sign-in | Keycloak talks to your Entra ID tenant |
| SharePoint sync | The portal reads the SharePoint folders you connect to knowledge bases through Microsoft Graph |
| Connected apps | An application served from outside your account receives the calls a person allows |
| Web search and research | Only if your organisation turns these tools on |
| Pasted provider keys | Only when a department or person adds a key for a provider outside Bedrock |
Why it works this way
Why one deployment per customer?
Your data, keys and audit trail live in an account you control, under your own AWS policies. Every deployment is the same code and the same infrastructure templates, so a new customer is configuration — identity provider, branding, models, budgets — not a code change.
Why route every model call through LiteLLM?
It is the one place where budgets, rate limits, model access and cost attribution are enforced. Every call carries a department key, so nothing — not the app, not an agent — reaches Bedrock without being counted and capped.
Why one shared agent host instead of one per agent?
An agent is configuration — instructions, tools, knowledge and a model — stored in the portal and loaded each time it runs. One host means creating or editing an agent never deploys anything, and every agent gets the same security checks and updates at once.
Under the hood
Local development stack
Local development runs Postgres, Redis, RustFS (in place of S3), Keycloak and LiteLLM from Docker Compose.