Model catalog — AI Portal Docs

The catalog is one table with one row per model. The portal owns that truth, and the LiteLLM gateway is reconciled to serve it. Which model does each of the portal’s own tasks is a separate answer: two model assignments on the organisation.

Key rules

The portal is the source of truth. The gateway is reconciled to the catalog; nobody publishes it by hand.
A model has no approval state. Ticking one in the picker catalogues it and leaves it serving.
Nothing is deleted. Withdrawing switches a model off, because past audit rows and invoices name it.
A new chat model reaches every department at once. Departments block models on their own screen.
A key pasted for another provider (BYOK) waits for approval and serves only its owner.

What a row holds

Field	Meaning
Kind	Text or vectors
Capabilities	What the model accepts or does on the way
Provider	Which upstream the row reaches
Vendor	Who made the model
Served name	The name the gateway serves it under
Upstream identifier	The identifier the gateway invokes
Prices	Copied from the gateway’s rate card
Active, served	What the admin asked for, and what the gateway answered

The gateway writes the provider’s prefix in front of the upstream identifier: bedrock/, gemini/, openai/ or anthropic/. Bedrock hosts many vendors, so “Bedrock” is how a model is served; who made it is the vendor, read from the provider when the model is added.

Kinds, capabilities and assignments

Term	Meaning	Where the answer comes from
Kind	What the model produces: text or vectors. It never changes and filters every picker	The provider, which publishes which models emit vectors
Capabilities	What the model accepts or does on the way: reads pictures, calls tools, reasons at depth. A set on the row, so a new one costs one value rather than a column	The provider first: Bedrock publishes image input, and tool use runs through the conversation API. Reasoning depth and prompt caching come from a table the portal keeps, matched on the identifier
Model assignment	What this organisation uses a model for. Two, each naming at most one model	The organisation

A new output is a kind; a new input is a capability. Provider and portal answers are merged when the model is added and written onto the row, so a later release cannot change a model already in use, and an admin can correct a wrong answer. A model the portal’s table has never heard of is added with the provider’s answer alone.

The two model assignments

The Admin console calls these the portal’s own jobs — not to be confused with background jobs on the work queue.

Assignment	Model work	When it names no model
Document search (search model)	Makes documents and memories searchable	Nothing is searchable, except files with semantic search off (Keyword only)
Background tasks (background model)	Describes the pictures on a page at Text + figures, writes a page out at Full rewrite, and writes thinking summaries	A file asking for a model reading has those pages refused, with a reason naming the assignment. OCR still reads every page a text layer cannot serve

Chat is not an assignment. Chat offers every active text model minus what the person’s department blocks, because an operator catalogues several so an agent can be moved between them.
An empty assignment is a real state. An organisation with no embedding model assigned indexes nothing until one is.
Reading pictures needs an image-reading model. A page is read by a model only when its file asks for Text + figures or Full rewrite. The portal checks the catalog and refuses those pages with a reason when it can’t. The capability isn’t required of the assignment itself, because an organisation with no scans never needs it.
The embedding width is fixed. Every vector column has a fixed width, so a replacement embedding model must answer at the same number of dimensions, or the knowledge base is re-embedded.

Reconciliation

The picker lists Bedrock models

The admin ticks any number. Rows are written, active.
Routes are pushed

Switching a model on pushes that one route to the gateway.
Rates are copied

The gateway’s published rates are copied onto the rows.
A sweep keeps it in step

A sweep runs on its own: it pushes routes again, asks the gateway what it serves and which routes answer an upstream probe, and writes both answers onto the rows.

There is no publish button and no publish route. A push that failed while the gateway restarted is retried by the next sweep. Reconciling derives what to push from the rows and is idempotent, so there is no queue, and a catalog that is already correct costs one read and no writes.

Replacing the organisation’s credential pushes every route it holds, because each gateway deployment keeps its own copy of the key and the gateway never returns a stored key for comparison.

Catalog screens read the columns the sweep wrote, so a screen costs one query and waits on neither the gateway nor Bedrock. Both gateway reads answer for the whole gateway, so one pass covers the whole catalog.

Status	Meaning
`active`	What the admin asked for
`served`	What the gateway answered at the last sweep. It differs from `active` while a push catches up, and stays apart when something is misconfigured
`healthy`	Served, priced, and answering an upstream probe at the last sweep
`observed at`	When the sweep last looked. Empty for the first seconds of a model’s life, which a screen shows as syncing rather than as a fault

Adding and withdrawing

Adding. Picking a model from what the organisation’s credential can reach catalogues it and leaves it serving, in one request.
Withdrawing is the model’s own switch. A model switched off leaves every picker, and a turn that names it is refused at the gateway.
Department blocks survive. Blocks naming a withdrawn model are left alone, so switching it back on restores what it had.
Nothing is deleted, because last month’s audit rows and invoice name the model.
New models reach everyone. A department reaches every chat model until it blocks one. Blocking is set on the department’s own screen — see Departments.
Assigned models stay on. A model holding an assignment cannot be switched off until the assignment points elsewhere. An assignment may name only a catalogued model whose kind fits and which is on.
The catalog decides. The model an assignment names is read from the organisation’s catalog, never from the deployment’s environment.
Embedding models are priced and switched off like any other, cannot be blocked by a department, and never appear in a chat picker.

Prices and prompt caching

The gateway holds the vendor’s rate card and bills, so reconciliation copies its published rates onto the row. A model is created with no prices and is unusable until that copy — the moment it becomes servable. What each rate pays for, and why a cached prompt needs four rates, is in Audit and cost.

Prompt-cache checkpoints are configured on the served model. The agent talks to the gateway through an OpenAI-style chat adapter that does not forward cache instructions, so the gateway injects checkpoints on the system prompt and the last message, and later calls reuse a matching prefix.

A route gets checkpoints only when the catalog holds both answers about it:

The PROMPT_CACHE capability — the provider caches prompts on this model. The portal’s table sets it for the Claude models Bedrock lists as generally available for explicit caching; an admin can set or clear it on any model. Claude 3.5 Sonnet v2 is left out because Bedrock lists it as a preview. A published cache price is not this answer: LiteLLM’s catalogue carries cache rates for routes Bedrock refuses cache controls on.
Both cache rates — an audit row refuses to price cache tokens it has no rate for, and an unbillable call is a failed one. Rates are mirrored from the gateway after the first push, so a newly catalogued model starts without checkpoints and gains them on the next sweep.

Rule	Detail
Request-level opt-in	None. The gateway strips any caching switch from a request body as an untrusted control field and reads it only from key metadata
Request-level opt-out	A caller with nothing to reuse sends an empty injection list, which the gateway merges over the route’s own
Refused checkpoints	A call Bedrock refuses for carrying cache controls fails like any other refusal. The fix is to clear `PROMPT_CACHE` on that model; the next sweep removes its checkpoints
Older or separately configured routes	May still have checkpoints. The deployed route says whether caching is requested; its cache-read and cache-write tokens say whether it happened
Document processing	Can request caching on individual content blocks, independently of the agent’s gateway configuration
Page reading	Declines checkpoints outright. Each call carries its own page image ahead of its text, so no prefix repeats, and the shared instruction is 124 tokens where the provider’s smallest cacheable prefix is 512. Over 270 calls before the opt-out: 836,577 tokens written to the cache, none read back, at a write’s price

Tool argument streaming

Claude writes a tool call’s arguments as it goes, but Bedrock holds each argument back until the value is complete unless the request asks for Anthropic’s fine-grained tool streaming. A file the agent writes is one long string, so without it the chat’s progress count sits at the call’s first characters and then jumps to the whole file at once.

The TOOL_STREAMING capability says a model accepts the flag. The portal’s table sets it for Claude 3.7 Sonnet and the Claude 4 and 5 families; an admin can set or clear it on any model. Claude 3 Haiku refuses the flag, and Bedrock refuses the parameter itself for any model other than Claude.

Rule	Detail
Reconciliation	Sets the fine-grained tool streaming flag on a route holding `TOOL_STREAMING`, and leaves the parameter out everywhere else
Dropping it	Replaces the route rather than updating it. An update merges parameters and cannot remove one, and an empty list is still sent, which a model other than Claude refuses
Unfinished arguments	Arguments that stop before they are valid JSON, because the output limit fell, come back to the model as a failed call

Bring your own key (BYOK)

A provider outside Bedrock is reached with a key its owner supplies — a department or one person. People manage their own in Settings › Model keys; admins in Models.

One credential brings its own catalog rows. Two departments that each paste a Gemini key hold two rows, two route names and two gateway deployments, each carrying its own key. That keeps public route names unique, and leaves reconciliation, the serving sweep, the price mirror, the audit row and department blocks working on rows as they always do.

PENDING — the key is pasted and models are ticked. No route is served until somebody holding model_catalog.manage decides.
Approved — the routes are served, to the owner and nobody else.
Rotated — the key is replaced under the same reference and the row returns to PENDING, because a replaced key is a new key.
Withdrawn — the routes are taken away and the catalog rows are kept, so approving again restores what the credential had.

Rule	Detail
Owner	Exactly one department or one person — also who the rows are offered to, and nobody else
The key itself	Sealed by the portal and referenced, never stored on the row and never read back to a screen
Cataloguing	The provider is asked what the pasted key reaches, and models are ticked from that list. That is the only real test of the key: reading back a sealed secret proves nothing about whether the provider accepts it. Gemini and Anthropic publish a display name and both token limits; OpenAI publishes identifiers alone, so its models are named after their identifier and take the catalog’s default limits
Kind	Always text. What embeds documents is one assignment for the whole organisation, which a key owned by one department cannot answer for everybody
Prices	Mirrored from the gateway like any other row, so a model the gateway publishes no rates for stays unpriced and therefore unusable

Spend on these rows is outside the department’s budget pool — see Budgets and policy.

Credentials

The organisation holds one credential per purpose: Bedrock, Firecrawl, Tavily and SharePoint. Every in-account model is served with the Bedrock credential unless its row names a pasted key.

Rule	Detail
Reference, not secret	Either a secret the portal seals, or the full ARN of a Secrets Manager secret the AWS account owns. Resolved at the moment of the call and never shown on a screen
Rotation	Replace the reference in place
Disabling	Stops every call resolving through it; reconciliation reports each model it can no longer provision
Host IAM role	A Bedrock credential may hold no reference at all. The backend then signs catalogue reads and connection tests with SigV4 from its own credential chain, and supplies its temporary access key, secret and session token to LiteLLM. Offered by default; the deployment can turn it off
Switching method	Replaces every gateway deployment the organisation holds. An update cannot remove a stored key, so each deployment is marked with the method it used, and one whose mark differs is replaced. A deployment from before the mark counts as carrying a key

With the host IAM role, the AWS SDK caches and refreshes the backend’s role credentials. Each sweep compares an opaque session version — a hash — and updates routes when the session changes. Resolved credentials stay out of reconciliation reports and API responses. The gateway holds the temporary session until the next update, is trusted with it, and inherits the backend role’s IAM permissions for its lifetime.

Only the backend must reach instance metadata. The gateway uses the supplied session for both chat and embeddings, so a separate gateway host can keep its existing metadata configuration. Credentials without a session token or a valid expiry are refused. LiteLLM’s Bedrock authentication supports these AWS parameters, and the AWS SDK credential chain resolves EC2 instance profiles.

EC2 role permissions

The role attached to the backend needs catalogue reads, availability reads and model invocation. This policy keeps invocation to the Claude, Cohere embedding and Titan embedding families. Replace <account-id> with the deployment’s account and request the grants from its infrastructure owner.

IAM policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BedrockCatalogRead",
      "Effect": "Allow",
      "Action": [
        "bedrock:ListFoundationModels",
        "bedrock:ListInferenceProfiles",
        "bedrock:GetFoundationModelAvailability"
      ],
      "Resource": "*"
    },
    {
      "Sid": "BedrockModelInvoke",
      "Effect": "Allow",
      "Action": [
        "bedrock:InvokeModel",
        "bedrock:InvokeModelWithResponseStream"
      ],
      "Resource": [
        "arn:aws:bedrock:*::foundation-model/anthropic.claude-*",
        "arn:aws:bedrock:*::foundation-model/cohere.embed-*",
        "arn:aws:bedrock:*::foundation-model/amazon.titan-embed-*",
        "arn:aws:bedrock:*:<account-id>:inference-profile/*"
      ]
    }
  ]
}

The region wildcard on foundation models lets regional and global inference profiles invoke their destination models. Another model family needs its foundation-model ARN added to the invocation grant. Guardrails need one more statement for the organisation’s Bedrock guardrail — see Guardrails.

The picker

How an admin adds models is in Models. Behind the screen:

When the picker opens, the console asks Bedrock what this organisation can see, with its own credential in the region it bills. Nothing is stored, so a model AWS has withdrawn is absent next time.
The admin ticks any number of models. A model reachable both on demand and through an inference profile is two identifiers and two rows; the second name is suffixed to avoid a collision.
One request writes every ticked model, each left serving. The provider is read once for the batch, names are allocated against one read of the catalog, and an entry the provider does not offer writes none of the batch. The identifier comes from the picker rather than a form; a pasted key for another provider is picked from the same way.
The context window and output ceiling start at a default, edited on the rare model where either matters. A wrong context window narrows what is sent and never breaks a call.

Tool use is derived: Bedrock publishes only whether a model serves the API that tool use runs through, so the console reads an answer off that and lets an admin overrule it.

Bedrock region and inference profiles

Most current models are served only through an inference profile. A profile spans regions: regional ones route to a neighbour, global ones carry no region.

Who may use a model is on Budgets and policy.

Under the hood

Error codes from reconciliation

Code	Means
`byok_credential_not_approved`	A pasted key’s routes are withheld until it is approved
`org_has_no_bedrock_credential`	The Bedrock credential is disabled, so the model cannot be provisioned

Secrets, settings and gateway parameters

Name	Role
`governance.managed_secrets`	Where sealed secrets live, referenced as `portal:<id>`
`BEDROCK_INSTANCE_ROLE_ENABLED`	Overrides the default that offers the host IAM role method
`litellm_params.aws_session_name`	Holds the session version each sweep compares
`cachePrompt`	Requested `true` for a route holding `PROMPT_CACHE` and both cache rates, `false` otherwise
`enable_prompt_caching`	Stripped from request bodies; read only from key metadata
`anthropic_beta`	Carries the fine-grained tool streaming flag on `TOOL_STREAMING` routes

The agent uses the Strands SDK’s OpenAI chat adapter, which is why the gateway adapter injects cache checkpoints itself.

Embedding width and encoding

The width and the encoding are asked for on every embedding call. A client left to choose may pick packed binary and decode it as bytes while the gateway answers plain numbers, and the vector arrives at a quarter of its width. The wrong width is refused where it is produced, and the test fixture for that check reproduces the gateway’s real answer.

Development stack model

The development stack runs an older chat model because of the region-policy refusal above, and the seeder says so where the identifiers are declared.

Models Add models, set assignments and connect Bedrock. Budgets and policy Who may use a model, and BYOK spend. Audit and cost What each rate pays for.