toallab-automation/RHOAI_MAAS.md

# OpenShift AI Models-as-a-Service

This deploys Red Hat OpenShift AI (RHOAI) 3.5 with **Models-as-a-Service (MaaS)** and
publishes the existing `llama-coder` server (llama.cpp running Qwen3.5-9B) through it.

MaaS puts one authenticated, rate-limited, OpenAI-compatible endpoint in front of the
cluster's models:
- users get their own API keys;
- token quotas apply per user;
- access is granted by group.

Nothing about llama-coder itself changes. Clients that call it directly keep working.

- Playbook: [playbooks/deploy_rhoai_maas.yml](playbooks/deploy_rhoai_maas.yml)
- Role: [roles/rhoai_maas/](roles/rhoai_maas/) (database: [roles/cloudnativepg_operator/](roles/cloudnativepg_operator/))
- Official docs: [Govern LLM access with Models-as-a-Service (RHOAI 3.5)](https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/3.5/pdf/govern_llm_access_with_models-as-a-service/Red_Hat_OpenShift_AI_Self-Managed-3.5-Govern_LLM_access_with_Models-as-a-Service-en-US.pdf),
  plus the [companion guide](https://github.com/jharmison-redhat/rhoai-maas-guide)

> **Support status:** MaaS itself is GA in RHOAI 3.5. **External models**, which this uses
> to front llama.cpp instead of a KServe `LLMInferenceService`, are **Technology Preview**.

## How it works

```
 client ──HTTPS──► Route maas.apps.openshift.toal.ca (passthrough)
   (sk-oai-… key)        │
                         ▼
                  Gateway maas-default-gateway (openshift-ingress, Istio, ClusterIP)
                         │  Authorino: validate API key + MaaSAuthPolicy
                         │  Limitador: enforce MaaSSubscription token limits
                         ▼
                  ExternalModel qwen35-9b-coder ──HTTP:8080 + llama-coder API key──►
                  Service llama-coder (llm-coding-assistant) ──► llama.cpp on the desktop GPU node
```

1. A user creates a MaaS API key (prefix `sk-oai-`) for a **subscription**.
2. The client sends OpenAI-format requests to `https://maas.apps.openshift.toal.ca/v1/...`,
   using the `model` field to pick the model.
3. The gateway checks three things:
   - the key is valid;
   - a **MaaSAuthPolicy** grants the user's groups access to that model;
   - the user's subscription still has **token quota** left.
4. MaaS forwards the request to llama-coder over in-cluster HTTP, adding llama-coder's own API
   key. Users never see that key.

### What gets deployed

| Layer | Resources |
|---|---|
| Operators (OLM) | Red Hat Connectivity Link 1.4 (Kuadrant, Authorino, Limitador, DNS) in `openshift-operators`, Leader Worker Set in `openshift-lws-operator`, RHOAI `stable-3.5` in `redhat-ods-operator` |
| Monitoring | User Workload Monitoring turned on (`cluster-monitoring-config`). MaaS reports *Degraded* without it. |
| Gateway | GatewayClass `openshift-default` (makes the cluster install Service Mesh 3/Istio), Certificate `maas-default-gateway-cert` (Let's Encrypt, `maas.apps…` only), Gateway `maas-default-gateway` using it, Route `maas-default-gateway-https` (passthrough) |
| Auth and limits | `Kuadrant` and `Authorino` in `kuadrant-system`, with Authorino TLS using a certificate from OpenShift's internal service CA |
| Database | Single-instance CloudNativePG cluster `maas-postgres` in `rhoai-maas-db`, plus the `maas-db-config` secret in `redhat-ai-gateway-infra` |
| RHOAI | `DSCInitialization default-dsci` and `DataScienceCluster default-dsc` (minimal component set, below), with the dashboard's MaaS and external-models views turned on |
| Model | ExternalProvider `llama-coder-provider`, ExternalModel and MaaSModelRef `qwen35-9b-coder` in `llm-coding-assistant` |
| Access | MaaSAuthPolicy `llama-coder-access` and MaaSSubscription `llama-coder-default` in `models-as-a-service` |

**DataScienceCluster components:**
- **Managed:** `dashboard`, `kserve`, `aigateway` (with `modelsAsAService`), and `ogx` (Gen AI Studio).
- **Removed:** everything else (pipelines, workbenches, model registry, Ray, TrustyAI, MLflow,
  Feast, Kueue, trainer, training operator).

### Why an external model, not KServe

The cluster has one GPU: an RTX 4060 with 8 GB of VRAM and no sharing, already in use by
llama-coder. Serving through KServe/vLLM would mean shutting down llama.cpp and dropping to a
much smaller model. Using `ExternalModel` with `maas.opendatahub.io/tls: "false"` and
`maas.opendatahub.io/port: "8080"` lets MaaS govern the existing server in place.

## Prerequisites

- OpenShift 4.19.9 or later (the cluster runs 4.22) with cert-manager installed.
- The `letsencrypt-production` ClusterIssuer (DNS-01). The role issues a dedicated cert for
  `maas.apps.openshift.toal.ca`. Don't put the `*.apps` wildcard on the gateway: browsers reuse an
  HTTP/2 connection for any host its certificate covers, so after you visit MaaS, the console
  and other `*.apps` sites would be sent to the MaaS gateway and get 404s.
- llama-coder is running in `llm-coding-assistant`, with secret `llama-coder-api-key` (key `api-key`).
- A `rhoai-deployer` ServiceAccount, provisioned with the permissions in
  [deploy_openshift.yml](playbooks/deploy_openshift.yml). Re-run this after any change to those
  permissions:

  ```bash
  ansible-navigator run playbooks/deploy_openshift.yml --tags sno_deploy_service_accounts
  ```

- Its token is referenced in the inventory at
  `host_vars/sno.openshift.toal.ca/hashivault_secrets.yml`:

  ```yaml
  vault_rhoai_deployer_token: "{{ lookup('community.hashi_vault.hashi_vault', 'kv/data/toallab/rhoai-deployer:token') }}"
  ```

## Deploy

```bash
source ~/.venv/ansible/bin/activate
op run --env-file=$HOME/.ansible.zshenv -- \
  ansible-navigator run playbooks/deploy_rhoai_maas.yml --mode stdout
```

A first run takes roughly 15–25 minutes, mostly waiting on operators. It runs these steps in
order:

1. **CloudNativePG:** creates the `maas-postgres` cluster and its `maas` database. The password
   is generated once and stored in Vault at `kv/cloudnativepg/maas-postgres`.
2. **Operators:** installs Connectivity Link, Leader Worker Set, and RHOAI, waiting for each
   to finish installing.
3. **Platform:**
   - turns on User Workload Monitoring;
   - creates the GatewayClass, Gateway, and Route;
   - creates Kuadrant and restarts its operator once Istio exists;
   - turns on Authorino TLS.
4. **Database secret:** builds `maas-db-config` from the CloudNativePG credentials.
5. **RHOAI:**
   - applies the DSCInitialization and DataScienceCluster;
   - waits for the MaaS resource types to be registered and for `maas-api` to become available;
   - turns on the dashboard's MaaS views.
6. **Models:**
   - labels the model namespace and API-key secret;
   - creates the ExternalProvider, ExternalModel, and MaaSModelRef;
   - applies the auth policy and subscription.

Each step is safe to re-run. If a run fails, fix the cause and run the playbook again; it
continues from where things stand.

### Verify the deployment

```bash
oc get datasciencecluster default-dsc -o jsonpath='{.status.phase}{"\n"}'     # Ready
oc get kuadrant kuadrant -n kuadrant-system \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}'          # True
oc get gateway maas-default-gateway -n openshift-ingress                       # PROGRAMMED True
oc get deploy maas-api -n redhat-ai-gateway-infra                              # 1/1
oc get externalprovider,externalmodel,maasmodelref -n llm-coding-assistant
oc get maasauthpolicy,maassubscription -n models-as-a-service
curl -s -o /dev/null -w '%{http_code}\n' https://maas.apps.openshift.toal.ca/maas-api/health
```

## Use it

### 1. Get an API key

You must belong to a group named in a MaaSSubscription. The default subscription covers every
authenticated user.

**From the dashboard:**

1. Open **OpenShift AI → Gen AI studio → API keys → Create API key**.
2. Pick the `llama-coder default` subscription and an expiry (the default is 90 days).
3. Copy the key. It's shown **only once**.

**From the CLI**, using your OpenShift login token:

```bash
MAAS=https://maas.apps.openshift.toal.ca
curl -s -X POST "$MAAS/maas-api/v1/api-keys" \
  -H "Authorization: Bearer $(oc whoami -t)" \
  -H "Content-Type: application/json" \
  -d '{"name": "laptop", "expiresIn": "720h", "subscription": "llama-coder-default"}' | jq -r .key
```

An API key records your group memberships **when it is created**. If your groups change,
revoke the key and create a new one.

### 2. Call the model

The endpoint is OpenAI-compatible. Use the model name `qwen35-9b-coder`.

```bash
export MAAS_URL=https://maas.apps.openshift.toal.ca
export MAAS_API_KEY=sk-oai-...

curl -s "$MAAS_URL/v1/models" -H "Authorization: Bearer $MAAS_API_KEY"

curl -s "$MAAS_URL/v1/chat/completions" \
  -H "Authorization: Bearer $MAAS_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "qwen35-9b-coder", "messages": [{"role": "user", "content": "Write a haiku about Ansible"}]}'
```

Any OpenAI SDK or coding assistant works with these values:

| Setting | Value |
|---|---|
| Base URL | `https://maas.apps.openshift.toal.ca/v1` |
| API key | your `sk-oai-…` key |
| Model | `qwen35-9b-coder` |

```python
from openai import OpenAI

client = OpenAI(base_url="https://maas.apps.openshift.toal.ca/v1", api_key="sk-oai-...")
print(client.chat.completions.create(
    model="qwen35-9b-coder",
    messages=[{"role": "user", "content": "Hello"}],
).choices[0].message.content)
```

### Limits

- The default subscription allows each user **200,000 tokens per hour**. A user who exceeds
  it gets HTTP 429 until the hour window resets.
- Token limits are counted only for OpenAI Chat Completions requests (`openai-chat`).
- llama.cpp handles one model with a 48k-token context and queues requests on a single GPU.
  MaaS adds governance, not capacity.

## Administer

All access settings live in role variables. Change them in the inventory or with `-e`, then
re-run the playbook. The role creates and updates resources but never deletes them; see
[Remove a model or policy](#remove-a-model-or-policy).

### Grant access to specific groups

The OpenShift groups come from the Keycloak OIDC login. For example, to replace "all
authenticated users" with two tiers:

```yaml
rhoai_maas_auth_policies:
  - name: llama-coder-access
    groups: [developers, platform-admins]
    model_refs:
      - {name: qwen35-9b-coder, namespace: llm-coding-assistant}

rhoai_maas_subscriptions:
  - name: llama-coder-standard
    display_name: Standard
    priority: 10
    groups: [developers]
    model_refs:
      - name: qwen35-9b-coder
        namespace: llm-coding-assistant
        tokenRateLimits: [{limit: 100000, window: 1h}]
  - name: llama-coder-premium
    display_name: Premium
    priority: 20            # wins for users in both groups
    groups: [platform-admins]
    model_refs:
      - name: qwen35-9b-coder
        namespace: llm-coding-assistant
        tokenRateLimits: [{limit: 1000000, window: 1h}]
```

- An **auth policy** decides *who may call* a model.
- A **subscription** decides *how many tokens* they get.

A user needs both. `users:` (a list of usernames) works alongside `groups:` in both resources.

### Add another model

Any OpenAI-compatible endpoint can be added to `rhoai_maas_external_models`. That includes
another in-cluster server or a cloud provider. Then reference it from an auth policy and a
subscription.

```yaml
rhoai_maas_external_models:
  - name: qwen35-9b-coder
    # ... existing entry ...
  - name: gpt-4o-mini
    namespace: llm-coding-assistant
    display_name: GPT-4o mini (OpenAI)
    provider_name: openai-provider
    provider_type: openai           # openai | anthropic | azure | vertex | bedrock
    endpoint: api.openai.com        # hostname only, no scheme
    port: 443
    tls: true
    api_key_secret: openai-api-key  # create first: key "api-key"
    target_model: gpt-4o-mini
    api_format: openai-chat
    path: /v1/chat/completions
```

Rules for model entries:
- **Names:** must be DNS-safe and must not collide with an existing Service in the namespace.
  The controller creates a Service for each provider and model. That's why the provider is
  called `llama-coder-provider` rather than `llama-coder`.
- **Secrets:** the API-key secret must exist in the model's namespace before you run the
  playbook. The role only adds the `inference.llm-d.ai/ipp-managed` label to it.
- **Shared provider limits:** for a cloud provider, every user shares the provider's key, so
  its rate limits apply to all MaaS users combined.

### Revoke keys

Use **Gen AI studio → API keys** in the dashboard. Administrators can revoke one key or all of
a user's keys. Revoke a user's keys after removing them from a group, because existing keys
keep the old memberships.

### Remove a model or policy

The role never deletes resources. Delete them by hand, then remove the entry from the
variables:

```bash
oc delete maassubscription <name> -n models-as-a-service
oc delete maasauthpolicy <name> -n models-as-a-service
oc delete maasmodelref,externalmodel <model> -n llm-coding-assistant
oc delete externalprovider <provider> -n llm-coding-assistant
```

## Configuration reference

All variables are in [roles/rhoai_maas/defaults/main.yml](roles/rhoai_maas/defaults/main.yml).
[meta/argument_specs.yml](roles/rhoai_maas/meta/argument_specs.yml) documents each one.

| Variable | Default | Notes |
|---|---|---|
| `rhoai_maas_operators` | RHCL `stable-1.4`, LWS `stable-v1.0`, RHOAI `stable-3.5` | RHOAI is pinned because the MaaS settings layout changed between 3.4 and 3.5 |
| `rhoai_maas_hostname` | `""` → `maas.<apps domain>` | |
| `rhoai_maas_gateway_cert_secret` | `maas-gateway-tls` | Single-host cert in `openshift-ingress`; never the `*.apps` wildcard |
| `rhoai_maas_gateway_cert_issuer` | `letsencrypt-production` | ClusterIssuer for that cert; `""` to bring your own secret |
| `rhoai_maas_gateway_service_type` | `ClusterIP` | Use `LoadBalancer` if MetalLB is added |
| `rhoai_maas_gateway_create_route` | `true` | Passthrough Route on the default router |
| `rhoai_maas_db_secret_name` / `_namespace` | `maas-postgres-secret` / `rhoai-maas-db` | Basic-auth secret from CloudNativePG |
| `rhoai_maas_db_sslmode` | `require` | |
| `rhoai_maas_dsc_managed_components` / `_removed_components` | see above | |
| `rhoai_maas_external_models` | llama-coder | |
| `rhoai_maas_auth_policies` | all authenticated users | |
| `rhoai_maas_subscriptions` | 200k tokens/h per user | |
| `rhoai_maas_wait_timeout` | `900` | Seconds, per wait |

The database is sized in the playbook's `cloudnativepg_operator` variables: 1 instance, 5 Gi,
100 connections.

## Operations

### Troubleshooting

| Symptom | Cause and fix |
|---|---|
| Run hangs at **Wait for MaaS CRDs**, and the DSC shows `ModelsAsAServiceReady=False … Removed` | MaaS isn't turned on. In 3.5 the field is `aigateway.modelsAsAService` (note "AsA"). The older `modelsAsService` spelling is silently dropped by the API server. Check with `oc get dsc default-dsc -o jsonpath='{.spec.components.aigateway}'`. |
| Run stops at **Wait for Kuadrant to be ready**, and Kuadrant shows `MissingDependency` | The Kuadrant operator started before Istio existed. The role restarts it automatically. To fix it by hand: `oc delete pod -n openshift-operators -l app=kuadrant,control-plane=controller-manager` |
| Gateway not `Programmed` | Check `oc describe gateway maas-default-gateway -n openshift-ingress`. `LoadBalancerPending` and `DNSReady=False` are expected with ClusterIP plus a Route and do no harm. |
| `maas-api` never appears | Check `oc get datasciencecluster default-dsc -o yaml` (status conditions) and that `maas-db-config` exists in `redhat-ai-gateway-infra`. |
| DSC shows *Degraded*, mentioning monitoring | User Workload Monitoring is off. Check `cluster-monitoring-config`. |
| `401` from `/v1/...` | The key is missing, expired, or revoked, or you used an OpenShift token instead of an `sk-oai-` key. |
| `403` | No MaaSAuthPolicy grants your groups that model. Check that `model_refs` names and namespaces match. |
| `429` | Subscription token limit reached. Wait for the window to reset, or raise `tokenRateLimits`. |
| `ExternalProvider` not `Ready` | Check the API-key secret exists with key `api-key`, and that the Service, ServiceEntry, and DestinationRule were created in the model namespace. |
| Long prompts fail or are cut off | Requests over 16 KB can be truncated on Service Mesh before 3.3.1. Check the installed version with `oc get csv -n openshift-operators \| grep servicemesh`. |
| Model timeouts under load | llama.cpp is serving on one GPU. Check `oc logs -n llm-coding-assistant deploy/llama-coder`. |

### Usage metrics

MaaS exports token and request metrics to User Workload Monitoring. Query them under
**Observe → Metrics** in the console, or use the dashboard's MaaS views. The full
observability stack (Tempo, OpenTelemetry, Loki) isn't deployed here. See phase 7 of the
[companion guide](https://github.com/jharmison-redhat/rhoai-maas-guide).

### Upgrades

RHOAI follows the `stable-3.5` channel with Automatic approval, so 3.5.z updates install on
their own. Moving to 3.6 or later means changing the channel in `rhoai_maas_operators`,
after checking the release notes. MaaS settings have changed between minor versions before:
3.4 configured MaaS under `kserve`, 3.5 moved it to `aigateway`.

### Remove

Removing MaaS deletes the database and every API key. Work in this order:

```bash
oc delete maassubscription,maasauthpolicy --all -n models-as-a-service
oc delete maasmodelref,externalmodel,externalprovider --all -n llm-coding-assistant
oc delete datasciencecluster default-dsc
oc delete dscinitialization default-dsci
oc delete gateway maas-default-gateway -n openshift-ingress
oc delete route maas-default-gateway-https -n openshift-ingress
oc delete kuadrant kuadrant -n kuadrant-system
oc delete cluster.postgresql.cnpg.io maas-postgres -n rhoai-maas-db
# then uninstall the operators (Subscriptions + CSVs) from the console
```

Leave the `openshift-default` GatewayClass in place if anything else uses Gateway API.