ptoal/toallab-automation
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add OPA and Openshift AI

Browse Source
This commit is contained in:
Patrick Toal
2026-10-20 14:08:57 -04:00
parent 7df35b9dd4
commit 5b46ba7c74
27 changed files with 2634 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

372
RHOAI_MAAS.md Normal file
View File

@@ -0,0 +1,372 @@
# OpenShift AI Models-as-a-Service
This deploys Red Hat OpenShift AI (RHOAI) 3.5 with **Models-as-a-Service (MaaS)** and
publishes the existing `llama-coder` server (llama.cpp running Qwen3.5-9B) through it.
MaaS puts one authenticated, rate-limited, OpenAI-compatible endpoint in front of the
cluster's models:
- users get their own API keys;
- token quotas apply per user;
- access is granted by group.
Nothing about llama-coder itself changes. Clients that call it directly keep working.
- Playbook: [playbooks/deploy_rhoai_maas.yml](playbooks/deploy_rhoai_maas.yml)
- Role: [roles/rhoai_maas/](roles/rhoai_maas/) (database: [roles/cloudnativepg_operator/](roles/cloudnativepg_operator/))
- Official docs: [Govern LLM access with Models-as-a-Service (RHOAI 3.5)](https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/3.5/pdf/govern_llm_access_with_models-as-a-service/Red_Hat_OpenShift_AI_Self-Managed-3.5-Govern_LLM_access_with_Models-as-a-Service-en-US.pdf),
plus the [companion guide](https://github.com/jharmison-redhat/rhoai-maas-guide)
> **Support status:** MaaS itself is GA in RHOAI 3.5. **External models**, which this uses
> to front llama.cpp instead of a KServe `LLMInferenceService`, are **Technology Preview**.
## How it works
```
client ──HTTPS──► Route maas.apps.openshift.toal.ca (passthrough)
(sk-oai-… key) │
▼
Gateway maas-default-gateway (openshift-ingress, Istio, ClusterIP)
│ Authorino: validate API key + MaaSAuthPolicy
│ Limitador: enforce MaaSSubscription token limits
▼
ExternalModel qwen35-9b-coder ──HTTP:8080 + llama-coder API key──►
Service llama-coder (llm-coding-assistant) ──► llama.cpp on the desktop GPU node
```
1. A user creates a MaaS API key (prefix `sk-oai-`) for a **subscription**.
2. The client sends OpenAI-format requests to `https://maas.apps.openshift.toal.ca/v1/...`,
using the `model` field to pick the model.
3. The gateway checks three things:
- the key is valid;
- a **MaaSAuthPolicy** grants the user's groups access to that model;
- the user's subscription still has **token quota** left.
4. MaaS forwards the request to llama-coder over in-cluster HTTP, adding llama-coder's own API
key. Users never see that key.
### What gets deployed
| Layer | Resources |
|---|---|
| Operators (OLM) | Red Hat Connectivity Link 1.4 (Kuadrant, Authorino, Limitador, DNS) in `openshift-operators`, Leader Worker Set in `openshift-lws-operator`, RHOAI `stable-3.5` in `redhat-ods-operator` |
| Monitoring | User Workload Monitoring turned on (`cluster-monitoring-config`). MaaS reports *Degraded* without it. |
| Gateway | GatewayClass `openshift-default` (makes the cluster install Service Mesh 3/Istio), Certificate `maas-default-gateway-cert` (Let's Encrypt, `maas.apps…` only), Gateway `maas-default-gateway` using it, Route `maas-default-gateway-https` (passthrough) |
| Auth and limits | `Kuadrant` and `Authorino` in `kuadrant-system`, with Authorino TLS using a certificate from OpenShift's internal service CA |
| Database | Single-instance CloudNativePG cluster `maas-postgres` in `rhoai-maas-db`, plus the `maas-db-config` secret in `redhat-ai-gateway-infra` |
| RHOAI | `DSCInitialization default-dsci` and `DataScienceCluster default-dsc` (minimal component set, below), with the dashboard's MaaS and external-models views turned on |
| Model | ExternalProvider `llama-coder-provider`, ExternalModel and MaaSModelRef `qwen35-9b-coder` in `llm-coding-assistant` |
| Access | MaaSAuthPolicy `llama-coder-access` and MaaSSubscription `llama-coder-default` in `models-as-a-service` |
**DataScienceCluster components:**
- **Managed:** `dashboard`, `kserve`, `aigateway` (with `modelsAsAService`), and `ogx` (Gen AI Studio).
- **Removed:** everything else (pipelines, workbenches, model registry, Ray, TrustyAI, MLflow,
Feast, Kueue, trainer, training operator).
### Why an external model, not KServe
The cluster has one GPU: an RTX 4060 with 8 GB of VRAM and no sharing, already in use by
llama-coder. Serving through KServe/vLLM would mean shutting down llama.cpp and dropping to a
much smaller model. Using `ExternalModel` with `maas.opendatahub.io/tls: "false"` and
`maas.opendatahub.io/port: "8080"` lets MaaS govern the existing server in place.
## Prerequisites
- OpenShift 4.19.9 or later (the cluster runs 4.22) with cert-manager installed.
- The `letsencrypt-production` ClusterIssuer (DNS-01). The role issues a dedicated cert for
`maas.apps.openshift.toal.ca`. Don't put the `*.apps` wildcard on the gateway: browsers reuse an
HTTP/2 connection for any host its certificate covers, so after you visit MaaS, the console
and other `*.apps` sites would be sent to the MaaS gateway and get 404s.
- llama-coder is running in `llm-coding-assistant`, with secret `llama-coder-api-key` (key `api-key`).
- A `rhoai-deployer` ServiceAccount, provisioned with the permissions in
[deploy_openshift.yml](playbooks/deploy_openshift.yml). Re-run this after any change to those
permissions:
```bash
ansible-navigator run playbooks/deploy_openshift.yml --tags sno_deploy_service_accounts
```
- Its token is referenced in the inventory at
`host_vars/sno.openshift.toal.ca/hashivault_secrets.yml`:
```yaml
vault_rhoai_deployer_token: "{{ lookup('community.hashi_vault.hashi_vault', 'kv/data/toallab/rhoai-deployer:token') }}"
```
## Deploy
```bash
source ~/.venv/ansible/bin/activate
op run --env-file=$HOME/.ansible.zshenv -- \
ansible-navigator run playbooks/deploy_rhoai_maas.yml --mode stdout
```
A first run takes roughly 15–25 minutes, mostly waiting on operators. It runs these steps in
order:
1. **CloudNativePG:** creates the `maas-postgres` cluster and its `maas` database. The password
is generated once and stored in Vault at `kv/cloudnativepg/maas-postgres`.
2. **Operators:** installs Connectivity Link, Leader Worker Set, and RHOAI, waiting for each
to finish installing.
3. **Platform:**
- turns on User Workload Monitoring;
- creates the GatewayClass, Gateway, and Route;
- creates Kuadrant and restarts its operator once Istio exists;
- turns on Authorino TLS.
4. **Database secret:** builds `maas-db-config` from the CloudNativePG credentials.
5. **RHOAI:**
- applies the DSCInitialization and DataScienceCluster;
- waits for the MaaS resource types to be registered and for `maas-api` to become available;
- turns on the dashboard's MaaS views.
6. **Models:**
- labels the model namespace and API-key secret;
- creates the ExternalProvider, ExternalModel, and MaaSModelRef;
- applies the auth policy and subscription.
Each step is safe to re-run. If a run fails, fix the cause and run the playbook again; it
continues from where things stand.
### Verify the deployment
```bash
oc get datasciencecluster default-dsc -o jsonpath='{.status.phase}{"\n"}' # Ready
oc get kuadrant kuadrant -n kuadrant-system \
-o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}' # True
oc get gateway maas-default-gateway -n openshift-ingress # PROGRAMMED True
oc get deploy maas-api -n redhat-ai-gateway-infra # 1/1
oc get externalprovider,externalmodel,maasmodelref -n llm-coding-assistant
oc get maasauthpolicy,maassubscription -n models-as-a-service
curl -s -o /dev/null -w '%{http_code}\n' https://maas.apps.openshift.toal.ca/maas-api/health
```
## Use it
### 1. Get an API key
You must belong to a group named in a MaaSSubscription. The default subscription covers every
authenticated user.
**From the dashboard:**
1. Open **OpenShift AI → Gen AI studio → API keys → Create API key**.
2. Pick the `llama-coder default` subscription and an expiry (the default is 90 days).
3. Copy the key. It's shown **only once**.
**From the CLI**, using your OpenShift login token:
```bash
MAAS=https://maas.apps.openshift.toal.ca
curl -s -X POST "$MAAS/maas-api/v1/api-keys" \
-H "Authorization: Bearer $(oc whoami -t)" \
-H "Content-Type: application/json" \
-d '{"name": "laptop", "expiresIn": "720h", "subscription": "llama-coder-default"}' | jq -r .key
```
An API key records your group memberships **when it is created**. If your groups change,
revoke the key and create a new one.
### 2. Call the model
The endpoint is OpenAI-compatible. Use the model name `qwen35-9b-coder`.
```bash
export MAAS_URL=https://maas.apps.openshift.toal.ca
export MAAS_API_KEY=sk-oai-...
curl -s "$MAAS_URL/v1/models" -H "Authorization: Bearer $MAAS_API_KEY"
curl -s "$MAAS_URL/v1/chat/completions" \
-H "Authorization: Bearer $MAAS_API_KEY" -H "Content-Type: application/json" \
-d '{"model": "qwen35-9b-coder", "messages": [{"role": "user", "content": "Write a haiku about Ansible"}]}'
```
Any OpenAI SDK or coding assistant works with these values:
| Setting | Value |
|---|---|
| Base URL | `https://maas.apps.openshift.toal.ca/v1` |
| API key | your `sk-oai-…` key |
| Model | `qwen35-9b-coder` |
```python
from openai import OpenAI
client = OpenAI(base_url="https://maas.apps.openshift.toal.ca/v1", api_key="sk-oai-...")
print(client.chat.completions.create(
model="qwen35-9b-coder",
messages=[{"role": "user", "content": "Hello"}],
).choices[0].message.content)
```
### Limits
- The default subscription allows each user **200,000 tokens per hour**. A user who exceeds
it gets HTTP 429 until the hour window resets.
- Token limits are counted only for OpenAI Chat Completions requests (`openai-chat`).
- llama.cpp handles one model with a 48k-token context and queues requests on a single GPU.
MaaS adds governance, not capacity.
## Administer
All access settings live in role variables. Change them in the inventory or with `-e`, then
re-run the playbook. The role creates and updates resources but never deletes them; see
[Remove a model or policy](#remove-a-model-or-policy).
### Grant access to specific groups
The OpenShift groups come from the Keycloak OIDC login. For example, to replace "all
authenticated users" with two tiers:
```yaml
rhoai_maas_auth_policies:
- name: llama-coder-access
groups: [developers, platform-admins]
model_refs:
- {name: qwen35-9b-coder, namespace: llm-coding-assistant}
rhoai_maas_subscriptions:
- name: llama-coder-standard
display_name: Standard
priority: 10
groups: [developers]
model_refs:
- name: qwen35-9b-coder
namespace: llm-coding-assistant
tokenRateLimits: [{limit: 100000, window: 1h}]
- name: llama-coder-premium
display_name: Premium
priority: 20 # wins for users in both groups
groups: [platform-admins]
model_refs:
- name: qwen35-9b-coder
namespace: llm-coding-assistant
tokenRateLimits: [{limit: 1000000, window: 1h}]
```
- An **auth policy** decides *who may call* a model.
- A **subscription** decides *how many tokens* they get.
A user needs both. `users:` (a list of usernames) works alongside `groups:` in both resources.
### Add another model
Any OpenAI-compatible endpoint can be added to `rhoai_maas_external_models`. That includes
another in-cluster server or a cloud provider. Then reference it from an auth policy and a
subscription.
```yaml
rhoai_maas_external_models:
- name: qwen35-9b-coder
# ... existing entry ...
- name: gpt-4o-mini
namespace: llm-coding-assistant
display_name: GPT-4o mini (OpenAI)
provider_name: openai-provider
provider_type: openai # openai | anthropic | azure | vertex | bedrock
endpoint: api.openai.com # hostname only, no scheme
port: 443
tls: true
api_key_secret: openai-api-key # create first: key "api-key"
target_model: gpt-4o-mini
api_format: openai-chat
path: /v1/chat/completions
```
Rules for model entries:
- **Names:** must be DNS-safe and must not collide with an existing Service in the namespace.
The controller creates a Service for each provider and model. That's why the provider is
called `llama-coder-provider` rather than `llama-coder`.
- **Secrets:** the API-key secret must exist in the model's namespace before you run the
playbook. The role only adds the `inference.llm-d.ai/ipp-managed` label to it.
- **Shared provider limits:** for a cloud provider, every user shares the provider's key, so
its rate limits apply to all MaaS users combined.
### Revoke keys
Use **Gen AI studio → API keys** in the dashboard. Administrators can revoke one key or all of
a user's keys. Revoke a user's keys after removing them from a group, because existing keys
keep the old memberships.
### Remove a model or policy
The role never deletes resources. Delete them by hand, then remove the entry from the
variables:
```bash
oc delete maassubscription <name> -n models-as-a-service
oc delete maasauthpolicy <name> -n models-as-a-service
oc delete maasmodelref,externalmodel <model> -n llm-coding-assistant
oc delete externalprovider <provider> -n llm-coding-assistant
```
## Configuration reference
All variables are in [roles/rhoai_maas/defaults/main.yml](roles/rhoai_maas/defaults/main.yml).
[meta/argument_specs.yml](roles/rhoai_maas/meta/argument_specs.yml) documents each one.
| Variable | Default | Notes |
|---|---|---|
| `rhoai_maas_operators` | RHCL `stable-1.4`, LWS `stable-v1.0`, RHOAI `stable-3.5` | RHOAI is pinned because the MaaS settings layout changed between 3.4 and 3.5 |
| `rhoai_maas_hostname` | `""` → `maas.<apps domain>` | |
| `rhoai_maas_gateway_cert_secret` | `maas-gateway-tls` | Single-host cert in `openshift-ingress`; never the `*.apps` wildcard |
| `rhoai_maas_gateway_cert_issuer` | `letsencrypt-production` | ClusterIssuer for that cert; `""` to bring your own secret |
| `rhoai_maas_gateway_service_type` | `ClusterIP` | Use `LoadBalancer` if MetalLB is added |
| `rhoai_maas_gateway_create_route` | `true` | Passthrough Route on the default router |
| `rhoai_maas_db_secret_name` / `_namespace` | `maas-postgres-secret` / `rhoai-maas-db` | Basic-auth secret from CloudNativePG |
| `rhoai_maas_db_sslmode` | `require` | |
| `rhoai_maas_dsc_managed_components` / `_removed_components` | see above | |
| `rhoai_maas_external_models` | llama-coder | |
| `rhoai_maas_auth_policies` | all authenticated users | |
| `rhoai_maas_subscriptions` | 200k tokens/h per user | |
| `rhoai_maas_wait_timeout` | `900` | Seconds, per wait |
The database is sized in the playbook's `cloudnativepg_operator` variables: 1 instance, 5 Gi,
100 connections.
## Operations
### Troubleshooting
| Symptom | Cause and fix |
|---|---|
| Run hangs at **Wait for MaaS CRDs**, and the DSC shows `ModelsAsAServiceReady=False … Removed` | MaaS isn't turned on. In 3.5 the field is `aigateway.modelsAsAService` (note "AsA"). The older `modelsAsService` spelling is silently dropped by the API server. Check with `oc get dsc default-dsc -o jsonpath='{.spec.components.aigateway}'`. |
| Run stops at **Wait for Kuadrant to be ready**, and Kuadrant shows `MissingDependency` | The Kuadrant operator started before Istio existed. The role restarts it automatically. To fix it by hand: `oc delete pod -n openshift-operators -l app=kuadrant,control-plane=controller-manager` |
| Gateway not `Programmed` | Check `oc describe gateway maas-default-gateway -n openshift-ingress`. `LoadBalancerPending` and `DNSReady=False` are expected with ClusterIP plus a Route and do no harm. |
| `maas-api` never appears | Check `oc get datasciencecluster default-dsc -o yaml` (status conditions) and that `maas-db-config` exists in `redhat-ai-gateway-infra`. |
| DSC shows *Degraded*, mentioning monitoring | User Workload Monitoring is off. Check `cluster-monitoring-config`. |
| `401` from `/v1/...` | The key is missing, expired, or revoked, or you used an OpenShift token instead of an `sk-oai-` key. |
| `403` | No MaaSAuthPolicy grants your groups that model. Check that `model_refs` names and namespaces match. |
| `429` | Subscription token limit reached. Wait for the window to reset, or raise `tokenRateLimits`. |
| `ExternalProvider` not `Ready` | Check the API-key secret exists with key `api-key`, and that the Service, ServiceEntry, and DestinationRule were created in the model namespace. |
| Long prompts fail or are cut off | Requests over 16 KB can be truncated on Service Mesh before 3.3.1. Check the installed version with `oc get csv -n openshift-operators \| grep servicemesh`. |
| Model timeouts under load | llama.cpp is serving on one GPU. Check `oc logs -n llm-coding-assistant deploy/llama-coder`. |
### Usage metrics
MaaS exports token and request metrics to User Workload Monitoring. Query them under
**Observe → Metrics** in the console, or use the dashboard's MaaS views. The full
observability stack (Tempo, OpenTelemetry, Loki) isn't deployed here. See phase 7 of the
[companion guide](https://github.com/jharmison-redhat/rhoai-maas-guide).
### Upgrades
RHOAI follows the `stable-3.5` channel with Automatic approval, so 3.5.z updates install on
their own. Moving to 3.6 or later means changing the channel in `rhoai_maas_operators`,
after checking the release notes. MaaS settings have changed between minor versions before:
3.4 configured MaaS under `kserve`, 3.5 moved it to `aigateway`.
### Remove
Removing MaaS deletes the database and every API key. Work in this order:
```bash
oc delete maassubscription,maasauthpolicy --all -n models-as-a-service
oc delete maasmodelref,externalmodel,externalprovider --all -n llm-coding-assistant
oc delete datasciencecluster default-dsc
oc delete dscinitialization default-dsci
oc delete gateway maas-default-gateway -n openshift-ingress
oc delete route maas-default-gateway-https -n openshift-ingress
oc delete kuadrant kuadrant -n kuadrant-system
oc delete cluster.postgresql.cnpg.io maas-postgres -n rhoai-maas-db
# then uninstall the operators (Subscriptions + CSVs) from the console
```
Leave the `openshift-default` GatewayClass in place if anything else uses Gateway API.