ptoal/toallab-automation

Fork 0

Files

Patrick Toal 5b46ba7c74

Add OPA and Openshift AI

2026-10-20 14:08:57 -04:00

18 KiB

Raw Blame History

OpenShift AI Models-as-a-Service

This deploys Red Hat OpenShift AI (RHOAI) 3.5 with Models-as-a-Service (MaaS) and publishes the existing llama-coder server (llama.cpp running Qwen3.5-9B) through it.

MaaS puts one authenticated, rate-limited, OpenAI-compatible endpoint in front of the cluster's models:

users get their own API keys;
token quotas apply per user;
access is granted by group.

Nothing about llama-coder itself changes. Clients that call it directly keep working.

Playbook: playbooks/deploy_rhoai_maas.yml
Role: roles/rhoai_maas/ (database: roles/cloudnativepg_operator/)
Official docs: Govern LLM access with Models-as-a-Service (RHOAI 3.5), plus the companion guide

Support status: MaaS itself is GA in RHOAI 3.5. External models, which this uses to front llama.cpp instead of a KServe LLMInferenceService, are Technology Preview.

How it works

 client ──HTTPS──► Route maas.apps.openshift.toal.ca (passthrough)
   (sk-oai-… key)        │
                         ▼
                  Gateway maas-default-gateway (openshift-ingress, Istio, ClusterIP)
                         │  Authorino: validate API key + MaaSAuthPolicy
                         │  Limitador: enforce MaaSSubscription token limits
                         ▼
                  ExternalModel qwen35-9b-coder ──HTTP:8080 + llama-coder API key──►
                  Service llama-coder (llm-coding-assistant) ──► llama.cpp on the desktop GPU node

A user creates a MaaS API key (prefix sk-oai-) for a subscription.
The client sends OpenAI-format requests to https://maas.apps.openshift.toal.ca/v1/..., using the model field to pick the model.
The gateway checks three things:
- the key is valid;
- a MaaSAuthPolicy grants the user's groups access to that model;
- the user's subscription still has token quota left.
MaaS forwards the request to llama-coder over in-cluster HTTP, adding llama-coder's own API key. Users never see that key.

What gets deployed

Layer	Resources
Operators (OLM)	Red Hat Connectivity Link 1.4 (Kuadrant, Authorino, Limitador, DNS) in `openshift-operators`, Leader Worker Set in `openshift-lws-operator`, RHOAI `stable-3.5` in `redhat-ods-operator`
Monitoring	User Workload Monitoring turned on (`cluster-monitoring-config`). MaaS reports Degraded without it.
Gateway	GatewayClass `openshift-default` (makes the cluster install Service Mesh 3/Istio), Certificate `maas-default-gateway-cert` (Let's Encrypt, `maas.apps…` only), Gateway `maas-default-gateway` using it, Route `maas-default-gateway-https` (passthrough)
Auth and limits	`Kuadrant` and `Authorino` in `kuadrant-system`, with Authorino TLS using a certificate from OpenShift's internal service CA
Database	Single-instance CloudNativePG cluster `maas-postgres` in `rhoai-maas-db`, plus the `maas-db-config` secret in `redhat-ai-gateway-infra`
RHOAI	`DSCInitialization default-dsci` and `DataScienceCluster default-dsc` (minimal component set, below), with the dashboard's MaaS and external-models views turned on
Model	ExternalProvider `llama-coder-provider`, ExternalModel and MaaSModelRef `qwen35-9b-coder` in `llm-coding-assistant`
Access	MaaSAuthPolicy `llama-coder-access` and MaaSSubscription `llama-coder-default` in `models-as-a-service`

DataScienceCluster components:

Managed: dashboard, kserve, aigateway (with modelsAsAService), and ogx (Gen AI Studio).
Removed: everything else (pipelines, workbenches, model registry, Ray, TrustyAI, MLflow, Feast, Kueue, trainer, training operator).

Why an external model, not KServe

The cluster has one GPU: an RTX 4060 with 8 GB of VRAM and no sharing, already in use by llama-coder. Serving through KServe/vLLM would mean shutting down llama.cpp and dropping to a much smaller model. Using ExternalModel with maas.opendatahub.io/tls: "false" and maas.opendatahub.io/port: "8080" lets MaaS govern the existing server in place.

Prerequisites

OpenShift 4.19.9 or later (the cluster runs 4.22) with cert-manager installed.
The letsencrypt-production ClusterIssuer (DNS-01). The role issues a dedicated cert for maas.apps.openshift.toal.ca. Don't put the *.apps wildcard on the gateway: browsers reuse an HTTP/2 connection for any host its certificate covers, so after you visit MaaS, the console and other *.apps sites would be sent to the MaaS gateway and get 404s.
llama-coder is running in llm-coding-assistant, with secret llama-coder-api-key (key api-key).
A rhoai-deployer ServiceAccount, provisioned with the permissions in deploy_openshift.yml. Re-run this after any change to those permissions:
```
ansible-navigator run playbooks/deploy_openshift.yml --tags sno_deploy_service_accounts
```

Its token is referenced in the inventory at host_vars/sno.openshift.toal.ca/hashivault_secrets.yml:

vault_rhoai_deployer_token: "{{ lookup('community.hashi_vault.hashi_vault', 'kv/data/toallab/rhoai-deployer:token') }}"

Deploy

source ~/.venv/ansible/bin/activate
op run --env-file=$HOME/.ansible.zshenv -- \
  ansible-navigator run playbooks/deploy_rhoai_maas.yml --mode stdout

A first run takes roughly 15–25 minutes, mostly waiting on operators. It runs these steps in order:

CloudNativePG: creates the maas-postgres cluster and its maas database. The password is generated once and stored in Vault at kv/cloudnativepg/maas-postgres.
Operators: installs Connectivity Link, Leader Worker Set, and RHOAI, waiting for each to finish installing.
Platform:
- turns on User Workload Monitoring;
- creates the GatewayClass, Gateway, and Route;
- creates Kuadrant and restarts its operator once Istio exists;
- turns on Authorino TLS.
Database secret: builds maas-db-config from the CloudNativePG credentials.
RHOAI:
- applies the DSCInitialization and DataScienceCluster;
- waits for the MaaS resource types to be registered and for maas-api to become available;
- turns on the dashboard's MaaS views.
Models:
- labels the model namespace and API-key secret;
- creates the ExternalProvider, ExternalModel, and MaaSModelRef;
- applies the auth policy and subscription.

Each step is safe to re-run. If a run fails, fix the cause and run the playbook again; it continues from where things stand.

Verify the deployment

oc get datasciencecluster default-dsc -o jsonpath='{.status.phase}{"\n"}'     # Ready
oc get kuadrant kuadrant -n kuadrant-system \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}'          # True
oc get gateway maas-default-gateway -n openshift-ingress                       # PROGRAMMED True
oc get deploy maas-api -n redhat-ai-gateway-infra                              # 1/1
oc get externalprovider,externalmodel,maasmodelref -n llm-coding-assistant
oc get maasauthpolicy,maassubscription -n models-as-a-service
curl -s -o /dev/null -w '%{http_code}\n' https://maas.apps.openshift.toal.ca/maas-api/health

Use it

1. Get an API key

You must belong to a group named in a MaaSSubscription. The default subscription covers every authenticated user.

From the dashboard:

Open OpenShift AI → Gen AI studio → API keys → Create API key.
Pick the llama-coder default subscription and an expiry (the default is 90 days).
Copy the key. It's shown only once.

From the CLI, using your OpenShift login token:

MAAS=https://maas.apps.openshift.toal.ca
curl -s -X POST "$MAAS/maas-api/v1/api-keys" \
  -H "Authorization: Bearer $(oc whoami -t)" \
  -H "Content-Type: application/json" \
  -d '{"name": "laptop", "expiresIn": "720h", "subscription": "llama-coder-default"}' | jq -r .key

An API key records your group memberships when it is created. If your groups change, revoke the key and create a new one.

2. Call the model

The endpoint is OpenAI-compatible. Use the model name qwen35-9b-coder.

export MAAS_URL=https://maas.apps.openshift.toal.ca
export MAAS_API_KEY=sk-oai-...

curl -s "$MAAS_URL/v1/models" -H "Authorization: Bearer $MAAS_API_KEY"

curl -s "$MAAS_URL/v1/chat/completions" \
  -H "Authorization: Bearer $MAAS_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "qwen35-9b-coder", "messages": [{"role": "user", "content": "Write a haiku about Ansible"}]}'

Any OpenAI SDK or coding assistant works with these values:

Setting	Value
Base URL	`https://maas.apps.openshift.toal.ca/v1`
API key	your `sk-oai-…` key
Model	`qwen35-9b-coder`

from openai import OpenAI

client = OpenAI(base_url="https://maas.apps.openshift.toal.ca/v1", api_key="sk-oai-...")
print(client.chat.completions.create(
    model="qwen35-9b-coder",
    messages=[{"role": "user", "content": "Hello"}],
).choices[0].message.content)

Limits

The default subscription allows each user 200,000 tokens per hour. A user who exceeds it gets HTTP 429 until the hour window resets.
Token limits are counted only for OpenAI Chat Completions requests (openai-chat).
llama.cpp handles one model with a 48k-token context and queues requests on a single GPU. MaaS adds governance, not capacity.

Administer

All access settings live in role variables. Change them in the inventory or with -e, then re-run the playbook. The role creates and updates resources but never deletes them; see Remove a model or policy.

Grant access to specific groups

The OpenShift groups come from the Keycloak OIDC login. For example, to replace "all authenticated users" with two tiers:

rhoai_maas_auth_policies:
  - name: llama-coder-access
    groups: [developers, platform-admins]
    model_refs:
      - {name: qwen35-9b-coder, namespace: llm-coding-assistant}

rhoai_maas_subscriptions:
  - name: llama-coder-standard
    display_name: Standard
    priority: 10
    groups: [developers]
    model_refs:
      - name: qwen35-9b-coder
        namespace: llm-coding-assistant
        tokenRateLimits: [{limit: 100000, window: 1h}]
  - name: llama-coder-premium
    display_name: Premium
    priority: 20            # wins for users in both groups
    groups: [platform-admins]
    model_refs:
      - name: qwen35-9b-coder
        namespace: llm-coding-assistant
        tokenRateLimits: [{limit: 1000000, window: 1h}]

An auth policy decides who may call a model.
A subscription decides how many tokens they get.

A user needs both. users: (a list of usernames) works alongside groups: in both resources.

Add another model

Any OpenAI-compatible endpoint can be added to rhoai_maas_external_models. That includes another in-cluster server or a cloud provider. Then reference it from an auth policy and a subscription.

rhoai_maas_external_models:
  - name: qwen35-9b-coder
    # ... existing entry ...
  - name: gpt-4o-mini
    namespace: llm-coding-assistant
    display_name: GPT-4o mini (OpenAI)
    provider_name: openai-provider
    provider_type: openai           # openai | anthropic | azure | vertex | bedrock
    endpoint: api.openai.com        # hostname only, no scheme
    port: 443
    tls: true
    api_key_secret: openai-api-key  # create first: key "api-key"
    target_model: gpt-4o-mini
    api_format: openai-chat
    path: /v1/chat/completions

Rules for model entries:

Names: must be DNS-safe and must not collide with an existing Service in the namespace. The controller creates a Service for each provider and model. That's why the provider is called llama-coder-provider rather than llama-coder.
Secrets: the API-key secret must exist in the model's namespace before you run the playbook. The role only adds the inference.llm-d.ai/ipp-managed label to it.
Shared provider limits: for a cloud provider, every user shares the provider's key, so its rate limits apply to all MaaS users combined.

Revoke keys

Use Gen AI studio → API keys in the dashboard. Administrators can revoke one key or all of a user's keys. Revoke a user's keys after removing them from a group, because existing keys keep the old memberships.

Remove a model or policy

The role never deletes resources. Delete them by hand, then remove the entry from the variables:

oc delete maassubscription <name> -n models-as-a-service
oc delete maasauthpolicy <name> -n models-as-a-service
oc delete maasmodelref,externalmodel <model> -n llm-coding-assistant
oc delete externalprovider <provider> -n llm-coding-assistant

Configuration reference

All variables are in roles/rhoai_maas/defaults/main.yml. meta/argument_specs.yml documents each one.

Variable	Default	Notes
`rhoai_maas_operators`	RHCL `stable-1.4`, LWS `stable-v1.0`, RHOAI `stable-3.5`	RHOAI is pinned because the MaaS settings layout changed between 3.4 and 3.5
`rhoai_maas_hostname`	`""` → `maas.<apps domain>`
`rhoai_maas_gateway_cert_secret`	`maas-gateway-tls`	Single-host cert in `openshift-ingress`; never the `*.apps` wildcard
`rhoai_maas_gateway_cert_issuer`	`letsencrypt-production`	ClusterIssuer for that cert; `""` to bring your own secret
`rhoai_maas_gateway_service_type`	`ClusterIP`	Use `LoadBalancer` if MetalLB is added
`rhoai_maas_gateway_create_route`	`true`	Passthrough Route on the default router
`rhoai_maas_db_secret_name` / `_namespace`	`maas-postgres-secret` / `rhoai-maas-db`	Basic-auth secret from CloudNativePG
`rhoai_maas_db_sslmode`	`require`
`rhoai_maas_dsc_managed_components` / `_removed_components`	see above
`rhoai_maas_external_models`	llama-coder
`rhoai_maas_auth_policies`	all authenticated users
`rhoai_maas_subscriptions`	200k tokens/h per user
`rhoai_maas_wait_timeout`	`900`	Seconds, per wait

The database is sized in the playbook's cloudnativepg_operator variables: 1 instance, 5 Gi, 100 connections.

Operations

Troubleshooting

Symptom	Cause and fix
Run hangs at Wait for MaaS CRDs, and the DSC shows `ModelsAsAServiceReady=False … Removed`	MaaS isn't turned on. In 3.5 the field is `aigateway.modelsAsAService` (note "AsA"). The older `modelsAsService` spelling is silently dropped by the API server. Check with `oc get dsc default-dsc -o jsonpath='{.spec.components.aigateway}'`.
Run stops at Wait for Kuadrant to be ready, and Kuadrant shows `MissingDependency`	The Kuadrant operator started before Istio existed. The role restarts it automatically. To fix it by hand: `oc delete pod -n openshift-operators -l app=kuadrant,control-plane=controller-manager`
Gateway not `Programmed`	Check `oc describe gateway maas-default-gateway -n openshift-ingress`. `LoadBalancerPending` and `DNSReady=False` are expected with ClusterIP plus a Route and do no harm.
`maas-api` never appears	Check `oc get datasciencecluster default-dsc -o yaml` (status conditions) and that `maas-db-config` exists in `redhat-ai-gateway-infra`.
DSC shows Degraded, mentioning monitoring	User Workload Monitoring is off. Check `cluster-monitoring-config`.
`401` from `/v1/...`	The key is missing, expired, or revoked, or you used an OpenShift token instead of an `sk-oai-` key.
`403`	No MaaSAuthPolicy grants your groups that model. Check that `model_refs` names and namespaces match.
`429`	Subscription token limit reached. Wait for the window to reset, or raise `tokenRateLimits`.
`ExternalProvider` not `Ready`	Check the API-key secret exists with key `api-key`, and that the Service, ServiceEntry, and DestinationRule were created in the model namespace.
Long prompts fail or are cut off	Requests over 16 KB can be truncated on Service Mesh before 3.3.1. Check the installed version with `oc get csv -n openshift-operators \| grep servicemesh`.
Model timeouts under load	llama.cpp is serving on one GPU. Check `oc logs -n llm-coding-assistant deploy/llama-coder`.

Usage metrics

MaaS exports token and request metrics to User Workload Monitoring. Query them under Observe → Metrics in the console, or use the dashboard's MaaS views. The full observability stack (Tempo, OpenTelemetry, Loki) isn't deployed here. See phase 7 of the companion guide.

Upgrades

RHOAI follows the stable-3.5 channel with Automatic approval, so 3.5.z updates install on their own. Moving to 3.6 or later means changing the channel in rhoai_maas_operators, after checking the release notes. MaaS settings have changed between minor versions before: 3.4 configured MaaS under kserve, 3.5 moved it to aigateway.

Remove

Removing MaaS deletes the database and every API key. Work in this order:

oc delete maassubscription,maasauthpolicy --all -n models-as-a-service
oc delete maasmodelref,externalmodel,externalprovider --all -n llm-coding-assistant
oc delete datasciencecluster default-dsc
oc delete dscinitialization default-dsci
oc delete gateway maas-default-gateway -n openshift-ingress
oc delete route maas-default-gateway-https -n openshift-ingress
oc delete kuadrant kuadrant -n kuadrant-system
oc delete cluster.postgresql.cnpg.io maas-postgres -n rhoai-maas-db
# then uninstall the operators (Subscriptions + CSVs) from the console

Leave the openshift-default GatewayClass in place if anything else uses Gateway API.

18 KiB Raw Blame History Unescape Escape