# OpenShift AI Models-as-a-Service This deploys Red Hat OpenShift AI (RHOAI) 3.5 with **Models-as-a-Service (MaaS)** and publishes the existing `llama-coder` server (llama.cpp running Qwen3.5-9B) through it. MaaS puts one authenticated, rate-limited, OpenAI-compatible endpoint in front of the cluster's models: - users get their own API keys; - token quotas apply per user; - access is granted by group. Nothing about llama-coder itself changes. Clients that call it directly keep working. - Playbook: [playbooks/deploy_rhoai_maas.yml](playbooks/deploy_rhoai_maas.yml) - Role: [roles/rhoai_maas/](roles/rhoai_maas/) (database: [roles/cloudnativepg_operator/](roles/cloudnativepg_operator/)) - Official docs: [Govern LLM access with Models-as-a-Service (RHOAI 3.5)](https://docs.redhat.com/en/documentation/red_hat_openshift_ai_self-managed/3.5/pdf/govern_llm_access_with_models-as-a-service/Red_Hat_OpenShift_AI_Self-Managed-3.5-Govern_LLM_access_with_Models-as-a-Service-en-US.pdf), plus the [companion guide](https://github.com/jharmison-redhat/rhoai-maas-guide) > **Support status:** MaaS itself is GA in RHOAI 3.5. **External models**, which this uses > to front llama.cpp instead of a KServe `LLMInferenceService`, are **Technology Preview**. ## How it works ``` client ──HTTPS──► Route maas.apps.openshift.toal.ca (passthrough) (sk-oai-… key) │ ▼ Gateway maas-default-gateway (openshift-ingress, Istio, ClusterIP) │ Authorino: validate API key + MaaSAuthPolicy │ Limitador: enforce MaaSSubscription token limits ▼ ExternalModel qwen35-9b-coder ──HTTP:8080 + llama-coder API key──► Service llama-coder (llm-coding-assistant) ──► llama.cpp on the desktop GPU node ``` 1. A user creates a MaaS API key (prefix `sk-oai-`) for a **subscription**. 2. The client sends OpenAI-format requests to `https://maas.apps.openshift.toal.ca/v1/...`, using the `model` field to pick the model. 3. The gateway checks three things: - the key is valid; - a **MaaSAuthPolicy** grants the user's groups access to that model; - the user's subscription still has **token quota** left. 4. MaaS forwards the request to llama-coder over in-cluster HTTP, adding llama-coder's own API key. Users never see that key. ### What gets deployed | Layer | Resources | |---|---| | Operators (OLM) | Red Hat Connectivity Link 1.4 (Kuadrant, Authorino, Limitador, DNS) in `openshift-operators`, Leader Worker Set in `openshift-lws-operator`, RHOAI `stable-3.5` in `redhat-ods-operator` | | Monitoring | User Workload Monitoring turned on (`cluster-monitoring-config`). MaaS reports *Degraded* without it. | | Gateway | GatewayClass `openshift-default` (makes the cluster install Service Mesh 3/Istio), Certificate `maas-default-gateway-cert` (Let's Encrypt, `maas.apps…` only), Gateway `maas-default-gateway` using it, Route `maas-default-gateway-https` (passthrough) | | Auth and limits | `Kuadrant` and `Authorino` in `kuadrant-system`, with Authorino TLS using a certificate from OpenShift's internal service CA | | Database | Single-instance CloudNativePG cluster `maas-postgres` in `rhoai-maas-db`, plus the `maas-db-config` secret in `redhat-ai-gateway-infra` | | RHOAI | `DSCInitialization default-dsci` and `DataScienceCluster default-dsc` (minimal component set, below), with the dashboard's MaaS and external-models views turned on | | Model | ExternalProvider `llama-coder-provider`, ExternalModel and MaaSModelRef `qwen35-9b-coder` in `llm-coding-assistant` | | Access | MaaSAuthPolicy `llama-coder-access` and MaaSSubscription `llama-coder-default` in `models-as-a-service` | **DataScienceCluster components:** - **Managed:** `dashboard`, `kserve`, `aigateway` (with `modelsAsAService`), and `ogx` (Gen AI Studio). - **Removed:** everything else (pipelines, workbenches, model registry, Ray, TrustyAI, MLflow, Feast, Kueue, trainer, training operator). ### Why an external model, not KServe The cluster has one GPU: an RTX 4060 with 8 GB of VRAM and no sharing, already in use by llama-coder. Serving through KServe/vLLM would mean shutting down llama.cpp and dropping to a much smaller model. Using `ExternalModel` with `maas.opendatahub.io/tls: "false"` and `maas.opendatahub.io/port: "8080"` lets MaaS govern the existing server in place. ## Prerequisites - OpenShift 4.19.9 or later (the cluster runs 4.22) with cert-manager installed. - The `letsencrypt-production` ClusterIssuer (DNS-01). The role issues a dedicated cert for `maas.apps.openshift.toal.ca`. Don't put the `*.apps` wildcard on the gateway: browsers reuse an HTTP/2 connection for any host its certificate covers, so after you visit MaaS, the console and other `*.apps` sites would be sent to the MaaS gateway and get 404s. - llama-coder is running in `llm-coding-assistant`, with secret `llama-coder-api-key` (key `api-key`). - A `rhoai-deployer` ServiceAccount, provisioned with the permissions in [deploy_openshift.yml](playbooks/deploy_openshift.yml). Re-run this after any change to those permissions: ```bash ansible-navigator run playbooks/deploy_openshift.yml --tags sno_deploy_service_accounts ``` - Its token is referenced in the inventory at `host_vars/sno.openshift.toal.ca/hashivault_secrets.yml`: ```yaml vault_rhoai_deployer_token: "{{ lookup('community.hashi_vault.hashi_vault', 'kv/data/toallab/rhoai-deployer:token') }}" ``` ## Deploy ```bash source ~/.venv/ansible/bin/activate op run --env-file=$HOME/.ansible.zshenv -- \ ansible-navigator run playbooks/deploy_rhoai_maas.yml --mode stdout ``` A first run takes roughly 15–25 minutes, mostly waiting on operators. It runs these steps in order: 1. **CloudNativePG:** creates the `maas-postgres` cluster and its `maas` database. The password is generated once and stored in Vault at `kv/cloudnativepg/maas-postgres`. 2. **Operators:** installs Connectivity Link, Leader Worker Set, and RHOAI, waiting for each to finish installing. 3. **Platform:** - turns on User Workload Monitoring; - creates the GatewayClass, Gateway, and Route; - creates Kuadrant and restarts its operator once Istio exists; - turns on Authorino TLS. 4. **Database secret:** builds `maas-db-config` from the CloudNativePG credentials. 5. **RHOAI:** - applies the DSCInitialization and DataScienceCluster; - waits for the MaaS resource types to be registered and for `maas-api` to become available; - turns on the dashboard's MaaS views. 6. **Models:** - labels the model namespace and API-key secret; - creates the ExternalProvider, ExternalModel, and MaaSModelRef; - applies the auth policy and subscription. Each step is safe to re-run. If a run fails, fix the cause and run the playbook again; it continues from where things stand. ### Verify the deployment ```bash oc get datasciencecluster default-dsc -o jsonpath='{.status.phase}{"\n"}' # Ready oc get kuadrant kuadrant -n kuadrant-system \ -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}' # True oc get gateway maas-default-gateway -n openshift-ingress # PROGRAMMED True oc get deploy maas-api -n redhat-ai-gateway-infra # 1/1 oc get externalprovider,externalmodel,maasmodelref -n llm-coding-assistant oc get maasauthpolicy,maassubscription -n models-as-a-service curl -s -o /dev/null -w '%{http_code}\n' https://maas.apps.openshift.toal.ca/maas-api/health ``` ## Use it ### 1. Get an API key You must belong to a group named in a MaaSSubscription. The default subscription covers every authenticated user. **From the dashboard:** 1. Open **OpenShift AI → Gen AI studio → API keys → Create API key**. 2. Pick the `llama-coder default` subscription and an expiry (the default is 90 days). 3. Copy the key. It's shown **only once**. **From the CLI**, using your OpenShift login token: ```bash MAAS=https://maas.apps.openshift.toal.ca curl -s -X POST "$MAAS/maas-api/v1/api-keys" \ -H "Authorization: Bearer $(oc whoami -t)" \ -H "Content-Type: application/json" \ -d '{"name": "laptop", "expiresIn": "720h", "subscription": "llama-coder-default"}' | jq -r .key ``` An API key records your group memberships **when it is created**. If your groups change, revoke the key and create a new one. ### 2. Call the model The endpoint is OpenAI-compatible. Use the model name `qwen35-9b-coder`. ```bash export MAAS_URL=https://maas.apps.openshift.toal.ca export MAAS_API_KEY=sk-oai-... curl -s "$MAAS_URL/v1/models" -H "Authorization: Bearer $MAAS_API_KEY" curl -s "$MAAS_URL/v1/chat/completions" \ -H "Authorization: Bearer $MAAS_API_KEY" -H "Content-Type: application/json" \ -d '{"model": "qwen35-9b-coder", "messages": [{"role": "user", "content": "Write a haiku about Ansible"}]}' ``` Any OpenAI SDK or coding assistant works with these values: | Setting | Value | |---|---| | Base URL | `https://maas.apps.openshift.toal.ca/v1` | | API key | your `sk-oai-…` key | | Model | `qwen35-9b-coder` | ```python from openai import OpenAI client = OpenAI(base_url="https://maas.apps.openshift.toal.ca/v1", api_key="sk-oai-...") print(client.chat.completions.create( model="qwen35-9b-coder", messages=[{"role": "user", "content": "Hello"}], ).choices[0].message.content) ``` ### Limits - The default subscription allows each user **200,000 tokens per hour**. A user who exceeds it gets HTTP 429 until the hour window resets. - Token limits are counted only for OpenAI Chat Completions requests (`openai-chat`). - llama.cpp handles one model with a 48k-token context and queues requests on a single GPU. MaaS adds governance, not capacity. ## Administer All access settings live in role variables. Change them in the inventory or with `-e`, then re-run the playbook. The role creates and updates resources but never deletes them; see [Remove a model or policy](#remove-a-model-or-policy). ### Grant access to specific groups The OpenShift groups come from the Keycloak OIDC login. For example, to replace "all authenticated users" with two tiers: ```yaml rhoai_maas_auth_policies: - name: llama-coder-access groups: [developers, platform-admins] model_refs: - {name: qwen35-9b-coder, namespace: llm-coding-assistant} rhoai_maas_subscriptions: - name: llama-coder-standard display_name: Standard priority: 10 groups: [developers] model_refs: - name: qwen35-9b-coder namespace: llm-coding-assistant tokenRateLimits: [{limit: 100000, window: 1h}] - name: llama-coder-premium display_name: Premium priority: 20 # wins for users in both groups groups: [platform-admins] model_refs: - name: qwen35-9b-coder namespace: llm-coding-assistant tokenRateLimits: [{limit: 1000000, window: 1h}] ``` - An **auth policy** decides *who may call* a model. - A **subscription** decides *how many tokens* they get. A user needs both. `users:` (a list of usernames) works alongside `groups:` in both resources. ### Add another model Any OpenAI-compatible endpoint can be added to `rhoai_maas_external_models`. That includes another in-cluster server or a cloud provider. Then reference it from an auth policy and a subscription. ```yaml rhoai_maas_external_models: - name: qwen35-9b-coder # ... existing entry ... - name: gpt-4o-mini namespace: llm-coding-assistant display_name: GPT-4o mini (OpenAI) provider_name: openai-provider provider_type: openai # openai | anthropic | azure | vertex | bedrock endpoint: api.openai.com # hostname only, no scheme port: 443 tls: true api_key_secret: openai-api-key # create first: key "api-key" target_model: gpt-4o-mini api_format: openai-chat path: /v1/chat/completions ``` Rules for model entries: - **Names:** must be DNS-safe and must not collide with an existing Service in the namespace. The controller creates a Service for each provider and model. That's why the provider is called `llama-coder-provider` rather than `llama-coder`. - **Secrets:** the API-key secret must exist in the model's namespace before you run the playbook. The role only adds the `inference.llm-d.ai/ipp-managed` label to it. - **Shared provider limits:** for a cloud provider, every user shares the provider's key, so its rate limits apply to all MaaS users combined. ### Revoke keys Use **Gen AI studio → API keys** in the dashboard. Administrators can revoke one key or all of a user's keys. Revoke a user's keys after removing them from a group, because existing keys keep the old memberships. ### Remove a model or policy The role never deletes resources. Delete them by hand, then remove the entry from the variables: ```bash oc delete maassubscription -n models-as-a-service oc delete maasauthpolicy -n models-as-a-service oc delete maasmodelref,externalmodel -n llm-coding-assistant oc delete externalprovider -n llm-coding-assistant ``` ## Configuration reference All variables are in [roles/rhoai_maas/defaults/main.yml](roles/rhoai_maas/defaults/main.yml). [meta/argument_specs.yml](roles/rhoai_maas/meta/argument_specs.yml) documents each one. | Variable | Default | Notes | |---|---|---| | `rhoai_maas_operators` | RHCL `stable-1.4`, LWS `stable-v1.0`, RHOAI `stable-3.5` | RHOAI is pinned because the MaaS settings layout changed between 3.4 and 3.5 | | `rhoai_maas_hostname` | `""` → `maas.` | | | `rhoai_maas_gateway_cert_secret` | `maas-gateway-tls` | Single-host cert in `openshift-ingress`; never the `*.apps` wildcard | | `rhoai_maas_gateway_cert_issuer` | `letsencrypt-production` | ClusterIssuer for that cert; `""` to bring your own secret | | `rhoai_maas_gateway_service_type` | `ClusterIP` | Use `LoadBalancer` if MetalLB is added | | `rhoai_maas_gateway_create_route` | `true` | Passthrough Route on the default router | | `rhoai_maas_db_secret_name` / `_namespace` | `maas-postgres-secret` / `rhoai-maas-db` | Basic-auth secret from CloudNativePG | | `rhoai_maas_db_sslmode` | `require` | | | `rhoai_maas_dsc_managed_components` / `_removed_components` | see above | | | `rhoai_maas_external_models` | llama-coder | | | `rhoai_maas_auth_policies` | all authenticated users | | | `rhoai_maas_subscriptions` | 200k tokens/h per user | | | `rhoai_maas_wait_timeout` | `900` | Seconds, per wait | The database is sized in the playbook's `cloudnativepg_operator` variables: 1 instance, 5 Gi, 100 connections. ## Operations ### Troubleshooting | Symptom | Cause and fix | |---|---| | Run hangs at **Wait for MaaS CRDs**, and the DSC shows `ModelsAsAServiceReady=False … Removed` | MaaS isn't turned on. In 3.5 the field is `aigateway.modelsAsAService` (note "AsA"). The older `modelsAsService` spelling is silently dropped by the API server. Check with `oc get dsc default-dsc -o jsonpath='{.spec.components.aigateway}'`. | | Run stops at **Wait for Kuadrant to be ready**, and Kuadrant shows `MissingDependency` | The Kuadrant operator started before Istio existed. The role restarts it automatically. To fix it by hand: `oc delete pod -n openshift-operators -l app=kuadrant,control-plane=controller-manager` | | Gateway not `Programmed` | Check `oc describe gateway maas-default-gateway -n openshift-ingress`. `LoadBalancerPending` and `DNSReady=False` are expected with ClusterIP plus a Route and do no harm. | | `maas-api` never appears | Check `oc get datasciencecluster default-dsc -o yaml` (status conditions) and that `maas-db-config` exists in `redhat-ai-gateway-infra`. | | DSC shows *Degraded*, mentioning monitoring | User Workload Monitoring is off. Check `cluster-monitoring-config`. | | `401` from `/v1/...` | The key is missing, expired, or revoked, or you used an OpenShift token instead of an `sk-oai-` key. | | `403` | No MaaSAuthPolicy grants your groups that model. Check that `model_refs` names and namespaces match. | | `429` | Subscription token limit reached. Wait for the window to reset, or raise `tokenRateLimits`. | | `ExternalProvider` not `Ready` | Check the API-key secret exists with key `api-key`, and that the Service, ServiceEntry, and DestinationRule were created in the model namespace. | | Long prompts fail or are cut off | Requests over 16 KB can be truncated on Service Mesh before 3.3.1. Check the installed version with `oc get csv -n openshift-operators \| grep servicemesh`. | | Model timeouts under load | llama.cpp is serving on one GPU. Check `oc logs -n llm-coding-assistant deploy/llama-coder`. | ### Usage metrics MaaS exports token and request metrics to User Workload Monitoring. Query them under **Observe → Metrics** in the console, or use the dashboard's MaaS views. The full observability stack (Tempo, OpenTelemetry, Loki) isn't deployed here. See phase 7 of the [companion guide](https://github.com/jharmison-redhat/rhoai-maas-guide). ### Upgrades RHOAI follows the `stable-3.5` channel with Automatic approval, so 3.5.z updates install on their own. Moving to 3.6 or later means changing the channel in `rhoai_maas_operators`, after checking the release notes. MaaS settings have changed between minor versions before: 3.4 configured MaaS under `kserve`, 3.5 moved it to `aigateway`. ### Remove Removing MaaS deletes the database and every API key. Work in this order: ```bash oc delete maassubscription,maasauthpolicy --all -n models-as-a-service oc delete maasmodelref,externalmodel,externalprovider --all -n llm-coding-assistant oc delete datasciencecluster default-dsc oc delete dscinitialization default-dsci oc delete gateway maas-default-gateway -n openshift-ingress oc delete route maas-default-gateway-https -n openshift-ingress oc delete kuadrant kuadrant -n kuadrant-system oc delete cluster.postgresql.cnpg.io maas-postgres -n rhoai-maas-db # then uninstall the operators (Subscriptions + CSVs) from the console ``` Leave the `openshift-default` GatewayClass in place if anything else uses Gateway API.