toallab-automation/OPA_POLICY_ENFORCEMENT.md

OPA Policy Enforcement for AAP

This deploys an Open Policy Agent (OPA) server on the OpenShift cluster and connects Ansible Automation Platform 2.7 to it. Once connected, AAP asks OPA whether each job may run before launching it.

• Playbook: playbooks/deploy_opa.yml
• Role: roles/opa_server/
• Policies: roles/opa_server/files/policies/
• Policy tests: roles/opa_server/files/tests/

How it works

 job launch ──► AAP controller ──HTTPS + bearer token──► OPA (opa.opa.svc:8443)
                      │                                         │
                      │   POST /v1/data/<query path>            │  Rego policies
                      │   {"input": <serialized job>}           │  + data.aap_policy_config
                      ◄──── {"allowed": bool, "violations": [...]}

1. A job launches for a job template. That template, its inventory, or its organization has an OPA query path set (for example aap/policy/decision).
2. AAP sends the job's details to OPA once for each of those resources that has a query path.
3. OPA evaluates the Rego policy at that path and returns allowed plus a list of violations.
4. If any policy denies the job, AAP blocks it and shows the violation messages on the job.

AAP fails closed. Once a resource has a query path, the job is also blocked when:

• OPA can't be reached or times out;
• the path doesn't exist in OPA;
• the response isn't in the {allowed, violations} shape.

Resources with no query path are never checked.

What gets deployed

Resource	Purpose
Namespace opa	Holds the OPA server
Deployment opa	OPA v1.21.1 (docker.io/openpolicyagent/opa:1.21.1-static), non-root, read-only filesystem
Service opa (port 8443)	HTTPS, using a certificate issued by OpenShift's internal service CA
ConfigMap opa-policies	Every *.rego file in the policy directory, plus aap_policy_config.json
Secret opa-authz	The bearer token, which the server's authorization policy checks
AAP setting changes	The OPA_* settings under Settings → Policy as Code

Security model

• TLS: OPA serves TLS with a certificate from OpenShift's internal service CA. AAP is given that CA bundle so it can verify the server.
• Authentication: every request except GET /health needs the bearer token. This is enforced by opa run --authentication=token --authorization=basic together with authz.rego.
• Token storage: the token is generated on the first run and stored in Vault at kv/toallab/opa-server (key token). Later runs reuse it.
• Decision logs: every decision is written to the OPA pod log for auditing, including the job's extra vars. Turn this off with opa_server_decision_logs: false.

Prerequisites

• AAP is deployed on the cluster (deploy_aap.yml).

• The aap host has aap_gateway_url and aap_k8s_api_url set in the inventory.

• vault_aap_deployer_token resolves, and the aap-deployer ServiceAccount has the current permissions. The playbook needs to create ConfigMaps, Services, and Deployments, which were added for this. Re-run this after any change to those permissions:

  ansible-navigator run playbooks/deploy_openshift.yml --tags sno_deploy_service_accounts
  

Deploy

source ~/.venv/ansible/bin/activate
op run --env-file=$HOME/.ansible.zshenv -- \
  ansible-navigator run playbooks/deploy_opa.yml --mode stdout

The playbook:

1. Reads the token from Vault, or generates and stores one.
2. Applies the namespace, policy ConfigMap, token Secret, Service, and Deployment.
3. Waits for the TLS certificate and for the pod to become available.
4. Updates AAP's OPA_* settings.
5. Attaches any query paths listed in opa_server_aap_query_paths. This list is empty by default.

Re-running is safe. Pods restart only when the policies, policy config, or token change.

Verify the deployment

oc get pods -n opa
oc logs -n opa deploy/opa | head

# Query OPA directly
oc port-forward -n opa svc/opa 8443:8443 &
TOKEN=$(vault kv get -field=token kv/toallab/opa-server)

curl -sk https://localhost:8443/health                       # {} (no token needed)
curl -sk -o /dev/null -w '%{http_code}\n' https://localhost:8443/v1/data   # 401
curl -sk -H "Authorization: Bearer $TOKEN" \
  -d '{"input":{"job_type":"run","execution_environment":{"image":"quay.io/x/ee:latest"},"extra_vars":{}}}' \
  https://localhost:8443/v1/data/aap/policy/decision

In AAP, open Settings → Policy as Code. You should see:

• OPA host opa.opa.svc
• port 8443
• SSL enabled
• auth type Token

Use it

Available query paths

Query path	Behavior
aap/smoke/allow_all	Always allows. Use it to attach safely and confirm AAP can reach OPA.
aap/smoke/deny_all	Always denies. Use it to confirm enforcement is active.
aap/policy/decision	The baseline policy described below.

Baseline policy (aap/policy/decision)

aap_policy.rego reports these violations:

Rule	Denies when	Setting in opa_server_policy_config
Approved registries	The execution environment image doesn't start with an allowed prefix	allowed_ee_registries (default: aap.toal.ca/, registry.redhat.io/)
No credentials in extra vars	A launch-time extra var matches a forbidden name (case-insensitive)	forbidden_extra_vars (default: ansible_password, ansible_become_password, ansible_ssh_pass, ansible_become_pass)
Change freeze	change_freeze is true and the job runs in run mode (not check mode), unless the launcher is a superuser or in an exempt team	change_freeze, change_freeze_exempt_teams

An empty allowed_ee_registries list turns the registry check off.

Scheduled jobs have no launching user, so a change freeze blocks them too.

The Cloud Services Execution Environment (quay.io/scottharwell/cloud-ee) fails the registry rule as configured. That makes it a convenient demo.

Attach a policy

Attach the smoke test first. Then switch to the real policy once AAP shows it reaching OPA.

1. Confirm AAP can reach OPA. The job should run normally:

   ansible-navigator run playbooks/deploy_opa.yml --mode stdout -e '{
     "opa_server_aap_query_paths": [
       {"type": "job_templates", "name": "Demo Job Template", "query_path": "aap/smoke/allow_all"}
     ]}'
   

2. Confirm enforcement. Switch the path to aap/smoke/deny_all, then launch the template. The job should fail with aap.smoke.deny_all: policy enforcement is active.

3. Turn on the real policy by switching the path to aap/policy/decision.

type is one of organizations, inventories, or job_templates. name must match exactly one resource.

To keep the attachments permanently, put opa_server_aap_query_paths in the aap host's host_vars instead of passing -e. You can also set the path by hand on the resource's edit form in the AAP UI, or with the API field opa_query_path on /api/controller/v2/{organizations,inventories,job_templates}/<id>/.

Detach a policy

Set the query path to an empty string:

-e '{"opa_server_aap_query_paths": [{"type": "job_templates", "name": "Demo Job Template", "query_path": ""}]}'

Removing an entry from opa_server_aap_query_paths does not detach it. The role only sets the paths it's given.

Change the policy config

Override opa_server_policy_config in host_vars or with -e, then re-run the playbook. For example, to start a change freeze:

opa_server_policy_config:
  allowed_ee_registries: [aap.toal.ca/, registry.redhat.io/]
  forbidden_extra_vars: [ansible_password, ansible_become_password, ansible_ssh_pass, ansible_become_pass]
  change_freeze: true
  change_freeze_exempt_teams: [Platform]

Lists replace the defaults rather than merging with them, so give the whole object.

Write your own policies

AAP needs the rule at the query path to return exactly this shape:

{"allowed": true, "violations": []}

violations is a list of strings shown to the user. It's ignored when allowed is true.

The input AAP sends

input is the serialized job (AWX policy.py):

Field	Contents
id, name, created, job_type (run/check), launch_type, launched_by	Job basics
limit, forks, playbook, scm_branch, scm_revision, hosts_count	Run details
extra_vars	Launch extra vars (survey passwords masked)
created_by	{id, username, is_superuser, teams: [{id, name}]}; null for scheduled jobs
execution_environment	{id, name, image, pull}
inventory	{id, name, kind, total_hosts, has_active_failures, inventory_sources, ...}
job_template	{id, name, job_type}
organization	{id, name}
project	{id, name, scm_type, scm_url, scm_branch, ...}
credentials	[{id, name, credential_type, kind, cloud, kubernetes, organization, ...}]
labels	[{id, name, organization}]
instance_group	{id, name, capacity, max_forks, ...}
workflow_job, workflow_job_template	{id, name, ...}, or null if not run from a workflow

Add a policy

1. Add a .rego file to roles/opa_server/files/policies/. Use import rego.v1; OPA 1.x requires Rego v1 syntax. The package determines the query path: package aap.network maps to aap/network/<rule>.

   package aap.network
   
   import rego.v1
   
   decision := {"allowed": count(violations) == 0, "violations": [m | some m in violations]}
   
   violations contains "Production jobs require a --limit" if {
       input.inventory.name == "Production"
       input.limit == ""
   }
   

2. Add tests to roles/opa_server/files/tests/ and run them:

   podman run --rm -v ./roles/opa_server/files:/files:Z \
     docker.io/openpolicyagent/opa:1.21.1-static test /files/policies /files/tests -v
   
   podman run --rm -v ./roles/opa_server/files:/files:Z \
     docker.io/openpolicyagent/opa:1.21.1-static check --strict /files/policies
   

3. Re-run the playbook. The ConfigMap is replaced in full, so deleted .rego files are removed too, and the pod restarts to load the changes.

Reference data you need in a policy through data.aap_policy_config. To read data, define a defaulted rule such as default user_config := {} and assign data.aap_policy_config to it. Don't use object.get(data, ...): referencing all of data from inside a policy is treated as recursion.

Configuration reference

All variables are in roles/opa_server/defaults/main.yml. meta/argument_specs.yml documents each one.

Variable	Default	Notes
opa_server_namespace	opa
opa_server_image	docker.io/openpolicyagent/opa:1.21.1-static	Pin exact versions
opa_server_replicas	1	Increase for availability (AAP fails closed)
opa_server_port	8443
opa_server_decision_logs	true	Logs inputs, including extra vars
opa_server_policy_dir	files/policies in the role
opa_server_policy_config	see above	Exposed as data.aap_policy_config
opa_server_vault_path	toallab/opa-server	Mount kv
opa_server_configure_aap	true	false deploys OPA only
opa_server_aap_request_timeout / _retries	1.5 / 2	AAP's OPA_REQUEST_TIMEOUT / OPA_REQUEST_RETRIES
opa_server_aap_query_paths	[]	Attachments

Operations

Rotate the token

vault kv delete kv/toallab/opa-server
ansible-navigator run playbooks/deploy_opa.yml --mode stdout

The playbook generates a new token, updates the Secret (which restarts the pod), and updates AAP's OPA_AUTH_TOKEN.

Warning: for a few seconds AAP and OPA hold different tokens. Jobs on resources with a policy attached fail during that window, so rotate outside busy periods.

Troubleshooting

Symptom on the AAP job	Cause
Call to OPA failed ... 401	AAP's token doesn't match the Secret. Re-run the playbook. If that doesn't fix it, delete the Vault key to force a new token.
Call to OPA failed ... Connection or a timeout	The OPA pod is down. Check oc get pods -n opa.
did not return a "result" property	The query path doesn't exist. Check the package and rule name, and use / or . as separators.
OPA policy returned invalid result	The rule doesn't return {allowed: bool, violations: [string]}.
A job runs when it should be blocked	No query path is attached to its organization, inventory, or template, or the rule returned allowed: true.

See exactly what OPA decided:

oc logs -n opa deploy/opa | grep '"decision_id"' | tail -5

Remove

Detach every query path first. Otherwise jobs on those resources fail once OPA is gone.

# In AAP: Settings → Policy as Code → clear OPA host (an empty host disables evaluation)
oc delete namespace opa
vault kv metadata delete kv/toallab/opa-server