toallab-automation/OPA_POLICY_ENFORCEMENT.md

# OPA Policy Enforcement for AAP

This deploys an [Open Policy Agent](https://www.openpolicyagent.org/) (OPA) server on the
OpenShift cluster and connects Ansible Automation Platform 2.7 to it. Once connected, AAP
asks OPA whether each job may run before launching it.

- Playbook: [playbooks/deploy_opa.yml](playbooks/deploy_opa.yml)
- Role: [roles/opa_server/](roles/opa_server/)
- Policies: [roles/opa_server/files/policies/](roles/opa_server/files/policies/)
- Policy tests: [roles/opa_server/files/tests/](roles/opa_server/files/tests/)

## How it works

```
 job launch ──► AAP controller ──HTTPS + bearer token──► OPA (opa.opa.svc:8443)
                      │                                         │
                      │   POST /v1/data/<query path>            │  Rego policies
                      │   {"input": <serialized job>}           │  + data.aap_policy_config
                      ◄──── {"allowed": bool, "violations": [...]}
```

1. A job launches for a job template. That template, its inventory, or its organization has an
   **OPA query path** set (for example `aap/policy/decision`).
2. AAP sends the job's details to OPA once for each of those resources that has a query path.
3. OPA evaluates the Rego policy at that path and returns `allowed` plus a list of
   `violations`.
4. If any policy denies the job, AAP blocks it and shows the violation messages on the job.

**AAP fails closed.** Once a resource has a query path, the job is also blocked when:
- OPA can't be reached or times out;
- the path doesn't exist in OPA;
- the response isn't in the `{allowed, violations}` shape.

Resources with no query path are never checked.

### What gets deployed

| Resource | Purpose |
|---|---|
| Namespace `opa` | Holds the OPA server |
| Deployment `opa` | OPA v1.21.1 (`docker.io/openpolicyagent/opa:1.21.1-static`), non-root, read-only filesystem |
| Service `opa` (port 8443) | HTTPS, using a certificate issued by OpenShift's internal service CA |
| ConfigMap `opa-policies` | Every `*.rego` file in the policy directory, plus `aap_policy_config.json` |
| Secret `opa-authz` | The bearer token, which the server's authorization policy checks |
| AAP setting changes | The `OPA_*` settings under **Settings → Policy as Code** |

### Security model

- **TLS:** OPA serves TLS with a certificate from OpenShift's internal service CA. AAP is given
  that CA bundle so it can verify the server.
- **Authentication:** every request except `GET /health` needs the bearer token. This is
  enforced by `opa run --authentication=token --authorization=basic` together with
  [authz.rego](roles/opa_server/files/policies/authz.rego).
- **Token storage:** the token is generated on the first run and stored in Vault at
  `kv/toallab/opa-server` (key `token`). Later runs reuse it.
- **Decision logs:** every decision is written to the OPA pod log for auditing, including the
  job's extra vars. Turn this off with `opa_server_decision_logs: false`.

## Prerequisites

- AAP is deployed on the cluster ([deploy_aap.yml](playbooks/deploy_aap.yml)).
- The `aap` host has `aap_gateway_url` and `aap_k8s_api_url` set in the inventory.
- `vault_aap_deployer_token` resolves, and the `aap-deployer` ServiceAccount has the current
  permissions. The playbook needs to create ConfigMaps, Services, and Deployments, which were
  added for this. Re-run this after any change to those permissions:

  ```bash
  ansible-navigator run playbooks/deploy_openshift.yml --tags sno_deploy_service_accounts
  ```

## Deploy

```bash
source ~/.venv/ansible/bin/activate
op run --env-file=$HOME/.ansible.zshenv -- \
  ansible-navigator run playbooks/deploy_opa.yml --mode stdout
```

The playbook:

1. Reads the token from Vault, or generates and stores one.
2. Applies the namespace, policy ConfigMap, token Secret, Service, and Deployment.
3. Waits for the TLS certificate and for the pod to become available.
4. Updates AAP's `OPA_*` settings.
5. Attaches any query paths listed in `opa_server_aap_query_paths`. This list is empty by
   default.

Re-running is safe. Pods restart only when the policies, policy config, or token change.

### Verify the deployment

```bash
oc get pods -n opa
oc logs -n opa deploy/opa | head

# Query OPA directly
oc port-forward -n opa svc/opa 8443:8443 &
TOKEN=$(vault kv get -field=token kv/toallab/opa-server)

curl -sk https://localhost:8443/health                       # {} (no token needed)
curl -sk -o /dev/null -w '%{http_code}\n' https://localhost:8443/v1/data   # 401
curl -sk -H "Authorization: Bearer $TOKEN" \
  -d '{"input":{"job_type":"run","execution_environment":{"image":"quay.io/x/ee:latest"},"extra_vars":{}}}' \
  https://localhost:8443/v1/data/aap/policy/decision
```

In AAP, open **Settings → Policy as Code**. You should see:
- OPA host `opa.opa.svc`
- port `8443`
- SSL enabled
- auth type `Token`

## Use it

### Available query paths

| Query path | Behavior |
|---|---|
| `aap/smoke/allow_all` | Always allows. Use it to attach safely and confirm AAP can reach OPA. |
| `aap/smoke/deny_all` | Always denies. Use it to confirm enforcement is active. |
| `aap/policy/decision` | The baseline policy described below. |

### Baseline policy (`aap/policy/decision`)

[aap_policy.rego](roles/opa_server/files/policies/aap_policy.rego) reports these violations:

| Rule | Denies when | Setting in `opa_server_policy_config` |
|---|---|---|
| Approved registries | The execution environment image doesn't start with an allowed prefix | `allowed_ee_registries` (default: `aap.toal.ca/`, `registry.redhat.io/`) |
| No credentials in extra vars | A launch-time extra var matches a forbidden name (case-insensitive) | `forbidden_extra_vars` (default: `ansible_password`, `ansible_become_password`, `ansible_ssh_pass`, `ansible_become_pass`) |
| Change freeze | `change_freeze` is true and the job runs in `run` mode (not check mode), unless the launcher is a superuser or in an exempt team | `change_freeze`, `change_freeze_exempt_teams` |

An empty `allowed_ee_registries` list turns the registry check off.

Scheduled jobs have no launching user, so a change freeze blocks them too.

The **Cloud Services Execution Environment** (`quay.io/scottharwell/cloud-ee`) fails the
registry rule as configured. That makes it a convenient demo.

### Attach a policy

Attach the smoke test first. Then switch to the real policy once AAP shows it reaching OPA.

1. Confirm AAP can reach OPA. The job should run normally:

   ```bash
   ansible-navigator run playbooks/deploy_opa.yml --mode stdout -e '{
     "opa_server_aap_query_paths": [
       {"type": "job_templates", "name": "Demo Job Template", "query_path": "aap/smoke/allow_all"}
     ]}'
   ```

2. Confirm enforcement. Switch the path to `aap/smoke/deny_all`, then launch the template.
   The job should fail with
   `aap.smoke.deny_all: policy enforcement is active`.

3. Turn on the real policy by switching the path to `aap/policy/decision`.

`type` is one of `organizations`, `inventories`, or `job_templates`. `name` must match
exactly one resource.

To keep the attachments permanently, put `opa_server_aap_query_paths` in the `aap` host's
`host_vars` instead of passing `-e`. You can also set the path by hand on the resource's edit form in the AAP UI, or with the
API field `opa_query_path` on `/api/controller/v2/{organizations,inventories,job_templates}/<id>/`.

### Detach a policy

Set the query path to an empty string:

```bash
-e '{"opa_server_aap_query_paths": [{"type": "job_templates", "name": "Demo Job Template", "query_path": ""}]}'
```

Removing an entry from `opa_server_aap_query_paths` does **not** detach it. The role only
sets the paths it's given.

### Change the policy config

Override `opa_server_policy_config` in `host_vars` or with `-e`, then re-run the playbook.
For example, to start a change freeze:

```yaml
opa_server_policy_config:
  allowed_ee_registries: [aap.toal.ca/, registry.redhat.io/]
  forbidden_extra_vars: [ansible_password, ansible_become_password, ansible_ssh_pass, ansible_become_pass]
  change_freeze: true
  change_freeze_exempt_teams: [Platform]
```

Lists replace the defaults rather than merging with them, so give the whole object.

## Write your own policies

AAP needs the rule at the query path to return exactly this shape:

```json
{"allowed": true, "violations": []}
```

`violations` is a list of strings shown to the user. It's ignored when `allowed` is true.

### The input AAP sends

`input` is the serialized job ([AWX policy.py](https://github.com/ansible/awx/blob/devel/awx/main/tasks/policy.py)):

| Field | Contents |
|---|---|
| `id`, `name`, `created`, `job_type` (`run`/`check`), `launch_type`, `launched_by` | Job basics |
| `limit`, `forks`, `playbook`, `scm_branch`, `scm_revision`, `hosts_count` | Run details |
| `extra_vars` | Launch extra vars (survey passwords masked) |
| `created_by` | `{id, username, is_superuser, teams: [{id, name}]}`; `null` for scheduled jobs |
| `execution_environment` | `{id, name, image, pull}` |
| `inventory` | `{id, name, kind, total_hosts, has_active_failures, inventory_sources, ...}` |
| `job_template` | `{id, name, job_type}` |
| `organization` | `{id, name}` |
| `project` | `{id, name, scm_type, scm_url, scm_branch, ...}` |
| `credentials` | `[{id, name, credential_type, kind, cloud, kubernetes, organization, ...}]` |
| `labels` | `[{id, name, organization}]` |
| `instance_group` | `{id, name, capacity, max_forks, ...}` |
| `workflow_job`, `workflow_job_template` | `{id, name, ...}`, or `null` if not run from a workflow |

### Add a policy

1. Add a `.rego` file to [roles/opa_server/files/policies/](roles/opa_server/files/policies/).
   Use `import rego.v1`; OPA 1.x requires Rego v1 syntax. The package determines the query
   path: `package aap.network` maps to `aap/network/<rule>`.

   ```rego
   package aap.network

   import rego.v1

   decision := {"allowed": count(violations) == 0, "violations": [m | some m in violations]}

   violations contains "Production jobs require a --limit" if {
       input.inventory.name == "Production"
       input.limit == ""
   }
   ```

2. Add tests to [roles/opa_server/files/tests/](roles/opa_server/files/tests/) and run them:

   ```bash
   podman run --rm -v ./roles/opa_server/files:/files:Z \
     docker.io/openpolicyagent/opa:1.21.1-static test /files/policies /files/tests -v

   podman run --rm -v ./roles/opa_server/files:/files:Z \
     docker.io/openpolicyagent/opa:1.21.1-static check --strict /files/policies
   ```

3. Re-run the playbook. The ConfigMap is replaced in full, so deleted `.rego` files are
   removed too, and the pod restarts to load the changes.

Reference data you need in a policy through `data.aap_policy_config`. To read data, define a
defaulted rule such as `default user_config := {}` and assign `data.aap_policy_config` to it.
Don't use `object.get(data, ...)`: referencing all of `data` from inside a policy is treated
as recursion.

## Configuration reference

All variables are in [roles/opa_server/defaults/main.yml](roles/opa_server/defaults/main.yml).
[meta/argument_specs.yml](roles/opa_server/meta/argument_specs.yml) documents each one.

| Variable | Default | Notes |
|---|---|---|
| `opa_server_namespace` | `opa` | |
| `opa_server_image` | `docker.io/openpolicyagent/opa:1.21.1-static` | Pin exact versions |
| `opa_server_replicas` | `1` | Increase for availability (AAP fails closed) |
| `opa_server_port` | `8443` | |
| `opa_server_decision_logs` | `true` | Logs inputs, including extra vars |
| `opa_server_policy_dir` | `files/policies` in the role | |
| `opa_server_policy_config` | see above | Exposed as `data.aap_policy_config` |
| `opa_server_vault_path` | `toallab/opa-server` | Mount `kv` |
| `opa_server_configure_aap` | `true` | `false` deploys OPA only |
| `opa_server_aap_request_timeout` / `_retries` | `1.5` / `2` | AAP's `OPA_REQUEST_TIMEOUT` / `OPA_REQUEST_RETRIES` |
| `opa_server_aap_query_paths` | `[]` | Attachments |

## Operations

### Rotate the token

```bash
vault kv delete kv/toallab/opa-server
ansible-navigator run playbooks/deploy_opa.yml --mode stdout
```

The playbook generates a new token, updates the Secret (which restarts the pod), and updates
AAP's `OPA_AUTH_TOKEN`.

> **Warning:** for a few seconds AAP and OPA hold different tokens. Jobs on resources with a
> policy attached fail during that window, so rotate outside busy periods.

### Troubleshooting

| Symptom on the AAP job | Cause |
|---|---|
| `Call to OPA failed ... 401` | AAP's token doesn't match the Secret. Re-run the playbook. If that doesn't fix it, delete the Vault key to force a new token. |
| `Call to OPA failed ... Connection` or a timeout | The OPA pod is down. Check `oc get pods -n opa`. |
| `did not return a "result" property` | The query path doesn't exist. Check the package and rule name, and use `/` or `.` as separators. |
| `OPA policy returned invalid result` | The rule doesn't return `{allowed: bool, violations: [string]}`. |
| A job runs when it should be blocked | No query path is attached to its organization, inventory, or template, or the rule returned `allowed: true`. |

See exactly what OPA decided:

```bash
oc logs -n opa deploy/opa | grep '"decision_id"' | tail -5
```

### Remove

Detach every query path first. Otherwise jobs on those resources fail once OPA is gone.

```bash
# In AAP: Settings → Policy as Code → clear OPA host (an empty host disables evaluation)
oc delete namespace opa
vault kv metadata delete kv/toallab/opa-server
```