SPEC-TF-005 — Terraform-Driven MkDocs Documentation Generation

Status: Proposed — v1.0.0-draft Owner: Documentation generation pipeline (SPEC-TF-004 continuation) Depends on: SPEC-TF-004 (output contract), ADR-TF-008/009 (contract drift / page lifecycle), ADR-TF-003 (local backend) Related: SPEC-TF-001 (scaffold), mkdocs-material, SAP BTP Terraform provider ≥1.14

1. Purpose

SPEC-TF-005 specifies the end-to-end process that converts live Terraform state in the environments/dev root module into rendered MkDocs Material documentation, and (optionally) copies that site to the configured media server. It is the authoritative reference for the make docs-* targets, the scripts/generate_docs.py generator, and the deployment path — the intended replacement for ad-hoc documentation updates.

The process is deterministic and idempotent: given the same Terraform output, it produces the same set of pages every time, overwrites stale on-disk clones, and never deletes a nav-referenced page simply because its section happens to be empty in the current state.

2. Scope

In scope:

The environments/dev root module as the generation source of truth (the only Terraform backend in this project at this stage).
The globalaccount_metadata consolidated output as the single contract surface between Terraform and the generator.
The scripts/generate_docs.py Python generator and every page type it currently renders for the Vine House Solutions Ltd global account.
The mkdocs build invocation and the optional scp copy to the media server (make docs-deploy).
The behaviour required when a wired data source returns no rows (the "No data available" Material alert requirement, ADR-TF-009 §5).

Out of scope (for now):

Other global accounts or subaccounts beyond Vine House Solutions Ltd and its shared subaccount.
ReadTheDocs or any hosted deployment other than the media server scp.
Static analysis, linting, or testing of the generated Markdown content beyond what mkdocs build --strict already enforces.

3. Pipeline phases

The full pipeline has four sequential phases. Each phase is a discrete, verifiable step; any phase may be run in isolation for debugging.

Phase 0 — Prerequisites

Before any phase runs the project root must satisfy:

make (GNU Make) on PATH.
python3 + a project virtual environment at .venv/ containing mkdocs>=1.6,<2 and mkdocs-material>=9.0 (see requirements.txt).
scp on PATH (the docs-deploy phase requires it; the earlier rsync dependency was replaced by scp because the agent Docker image did not include rsync and installing it was blocked by root-only apt access).
terraform ≥1.14 on PATH (or invocable via /tmp/terraform) for the extract phase when run inline rather than through make docs-extract.

Phase 1 — Extract (INFO+Tf state → JSON)

Command: make docs-extract or equivalently:

cd environments/dev
terraform refresh -input=false
terraform output -json globalaccount_metadata \
    > ../../.cache/globalaccount_metadata.json

Effect: refreshes the environments/dev state against the live BTP API and writes the consolidated globalaccount_metadata output into the untracked cache at .cache/globalaccount_metadata.json.

The cache file is untracked (.gitignore), regenerated on every make docs run, and never edited by hand.

Phase 2 — Generate (JSON → Markdown pages)

Command: make docs-generate or equivalently:

python3 scripts/generate_docs.py \
    --input .cache/globalaccount_metadata.json \
    --output docs/ \
    --ga-name "Vine House Solutions Ltd"

Effect: reads the cache JSON, validates it against the generator contract (§6), and writes a set of MyST Markdown pages into docs/globalaccounts/<slug>/ where <slug> is the slugified GA name.

The generator is destructive-over-generative for the pages it knows about: if a page exists on disk from a previous run and the generator no longer emits that section, the stale page is overwritten by the new output. This prevents orphaned pages (§7).

Phase 3 — Build (Markdown + mkdocs.yml → static site)

Command: make docs-build or equivalently:

.make/bin/python3 -m mkdocs build --strict --site-dir site

Effect: MkDocs Material renders docs/ into site/ using the project's mkdocs.yml. The --strict flag turns Markdown warnings into build failures so the pipeline does not ship a site with broken references.

site/ is a build artifact; it is not committed and is overwritten on every build.

Phase 4 — Deploy (site/ → media server, optional)

Command: make docs-deploy (depends on docs-build) or equivalently:

python3 scripts/generate_docs.py --input ... --output ... --ga-name ... --deploy

Effect: after a successful build, copies site/ to the remote host via scp -r using the deployment environment variables (§8).

This phase is conditional: when DEPLOY_REMOTE_HOST is unset the deploy step prints a notice and skips rather than failing.

4. Data flow

BTP API
  │
  ▼
environments/dev/terraform.tfstate   ← terraform refresh (Phase 1)
  │
  ▼
terraform output -json globalaccount_metadata
  │
  ▼
.cache/globalaccount_metadata.json     ← single contract JSON (untracked)
  │
  ▼
scripts/generate_docs.py  ──▶  docs/globalaccounts/Vine_House_Solutions_Ltd/*.md
  │                             (Phase 2; contract-validated)
  ▼
mkdocs build --strict --site-dir site  ← Phase 3
  │
  ▼
scp -r site/ roger@media:/config/website/content/  ← Phase 4 (optional)

Every downstream stage consumes only what the stage before it produced. The generator never reaches directly into Terraform state; it only reads the cache JSON.

5. Page inventory

All pages below are generated into docs/globalaccounts/Vine_House_Solutions_Ltd/ (or the equivalent slug for the named GA). Pages marked [sub] are written under a subaccounts/<id>/ directory rather than the GA root. Pages marked [dir] are written under a directories/ directory in the GA root.

Page file	Section key in metadata	Content	Source data
`index.md`	`identity`	Global Account identity overview, generated-at timestamp	`globalaccount_metadata.identity`
`architecture.md`	(static)	Static architecture doc, not generated per-run	written once by scaffold; not regenerated
`operations.md`	(static)	Static operations runbook, not generated per-run	written once by scaffold
`account-overview.md`	`identity`	Expanded GA overview with display name, subdomain, ID	`globalaccount_metadata.identity`
`users.md`	`users`	List of users on the default IdP	`globalaccount_users.this` → `local.globalaccount_users`
`user-details.md`	`user_details`	Per-user detail table for all users across every IdP (concat of `globalaccount_user_details` + `globalaccount_arsucc_user_details`)	`btp_globalaccount_user.*` per-user lookups
`arsucc-users.md`	`arsucc_users`	Users on the Platform-type trusted IdP, with per-user details	`globalaccount_users.arsucc` + `globalaccount_arsucc_user_details`
`roles.md`	`roles`	Role collections available in the GA	`globalaccount_role_collections`
`role-inventory.md`	`role_inventory`	Per-role detail listing	`globalaccount_roles`
`entitlements.md`	`entitlements`	Global Account-level entitlements	`globalaccount_entitlements`
`trust-configurations.md`	`trust_configurations`	Active trust configurations (IdPs) with dynamic type/provider detection	`globalaccount_trust_configurations` (now includes `type`)
`platform-users.md`	(no section)	Pre-existing artifact; renders empty because `platform_users` is not in metadata. Not in nav. Not from this generator.	—
`subaccounts/index.md`	`subaccounts`	Index of subaccounts under this GA	`globalaccount_metadata.subaccounts`
`subaccounts/<id>/subaccount-info.md` [sub]	per-subaccount	Subaccount identity + type + URLs	`btp_subaccount` data source
`subaccounts/<id>/subaccount-entitlements.md` [sub]	(empty → alert)	Entitlements for the subaccount, or "No data available" alert if empty	`btp_subaccount_entitlements`
`subaccounts/<id>/subaccount-labels.md` [sub]	(empty → alert)	Labels for the subaccount, or "No data available" alert if empty	`btp_subaccount_labels`
`directories/index.md` [dir]	`directory_labels` (optional)	Resource labels of every directory in the hierarchy, grouped per directory, one `\\| Key \\| Value \\|` row per (directory, key, value) pair, sorted by directory name then key then value; Total labels counts individual label values, not rows. Renders the raw directory ID as the group heading where the hierarchy walk supplied no display name. Falls back to a "No data available" alert when the section is empty or absent	`btp_directory_labels`

The generator unconditionally writes the empty-section pages (subaccount entitlements and labels) so the nav never breaks (ADR-TF-009 §5); when data is empty the page contains a Material alert box instead of a table.

directories/index.md is written unconditionally for the same reason, but by a different mechanism: directory_labels is an optional contract section (§6), so a valid cache may not carry the key at all, and unlike the subaccount pages there is no list of directories to loop over to prove the page is needed. The Directories dispatch in main() is therefore outside every isinstance(...) guard and always writes the page, rendering the alert when metadata.get("directory_labels") is None. A single index page is used rather than a *[dir]* per-directory detail page: the only directory data in the contract is the label rows themselves, so a per-directory page would carry nothing that the grouped index does not. If a future contract section exposes the hierarchy itself, per-directory pages become worthwhile and should follow the subaccounts/<id>/ pattern.

6. Contract validation

The generator validates the cache JSON against a section contract on load before it writes any pages:

EXPECTED_SECTIONS = {
    "identity", "users", "user_details", "roles", "role_inventory",
    "entitlements", "trust_configurations", "subaccounts",
    "arsucc_users", "arsucc_user_details",
}
OPTIONAL_SECTIONS = {
    "user_details", "arsucc_users", "arsucc_user_details",
    "subaccount_users", "subaccount_user_details",
    "subaccount_arsucc_users", "subaccount_arsucc_user_details",
    "directory_labels",
}

Two shapes appear in OPTIONAL_SECTIONS:

In both sets (user_details) — required in principle, absence tolerated. A key listed in OPTIONAL_SECTIONS relaxes the missing check only if it is also in EXPECTED_SECTIONS.
Optional only (arsucc_users, arsucc_user_details, subaccount_users, subaccount_user_details, subaccount_arsucc_users, subaccount_arsucc_user_details, directory_labels) — recognised, never required. The unexpected check accepts any key in either set, which is what makes an optional-only key legal at all.

directory_labels is optional-only: a metadata capture taken before btp_directory_labels was wired into data.tf legitimately has no such key. Its page is written unconditionally (§5) so this toleration cannot turn into a --strict build failure. directory_objects is deliberately not a contract key — it is exposed as a separate Terraform output, not added to globalaccount_metadata, so the contract delta stays at one section; the display names reach the generator through directory_name on each label row.

check_contract(metadata) raises ContractDriftError if the cache contains a top-level key that is in neither set. The required-vs-optional split exists so that adding a new section to the Terraform output does not break the generator until the generator is updated — but unknown extra keys still fail fast because they are almost certainly a mistake.

Contract validation covers only the top-level keys present in the cache. It does not validate the shape of each section's rows; row-level shape is the generator's responsibility and is validated at render time by defensive default values (e.g. or [], or {}, or "") within each render function.

7. Idempotency and regeneration invariants

Re-running make docs is safe. It refreshes state, regenerates pages, and rebuilds the site with no manual cleanup step required. Old pages the generator no longer emits are overwritten by new output; pages that should not exist anymore (e.g. a page for a global account that was deleted from state) are the responsibility of the operator to remove from disk or from mkdocs.yml.
Stale on-disk pages for empty sections are not removed. If a subaccount gains entitlements in a later refresh, re-running make docs overwrites the "No data available" alert page with the populated table. The alert is a rendering choice, not a deletion signal.
The cache is the only non-reproducible local artifact. Given a fresh terraform refresh, .cache/globalaccount_metadata.json is regenerated identically; nothing downstream depends on the cache file's modification time or previous contents.
mkdocs build --strict failures block the pipeline. A broken link, a missing nav entry, or a Markdown syntax error in generated content fails the build and prevents deployment. This is deliberate — a documentation site must not ship in a broken state.

8. Environment variables

Terraform / extract phase

Variable	Used by	Meaning
`ENV`	`make docs-extract`	Environment root to extract from (default `dev` → `environments/dev`)

Generator

Variable	Used by	Meaning
(CLI) `--input`	`generate_docs.py`	Path to the cache JSON (required)
(CLI) `--output`	`generate_docs.py`	Root directory to write pages into (required)
(CLI) `--ga-name`	`generate_docs.py`	Display name of the GA; slugified into a subdirectory (required)
(CLI) `--deploy`	`generate_docs.py`	After generation, build + copy to media server

Deploy phase

Variable	Default	Meaning
`DEPLOY_REMOTE_HOST`	`media`	Remote host for `scp`. When unset the deploy step is skipped.
`DEPLOY_SSH_USER`	`roger`	SSH user for the `scp` destination.
`DEPLOY_REMOTE_PATH`	`/config/website/content`	Remote directory to copy `site/` into.
`DEPLOY_SSH_KEY`	(empty)	Optional SSH private key path; when set, `scp` is invoked with `-i <key>`.

The deploy-phase defaults are chosen for the current media-server target (roger@media:/config/website/content). They are documented in the Makefile and in generate_docs.py so they are visible without reading code.

ADR-TF-001 — local Terraform backend strategy (state layout, single operator per root module).
ADR-TF-003 — local backend rationale and backup procedure.
ADR-TF-004 — generation pipeline authorship and the make docs target.
ADR-TF-006 — plural-to-singular linkage (btp_globalaccount_users → btp_globalaccount_user per username) and the origin parameter that makes it work for non-default IdPs.
ADR-TF-008 — output contract drift handling and the EXPECTED_SECTIONS / OPTIONAL_SECTIONS split.
ADR-TF-009 — no-data page lifecycle: Material alert boxes, unconditional page writing, and the generator comment that explains why wired sections always get a page.
mkdocs.yml (project root only) — the nav configuration that determines which generated pages are reachable in the built site. The build runs mkdocs build from this directory with no -f flag, so this is the only config that is read. A duplicate docs/mkdocs.yml existed but was never read by the build and had drifted; it was removed rather than maintained. Edit the root file.
requirements.txt — pinned package list for the project venv.

10. Status and approval

Field	Value
Document ID	SPEC-TF-005
Version	1.0.0-draft
Created	2026-09-30
Author	Hermes Agent (coder-doc6 profile), per user request
Status	Proposed — reflects the current implementation as of the 2026-09-30 refresh + generate run
Implementation location	`Makefile` (docs-* targets), `scripts/generate_docs.py`, `requirements.txt`, `docs/requirements.txt`, `site/requirements.txt`

This spec is written to match the current implementation. Any future change to the pipeline (new page type, new environment variable, new deploy method, new contract key) should update this spec as part of the same change.