Annex — Tier 2 / 2.5: hosted pipelines

Applies when something runs on a schedule: reads client data from a source system, aggregates it, and publishes an output the client opens. This annex adds to baseline-runbook.md.

Tier 2 is where the honest answer to "does our data leave our environment?" changes from "no" to "it depends how we set it up" — so the first section is the one that matters most.

Where the code runs

I decide this with the client before building, and I write the answer into the data-flow diagram. Preference order:

1. Inside the client's tenant — their Azure subscription, their Function App or scheduled job, their storage. Data never crosses a boundary, they can see and kill it, and there is no bus-factor problem if I disappear. This is the default I argue for.
2. My EU-region infrastructure, client-owned credentials — used when the client has no place to run it or does not want to manage one. Data crosses to me, so it goes in the Art. 30 register and the DPA has to be in place first. Nothing persists between runs: the source extract is deleted once the aggregate is built, and every copy is gone within seven days of the engagement ending. That is a real constraint on me as well as a promise to them — it means a disputed number cannot be re-checked here without a fresh extract, and I say so up front rather than discovering it during an incident.
3. My VPS — only when the first two do not fit. See annex-selfhosted.md.

Whichever it is, I state it plainly in writing rather than letting the client assume tier-1 rules still apply. Compute and storage regions are EU/EEA and I record the configured region string, not the vendor's default.

Read-only API scopes

• The pipeline reads. It does not write back to the source system, and it does not hold a credential that could.
• Scopes are the narrowest the platform allows: on Microsoft Graph that means application permissions limited to one site or one folder rather than tenant-wide read, granted via a client-owned app registration.
• The app registration belongs to the client, not to me. They can revoke it in a click, without asking me and without breaking anything else they own.
• Credentials are per-project. No credential is reused across clients, ever.
• Every scope is written into templates/access-register.md with the date it was granted and, later, the date it was revoked.
• I check the granted scopes in the platform console after consent, because what was requested and what was granted are not always the same thing.

Vault choice

• Client on Microsoft → Azure Key Vault in their tenant, EU region, access granted to the job's managed identity rather than to a person.
• Client elsewhere → that platform's native secret manager, in an EU region.
• My own infrastructure → the platform secret store, or an encrypted secrets file with the key held separately if the platform has none.

Rules that do not change with the choice: secrets are injected at runtime as environment variables, never committed, never logged, never pasted into chat or a ticket. Rotation is possible without a code change — if rotating a key means editing code, the design is wrong. Keys are rotated at handover and whenever anyone with access leaves.

Backup and restore

What gets backed up:

• Code and configuration — Git, EU-hosted, tagged at each delivered version.
• Infrastructure definition — whatever is needed to stand the job back up: environment variables (names, not values), schedule, permissions, runtime version.
• The published output, versioned, so yesterday's good version survives today's bad run.
• Any state the client depends on that does not exist elsewhere. Source data itself is not backed up by me — it lives in their system, and copying it would create a second place to protect.

Restore test: at least once per project, before handover, I rebuild the pipeline from the backup into a scratch environment and run it end to end. I write down the date, what I restored, how long it took, and what was missing — a restore test that finds nothing missing usually means I did not really do it.

Monitoring

• Every run logs start, end, row counts in and out, and outcome. Logs record events, not payloads — no personal data, no secrets in log lines.
• Failures alert me on the day, through a channel I actually read.
• A missed run alerts too. Silence is the failure mode that hurts, because a stale dashboard looks exactly like a fresh one.
• Sanity checks run before publish: row count within an expected band, no all-null key column, totals within tolerance of the previous run. If a check fails the job stops and keeps the previous good output rather than publishing something wrong.
• The output carries a visible "data as of" timestamp so a stale file is obvious to the person reading it, not just to me.
• I check the log and alert history at handover and hand the client the runbook for what to do when it fires after I am gone.