{"authors":[{"name":"Martin Stühmer","url":"https://daily-devops.net/authors/martin/"},{"name":"Jendrik Brack","url":"https://daily-devops.net/authors/jendrik/"}],"description":"Recent content in DevOps Practices That Actually Ship on Daily DevOps \u0026 .NET","favicon":"https://daily-devops.net/images/logo_hu_6465d873dfa490cf.png","feed_url":"https://daily-devops.net/tags/devops/feed.json","home_page_url":"https://daily-devops.net/tags/devops/","icon":"https://daily-devops.net/images/logo_hu_5926de77762241ba.png","items":[{"authors":[{"name":"Jendrik Brack","url":"https://daily-devops.net/authors/jendrik/"}],"content_html":"\u003cp\u003eEvery platform engineering conference talk in the last two years has had a Backstage slide. Glossy catalogue screenshot, a scaffolder demo that creates a repo in four clicks, a knowing nod about \u0026ldquo;developer experience\u0026rdquo;. What the slide never shows is the six months the team spent building plugins, the Postgres instance somebody now babysits, the TechDocs theme nobody asked for, and the 0.4 of an engineer permanently assigned to chasing Backstage\u0026rsquo;s two-week release cadence.\u003c/p\u003e\n\u003cp\u003eThere is no shame in any of this. Backstage is a serious project and serious teams run it well. The shame is treating it as the \u003cem\u003edefault\u003c/em\u003e (the thing you reach for on day one) when most teams could ship 80% of the value with a tenth of the effort and a fraction of the running cost. Backstage is a platform for building platforms. Most teams need a platform, not a platform-platform.\u003c/p\u003e\n\u003cp\u003eThis post is the Internal Developer Platform (IDP) I keep building when nobody is forcing me to use Backstage. It is small, opinionated, runs on Azure plumbing you already pay for, and ships value in the first quarter instead of the third year.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"what-an-idp-actually-needs-to-do\"\u003e\u003ca href=\"/posts/platform-engineering-without-backstage/#what-an-idp-actually-needs-to-do\" title=\"What an IDP Actually Needs to Do\"\u003eWhat an IDP Actually Needs to Do\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eBefore the tooling argument, the value list. An Internal Developer Platform exists to:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eLower the time to first deployment\u003c/strong\u003e for a new service. Day one, not day thirty.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eMake the right thing the easy thing.\u003c/strong\u003e Security, observability, and cost defaults arrive for free, not as a checklist the team forgets.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eProvide a catalogue\u003c/strong\u003e so anyone can answer \u0026ldquo;who owns this?\u0026rdquo; without a Slack archaeology session at 02:00.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eStop the SRE from writing the same answer a third time.\u003c/strong\u003e Every repeated question is a missing platform feature.\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eEverything else is decoration. If a feature doesn\u0026rsquo;t move one of those four numbers, skip it.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-pragmatic-stack-on-azure\"\u003e\u003ca href=\"/posts/platform-engineering-without-backstage/#the-pragmatic-stack-on-azure\" title=\"The Pragmatic Stack on Azure\"\u003eThe Pragmatic Stack on Azure\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eFive components. None of them require Backstage. All of them you can stand up with tools your org already owns.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eGolden-path repositories:\u003c/strong\u003e skeleton projects for the shapes you support (API, worker, frontend), each pre-wired to the platform.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eReusable GitHub Actions workflows:\u003c/strong\u003e a small set of \u003ccode\u003eworkflow_call\u003c/code\u003e templates that do the right thing so teams don\u0026rsquo;t copy-paste CI.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eOpinionated Bicep modules:\u003c/strong\u003e one module per platform service (Container App, Azure Kubernetes Service (AKS) namespace, Key Vault, Storage), with sane defaults and clearly marked escape hatches.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eA service catalogue:\u003c/strong\u003e one YAML file per service repo, aggregated into a queryable, static site. No database.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eScaffolding:\u003c/strong\u003e \u003ccode\u003egh repo create --template\u003c/code\u003e is enough for most teams. A CLI comes \u003cem\u003elater\u003c/em\u003e, if ever.\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch3 id=\"golden-path-repository-the-highest-leverage-artefact\"\u003e\u003ca href=\"/posts/platform-engineering-without-backstage/#golden-path-repository-the-highest-leverage-artefact\" title=\"Golden-Path Repository: The Highest-Leverage Artefact\"\u003eGolden-Path Repository: The Highest-Leverage Artefact\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eA new service starts as a fork of a template repo. Day one, the team has CI, CD, infrastructure-as-code, and platform integration already wired together.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003epayments-api/                      (created from my-org/template-dotnet-api)\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e├── .github/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│   └── workflows/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│       ├── ci.yml                 → calls the platform reusable workflow\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│       └── cd.yml                 → calls the platform reusable workflow\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e├── infra/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e│   └── main.bicep                 → consumes the platform Bicep module\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e├── src/                           → service code (the only part the team writes)\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e├── catalog-info.yaml              → one file, registers the service in the catalogue\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e├── .editorconfig                  → org formatting defaults\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e├── .gitignore\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e└── README.md                      → pre-filled: how to run, deploy, and page on-call\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe discipline that makes this work: the team owns \u003ccode\u003esrc/\u003c/code\u003e and almost nothing else. The moment teams start hand-editing the workflow files or the Bicep, the golden path is dead and you\u0026rsquo;re back to snowflakes. Keep the platform-owned files thin pointers to versioned platform artefacts, and the template stays maintainable for years.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"reusable-workflows-one-place-to-improve-everyones-ci\"\u003e\u003ca href=\"/posts/platform-engineering-without-backstage/#reusable-workflows-one-place-to-improve-everyones-ci\" title=\"Reusable Workflows: One Place to Improve Everyone\u0026rsquo;s CI\"\u003eReusable Workflows: One Place to Improve Everyone\u0026rsquo;s CI\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eThis is the highest-leverage line of code you will write all quarter. When you fix something in a reusable workflow (a vulnerable action version, a missing Software Bill of Materials (SBOM) step, a flaky cache), \u003cem\u003eevery\u003c/em\u003e service that calls it picks up the fix on its next run. No PR to fifty repos. No migration guide nobody reads.\u003c/p\u003e\n\u003cp\u003eThe platform owns the real workflow:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# my-org/platform-workflows/.github/workflows/dotnet-api.yml\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edotnet-api\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eon\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eworkflow_call\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003einputs\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003eservice\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erequired\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003etype\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003estring\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003eenvironment\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erequired\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003etype\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003estring\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003edefault\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003estaging\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003epermissions\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eid-token\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ewrite   \u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"c\"\u003e# OIDC federation to Azure — no long-lived secrets in any repo\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003econtents\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eread\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003epackages\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ewrite\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ejobs\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ebuild-test\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eruns-on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eubuntu-latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esteps\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/checkout@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/setup-dotnet@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ewith\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003edotnet-version\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;10.0.x\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edotnet restore\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edotnet build --no-restore -c Release\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edotnet test --no-build -c Release --logger trx\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"c\"\u003e# SBOM + image build are platform defaults, not per-team decisions.\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eBuild and push image\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          az acr build --registry ${{ vars.PLATFORM_ACR }} \\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            --image ${{ inputs.service }}:${{ github.sha }} .\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003edeploy\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eneeds\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ebuild-test\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eruns-on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eubuntu-latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eenvironment\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e${{ inputs.environment }}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esteps\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eazure/login@v2\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ewith\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003eclient-id\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e${{ secrets.AZURE_CLIENT_ID }}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003etenant-id\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e${{ vars.AZURE_TENANT_ID }}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003esubscription-id\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e${{ vars.AZURE_SUBSCRIPTION_ID }}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/checkout@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eDeploy infra + app\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          az deployment group create \\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            --resource-group rg-${{ inputs.service }}-${{ inputs.environment }} \\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            --template-file infra/main.bicep \\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            --parameters image=${{ vars.PLATFORM_ACR }}.azurecr.io/${{ inputs.service }}:${{ github.sha }}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe team\u0026rsquo;s entire deployment pipeline is then twelve lines that nobody needs to understand:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# payments-api/.github/workflows/cd.yml (complete file)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ecd\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eon\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003epush\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ebranches\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"l\"\u003emain]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ejobs\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eship\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003emy-org/platform-workflows/.github/workflows/dotnet-api.yml@v3\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ewith\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003eservice\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003epayments-api\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003eenvironment\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eproduction\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esecrets\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003einherit\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003ePin the \u003ccode\u003e@v3\u003c/code\u003e and treat platform releases like a product: a changelog, semantic versions, so teams adopt fixes on a tag bump instead of being silently broken by \u003ccode\u003e@main\u003c/code\u003e.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"bicep-modules-the-opinions-codified\"\u003e\u003ca href=\"/posts/platform-engineering-without-backstage/#bicep-modules-the-opinions-codified\" title=\"Bicep Modules: The Opinions, Codified\"\u003eBicep Modules: The Opinions, Codified\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eA Bicep module per platform service replaces hours of copy-paste, and more importantly, replaces hours of \u003cem\u003earguing about defaults\u003c/em\u003e. Teams call the module with the three or four parameters that matter for their service. The platform team owns the parameters that matter for everyone: tags, managed identity, network posture, the things an auditor will ask about. This is the same argument I make in \u003ca href=\"/posts/infrastructure-as-code-compliance-bicep/\"\u003eWhy Your Azure Portal Clicks Will Fail the Next Audit\u003c/a\u003e: defaults that live in code are defaults you can prove.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bicep\" data-lang=\"bicep\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e// modules/container-app.bicep — one module, sane defaults, marked escape hatches\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e@\u003c/span\u003e\u003cspan class=\"nf\"\u003edescription\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Service name. Drives naming, tags, and identity.\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eparam\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eservice\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003estring\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e@\u003c/span\u003e\u003cspan class=\"nf\"\u003edescription\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Container image including registry and tag.\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eparam\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eimage\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003estring\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eparam\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003elocation\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003estring\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003eresourceGroup\u003c/span\u003e\u003cspan class=\"p\"\u003e().\u003c/span\u003e\u003cspan class=\"nv\"\u003elocation\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e// --- Escape hatches: teams override only what their service genuinely needs. ---\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e@\u003c/span\u003e\u003cspan class=\"nf\"\u003eallowed\u003c/span\u003e\u003cspan class=\"p\"\u003e([\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;0.25\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;0.5\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;1.0\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;2.0\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e])\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eparam\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003ecpu\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003estring\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;0.5\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eparam\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003ememory\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003estring\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;1Gi\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eparam\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eminReplicas\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eint\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003e1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eparam\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003emaxReplicas\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eint\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003e10\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e// --- Platform-owned defaults teams do NOT get to weaken. ---\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003etags\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003eservice\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eservice\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;managed-by\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;platform\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;cost-center\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;engineering\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eresource\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eenv\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Microsoft.App/managedEnvironments@2024-03-01\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003eexisting\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;platform-aca-env\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eresource\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eidentity\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;id-\u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eservice\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003elocation\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003elocation\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003etags\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003etags\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eresource\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eapp\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Microsoft.App/containerApps@2024-03-01\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eservice\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003elocation\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003elocation\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003etags\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003etags\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003eidentity\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"kd\"\u003etype\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;UserAssigned\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003euserAssignedIdentities\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;\u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eidentity\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nv\"\u003eid\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{}\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003eproperties\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003emanagedEnvironmentId\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eenv\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nv\"\u003eid\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003econfiguration\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nv\"\u003eingress\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eexternal\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003etargetPort\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003e8080\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003etransport\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;auto\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003etemplate\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nv\"\u003econtainers\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eservice\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eimage\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eimage\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eresources\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003ecpu\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003ejson\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nv\"\u003ecpu\u003c/span\u003e\u003cspan class=\"p\"\u003e),\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003ememory\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003ememory\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nv\"\u003escale\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eminReplicas\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eminReplicas\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003emaxReplicas\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003emaxReplicas\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eoutput\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003efqdn\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003estring\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nv\"\u003eproperties\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nv\"\u003econfiguration\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nv\"\u003eingress\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nv\"\u003efqdn\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe escape hatches are explicit and few. If a team needs to override something that isn\u0026rsquo;t a parameter, it\u0026rsquo;s a signal: either the default is wrong for everyone (fix the module), or the service is a genuine snowflake. Never allow a quiet fork of the module into a service repo. The day that happens, you have N modules to patch, not one.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"service-catalogue-a-yaml-file-and-a-build-step\"\u003e\u003ca href=\"/posts/platform-engineering-without-backstage/#service-catalogue-a-yaml-file-and-a-build-step\" title=\"Service Catalogue: A YAML File and a Build Step\"\u003eService Catalogue: A YAML File and a Build Step\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eSkipping the catalogue is the most expensive shortcut on this list, and teams skip it first because it feels like paperwork. It\u0026rsquo;s the difference between a five-second answer and a forty-minute incident dig. You don\u0026rsquo;t need Postgres, a graph database, or Backstage to have one.\u003c/p\u003e\n\u003cp\u003eOne file per service repo:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# payments-api/catalog-info.yaml\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eservice\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003epayments-api\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eowner\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eteam-payments\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003etier\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e1\u003c/span\u003e\u003cspan class=\"w\"\u003e                       \u003c/span\u003e\u003cspan class=\"c\"\u003e# 1 = revenue-critical; drives alerting + review policy\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003elanguage\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edotnet\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ehosting\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003econtainer-apps\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eon_call\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003epayments-oncall@example.com\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003elinks\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003erepo\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ehttps://github.com/example/payments-api\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003erunbook\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ehttps://example.atlassian.net/wiki/payments-api\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003edashboard\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ehttps://example.grafana.net/d/payments\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003edepends_on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"l\"\u003epostgres-payments\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"l\"\u003eauth-api\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eA nightly job in the platform repo collects every one of those files across the org and emits a single static JSON the catalogue site reads. The whole \u0026ldquo;backend\u0026rdquo; is the GitHub search API, \u003ccode\u003ejq\u003c/code\u003e, and \u003ccode\u003eyq\u003c/code\u003e:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# my-org/platform/.github/workflows/build-catalogue.yml\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eon\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eschedule\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e- \u003cspan class=\"nt\"\u003ecron\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;0 3 * * *\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e     \u003c/span\u003e\u003cspan class=\"c\"\u003e# nightly; the org doesn\u0026#39;t reorganise faster than that\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eworkflow_dispatch\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ejobs\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eaggregate\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eruns-on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eubuntu-latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esteps\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eCollect every catalog-info.yaml in the org\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eenv\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003eGH_TOKEN\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e${{ secrets.CATALOG_READ_TOKEN }}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          gh search code --owner my-org --filename catalog-info.yaml \\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            --json path,repository \\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          | jq -r \u0026#39;.[] | \u0026#34;\\(.repository.nameWithOwner) \\(.path)\u0026#34;\u0026#39; \\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          | while read -r repo path; do\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e              gh api \u0026#34;repos/$repo/contents/$path\u0026#34; --jq \u0026#39;.content\u0026#39; | base64 -d\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            done \\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          | yq ea \u0026#39;[.]\u0026#39; -o=json \u0026gt; catalog.json\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ePublish to the static catalogue site\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          az storage blob upload --account-name platformcatalogue \\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            --container-name \u0026#39;$web\u0026#39; --name catalog.json --file catalog.json --overwrite\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThat \u003ccode\u003ecatalog.json\u003c/code\u003e lands in an Azure Storage static website (\u003ccode\u003e$web\u003c/code\u003e container) that costs roughly the price of a coffee per year to host. The catalogue front-end is a single page that fetches the JSON and renders a searchable table. \u0026ldquo;Who owns \u003ccode\u003epayments-api\u003c/code\u003e? What\u0026rsquo;s the runbook? What does it depend on?\u0026rdquo; All answered in one query, by anyone, without a database to back up or a service to keep alive at 02:00.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"from-new-service-to-first-deploy-under-30-minutes\"\u003e\u003ca href=\"/posts/platform-engineering-without-backstage/#from-new-service-to-first-deploy-under-30-minutes\" title=\"From New Service to First Deploy: Under 30 Minutes\"\u003eFrom New Service to First Deploy: Under 30 Minutes\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003e\u003ccode\u003egh repo create --template\u003c/code\u003e. Fill in four lines of \u003ccode\u003ecatalog-info.yaml\u003c/code\u003e. Write your logic. \u003ccode\u003egit push\u003c/code\u003e. The workflow builds, tests, and deploys with identity, tags, and labels already correct. The catalogue picks it up overnight. No ticket, no request to the platform team. The golden path \u003cem\u003eis\u003c/em\u003e the request system.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"when-backstage-actually-is-the-right-answer\"\u003e\u003ca href=\"/posts/platform-engineering-without-backstage/#when-backstage-actually-is-the-right-answer\" title=\"When Backstage Actually Is the Right Answer\"\u003eWhen Backstage Actually Is the Right Answer\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eBackstage is the right answer when:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eThe platform team is five-plus people\u003c/strong\u003e with genuine capacity to run Backstage as a product, including the upgrade treadmill.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eThe service count is large enough that catalogue queries genuinely matter:\u003c/strong\u003e hundreds of services, not dozens. At dozens, a YAML file and a static page answer every question Backstage would.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eYou need a dependency graph, not a list.\u003c/strong\u003e When \u0026ldquo;what breaks if \u003ccode\u003eauth-api\u003c/code\u003e goes down?\u0026rdquo; needs a traversable graph across hundreds of nodes, a flat YAML catalogue stops being enough.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eYou\u0026rsquo;ll get real use from existing Backstage plugins\u003c/strong\u003e for proprietary in-house systems, where the plugin ecosystem saves you building integrations from scratch.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eTwo or three engineers and a few dozen services? Backstage will eat your runway. Pick the tool that matches your headcount, not the one that matched the headcount of the team whose conference talk you watched.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"where-teams-go-wrong\"\u003e\u003ca href=\"/posts/platform-engineering-without-backstage/#where-teams-go-wrong\" title=\"Where Teams Go Wrong\"\u003eWhere Teams Go Wrong\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eTeams build what they want to operate rather than what developers need to ship through.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eToo much, too soon:\u003c/strong\u003e UI before catalogue (the data model is the hard part; the UI is a weekend). CLI before template (\u003ccode\u003egh repo create --template\u003c/code\u003e handles the first fifty services). Plugins for tools nobody uses yet. Speculative integration is speculative debt.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eNot enough:\u003c/strong\u003e The catalogue gets skipped because it feels like paperwork. Every incident without it costs you the forty minutes you \u0026ldquo;saved\u0026rdquo;. Documentation nobody can find at 02:00 is a graveyard. And the platform needs an on-call rota: no owner when it breaks means teams won\u0026rsquo;t depend on it. As I argue in \u003ca href=\"/posts/incident-response-github-actions/\"\u003eYour Incident Response Plan Is a Lie. Here\u0026rsquo;s How to Fix It.\u003c/a\u003e: undefined ownership is the failure, not the missing tool.\u003c/p\u003e\n\u003cp\u003eIf a feature doesn\u0026rsquo;t lower time-to-deploy, encode a good default, feed the catalogue, or kill a repeated SRE question: it waits.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"what-an-idp-cannot-fix\"\u003e\u003ca href=\"/posts/platform-engineering-without-backstage/#what-an-idp-cannot-fix\" title=\"What an IDP Cannot Fix\"\u003eWhat an IDP Cannot Fix\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eTooling is downstream of organisation. Team dynamics at war, a production swamp nobody has time to drain, leadership that won\u0026rsquo;t give teams space to adopt the golden path: these are org problems, not tool problems. It\u0026rsquo;s the same trap I describe in \u003ca href=\"/posts/kubernetes-not-platform-strategy/\"\u003eKubernetes Is Not a Platform Strategy\u003c/a\u003e. The tool is how you express the strategy, not a substitute for one.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"what-i-recommend\"\u003e\u003ca href=\"/posts/platform-engineering-without-backstage/#what-i-recommend\" title=\"What I Recommend\"\u003eWhat I Recommend\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eBuild one golden-path repo in two weeks: one service shape, CI, CD, Bicep, \u003ccode\u003ecatalog-info.yaml\u003c/code\u003e. Adopt it on two real services, not pilots. Add the catalogue in an afternoon. Only ask whether you need Backstage at month three or four, once service count and team size give you an actual answer.\u003c/p\u003e\n\u003cp\u003eEarn the right to operate Backstage by first delivering value without it.\u003c/p\u003e","date_modified":"2026-05-27T18:08:49+02:00","date_published":"2026-05-27T18:00:00+02:00","id":"https://daily-devops.net/posts/platform-engineering-without-backstage/","language":"en","summary":"Backstage is not the only path to an Internal Developer Platform. Pragmatic IDP patterns on Azure that ship value before the YAML eats your team.","tags":["platform-engineering","azure","devops","aks"],"title":"Platform Engineering Without Backstage: Pragmatic IDPs on Azure","url":"https://daily-devops.net/posts/platform-engineering-without-backstage/"},{"authors":[{"name":"Martin Stühmer","url":"https://daily-devops.net/authors/martin/"}],"content_html":"\u003cp\u003eGitHub Copilot code review is available in pull requests. Claude can review a diff. Cursor highlights issues as you type. Every major AI coding assistant now offers some form of review, and teams are using these tools to supplement (or in some cases replace) asynchronous human review on pull requests.\u003c/p\u003e\n\u003cp\u003eThis is not necessarily wrong. AI code review is genuinely useful. But there is a pattern to what it misses, and understanding that pattern matters more than debating whether to use these tools at all.\u003c/p\u003e\n\u003cp\u003eIn my experience, AI code reviewers behave like sycophants. They are good at finding small problems with how you built something. They are almost incapable of questioning whether you should have built it at all.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"what-ai-code-review-is-good-at\"\u003e\u003ca href=\"/posts/ai-code-review-is-a-sycophant/#what-ai-code-review-is-good-at\" title=\"What AI Code Review Is Good At\"\u003eWhat AI Code Review Is Good At\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eTo be clear: these tools are useful. Worth adding to your PR workflow.\u003c/p\u003e\n\u003cp\u003eAI reviews reliably catch:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eObvious bugs in isolation.\u003c/strong\u003e Null dereferences, off-by-one errors, incorrect operator precedence, missing \u003ccode\u003eawait\u003c/code\u003e, unchecked return values from methods that can fail. These are the bugs human reviewers also catch, and they slip through when reviewers are tired, rushed, or staring at a 500-line diff.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCommon anti-patterns.\u003c/strong\u003e \u003ccode\u003easync void\u003c/code\u003e, catching \u003ccode\u003eException\u003c/code\u003e without rethrowing, \u003ccode\u003eDateTime.Now\u003c/code\u003e instead of \u003ccode\u003eDateTime.UtcNow\u003c/code\u003e, string concatenation in loops, \u003ccode\u003eConfigureAwait(false)\u003c/code\u003e missing in library code. Pattern matching against known bad patterns is exactly what Large Language Models (LLMs) do well.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTrivial security issues.\u003c/strong\u003e SQL injection via string concatenation, hardcoded credentials, insecure random number generation. These appear in training data thousands of times.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eStyle consistency.\u003c/strong\u003e Naming inconsistencies, missing XML documentation, inconsistent error handling patterns relative to the rest of the file.\u003c/p\u003e\n\u003cp\u003eThese categories represent real value. A review pass that catches these before human review means human reviewers can spend their time on harder problems.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"what-ai-code-review-systematically-misses\"\u003e\u003ca href=\"/posts/ai-code-review-is-a-sycophant/#what-ai-code-review-systematically-misses\" title=\"What AI Code Review Systematically Misses\"\u003eWhat AI Code Review Systematically Misses\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eThis is where the sycophancy shows up.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWrong abstraction.\u003c/strong\u003e AI reviewers evaluate the code you wrote against its own internal logic. They rarely notice that the abstraction itself is wrong: that the \u003ccode\u003eOrderProcessor\u003c/code\u003e class is doing three different things and probably should not exist as a single class, that the interface design couples callers to implementation details, that the naming reveals a confused mental model of the domain. Recognizing a wrong abstraction requires understanding the system it lives in and the cost of fixing it later. AI reviewers do not have that context.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u0026ldquo;This should be deleted.\u0026rdquo;\u003c/strong\u003e The correct review comment for a surprising fraction of pull requests is something like: \u0026ldquo;This feature was not the right call, let\u0026rsquo;s talk before merging.\u0026rdquo; AI reviewers will not write that comment. They review code on its own terms. A well-implemented feature that solves the wrong problem gets a positive AI review, and that feedback loop, repeated over time, shapes how a team thinks about what quality means.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSystemic patterns across the codebase.\u003c/strong\u003e AI reviewers see the diff. They do not know that the same abstraction appeared in three other places and was wrong each time. They do not know that this exact approach was tried and reverted eight months ago, and that the revert commit explains why. Reviewers with codebase history catch this. AI reviewers cannot.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eBusiness logic correctness.\u003c/strong\u003e Is this the right formula for calculating the surcharge? Does this authorization check correctly represent the access control model? Is this state machine transition valid given how the domain actually works? AI reviewers can tell you the code is internally consistent. They cannot tell you it is correct relative to what the software is supposed to do. This is not a minor gap. Business logic bugs are often the costliest bugs, and they are invisible to a reviewer that does not understand the business.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePerformance under real load.\u003c/strong\u003e AI reviewers flag obvious O(n²) algorithms and missing database indexes in toy examples. They rarely have visibility into the data distribution, the access patterns, or the production load profile that determines whether the code will hold up at scale. The performance review that matters happens in load testing and production, not in the diff view.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-sycophancy-problem\"\u003e\u003ca href=\"/posts/ai-code-review-is-a-sycophant/#the-sycophancy-problem\" title=\"The Sycophancy Problem\"\u003eThe Sycophancy Problem\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eThe specific failure mode of AI code review is not that it misses things. Every review process misses things. The problem is the pattern of what it misses.\u003c/p\u003e\n\u003cp\u003eAI reviewers tend to approve the overall approach and find issues in the details. When a team leans heavily on AI review, there is a subtle risk: reviewers get better and better at fixing the details an AI flags, while the bigger structural questions get less attention over time. I have seen this happen, and it is not anyone\u0026rsquo;s fault. It is a natural response to the feedback signal you are getting.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"why-the-approval-bias-is-structural\"\u003e\u003ca href=\"/posts/ai-code-review-is-a-sycophant/#why-the-approval-bias-is-structural\" title=\"Why The Approval Bias Is Structural\"\u003eWhy The Approval Bias Is Structural\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eThe approval bias is structural. AI reviewers are trained on review data where most code in a diff is acceptable. The kind of feedback that says \u0026ldquo;the entire approach here is wrong, close this PR and start over\u0026rdquo; is rare in training data and produces outcomes that make the tool seem less useful. So the model optimizes away from it.\u003c/p\u003e\n\u003cp\u003eThe result: AI reviewers are systematically biased toward approving what you built and suggesting small improvements. They are not calibrated to recognize when the correct response is rejection.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"the-confidence-effect-on-developers\"\u003e\u003ca href=\"/posts/ai-code-review-is-a-sycophant/#the-confidence-effect-on-developers\" title=\"The Confidence Effect On Developers\"\u003eThe Confidence Effect On Developers\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eThere is also a confidence effect worth naming. A developer who ships a PR with zero AI findings tends to feel more confident that the code is solid. That confidence is not entirely wrong (the mechanical issues are likely clean), but it can crowd out the instinct to ask for a second human opinion. Over time, \u0026ldquo;the AI found nothing\u0026rdquo; starts to function as a substitute for \u0026ldquo;this is good code\u0026rdquo;, and that is a different claim entirely.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"what-ai-review-should-change-about-human-review\"\u003e\u003ca href=\"/posts/ai-code-review-is-a-sycophant/#what-ai-review-should-change-about-human-review\" title=\"What AI Review Should Change About Human Review\"\u003eWhat AI Review Should Change About Human Review\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eIf AI review is in your pipeline, it should shift what human reviewers focus on, not replace them.\u003c/p\u003e\n\u003cp\u003eAI reviewers handle the mechanical layer well: obvious bugs, pattern violations, style issues. That creates an opportunity for human reviewers to focus on what AI cannot do:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eIs this the right design?\u003c/li\u003e\n\u003cli\u003eDoes this code belong here at all?\u003c/li\u003e\n\u003cli\u003eDoes the naming suggest the author has a clear mental model of the domain?\u003c/li\u003e\n\u003cli\u003eIs this consistent with decisions made elsewhere in the system?\u003c/li\u003e\n\u003cli\u003eWhat will maintaining this cost in six months?\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch3 id=\"where-human-review-time-belongs\"\u003e\u003ca href=\"/posts/ai-code-review-is-a-sycophant/#where-human-review-time-belongs\" title=\"Where Human Review Time Belongs\"\u003eWhere Human Review Time Belongs\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eHuman review time is finite. If a human reviewer spends twenty minutes on a PR that an AI already reviewed and only surfaces style issues, something has gone wrong with how review time is being used. The value of human review is judgment, context, and the willingness to say \u0026ldquo;not yet.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eA team that uses AI review to reduce the need for human judgment does not end up with less review. It ends up with coverage that feels high but catches less of what actually matters.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-diff-problem\"\u003e\u003ca href=\"/posts/ai-code-review-is-a-sycophant/#the-diff-problem\" title=\"The Diff Problem\"\u003eThe Diff Problem\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eBoth AI and human review share a structural limitation: they evaluate changes, not outcomes.\u003c/p\u003e\n\u003cp\u003eA large refactor that genuinely improves a design looks messy as a diff: deletions everywhere, moved code, renamed concepts. A small change that introduces a subtle bug can look perfectly clean. Both human and AI reviewers are influenced by the shape of the change, not just its effect on the codebase.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"why-ai-cannot-step-outside-the-diff\"\u003e\u003ca href=\"/posts/ai-code-review-is-a-sycophant/#why-ai-cannot-step-outside-the-diff\" title=\"Why AI Cannot Step Outside The Diff\"\u003eWhy AI Cannot Step Outside The Diff\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eAI reviewers are more constrained here because they have no option to go beyond the diff. A human reviewer can pull the branch, run it, read the surrounding code, check git history. AI reviewers are limited to what is presented to them.\u003c/p\u003e\n\u003cp\u003eThis means AI review is structurally better suited to focused, contained changes, and less suited to catching problems that only become visible when you look at the broader context.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"a-concrete-library-migration-example\"\u003e\u003ca href=\"/posts/ai-code-review-is-a-sycophant/#a-concrete-library-migration-example\" title=\"A Concrete Library Migration Example\"\u003eA Concrete Library Migration Example\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eA concrete example: a PR that migrates a service to use a new internal library might look straightforward in the diff. The imports change, a few method calls are updated, tests pass. An AI reviewer sees nothing alarming. But a human who knows that the new library has different error propagation semantics, or that the migration breaks an assumption made elsewhere in the codebase, can catch that. The diff does not surface it. Context does.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"using-ai-review-without-becoming-dependent-on-it\"\u003e\u003ca href=\"/posts/ai-code-review-is-a-sycophant/#using-ai-review-without-becoming-dependent-on-it\" title=\"Using AI Review Without Becoming Dependent on It\"\u003eUsing AI Review Without Becoming Dependent on It\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eA few practices that have worked well in my experience:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eUse AI review as a pre-filter, not a gatekeeper.\u003c/strong\u003e Let it catch mechanical issues before human review. Humans then review for judgment, not syntax. An AI approval should not substitute for human review on anything that carries real risk.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTreat AI approval as a weak signal.\u003c/strong\u003e An AI saying \u0026ldquo;looks good\u0026rdquo; means it did not find a pattern match for common issues. That is useful information, but it is not an endorsement of the design.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eRead what the AI flagged, and what it did not.\u003c/strong\u003e If it found nothing interesting, that is not evidence the code is good. It may mean the problems are exactly the kind the AI cannot see.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eKeep humans in the design conversation.\u003c/strong\u003e Architecture decisions, new abstractions, changes to domain models: these all need human review from someone with context. No AI reviewer carries your system\u0026rsquo;s history, your domain knowledge, or the judgment to tell you a design direction is off before you build it out.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWatch for approval drift.\u003c/strong\u003e If PRs consistently get AI approval and human reviewers gradually stop questioning design decisions, that is a signal worth paying attention to. The human review may have been quietly degraded, not supplemented.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-honest-summary\"\u003e\u003ca href=\"/posts/ai-code-review-is-a-sycophant/#the-honest-summary\" title=\"The Honest Summary\"\u003eThe Honest Summary\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eAI code review tools are useful. Add them to your pipeline. Let them handle the mechanical layer.\u003c/p\u003e\n\u003cp\u003eBut they are not reviewers in the sense that actually matters. They do not have judgment. They do not know your system. They cannot tell you that you built the wrong thing. They are pattern matchers with a structural bias toward approving what you wrote.\u003c/p\u003e\n\u003cp\u003eThe risk is not that these tools make developers worse (most developers using AI review are thoughtful professionals who also get human review). The risk is subtler: over time, optimizing for what AI review catches can quietly shift attention away from the questions it cannot ask. Staying aware of that dynamic is enough to avoid it.\u003c/p\u003e\n\u003cp\u003eAI review is a useful tool. Keep it in that category.\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eAn AI reviewer that says \u0026ldquo;looks good\u0026rdquo; is not telling you the code is good. It is telling you it did not find a match.\u003c/p\u003e\n\u003c/blockquote\u003e\n","date_modified":"2026-05-25T22:27:03+02:00","date_published":"2026-05-12T17:00:00+02:00","id":"https://daily-devops.net/posts/ai-code-review-is-a-sycophant/","language":"en","summary":"Copilot and Claude find real bugs, but miss wrong abstractions and bad designs. Understanding that gap matters more than debating the tools.","tags":["ai","softwareengineering","codequality","bestpractices","devops"],"title":"AI Code Review Is a Sycophant: Why It Always Approves","url":"https://daily-devops.net/posts/ai-code-review-is-a-sycophant/"},{"authors":[{"name":"Martin Stühmer","url":"https://daily-devops.net/authors/martin/"}],"content_html":"\u003cp\u003eEvery time you drop into a Claude Code session, it starts indexing your project. In a .NET solution, that means \u003ccode\u003ebin/\u003c/code\u003e and \u003ccode\u003eobj/\u003c/code\u003e directories first: tens of thousands of compiled IL files, PDB symbols, generated Razor views, and NuGet extraction caches that serve exactly zero purpose in an AI context window. Then come the test coverage reports, the generated code, the environment files. All of it lands in context unless you tell Claude otherwise.\u003c/p\u003e\n\u003cp\u003eSo you ask Claude how to control which files it can see.\u003c/p\u003e\n\u003cp\u003eAnd Claude tells you about \u003ccode\u003e.claudeignore\u003c/code\u003e.\u003c/p\u003e\n\u003cp\u003eOf course it does. The concept is obvious: \u003ccode\u003e.gitignore\u003c/code\u003e keeps files out of version control, \u003ccode\u003e.dockerignore\u003c/code\u003e keeps files out of Docker build context, so naturally \u003ccode\u003e.claudeignore\u003c/code\u003e keeps files out of Claude\u0026rsquo;s context. Clean, consistent, intuitive. You add it to the project root, list your secrets and production config files, commit it alongside your \u003ccode\u003e.gitignore\u003c/code\u003e, and get on with your day.\u003c/p\u003e\n\u003cp\u003eThe AI assistant is properly restricted. Your teammates are protected. The security audit will go smoothly.\u003c/p\u003e\n\u003cp\u003eOne small problem.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-problem-claude-hallucinated-its-own-feature\"\u003e\u003ca href=\"/posts/claudeignore-dotnet/#the-problem-claude-hallucinated-its-own-feature\" title=\"The Problem: Claude Hallucinated Its Own Feature\"\u003eThe Problem: Claude Hallucinated Its Own Feature\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eHere\u0026rsquo;s what actually happened. Someone asked Claude how to prevent it from touching third-party licensed code in their repository. Claude, being helpful, explained that Claude Code supports a \u003ccode\u003e.claudeignore\u003c/code\u003e file with the same syntax as \u003ccode\u003e.gitignore\u003c/code\u003e: just drop it in the project root, and Claude will leave those files alone.\u003c/p\u003e\n\u003cp\u003eExcept that feature doesn\u0026rsquo;t exist. Claude invented it.\u003c/p\u003e\n\u003cp\u003eThe hallucination spread remarkably well. It turned up in blog posts, Stack Overflow answers, Reddit threads, and engineering wikis. Claude then read those discussions, concluded that \u003ccode\u003e.claudeignore\u003c/code\u003e must be real because everyone was talking about it, and started recommending it again. Confidently. Every time. An AI coding assistant invented a capability it doesn\u0026rsquo;t have, the made-up documentation spread across the internet, and now the AI keeps training on that documentation to reinvent the same capability. You couldn\u0026rsquo;t design a better hallucination feedback loop if you tried.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"how-a-false-sense-of-security-spread\"\u003e\u003ca href=\"/posts/claudeignore-dotnet/#how-a-false-sense-of-security-spread\" title=\"How A False Sense Of Security Spread\"\u003eHow A False Sense Of Security Spread\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eMeanwhile, developers were sleeping soundly. They had a \u003ccode\u003e.claudeignore\u003c/code\u003e in their repository. They had listed \u003ccode\u003eappsettings.Production.json\u003c/code\u003e. They had excluded \u003ccode\u003e.env.*\u003c/code\u003e. They had blocked \u003ccode\u003esecrets/\u003c/code\u003e. They had done the responsible thing, ticked the box, and assured their security teams the AI assistant was properly restricted.\u003c/p\u003e\n\u003cp\u003eProduction database connection strings: protected. API keys: safe. OAuth client secrets: completely off-limits.\u003c/p\u003e\n\u003cp\u003eNone of that was true. The file did nothing. It still does nothing. It is a text file with glob patterns that no tool reads.\u003c/p\u003e\n\u003cp\u003eTo be precise about the scope of the problem: \u003ccode\u003e.claudeignore\u003c/code\u003e is fiction, and it isn\u0026rsquo;t unique to Claude. Neither Claude Code, nor GitHub Copilot, nor OpenAI Codex offers a simple \u003ccode\u003e.claudeignore\u003c/code\u003e-style file at the project root. The mechanisms that actually exist are more fragmented, more verbose, tool-specific, and in several important cases considerably less capable than a plain ignore file would be.\u003c/p\u003e\n\u003cp\u003eHere\u0026rsquo;s what you\u0026rsquo;re actually working with.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"claude-code-permissionsdeny-in-settings\"\u003e\u003ca href=\"/posts/claudeignore-dotnet/#claude-code-permissionsdeny-in-settings\" title=\"Claude Code: permissions.deny in Settings\"\u003eClaude Code: \u003ccode\u003epermissions.deny\u003c/code\u003e in Settings\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eClaude Code\u0026rsquo;s real mechanism for restricting file access is the \u003ccode\u003epermissions\u003c/code\u003e section in \u003ccode\u003e.claude/settings.json\u003c/code\u003e. Instead of a clean, readable standalone file, you get JSON with tool permission syntax:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-json\" data-lang=\"json\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nt\"\u003e\u0026#34;permissions\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nt\"\u003e\u0026#34;deny\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e      \u003cspan class=\"s2\"\u003e\u0026#34;Read(./.env)\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e      \u003cspan class=\"s2\"\u003e\u0026#34;Read(./.env.*)\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e      \u003cspan class=\"s2\"\u003e\u0026#34;Read(./secrets/**)\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e      \u003cspan class=\"s2\"\u003e\u0026#34;Read(./appsettings.Production.json)\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e      \u003cspan class=\"s2\"\u003e\u0026#34;Read(./appsettings.Staging.json)\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eEach entry in the \u003ccode\u003edeny\u003c/code\u003e array is a tool call pattern. \u003ccode\u003eRead(path)\u003c/code\u003e denies Claude Code\u0026rsquo;s Read tool access to the matched path. The path supports glob patterns, so \u003ccode\u003eRead(./secrets/**)\u003c/code\u003e blocks the entire secrets directory tree.\u003c/p\u003e\n\u003cp\u003eThe settings file comes in two variants. \u003ccode\u003e.claude/settings.json\u003c/code\u003e is committed to version control and applies to the whole team. \u003ccode\u003e.claude/settings.local.json\u003c/code\u003e is personal, automatically gitignored by Claude Code when created. For security-relevant exclusions, use \u003ccode\u003esettings.json\u003c/code\u003e so restrictions are consistent across the team.\u003c/p\u003e\n\u003cp\u003eA practical baseline for .NET projects:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-json\" data-lang=\"json\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nt\"\u003e\u0026#34;permissions\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nt\"\u003e\u0026#34;deny\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e      \u003cspan class=\"s2\"\u003e\u0026#34;Read(./.env)\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e      \u003cspan class=\"s2\"\u003e\u0026#34;Read(./.env.*)\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e      \u003cspan class=\"s2\"\u003e\u0026#34;Read(./secrets.json)\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e      \u003cspan class=\"s2\"\u003e\u0026#34;Read(./appsettings.Production.json)\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e      \u003cspan class=\"s2\"\u003e\u0026#34;Read(./appsettings.Staging.json)\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e      \u003cspan class=\"s2\"\u003e\u0026#34;Read(./**/bin/**)\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e      \u003cspan class=\"s2\"\u003e\u0026#34;Read(./**/obj/**)\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\n\n\n\n\u003ch3 id=\"why-deny-rules-are-not-filesystem-blackouts\"\u003e\u003ca href=\"/posts/claudeignore-dotnet/#why-deny-rules-are-not-filesystem-blackouts\" title=\"Why Deny Rules Are Not Filesystem Blackouts\"\u003eWhy Deny Rules Are Not Filesystem Blackouts\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eNow here\u0026rsquo;s the part most writeups about this topic quietly skip: \u003cstrong\u003e\u003ccode\u003epermissions.deny\u003c/code\u003e rules are tool-level controls, not filesystem-level blackouts.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003ccode\u003eRead(./bin/**)\u003c/code\u003e blocks Claude Code\u0026rsquo;s Read tool from opening files in \u003ccode\u003ebin/\u003c/code\u003e. But Glob and Grep are separate tools, and \u003ccode\u003eRead\u003c/code\u003e deny rules don\u0026rsquo;t cover them. Claude can still discover those paths and search their contents. It just can\u0026rsquo;t read a file directly.\u003c/p\u003e\n\u003cp\u003eI asked Claude Code directly whether it had access to \u003ccode\u003ebin/\u003c/code\u003e folders after a \u003ccode\u003eRead\u003c/code\u003e deny rule was active. The answer: yes, Glob found them just fine.\u003c/p\u003e\n\u003cp\u003eSo \u003ccode\u003epermissions.deny\u003c/code\u003e alone does not make a directory invisible. For build artifacts like \u003ccode\u003ebin/\u003c/code\u003e and \u003ccode\u003eobj/\u003c/code\u003e, the more reliable mechanism is the \u003ccode\u003e.gitignore\u003c/code\u003e file you already have. Claude Code respects \u003ccode\u003e.gitignore\u003c/code\u003e by default (\u003ccode\u003erespectGitignore: true\u003c/code\u003e), and that exclusion applies across all discovery mechanisms, not just the Read tool. In practice, directories already in \u003ccode\u003e.gitignore\u003c/code\u003e are not surfaced by Glob or Grep in normal operation. Adding \u003ccode\u003eRead\u003c/code\u003e deny rules on top gives you a second layer, but \u003ccode\u003e.gitignore\u003c/code\u003e is doing the heavier lifting.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"what-this-means-for-secrets-in-the-repo\"\u003e\u003ca href=\"/posts/claudeignore-dotnet/#what-this-means-for-secrets-in-the-repo\" title=\"What This Means For Secrets In The Repo\"\u003eWhat This Means For Secrets In The Repo\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eFor \u003cstrong\u003esecrets and credentials\u003c/strong\u003e specifically, this distinction matters in a way that should make you uncomfortable. A production config file sitting in the repository root is not protected by a \u003ccode\u003eRead\u003c/code\u003e deny rule the way you might assume. Claude can still see that the file exists, discover its path with Glob, and match content patterns against it with Grep. The deny rule prevents reading the full file contents, but the file remains visible. The real answer, as always, is not having that file in the repository at all.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"github-copilot-exists-but-read-the-fine-print\"\u003e\u003ca href=\"/posts/claudeignore-dotnet/#github-copilot-exists-but-read-the-fine-print\" title=\"GitHub Copilot: Exists, But Read the Fine Print\"\u003eGitHub Copilot: Exists, But Read the Fine Print\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eGitHub Copilot does have a content exclusion feature. Before you get optimistic: it requires \u003cstrong\u003eCopilot Business or Copilot Enterprise\u003c/strong\u003e. Individual, Free, and Pro tiers don\u0026rsquo;t get it.\u003c/p\u003e\n\u003cp\u003eIt also isn\u0026rsquo;t a file you commit to your repository. There\u0026rsquo;s no \u003ccode\u003e.copilotignore\u003c/code\u003e. Exclusions are configured through GitHub\u0026rsquo;s web UI at the repository, organization, or enterprise level, and the configuration lives on GitHub\u0026rsquo;s end, not yours. If you want to know what\u0026rsquo;s excluded in a given repository, you navigate through GitHub\u0026rsquo;s admin interface to find out.\u003c/p\u003e\n\u003cp\u003eThe configuration format at least uses familiar YAML path patterns:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s2\"\u003e\u0026#34;*\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"s2\"\u003e\u0026#34;**/.env\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"s2\"\u003e\u0026#34;**/appsettings.Production.json\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"s2\"\u003e\u0026#34;**/secrets/**\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\n\n\n\n\u003ch3 id=\"why-copilot-exclusions-skip-agent-mode\"\u003e\u003ca href=\"/posts/claudeignore-dotnet/#why-copilot-exclusions-skip-agent-mode\" title=\"Why Copilot Exclusions Skip Agent Mode\"\u003eWhy Copilot Exclusions Skip Agent Mode\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eHere\u0026rsquo;s the part that really matters: these restrictions apply when Copilot is accessed through the IDE extension for code completion and inline chat. \u003cstrong\u003eThey do not apply to Agent mode, Copilot CLI, or the cloud agent.\u003c/strong\u003e GitHub states this explicitly in their documentation.\u003c/p\u003e\n\u003cp\u003eThe agentic workflows are exactly the ones where unrestricted file access is most concerning. They\u0026rsquo;re also the ones the exclusion feature doesn\u0026rsquo;t cover.\u003c/p\u003e\n\u003cp\u003eGitHub\u0026rsquo;s content exclusion is a governance feature for enterprise administrators, not a developer-facing tool for managing what an agent can see. The use case it serves is \u0026ldquo;prevent this entire codebase from reaching the AI at the org level.\u0026rdquo; The use case it doesn\u0026rsquo;t serve is \u0026ldquo;control what the agent accesses when I\u0026rsquo;m actively using it.\u0026rdquo;\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"openai-codex-nothing\"\u003e\u003ca href=\"/posts/claudeignore-dotnet/#openai-codex-nothing\" title=\"OpenAI Codex: Nothing\"\u003eOpenAI Codex: Nothing\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eAs of May 2026, OpenAI Codex has no built-in mechanism for file exclusion. Codex is an API. What goes into the context window is entirely the responsibility of the client making the call. If you\u0026rsquo;re using Codex through a third-party tool or IDE integration, that tool may or may not have its own ignore mechanism. The model itself is indifferent to what you send it.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"what-this-means-for-net-projects\"\u003e\u003ca href=\"/posts/claudeignore-dotnet/#what-this-means-for-net-projects\" title=\"What This Means for .NET Projects\"\u003eWhat This Means for .NET Projects\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eGiven the actual state of tooling, here\u0026rsquo;s where that leaves you:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eOn Claude Code\u003c/strong\u003e: use \u003ccode\u003epermissions.deny\u003c/code\u003e in \u003ccode\u003e.claude/settings.json\u003c/code\u003e, but understand you\u0026rsquo;re adding a tool-level restriction, not a filesystem blackout. Prioritize secrets and environment-specific config files. Build artifacts are better handled by \u003ccode\u003e.gitignore\u003c/code\u003e, which you almost certainly already have and which provides broader coverage across all discovery mechanisms.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eOn GitHub Copilot Business or Enterprise\u003c/strong\u003e: configure content exclusions at the organization level for sensitive repositories if you need the governance coverage. Accept that none of this applies to agent-based workflows. For individual context management in the IDE, the exclusion feature is your only option. For agents, there is no equivalent.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eOn Codex or API-based tools\u003c/strong\u003e: the exclusion problem is entirely at the integration layer. Read your tool\u0026rsquo;s documentation, not Codex\u0026rsquo;s.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-feature-that-should-exist-but-doesnt\"\u003e\u003ca href=\"/posts/claudeignore-dotnet/#the-feature-that-should-exist-but-doesnt\" title=\"The Feature That Should Exist But Doesn\u0026rsquo;t\"\u003eThe Feature That Should Exist But Doesn\u0026rsquo;t\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eThe \u003ccode\u003e.claudeignore\u003c/code\u003e hallucination spread so effectively because it named something developers genuinely need: a simple, version-controlled file that tells an AI coding assistant which files to leave alone. Same syntax as \u003ccode\u003e.gitignore\u003c/code\u003e. Lives in the project root. Gets committed like any other configuration. Readable by anyone on the team.\u003c/p\u003e\n\u003cp\u003eThat doesn\u0026rsquo;t exist. What exists instead is a JSON config with Read tool denial patterns that doesn\u0026rsquo;t cover Glob and Grep, a web-based admin feature that costs extra and doesn\u0026rsquo;t work for agents, and an API that does whatever the client tells it to.\u003c/p\u003e\n\u003cp\u003eThe gap between what developers want and what the tooling provides is real. Claude filling that gap by inventing a feature called \u003ccode\u003e.claudeignore\u003c/code\u003e is understandable: the concept is obvious, the need is legitimate, the name is perfect. It\u0026rsquo;s also deeply unhelpful, because developers have implemented \u003ccode\u003e.claudeignore\u003c/code\u003e files in their repositories, committed them, documented them in their onboarding guides, and assumed they were doing something. They weren\u0026rsquo;t.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"keep-credentials-out-of-the-repo\"\u003e\u003ca href=\"/posts/claudeignore-dotnet/#keep-credentials-out-of-the-repo\" title=\"Keep Credentials Out Of The Repo\"\u003eKeep Credentials Out Of The Repo\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eThe actual security takeaway\u003c/strong\u003e: don\u0026rsquo;t put credentials in your repository. Azure Key Vault, user secrets during development, environment variable injection in CI/CD: these are not just best practices, they\u0026rsquo;re the only mechanisms that actually guarantee your production credentials don\u0026rsquo;t end up in an AI context window. \u003ccode\u003epermissions.deny\u003c/code\u003e and GitHub\u0026rsquo;s content exclusion are useful layers on top. They are not a substitute for that foundation.\u003c/p\u003e\n\u003cp\u003eThe \u003ccode\u003e.claudeignore\u003c/code\u003e situation is a concrete illustration of a broader rule worth internalizing: AI tools recommend nonexistent features with the same confident tone they use for everything else. A hallucinated configuration file and a correct one read exactly the same way. Verify against official documentation before you trust it, especially for anything security-adjacent.\u003c/p\u003e\n\u003cp\u003eThe official documentation for Claude Code\u0026rsquo;s actual file exclusion mechanism is at \u003ca href=\"https://code.claude.com/docs/en/settings#excluding-sensitive-files\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003ecode.claude.com/docs/en/settings#excluding-sensitive-files\u003c/a\u003e. That\u0026rsquo;s where a search for \u003ccode\u003e.claudeignore\u003c/code\u003e should have taken you from the beginning.\u003c/p\u003e\n","date_modified":"2026-05-26T10:22:03+02:00","date_published":"2026-05-05T17:00:00+02:00","id":"https://daily-devops.net/posts/claudeignore-dotnet/","language":"en","summary":".claudeignore is a hallucination. Claude invented it, the internet spread it, and now Claude keeps recommending it. Here is what actually works in .NET.\n","tags":["ai-code-assistant","dotnet","github-copilot","devops","security"],"title":".claudeignore Doesn't Exist. Here's What Does.","url":"https://daily-devops.net/posts/claudeignore-dotnet/"},{"authors":[{"name":"Martin Stühmer","url":"https://daily-devops.net/authors/martin/"}],"content_html":"\u003cp\u003eEvery quarter, your compliance team gathers around conference tables reviewing spreadsheets that claim your organization processes personal data lawfully. Meanwhile, your production databases collect new PII fields nobody documented, consent records expire without notification, and deletion requests sit in ticketing systems with no automated verification they were honored. You\u0026rsquo;re not non-compliant because you don\u0026rsquo;t care. You\u0026rsquo;re non-compliant because manual privacy audits cannot keep pace with continuous deployment.\u003c/p\u003e\n\u003cp\u003eHere\u0026rsquo;s the uncomfortable truth: privacy compliance is a continuous state, not a quarterly checkpoint. Every code deployment potentially introduces new PII processing. Every schema migration might create compliance gaps. Every API endpoint modification could violate documented privacy purposes. Manual audits examine yesterday\u0026rsquo;s system while today\u0026rsquo;s changes accumulate unreviewed. By the time the next audit rolls around, you\u0026rsquo;ve already shipped three months of undocumented data collection.\u003c/p\u003e\n\u003cp\u003eThis article demonstrates building .NET CLI tools that automate privacy audit workflows. We\u0026rsquo;ll examine why manual approaches fail, then construct automated solutions using reflection for PII discovery, Entity Framework metadata for database schema analysis, and GitHub Actions for continuous compliance monitoring.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-fatal-manual-privacy-audit-pattern\"\u003e\u003ca href=\"/posts/privacy-audit-automation-dotnet-cli/#the-fatal-manual-privacy-audit-pattern\" title=\"The Fatal Manual Privacy Audit Pattern\"\u003eThe Fatal Manual Privacy Audit Pattern\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eLet\u0026rsquo;s examine what \u0026ldquo;compliance\u0026rdquo; looks like in organizations that rely on quarterly manual reviews:\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"schema-drift-the-silent-compliance-killer\"\u003e\u003ca href=\"/posts/privacy-audit-automation-dotnet-cli/#schema-drift-the-silent-compliance-killer\" title=\"Schema Drift: The Silent Compliance Killer\"\u003eSchema Drift: The Silent Compliance Killer\u003c/a\u003e\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-csharp\" data-lang=\"csharp\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"k\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003eUserProfile\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"n\"\u003eGuid\u003c/span\u003e \u003cspan class=\"n\"\u003eId\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \u003cspan class=\"k\"\u003eget\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003eset\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"kt\"\u003estring\u003c/span\u003e \u003cspan class=\"n\"\u003eEmail\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \u003cspan class=\"k\"\u003eget\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003eset\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"kt\"\u003estring\u003c/span\u003e \u003cspan class=\"n\"\u003eFullName\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \u003cspan class=\"k\"\u003eget\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003eset\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"c1\"\u003e// Added in sprint 47 - privacy docs? What privacy docs?\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"kt\"\u003estring\u003c/span\u003e \u003cspan class=\"n\"\u003eSocialSecurityNumber\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \u003cspan class=\"k\"\u003eget\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003eset\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"kt\"\u003estring\u003c/span\u003e \u003cspan class=\"n\"\u003eBiometricHash\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \u003cspan class=\"k\"\u003eget\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003eset\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eSensitive PII hits production while compliance reviews migration files from three months ago. By the next audit, you\u0026rsquo;ve collected biometric data for an entire quarter without documented legal basis. The spreadsheet says you\u0026rsquo;re compliant. Reality disagrees.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"the-consent-expiration-time-bomb\"\u003e\u003ca href=\"/posts/privacy-audit-automation-dotnet-cli/#the-consent-expiration-time-bomb\" title=\"The Consent Expiration Time Bomb\"\u003eThe Consent Expiration Time Bomb\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eYour consent records have expiration dates. Your tracking spreadsheet has a quarterly reminder to \u0026ldquo;check expired consents.\u0026rdquo; Meanwhile, 14,000 expired consent records remain marked active, and marketing emails continue flowing to users who withdrew consent months ago.\u003c/p\u003e\n\u003cp\u003eGDPR requires making consent withdrawal as easy as giving it. If your expiration tracking relies on someone remembering to run a database query every few months, you\u0026rsquo;re systematically violating this requirement. A daily automated check would catch this in hours, not quarters.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"deletion-requests-hope-is-not-a-strategy\"\u003e\u003ca href=\"/posts/privacy-audit-automation-dotnet-cli/#deletion-requests-hope-is-not-a-strategy\" title=\"Deletion Requests: Hope Is Not a Strategy\"\u003eDeletion Requests: Hope Is Not a Strategy\u003c/a\u003e\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-csharp\" data-lang=\"csharp\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"kd\"\u003easync\u003c/span\u003e \u003cspan class=\"n\"\u003eTask\u003c/span\u003e \u003cspan class=\"n\"\u003eDeleteUserAccount\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eGuid\u003c/span\u003e \u003cspan class=\"n\"\u003euserId\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"c1\"\u003e// Deletes from Users table\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003e_dbContext\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eUsers\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eWhere\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eu\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003eu\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eId\u003c/span\u003e \u003cspan class=\"p\"\u003e==\u003c/span\u003e \u003cspan class=\"n\"\u003euserId\u003c/span\u003e\u003cspan class=\"p\"\u003e).\u003c/span\u003e\u003cspan class=\"n\"\u003eExecuteDeleteAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"c1\"\u003e// But PII remains in: OrderHistory, AuditLogs, EmailCampaigns, \u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"c1\"\u003e// BackupSnapshots, and that analytics database nobody remembers exists\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eUser requests deletion. Support marks ticket resolved. Compliance assumes it happened. Meanwhile, email addresses live on in five different tables across three systems nobody checked. Without automated scanning, deletion requests become best-effort guesses.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"documentation-drift-accurate-records-of-a-system-that-no-longer-exists\"\u003e\u003ca href=\"/posts/privacy-audit-automation-dotnet-cli/#documentation-drift-accurate-records-of-a-system-that-no-longer-exists\" title=\"Documentation Drift: Accurate Records of a System That No Longer Exists\"\u003eDocumentation Drift: Accurate Records of a System That No Longer Exists\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eYour processing records say you collect \u0026ldquo;email addresses, names, and purchase history\u0026rdquo; with 36-month retention. Your actual system now processes health information, geolocation data, and biometric logs (none documented) with 60-month retention. The documentation was accurate. Eighteen months ago. Now it\u0026rsquo;s compliant fiction, and auditors are reviewing a system that no longer exists.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"building-the-fix-net-cli-tools-for-privacy-automation\"\u003e\u003ca href=\"/posts/privacy-audit-automation-dotnet-cli/#building-the-fix-net-cli-tools-for-privacy-automation\" title=\"Building the Fix: .NET CLI Tools for Privacy Automation\"\u003eBuilding the Fix: .NET CLI Tools for Privacy Automation\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eEnough problems. Let\u0026rsquo;s build solutions. The goal: make privacy compliance a build-time fact rather than a quarterly hope. These CLI tools integrate into your CI/CD pipeline, failing builds when privacy violations exist.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"pii-discovery-find-everything-you-forgot-to-document\"\u003e\u003ca href=\"/posts/privacy-audit-automation-dotnet-cli/#pii-discovery-find-everything-you-forgot-to-document\" title=\"PII Discovery: Find Everything You Forgot to Document\"\u003ePII Discovery: Find Everything You Forgot to Document\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eFirst, a CLI tool that scans your codebase and database schema for PII attributes:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-csharp\" data-lang=\"csharp\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"k\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003eDiscoverPiiCommand\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"kd\"\u003easync\u003c/span\u003e \u003cspan class=\"n\"\u003eTask\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003ePrivacyAuditReport\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003eExecuteAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003estring\u003c/span\u003e \u003cspan class=\"n\"\u003eassemblyPath\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"kt\"\u003estring\u003c/span\u003e \u003cspan class=\"n\"\u003econnectionString\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003ereport\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003ePrivacyAuditReport\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003eassembly\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eAssembly\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eLoadFrom\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eassemblyPath\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e// Find all properties marked with [PersonalData]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003epiiProperties\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eassembly\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetTypes\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eSelectMany\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003etype\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003etype\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetProperties\u003c/span\u003e\u003cspan class=\"p\"\u003e())\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eWhere\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eprop\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003eprop\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetCustomAttribute\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003ePersonalDataAttribute\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;()\u003c/span\u003e \u003cspan class=\"p\"\u003e!=\u003c/span\u003e \u003cspan class=\"kc\"\u003enull\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eSelect\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eprop\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003ePiiProperty\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"n\"\u003eTypeName\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eprop\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eDeclaringType\u003c/span\u003e\u003cspan class=\"p\"\u003e?.\u003c/span\u003e\u003cspan class=\"n\"\u003eFullName\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"n\"\u003ePropertyName\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eprop\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eName\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"n\"\u003eLegalBasis\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eprop\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetCustomAttribute\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eLegalBasisAttribute\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;()?.\u003c/span\u003e\u003cspan class=\"n\"\u003eBasis\u003c/span\u003e \u003cspan class=\"p\"\u003e??\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;NOT_DOCUMENTED\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e});\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003ereport\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eDiscoveredPiiProperties\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003epiiProperties\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eToList\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e// Scan EF metadata for database columns\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"k\"\u003eusing\u003c/span\u003e \u003cspan class=\"nn\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003edbContext\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eApplicationDbContext\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003econnectionString\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003eforeach\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003eentityType\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"n\"\u003edbContext\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eModel\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetEntityTypes\u003c/span\u003e\u003cspan class=\"p\"\u003e())\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003epiiColumns\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eentityType\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetProperties\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eWhere\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ep\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003ep\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eFindAnnotation\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Privacy:IsPII\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)?.\u003c/span\u003e\u003cspan class=\"n\"\u003eValue\u003c/span\u003e \u003cspan class=\"k\"\u003eas\u003c/span\u003e \u003cspan class=\"kt\"\u003ebool?\u003c/span\u003e \u003cspan class=\"p\"\u003e==\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"n\"\u003ereport\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eDatabasePiiColumns\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAddRange\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003epiiColumns\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eSelect\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ep\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eDatabasePiiColumn\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"n\"\u003eTableName\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eentityType\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetTableName\u003c/span\u003e\u003cspan class=\"p\"\u003e(),\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"n\"\u003eColumnName\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003ep\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetColumnName\u003c/span\u003e\u003cspan class=\"p\"\u003e(),\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"n\"\u003eLegalBasis\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003ep\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eFindAnnotation\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Privacy:LegalBasis\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)?.\u003c/span\u003e\u003cspan class=\"n\"\u003eValue\u003c/span\u003e \u003cspan class=\"k\"\u003eas\u003c/span\u003e \u003cspan class=\"kt\"\u003estring\u003c/span\u003e \u003cspan class=\"p\"\u003e??\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;UNDOCUMENTED\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e}));\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e// Cross-reference against processing records - anything missing is a compliance gap\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003edocumented\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003eLoadProcessingRecordsAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003ereport\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eUndocumentedPii\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003ereport\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eDatabasePiiColumns\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eWhere\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ecol\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"p\"\u003e!\u003c/span\u003e\u003cspan class=\"n\"\u003edocumented\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAny\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003erec\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003erec\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eCoversColumn\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ecol\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eTableName\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003ecol\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eColumnName\u003c/span\u003e\u003cspan class=\"p\"\u003e)))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eToList\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"n\"\u003ereport\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe tool scans code via reflection and database schema via EF metadata, cross-referencing against your processing records. Anything not documented shows up as a compliance gap. Run it in CI/CD and you\u0026rsquo;ll know within minutes when someone adds undocumented PII.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"consent-monitoring-catch-expirations-before-regulators-do\"\u003e\u003ca href=\"/posts/privacy-audit-automation-dotnet-cli/#consent-monitoring-catch-expirations-before-regulators-do\" title=\"Consent Monitoring: Catch Expirations Before Regulators Do\"\u003eConsent Monitoring: Catch Expirations Before Regulators Do\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eA CLI tool that runs daily to catch consent violations:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-csharp\" data-lang=\"csharp\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"k\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003eConsentAuditCommand\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"kd\"\u003easync\u003c/span\u003e \u003cspan class=\"n\"\u003eTask\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eConsentAuditReport\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003eExecuteAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003etoday\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eDateTime\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eUtcNow\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003ereport\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eConsentAuditReport\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \u003cspan class=\"n\"\u003eAuditDate\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003etoday\u003c/span\u003e \u003cspan class=\"p\"\u003e};\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e// Find expired consents still marked active\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003ereport\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eExpiredActiveConsents\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003e_dbContext\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eMarketingConsents\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eWhere\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ec\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003ec\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eIsActive\u003c/span\u003e \u003cspan class=\"p\"\u003e\u0026amp;\u0026amp;\u003c/span\u003e \u003cspan class=\"n\"\u003ec\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eExpirationDate\u003c/span\u003e \u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e \u003cspan class=\"n\"\u003etoday\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eSelect\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ec\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eExpiredConsent\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"n\"\u003eUserId\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003ec\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eUserId\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"n\"\u003ePurpose\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003ec\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003ePurpose\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"n\"\u003eDaysOverdue\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003eint\u003c/span\u003e\u003cspan class=\"p\"\u003e)(\u003c/span\u003e\u003cspan class=\"n\"\u003etoday\u003c/span\u003e \u003cspan class=\"p\"\u003e-\u003c/span\u003e \u003cspan class=\"n\"\u003ec\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eExpirationDate\u003c/span\u003e\u003cspan class=\"p\"\u003e).\u003c/span\u003e\u003cspan class=\"n\"\u003eTotalDays\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e})\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eToListAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e// Verify \u0026#34;completed\u0026#34; deletion requests actually deleted everything\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003edeletionRequests\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003e_dbContext\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGdprDeletionRequests\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eWhere\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003er\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003er\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eStatus\u003c/span\u003e \u003cspan class=\"p\"\u003e==\u003c/span\u003e \u003cspan class=\"n\"\u003eDeletionStatus\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eCompleted\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eSelect\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003er\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003er\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eUserId\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eToListAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003eforeach\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003euserId\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"n\"\u003edeletionRequests\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003eremainingData\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003eScanAllTablesForUser\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003euserId\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eremainingData\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAny\u003c/span\u003e\u003cspan class=\"p\"\u003e())\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"n\"\u003ereport\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eIncompleteDeletions\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAdd\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eIncompleteDeletion\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \u003cspan class=\"n\"\u003eUserId\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003euserId\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eLocations\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eremainingData\u003c/span\u003e \u003cspan class=\"p\"\u003e});\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"n\"\u003ereport\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eRun this daily. When it finds 14,000 expired consents still active, you\u0026rsquo;ll know about it the same day, not three months later during the quarterly audit.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"change-detection-know-when-privacy-impact-assessments-need-updates\"\u003e\u003ca href=\"/posts/privacy-audit-automation-dotnet-cli/#change-detection-know-when-privacy-impact-assessments-need-updates\" title=\"Change Detection: Know When Privacy Impact Assessments Need Updates\"\u003eChange Detection: Know When Privacy Impact Assessments Need Updates\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eGit integration catches privacy-impacting changes automatically:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-csharp\" data-lang=\"csharp\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"k\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003eDpiaChangeDetectionCommand\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"kd\"\u003easync\u003c/span\u003e \u003cspan class=\"n\"\u003eTask\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eDpiaChangeReport\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003eExecuteAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003estring\u003c/span\u003e \u003cspan class=\"n\"\u003ecurrentCommit\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"kt\"\u003estring\u003c/span\u003e \u003cspan class=\"n\"\u003epreviousCommit\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003ereport\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eDpiaChangeReport\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \u003cspan class=\"n\"\u003eCurrentCommit\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003ecurrentCommit\u003c/span\u003e \u003cspan class=\"p\"\u003e};\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003egitDiff\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003eGetGitDiffAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003epreviousCommit\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003ecurrentCommit\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003eforeach\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003echangedFile\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"n\"\u003egitDiff\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eChangedFiles\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eWhere\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ef\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003ef\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003ePath\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eEndsWith\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;.cs\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003ecurrentTree\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eCSharpSyntaxTree\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eParseText\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003eFile\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eReadAllTextAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003echangedFile\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003ePath\u003c/span\u003e\u003cspan class=\"p\"\u003e));\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003epreviousTree\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eCSharpSyntaxTree\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eParseText\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003eGetFileContentAtCommitAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003echangedFile\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003ePath\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003epreviousCommit\u003c/span\u003e\u003cspan class=\"p\"\u003e));\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"c1\"\u003e// Detect new [PersonalData] properties\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003enewPiiFields\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eGetPiiProperties\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ecurrentTree\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetRoot\u003c/span\u003e\u003cspan class=\"p\"\u003e())\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eExcept\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eGetPiiProperties\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003epreviousTree\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetRoot\u003c/span\u003e\u003cspan class=\"p\"\u003e()))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eToList\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003enewPiiFields\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAny\u003c/span\u003e\u003cspan class=\"p\"\u003e())\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"n\"\u003ereport\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eFilesWithPrivacyChanges\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAdd\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003ePrivacyChange\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                    \u003cspan class=\"n\"\u003eFilePath\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003echangedFile\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003ePath\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                    \u003cspan class=\"n\"\u003eNewPiiFields\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003enewPiiFields\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                    \u003cspan class=\"n\"\u003eRequiresDpiaUpdate\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"p\"\u003e});\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"n\"\u003ereport\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eNew PII field added in a commit? The tool flags it. External data flow changed? Flagged. Now your Data Protection Impact Assessment updates happen as part of code review, not eighteen months later when an auditor notices the mismatch.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"deletion-verification-test-that-it-actually-works\"\u003e\u003ca href=\"/posts/privacy-audit-automation-dotnet-cli/#deletion-verification-test-that-it-actually-works\" title=\"Deletion Verification: Test That It Actually Works\"\u003eDeletion Verification: Test That It Actually Works\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eDon\u0026rsquo;t trust that deletion requests were honored. Verify:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-csharp\" data-lang=\"csharp\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"k\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003eDeletionVerificationCommand\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"kd\"\u003easync\u003c/span\u003e \u003cspan class=\"n\"\u003eTask\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eDeletionReport\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003eExecuteAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e// Find all tables containing user PII\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003epiiTables\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003e_dbContext\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eModel\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetEntityTypes\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eWhere\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ee\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003ee\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetProperties\u003c/span\u003e\u003cspan class=\"p\"\u003e().\u003c/span\u003e\u003cspan class=\"n\"\u003eAny\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ep\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003ep\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eFindAnnotation\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Privacy:IsPII\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)?.\u003c/span\u003e\u003cspan class=\"n\"\u003eValue\u003c/span\u003e \u003cspan class=\"k\"\u003eas\u003c/span\u003e \u003cspan class=\"kt\"\u003ebool?\u003c/span\u003e \u003cspan class=\"p\"\u003e==\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\u003cspan class=\"p\"\u003e))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eSelect\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ee\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003ee\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetTableName\u003c/span\u003e\u003cspan class=\"p\"\u003e())\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eToList\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e// Create synthetic test user with data in all PII tables\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003etestUserId\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003eCreateSyntheticUserWithPii\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003e_deletionService\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eDeleteUserAccount\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003etestUserId\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e// Verify deletion actually worked\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003ereport\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eDeletionReport\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003eforeach\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003etable\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"n\"\u003epiiTables\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003eremainingRecords\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003eCountRecordsForUser\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003etable\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003etestUserId\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eremainingRecords\u003c/span\u003e \u003cspan class=\"p\"\u003e\u0026gt;\u003c/span\u003e \u003cspan class=\"m\"\u003e0\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"n\"\u003ereport\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eIssues\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAdd\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e$\u0026#34;Table {table} still contains {remainingRecords} records after deletion\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"n\"\u003ereport\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis creates a synthetic user, exercises deletion, then verifies every PII table is actually empty. No more assuming deletion worked. Prove it.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"processing-records-from-code-not-spreadsheets\"\u003e\u003ca href=\"/posts/privacy-audit-automation-dotnet-cli/#processing-records-from-code-not-spreadsheets\" title=\"Processing Records From Code, Not Spreadsheets\"\u003eProcessing Records From Code, Not Spreadsheets\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eGenerate compliance documentation from your actual system state:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-csharp\" data-lang=\"csharp\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"k\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003eGenerateProcessingRecordCommand\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"kd\"\u003easync\u003c/span\u003e \u003cspan class=\"n\"\u003eTask\u003c/span\u003e \u003cspan class=\"n\"\u003eExecuteAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"k\"\u003erecord\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eProcessingRecord\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \u003cspan class=\"n\"\u003eGeneratedDate\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eDateTime\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eUtcNow\u003c/span\u003e \u003cspan class=\"p\"\u003e};\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003eforeach\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003eentityType\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"n\"\u003e_dbContext\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eModel\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetEntityTypes\u003c/span\u003e\u003cspan class=\"p\"\u003e())\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003epiiProperties\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eentityType\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetProperties\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eWhere\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ep\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003ep\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eFindAnnotation\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Privacy:IsPII\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)?.\u003c/span\u003e\u003cspan class=\"n\"\u003eValue\u003c/span\u003e \u003cspan class=\"k\"\u003eas\u003c/span\u003e \u003cspan class=\"kt\"\u003ebool?\u003c/span\u003e \u003cspan class=\"p\"\u003e==\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eToList\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"p\"\u003e(!\u003c/span\u003e\u003cspan class=\"n\"\u003epiiProperties\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAny\u003c/span\u003e\u003cspan class=\"p\"\u003e())\u003c/span\u003e \u003cspan class=\"k\"\u003econtinue\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"n\"\u003erecord\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eActivities\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAdd\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eProcessingActivity\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"n\"\u003eName\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eentityType\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eClrType\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eName\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"n\"\u003eDataCategories\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003epiiProperties\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eSelect\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ep\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003ep\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eFindAnnotation\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Privacy:Category\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)?.\u003c/span\u003e\u003cspan class=\"n\"\u003eValue\u003c/span\u003e \u003cspan class=\"k\"\u003eas\u003c/span\u003e \u003cspan class=\"kt\"\u003estring\u003c/span\u003e\u003cspan class=\"p\"\u003e).\u003c/span\u003e\u003cspan class=\"n\"\u003eToList\u003c/span\u003e\u003cspan class=\"p\"\u003e(),\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"n\"\u003eLegalBasis\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eentityType\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eFindAnnotation\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Privacy:LegalBasis\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)?.\u003c/span\u003e\u003cspan class=\"n\"\u003eValue\u003c/span\u003e \u003cspan class=\"k\"\u003eas\u003c/span\u003e \u003cspan class=\"kt\"\u003estring\u003c/span\u003e \u003cspan class=\"p\"\u003e??\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;NOT_DOCUMENTED\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"n\"\u003ePurpose\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eentityType\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eFindAnnotation\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Privacy:Purpose\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)?.\u003c/span\u003e\u003cspan class=\"n\"\u003eValue\u003c/span\u003e \u003cspan class=\"k\"\u003eas\u003c/span\u003e \u003cspan class=\"kt\"\u003estring\u003c/span\u003e \u003cspan class=\"p\"\u003e??\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;NOT_DOCUMENTED\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e});\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003eFile\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eWriteAllTextAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;ProcessingRecord.json\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eJsonSerializer\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eSerialize\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003erecord\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eJsonSerializerOptions\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \u003cspan class=\"n\"\u003eWriteIndented\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e \u003cspan class=\"p\"\u003e}));\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eDocumentation generated from code annotations is always current. When someone adds a new PII field and forgets to annotate it, the tool flags it as undocumented. The processing record reflects reality, not eighteen-month-old assumptions.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"github-actions-integration\"\u003e\u003ca href=\"/posts/privacy-audit-automation-dotnet-cli/#github-actions-integration\" title=\"GitHub Actions Integration\"\u003eGitHub Actions Integration\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eWire everything into CI/CD:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ePrivacy Compliance Audit\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eon\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003epush\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ebranches\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"l\"\u003emain, develop]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003epull_request\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eschedule\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e- \u003cspan class=\"nt\"\u003ecron\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;0 2 * * *\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"c\"\u003e# Daily at 2 AM\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ejobs\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eprivacy-audit\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eruns-on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eubuntu-latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esteps\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/checkout@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ewith\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003efetch-depth\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e2\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/setup-dotnet@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ewith\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003edotnet-version\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;9.0.x\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eDiscover Undocumented PII\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edotnet run --project PrivacyAudit.Cli -- discover-pii --output pii-report.json\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eAudit Consent Records\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edotnet run --project PrivacyAudit.Cli -- audit-consent --output consent-report.json\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eDetect Privacy-Impacting Changes\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edotnet run --project PrivacyAudit.Cli -- detect-changes --output changes-report.json\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eVerify Deletion Completeness\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edotnet run --project PrivacyAudit.Cli -- verify-deletion --output deletion-report.json\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eFail on Violations\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edotnet run --project PrivacyAudit.Cli -- check-compliance --fail-on-violations true\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/upload-artifact@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eif\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ealways()\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ewith\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eprivacy-audit-reports\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;*-report.json\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eUndocumented PII? Build fails. Expired consents? Build fails. Incomplete deletions? Build fails. Compliance becomes a technical gate, not a quarterly prayer meeting.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"what-these-tools-actually-prove\"\u003e\u003ca href=\"/posts/privacy-audit-automation-dotnet-cli/#what-these-tools-actually-prove\" title=\"What These Tools Actually Prove\"\u003eWhat These Tools Actually Prove\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eLet\u0026rsquo;s map this back to what regulators care about:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eLegal basis documentation\u003c/strong\u003e: The PII discovery tool scans everything and flags undocumented processing. No more hoping developers remembered to update the spreadsheet.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent management\u003c/strong\u003e: Daily automated checks catch expired consents within 24 hours, not 90 days. When someone asks \u0026ldquo;how do you ensure consent is current?\u0026rdquo;, you have timestamped reports.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDeletion verification\u003c/strong\u003e: Synthetic user tests prove deletion works across all systems. When someone asks \u0026ldquo;how do you verify erasure requests?\u0026rdquo;, you have test results, not assurances.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eChange detection\u003c/strong\u003e: Git-integrated analysis flags privacy-impacting code changes at PR time. Impact assessments update when code changes, not whenever someone remembers.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eProcessing records\u003c/strong\u003e: Documentation generated from code annotations reflects current reality. The record you show auditors matches the system they\u0026rsquo;re auditing.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"getting-started-without-boiling-the-ocean\"\u003e\u003ca href=\"/posts/privacy-audit-automation-dotnet-cli/#getting-started-without-boiling-the-ocean\" title=\"Getting Started Without Boiling the Ocean\"\u003eGetting Started Without Boiling the Ocean\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eDon\u0026rsquo;t try to implement everything at once:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWeek 1-2\u003c/strong\u003e: Deploy PII discovery. Baseline your current state. You\u0026rsquo;ll find undocumented fields in 40-60% of your tables. That\u0026rsquo;s normal. Start documenting.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWeek 3-4\u003c/strong\u003e: Add consent monitoring. Run it daily. When you discover 14,000 expired consents nobody knew about, you\u0026rsquo;ll understand why quarterly audits don\u0026rsquo;t work.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWeek 5-6\u003c/strong\u003e: Integrate change detection into CI/CD. Make privacy compliance a build requirement. New undocumented PII stops reaching production.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWeek 7-8\u003c/strong\u003e: Deploy deletion verification. Create synthetic test users. Prove erasure actually works.\u003c/p\u003e\n\u003cp\u003eStart with non-production environments to tune false positive rates. Privacy audit automation reveals uncomfortable truths. Expect resistance from teams who prefer checkbox compliance to technical accountability.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-uncomfortable-truth\"\u003e\u003ca href=\"/posts/privacy-audit-automation-dotnet-cli/#the-uncomfortable-truth\" title=\"The Uncomfortable Truth\"\u003eThe Uncomfortable Truth\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eManual privacy audits are security theater. Quarterly spreadsheet reviews cannot detect PII added last Tuesday, consent that expired this morning, or deletion requests honored in some tables but not others.\u003c/p\u003e\n\u003cp\u003ePrivacy compliance is a continuous technical state, not a point-in-time assessment. Your compliance documentation is either generated from your actual system state, or it\u0026rsquo;s fiction. There\u0026rsquo;s no middle ground.\u003c/p\u003e\n\u003cp\u003eBuild .NET CLI tools that make compliance a build-time fact. Use reflection to discover all PII. Use Entity Framework metadata to scan schemas. Use Git integration to catch privacy-impacting changes. Fail builds when violations exist.\u003c/p\u003e\n\u003cp\u003eThe tools in this article are a starting point. Real implementations need customization for your specific architecture and regulatory requirements. But the fundamental principle stands: automate privacy audits or accept that your compliance documentation is wishful thinking.\u003c/p\u003e\n\u003cp\u003eStop reviewing spreadsheets. Start building proof.\u003c/p\u003e\n","date_modified":"2026-05-26T10:22:03+02:00","date_published":"2026-04-30T17:00:00+02:00","id":"https://daily-devops.net/posts/privacy-audit-automation-dotnet-cli/","language":"en","summary":"Quarterly audits can't catch PII added last Tuesday. Build .NET CLI tools that make compliance a build-time fact, not a spreadsheet fantasy.\n","tags":["iso-standards","privacy","cli","dotnet","automation","gdpr","compliance","security","codequality","devops"],"title":"Your Privacy Docs Are Fiction: Let's Fix That with .NET CLI Tools\n","url":"https://daily-devops.net/posts/privacy-audit-automation-dotnet-cli/"},{"authors":[{"name":"Martin Stühmer","url":"https://daily-devops.net/authors/martin/"}],"content_html":"\u003cp\u003eYour security tests pass. Great. But when did they actually run? Against which code version? Can you prove it wasn\u0026rsquo;t last Tuesday\u0026rsquo;s build you\u0026rsquo;re showing?\u003c/p\u003e\n\u003cp\u003eMost security testing lives in Word documents, Postman exports, and screenshot folders on SharePoint. The tests themselves might be perfectly valid. The problem is traceability: there\u0026rsquo;s no systematic link between test execution and the code being validated.\u003c/p\u003e\n\u003cp\u003eCLI-based security testing changes this equation. Instead of tests that produce reports, you build tests that prove themselves. Every execution generates structured logs with timestamps, correlation IDs, and commit hashes. The evidence trail isn\u0026rsquo;t something you create after the fact. It\u0026rsquo;s a byproduct of running the tests.\u003c/p\u003e\n\u003cp\u003eThis approach works whether you\u0026rsquo;re preparing for compliance reviews or simply want confidence that your security controls actually function in the code you\u0026rsquo;re about to deploy.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-documentation-problem\"\u003e\u003ca href=\"/posts/cli-security-testing-audit/#the-documentation-problem\" title=\"The Documentation Problem\"\u003eThe Documentation Problem\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eRecognize this pattern?\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eSecurity_Test_Report_Q2_2024.docx\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e✓ Authentication bypass: Tried /admin without token, got 401\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e✓ SQL injection: Tried \u0026#39; OR \u0026#39;1\u0026#39;=\u0026#39;1, got error message  \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e✓ Rate limiting: Sent 10 requests, got rate limited\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e✓ Authorization: User A couldn\u0026#39;t access User B\u0026#39;s data\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eEvidence: Screenshots in SharePoint\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eNext scheduled test: Q3 2024\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe tests are valid. The evidence isn\u0026rsquo;t.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eNo repeatability\u003c/strong\u003e: Manual tests run differently each time. Regression goes undetected.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eNo correlation\u003c/strong\u003e: Tests run quarterly. Code deploys daily. The gap between \u0026ldquo;tested\u0026rdquo; and \u0026ldquo;deployed\u0026rdquo; grows with every sprint.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eNo traceability\u003c/strong\u003e: Which deployment fixed which vulnerability? That question requires digging through months of documentation.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eNo automation\u003c/strong\u003e: Security validation waits for team availability instead of running with every build.\u003c/p\u003e\n\u003cp\u003eThe fix isn\u0026rsquo;t better documentation. It\u0026rsquo;s tests that document themselves.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"building-self-documenting-security-tests\"\u003e\u003ca href=\"/posts/cli-security-testing-audit/#building-self-documenting-security-tests\" title=\"Building Self-Documenting Security Tests\"\u003eBuilding Self-Documenting Security Tests\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eThe approach uses xUnit with ASP.NET Core\u0026rsquo;s \u003ccode\u003eWebApplicationFactory\u003c/code\u003e. This combination lets you test your application in-memory without deploying to actual infrastructure. More importantly, it integrates seamlessly with CI/CD pipelines that capture structured output.\u003c/p\u003e\n\u003cp\u003eThe key insight: every test should validate a specific security boundary and produce output that links execution to the code version being tested. You\u0026rsquo;re not writing tests that generate reports. You\u0026rsquo;re writing tests that generate evidence.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"the-core-pattern\"\u003e\u003ca href=\"/posts/cli-security-testing-audit/#the-core-pattern\" title=\"The Core Pattern\"\u003eThe Core Pattern\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eAuthentication boundaries are the natural starting point. They\u0026rsquo;re well-understood, frequently attacked, and straightforward to validate. A test for unauthenticated access checks three things: the response code, the presence of proper authentication headers, and the absence of sensitive information in error messages.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-csharp\" data-lang=\"csharp\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"k\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003eSecurityTests\u003c/span\u003e \u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"n\"\u003eIClassFixture\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eWebApplicationFactory\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eProgram\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003eprivate\u003c/span\u003e \u003cspan class=\"k\"\u003ereadonly\u003c/span\u003e \u003cspan class=\"n\"\u003eHttpClient\u003c/span\u003e \u003cspan class=\"n\"\u003e_client\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"n\"\u003eSecurityTests\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eWebApplicationFactory\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eProgram\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003efactory\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003e_client\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003efactory\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eCreateClient\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003e    [Fact]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003e    [Trait(\u0026#34;Category\u0026#34;, \u0026#34;Security\u0026#34;)]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"kd\"\u003easync\u003c/span\u003e \u003cspan class=\"n\"\u003eTask\u003c/span\u003e \u003cspan class=\"n\"\u003eProtectedEndpoint_NoToken_Returns401\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003eresponse\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003e_client\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;/api/users/profile\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003eAssert\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eEqual\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eHttpStatusCode\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eUnauthorized\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eresponse\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eStatusCode\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e// Error responses must not leak internal details\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003econtent\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003eresponse\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eContent\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eReadAsStringAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003eAssert\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eDoesNotContain\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;database\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003econtent\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eStringComparison\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eOrdinalIgnoreCase\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003eAssert\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eDoesNotContain\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;stack\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003econtent\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eStringComparison\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eOrdinalIgnoreCase\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003e    [Fact]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003e    [Trait(\u0026#34;Category\u0026#34;, \u0026#34;Security\u0026#34;)]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"kd\"\u003easync\u003c/span\u003e \u003cspan class=\"n\"\u003eTask\u003c/span\u003e \u003cspan class=\"n\"\u003eCrossUserAccess_Returns403\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003e_client\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eDefaultRequestHeaders\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAuthorization\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eAuthenticationHeaderValue\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Bearer\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;token-for-userA\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e// User A attempts to access User B\u0026#39;s data\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003eresponse\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003e_client\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;/api/users/userB-id/profile\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003eAssert\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eEqual\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eHttpStatusCode\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eForbidden\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eresponse\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eStatusCode\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003e    [Theory]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003e    [InlineData(\u0026#34;\u0026#39; OR \u0026#39;1\u0026#39;=\u0026#39;1\u0026#34;)]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003e    [InlineData(\u0026#34;\u0026lt;script\u0026gt;alert(\u0026#39;xss\u0026#39;)\u0026lt;/script\u0026gt;\u0026#34;)]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"na\"\u003e    [Trait(\u0026#34;Category\u0026#34;, \u0026#34;Security\u0026#34;)]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"kd\"\u003easync\u003c/span\u003e \u003cspan class=\"n\"\u003eTask\u003c/span\u003e \u003cspan class=\"n\"\u003eSearchEndpoint_MaliciousInput_Sanitized\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003estring\u003c/span\u003e \u003cspan class=\"n\"\u003epayload\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003eresponse\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003e_client\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"s\"\u003e$\u0026#34;/api/search?q={Uri.EscapeDataString(payload)}\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eresponse\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eIsSuccessStatusCode\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003econtent\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003eresponse\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eContent\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eReadAsStringAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"n\"\u003eAssert\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eDoesNotContain\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;\u0026lt;script\u0026gt;\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003econtent\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe \u003ccode\u003e[Trait(\u0026quot;Category\u0026quot;, \u0026quot;Security\u0026quot;)]\u003c/code\u003e attribute enables filtering. You can run \u003ccode\u003edotnet test --filter \u0026quot;Category=Security\u0026quot;\u003c/code\u003e to execute only security tests, which is useful for CI/CD pipelines where you want security validation as a separate gate.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"what-makes-this-self-documenting\"\u003e\u003ca href=\"/posts/cli-security-testing-audit/#what-makes-this-self-documenting\" title=\"What Makes This Self-Documenting\"\u003eWhat Makes This Self-Documenting\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eThe test output itself becomes evidence. When tests run in CI/CD, the execution context (commit hash, build number, timestamp) gets captured automatically in the pipeline logs. You don\u0026rsquo;t need to generate separate reports. The test run \u003cem\u003eis\u003c/em\u003e the report.\u003c/p\u003e\n\u003cp\u003eFor explicit logging, add a helper that writes structured output:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-csharp\" data-lang=\"csharp\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"kd\"\u003estatic\u003c/span\u003e \u003cspan class=\"k\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003eSecurityTestLog\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"kd\"\u003estatic\u003c/span\u003e \u003cspan class=\"k\"\u003evoid\u003c/span\u003e \u003cspan class=\"n\"\u003eWrite\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003estring\u003c/span\u003e \u003cspan class=\"n\"\u003etestName\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"kt\"\u003ebool\u003c/span\u003e \u003cspan class=\"n\"\u003epassed\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003eentry\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"n\"\u003eTimestamp\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eDateTimeOffset\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eUtcNow\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"n\"\u003eTestName\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003etestName\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"n\"\u003eResult\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003epassed\u003c/span\u003e \u003cspan class=\"p\"\u003e?\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;PASS\u0026#34;\u003c/span\u003e \u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;FAIL\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"n\"\u003eCommitSha\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eEnvironment\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetEnvironmentVariable\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;GITHUB_SHA\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"p\"\u003e??\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;local\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e};\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003eConsole\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eWriteLine\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e$\u0026#34;SECURITY_TEST: {JsonSerializer.Serialize(entry)}\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e// Save the entry to a file or database if needed\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"c1\"\u003e// ...\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis structured output gets captured in CI/CD logs, creating a searchable history of every security test execution across every deployment.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"running-tests-in-cicd\"\u003e\u003ca href=\"/posts/cli-security-testing-audit/#running-tests-in-cicd\" title=\"Running Tests in CI/CD\"\u003eRunning Tests in CI/CD\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eThe real value emerges when these tests run on every commit. In a CI/CD pipeline, each test execution automatically captures the commit hash, build number, and timestamp. This context transforms test results from \u0026ldquo;tests passed\u0026rdquo; into \u0026ldquo;tests passed for commit abc123 at 2024-06-15T14:32:00Z.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eA minimal GitHub Actions workflow runs security tests and preserves results:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eRun Security Tests\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edotnet test --filter \u0026#34;Category=Security\u0026#34; --logger \u0026#34;trx\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eenv\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eGITHUB_SHA\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e${{ github.sha }}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eUpload Results\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/upload-artifact@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ewith\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003esecurity-results-${{ github.run_number }}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e./TestResults/**/*.trx\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eretention-days\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e90\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe 90-day retention creates a historical record. When someone asks \u0026ldquo;was this tested before deployment?\u0026rdquo; you can point to specific artifacts linked to specific commits. The evidence exists independent of any documentation someone might have written.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"what-changes\"\u003e\u003ca href=\"/posts/cli-security-testing-audit/#what-changes\" title=\"What Changes\"\u003eWhat Changes\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eOnce security tests run in CI/CD with proper artifact retention, several things shift.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eRegression detection becomes automatic.\u003c/strong\u003e A vulnerability fixed in March stays fixed. If code changes reintroduce it in September, the test fails immediately rather than waiting for the next quarterly review.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eThe \u0026ldquo;tested vs. deployed\u0026rdquo; gap closes.\u003c/strong\u003e When tests run on every pull request, the code being deployed is the code that was tested. No more hoping that the security validation from three weeks ago still applies to today\u0026rsquo;s release.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEvidence generation becomes passive.\u003c/strong\u003e You\u0026rsquo;re not creating compliance documentation. The documentation creates itself as a byproduct of running tests. Pipeline logs, test artifacts, and commit history combine into an evidence trail that\u0026rsquo;s harder to fabricate than a Word document.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSecurity testing scales with development velocity.\u003c/strong\u003e The team deploys five times a day? Security tests run five times a day. No bottleneck waiting for security team availability.\u003c/p\u003e\n\u003cp\u003eThis matters for compliance reviews, certainly. But it also matters for the simpler question: \u0026ldquo;Do our security controls actually work in the code we\u0026rsquo;re shipping?\u0026rdquo; Automated tests answer that question continuously, not quarterly.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"where-to-start\"\u003e\u003ca href=\"/posts/cli-security-testing-audit/#where-to-start\" title=\"Where to Start\"\u003eWhere to Start\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eBegin with authentication tests. They validate the most commonly attacked boundary and demonstrate the pattern clearly. Use \u003ccode\u003eWebApplicationFactory\u0026lt;TProgram\u0026gt;\u003c/code\u003e to test your ASP.NET Core application in-memory. This requires no deployed infrastructure and runs fast enough for CI/CD feedback loops.\u003c/p\u003e\n\u003cp\u003eOrganize tests with \u003ccode\u003e[Trait(\u0026quot;Category\u0026quot;, \u0026quot;Security\u0026quot;)]\u003c/code\u003e from the start. This enables running security tests separately from unit tests, which is useful when you want security validation as a distinct pipeline gate. For teams seeking a cleaner approach, the open-source \u003ca href=\"https://www.nuget.org/packages/NetEvolve.Extensions.XUnit.V3\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eNetEvolve.Extensions.XUnit.V3\u003c/a\u003e package provides standardized attributes like \u003ccode\u003e[IntegrationTest]\u003c/code\u003e or \u003ccode\u003e[AcceptanceTest]\u003c/code\u003e that work consistently across xUnit, NUnit, MSTest, and TUnit with the same \u003ccode\u003edotnet test --filter TestCategory=...\u003c/code\u003e syntax.\u003c/p\u003e\n\u003ca href=\"https://github.com/dailydevops/extensions.test\" class=\"linked\" target=\"_blank\" rel=\"noopener external noreferrer\" title=\"Compatibility library for solutions using multiple .NET test frameworks.\"\u003e\n  \u003cimg src=\"/images/github-dailydevops-extensions.test.png\" class=\"repository\" width=\"1200\" height=\"630\" title=\"Compatibility library for solutions using multiple .NET test frameworks.\" alt=\"Compatibility library for solutions using multiple .NET test frameworks.\" /\u003e\n\u003c/a\u003e\n\u003cp\u003eConfigure artifact retention for at least 90 days. Shorter retention means you can\u0026rsquo;t demonstrate testing history during compliance reviews. Longer retention costs storage but provides deeper history.\u003c/p\u003e\n\u003cp\u003eStart small. Three or four authentication tests that run on every commit provide more value than 50 tests that run quarterly. The goal is continuous validation, not comprehensive coverage on day one.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-shift\"\u003e\u003ca href=\"/posts/cli-security-testing-audit/#the-shift\" title=\"The Shift\"\u003eThe Shift\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eManual testing produces documents. Automated testing produces evidence.\u003c/p\u003e\n\u003cp\u003eWhen someone asks \u0026ldquo;How do you verify security testing?\u0026rdquo; the answer changes. Instead of pointing to a quarterly report, you point to 847 test executions across 23 deployments, each linked to a specific commit and preserved in pipeline artifacts.\u003c/p\u003e\n\u003cp\u003eSecurity professionals still define what to test. The automation handles execution, logging, and retention. The result: security validation that runs continuously and proves itself without requiring anyone to write a report.\u003c/p\u003e\n","date_modified":"2026-05-26T10:22:03+02:00","date_published":"2026-04-28T17:00:00+02:00","id":"https://daily-devops.net/posts/cli-security-testing-audit/","language":"en","summary":"Build xUnit and WebApplicationFactory security tests that emit timestamped evidence tied to commit hashes. Retire the SharePoint screenshot folder.","tags":["iso-standards","security","cli","dotnet","testing","compliance","devops"],"title":"Security Tests That Prove Themselves","url":"https://daily-devops.net/posts/cli-security-testing-audit/"},{"authors":[{"name":"Martin Stühmer","url":"https://daily-devops.net/authors/martin/"}],"content_html":"\u003cp\u003eI\u0026rsquo;ve watched this trainwreck unfold a dozen times: Organization gets certified, consultants cash their checks, comprehensive documentation gets filed somewhere, and then\u0026hellip; compliance becomes a Word document ritual. \u0026ldquo;Just check the Azure portal\u0026rdquo; before each release. Screenshot some settings. Sign the checklist. Ship it.\u003c/p\u003e\n\u003cp\u003eThree months later, an audit exposes configuration drift, hardcoded secrets in production, and vulnerable dependencies nobody noticed. The compliance team swears everything was checked. The forensic evidence disagrees.\u003c/p\u003e\n\u003cp\u003eHere\u0026rsquo;s what nobody wants to admit: Manual compliance verification is fundamentally broken. Not \u0026ldquo;could be improved\u0026rdquo; broken. Broken in ways that would make your auditor weep if they understood software. Screenshots aren\u0026rsquo;t documentation. Different people finding different issues isn\u0026rsquo;t \u0026ldquo;thorough review,\u0026rdquo; it\u0026rsquo;s randomness. And \u0026ldquo;I checked it last Tuesday\u0026rdquo; isn\u0026rsquo;t an evidence trail.\u003c/p\u003e\n\u003cp\u003eThe fix isn\u0026rsquo;t more checklists or stricter sign-off procedures. It\u0026rsquo;s treating compliance as what it actually is: an engineering problem. One that demands automated tooling in your CI/CD pipeline, not a rotating sacrifice of whoever drew the short straw this sprint.\u003c/p\u003e\n\u003cp\u003e.NET gives you everything needed: \u003ccode\u003eSystem.CommandLine\u003c/code\u003e for CLI interfaces, Roslyn for code inspection, Azure SDK for infrastructure validation. Let me show you what real compliance automation looks like.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-checklist-ceremony-a-horror-story\"\u003e\u003ca href=\"/posts/compliance-verification-dotnet-cli/#the-checklist-ceremony-a-horror-story\" title=\"The Checklist Ceremony (A Horror Story)\"\u003eThe Checklist Ceremony (A Horror Story)\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eLet\u0026rsquo;s talk about what I see in organizations that claim to be \u0026ldquo;compliant.\u0026rdquo;\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"the-sacred-document\"\u003e\u003ca href=\"/posts/compliance-verification-dotnet-cli/#the-sacred-document\" title=\"The Sacred Document\"\u003eThe Sacred Document\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eA Word document titled \u003ccode\u003eSecurity_Compliance_Checklist_v3_FINAL_v2_REALLY_FINAL.docx\u003c/code\u003e gets circulated before each release. Someone (usually whoever lost the argument about who has to do it this time) manually verifies items:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e☐ Check code for hardcoded secrets (search GitHub for \u0026#34;password\u0026#34;, \u0026#34;apikey\u0026#34;)\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e☐ Verify all API endpoints require authentication (manually test 10 endpoints)\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e☐ Confirm Azure Key Vault configured correctly (log into portal, screenshot settings)\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e☐ Validate database encryption enabled (run SQL query, save results)\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e☐ Check dependency versions for known vulnerabilities (visit NuGet.org, manually search)\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e☐ Confirm HTTPS enforced on all services (curl a few endpoints)\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eEach check gets marked complete. Someone signs off. The deployment proceeds. Everyone feels very professional.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"what-actually-happens-spoiler-nothing-good\"\u003e\u003ca href=\"/posts/compliance-verification-dotnet-cli/#what-actually-happens-spoiler-nothing-good\" title=\"What Actually Happens (Spoiler: Nothing Good)\"\u003eWhat Actually Happens (Spoiler: Nothing Good)\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eInconsistent execution.\u003c/strong\u003e Different team members interpret \u0026ldquo;check for secrets\u0026rdquo; differently. One searches for \u0026ldquo;password\u0026rdquo; in the codebase. Another skips config files because \u0026ldquo;those don\u0026rsquo;t count, right?\u0026rdquo; A third forgets entirely because they\u0026rsquo;re three days behind on a feature that should have shipped last week.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eNo evidence trail.\u003c/strong\u003e The checklist says \u0026ldquo;completed\u0026rdquo; but provides zero forensic evidence. An auditor asks, \u0026ldquo;Can you prove API endpoints required authentication on March 15th?\u0026rdquo; The answer is a signature and a shrug. Maybe a vague memory of clicking around in the portal.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConfiguration drift goes unnoticed.\u003c/strong\u003e The Azure portal shows correct settings \u003cem\u003etoday\u003c/em\u003e. Last week when someone disabled encryption for \u0026ldquo;quick troubleshooting\u0026rdquo; and forgot to re-enable it? That shipped to production. Nobody knows.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eVulnerable dependencies everywhere.\u003c/strong\u003e Manually checking NuGet packages takes hours and misses transitive dependencies entirely. By the time someone googles the top-level packages, code with known CVEs is already running in prod.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEnvironments? What environments?\u003c/strong\u003e The checklist gets run in production. Maybe. Does staging match? Dev? Nobody\u0026rsquo;s manually checking every environment every deployment. That would be insane. (It would also be the actual requirement.)\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"why-auditors-should-be-furious\"\u003e\u003ca href=\"/posts/compliance-verification-dotnet-cli/#why-auditors-should-be-furious\" title=\"Why Auditors Should Be Furious\"\u003eWhy Auditors Should Be Furious\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eThe standard requires \u0026ldquo;regular\u0026rdquo; compliance reviews. Regular means every deployment, every config change, every commit. Not \u0026ldquo;quarterly when someone remembers.\u0026rdquo; Not \u0026ldquo;weekly if we\u0026rsquo;re feeling disciplined.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eIt also requires documented procedures. Screenshots in Word documents aren\u0026rsquo;t procedures. They\u0026rsquo;re artifacts from a specific moment that became lies the instant someone changed something.\u003c/p\u003e\n\u003cp\u003eManual compliance theater satisfies auditors who don\u0026rsquo;t understand software. It doesn\u0026rsquo;t satisfy the actual requirements. And increasingly, auditors \u003cem\u003edo\u003c/em\u003e understand software.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-fix-make-the-computer-do-it\"\u003e\u003ca href=\"/posts/compliance-verification-dotnet-cli/#the-fix-make-the-computer-do-it\" title=\"The Fix: Make the Computer Do It\"\u003eThe Fix: Make the Computer Do It\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eCompliance verification needs to be automated, repeatable, verifiable, fast, and comprehensive. Humans are bad at all of these things. Computers are great at them.\u003c/p\u003e\n\u003cp\u003e.NET CLI tools deliver exactly this. Here\u0026rsquo;s how to build them.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"building-your-compliance-scanner\"\u003e\u003ca href=\"/posts/compliance-verification-dotnet-cli/#building-your-compliance-scanner\" title=\"Building Your Compliance Scanner\"\u003eBuilding Your Compliance Scanner\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eWe\u0026rsquo;re creating a .NET Global Tool that scans code and infrastructure, then fails the pipeline when it finds problems. Simple concept, surprisingly effective.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003edotnet new tool -n ComplianceScanner\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003ecd\u003c/span\u003e ComplianceScanner\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003edotnet add package System.CommandLine --version 2.0.2\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003edotnet add package Microsoft.CodeAnalysis.CSharp --version 5.0.0\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003edotnet add package Azure.Identity --version 1.17.1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003edotnet add package Azure.ResourceManager --version 1.13.2\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe core CLI structure is straightforward:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-csharp\" data-lang=\"csharp\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eusing\u003c/span\u003e \u003cspan class=\"nn\"\u003eSystem.CommandLine\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003erootCommand\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eRootCommand\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Compliance verification tool\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003escanCommand\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eCommand\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;scan\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;Scan codebase for violations\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003epathOption\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eOption\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"kt\"\u003estring\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;--path\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"p\"\u003e()\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003eDirectory\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetCurrentDirectory\u003c/span\u003e\u003cspan class=\"p\"\u003e());\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003escanCommand\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAddOption\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003epathOption\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003escanCommand\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eSetHandler\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kd\"\u003easync\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003eviolations\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eList\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eComplianceViolation\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"n\"\u003eviolations\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAddRange\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003eSecretScanner\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eScanAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e));\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"n\"\u003eviolations\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAddRange\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003eAuthScanner\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eScanAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e));\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eviolations\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAny\u003c/span\u003e\u003cspan class=\"p\"\u003e())\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003eforeach\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003ev\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"n\"\u003eviolations\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"n\"\u003eConsole\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eWriteLine\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e$\u0026#34;[{v.Severity}] {v.Rule}: {v.Message}\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"m\"\u003e1\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"c1\"\u003e// Fails the pipeline\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"m\"\u003e0\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e},\u003c/span\u003e \u003cspan class=\"n\"\u003epathOption\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003erootCommand\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAddCommand\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003escanCommand\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003erootCommand\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eInvokeAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eargs\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe key insight: return non-zero exit codes. That\u0026rsquo;s what makes pipelines fail. Everything else is details.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"finding-secrets-the-ones-your-team-swears-arent-there\"\u003e\u003ca href=\"/posts/compliance-verification-dotnet-cli/#finding-secrets-the-ones-your-team-swears-arent-there\" title=\"Finding Secrets (The Ones Your Team Swears Aren\u0026rsquo;t There)\"\u003eFinding Secrets (The Ones Your Team Swears Aren\u0026rsquo;t There)\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eRoslyn makes this embarrassingly simple:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-csharp\" data-lang=\"csharp\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"k\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003eSecretScanner\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003eprivate\u003c/span\u003e \u003cspan class=\"kd\"\u003estatic\u003c/span\u003e \u003cspan class=\"k\"\u003ereadonly\u003c/span\u003e \u003cspan class=\"n\"\u003eRegex\u003c/span\u003e\u003cspan class=\"p\"\u003e[]\u003c/span\u003e \u003cspan class=\"n\"\u003ePatterns\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"p\"\u003e[]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eRegex\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e@\u0026#34;(?i)(password|pwd)\\s*=\\s*[\u0026#34;\u0026#34;\u0026#39;][^\u0026#34;\u0026#34;\u0026#39;]{8,}[\u0026#34;\u0026#34;\u0026#39;]\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e),\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eRegex\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e@\u0026#34;(?i)(api[_-]?key)\\s*=\\s*[\u0026#34;\u0026#34;\u0026#39;][^\u0026#34;\u0026#34;\u0026#39;]{20,}[\u0026#34;\u0026#34;\u0026#39;]\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e),\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eRegex\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e@\u0026#34;AKIA[0-9A-Z]{16}\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e),\u003c/span\u003e \u003cspan class=\"c1\"\u003e// AWS keys - always fun to find these\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e};\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"kd\"\u003estatic\u003c/span\u003e \u003cspan class=\"kd\"\u003easync\u003c/span\u003e \u003cspan class=\"n\"\u003eTask\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eList\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eViolation\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003eScanAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003estring\u003c/span\u003e \u003cspan class=\"n\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003eviolations\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eList\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eViolation\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003eforeach\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003efile\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"n\"\u003eDirectory\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetFiles\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;*.cs\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eSearchOption\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAllDirectories\u003c/span\u003e\u003cspan class=\"p\"\u003e))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003etree\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eCSharpSyntaxTree\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eParseText\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003eFile\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eReadAllTextAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003efile\u003c/span\u003e\u003cspan class=\"p\"\u003e));\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003eliterals\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003etree\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetRootAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e()).\u003c/span\u003e\u003cspan class=\"n\"\u003eDescendantNodes\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eOfType\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eLiteralExpressionSyntax\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eWhere\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003el\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003el\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eIsKind\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eSyntaxKind\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eStringLiteralExpression\u003c/span\u003e\u003cspan class=\"p\"\u003e));\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"k\"\u003eforeach\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003eliteral\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"n\"\u003eliterals\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ePatterns\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAny\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ep\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003ep\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eIsMatch\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eliteral\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eToken\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eValueText\u003c/span\u003e\u003cspan class=\"p\"\u003e)))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                    \u003cspan class=\"n\"\u003eviolations\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAdd\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;SECRET\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;CRITICAL\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003efile\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;Hardcoded secret\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e));\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"n\"\u003eviolations\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eYou\u0026rsquo;d be amazed how many \u0026ldquo;we definitely don\u0026rsquo;t have hardcoded secrets\u0026rdquo; codebases light up like Christmas trees when you run this.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"finding-naked-endpoints\"\u003e\u003ca href=\"/posts/compliance-verification-dotnet-cli/#finding-naked-endpoints\" title=\"Finding Naked Endpoints\"\u003eFinding Naked Endpoints\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eEvery ASP.NET Core controller endpoint should either have \u003ccode\u003e[Authorize]\u003c/code\u003e or an explicit \u003ccode\u003e[AllowAnonymous]\u003c/code\u003e with a documented reason. \u0026ldquo;We forgot\u0026rdquo; is not a documented reason.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-csharp\" data-lang=\"csharp\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"k\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003eAuthScanner\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"kd\"\u003estatic\u003c/span\u003e \u003cspan class=\"kd\"\u003easync\u003c/span\u003e \u003cspan class=\"n\"\u003eTask\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eList\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eViolation\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003eScanAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003estring\u003c/span\u003e \u003cspan class=\"n\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003eviolations\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eList\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eViolation\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003eforeach\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003efile\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"n\"\u003eDirectory\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetFiles\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;*.cs\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eSearchOption\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAllDirectories\u003c/span\u003e\u003cspan class=\"p\"\u003e))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003eroot\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003eCSharpSyntaxTree\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eParseText\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003eFile\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eReadAllTextAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003efile\u003c/span\u003e\u003cspan class=\"p\"\u003e)).\u003c/span\u003e\u003cspan class=\"n\"\u003eGetRootAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"k\"\u003eforeach\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003econtroller\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"n\"\u003eroot\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eDescendantNodes\u003c/span\u003e\u003cspan class=\"p\"\u003e().\u003c/span\u003e\u003cspan class=\"n\"\u003eOfType\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eClassDeclarationSyntax\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eWhere\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ec\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003ec\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eIdentifier\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eText\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eEndsWith\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Controller\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003ehasClassAuth\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003econtroller\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAttributeLists\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                    \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eSelectMany\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ea\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003ea\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAttributes\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                    \u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAny\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ea\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003ea\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eName\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eToString\u003c/span\u003e\u003cspan class=\"p\"\u003e().\u003c/span\u003e\u003cspan class=\"n\"\u003eContains\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Authorize\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e));\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"k\"\u003eforeach\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003emethod\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"n\"\u003econtroller\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eDescendantNodes\u003c/span\u003e\u003cspan class=\"p\"\u003e().\u003c/span\u003e\u003cspan class=\"n\"\u003eOfType\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eMethodDeclarationSyntax\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;())\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                    \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003eattrs\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003emethod\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAttributeLists\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eSelectMany\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ea\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003ea\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAttributes\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                    \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003eisEndpoint\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eattrs\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAny\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ea\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003ea\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eName\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eToString\u003c/span\u003e\u003cspan class=\"p\"\u003e().\u003c/span\u003e\u003cspan class=\"n\"\u003eStartsWith\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Http\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e));\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                    \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003ehasAuth\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eattrs\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAny\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003ea\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003ea\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eName\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eToString\u003c/span\u003e\u003cspan class=\"p\"\u003e().\u003c/span\u003e\u003cspan class=\"n\"\u003eContains\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;Authorize\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                                               \u003cspan class=\"p\"\u003e||\u003c/span\u003e \u003cspan class=\"n\"\u003ea\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eName\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eToString\u003c/span\u003e\u003cspan class=\"p\"\u003e().\u003c/span\u003e\u003cspan class=\"n\"\u003eContains\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;AllowAnonymous\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e));\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                    \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                    \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eisEndpoint\u003c/span\u003e \u003cspan class=\"p\"\u003e\u0026amp;\u0026amp;\u003c/span\u003e \u003cspan class=\"p\"\u003e!\u003c/span\u003e\u003cspan class=\"n\"\u003ehasClassAuth\u003c/span\u003e \u003cspan class=\"p\"\u003e\u0026amp;\u0026amp;\u003c/span\u003e \u003cspan class=\"p\"\u003e!\u003c/span\u003e\u003cspan class=\"n\"\u003ehasAuth\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                        \u003cspan class=\"n\"\u003eviolations\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAdd\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#34;AUTH\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;HIGH\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003efile\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                            \u003cspan class=\"s\"\u003e$\u0026#34;Endpoint \u0026#39;{method.Identifier}\u0026#39; has no auth attribute\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e));\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"n\"\u003eviolations\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis catches the \u0026ldquo;I\u0026rsquo;ll add authentication later\u0026rdquo; endpoints that somehow made it to production three years ago.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"infrastructure-reality-checks\"\u003e\u003ca href=\"/posts/compliance-verification-dotnet-cli/#infrastructure-reality-checks\" title=\"Infrastructure Reality Checks\"\u003eInfrastructure Reality Checks\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eAzure SDK lets you verify that what\u0026rsquo;s \u003cem\u003eactually\u003c/em\u003e deployed matches what everyone \u003cem\u003ethinks\u003c/em\u003e is deployed. Spoiler: it often doesn\u0026rsquo;t.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-csharp\" data-lang=\"csharp\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"k\"\u003eclass\u003c/span\u003e \u003cspan class=\"nc\"\u003eInfraValidator\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003eprivate\u003c/span\u003e \u003cspan class=\"k\"\u003ereadonly\u003c/span\u003e \u003cspan class=\"n\"\u003eArmClient\u003c/span\u003e \u003cspan class=\"n\"\u003e_client\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eDefaultAzureCredential\u003c/span\u003e\u003cspan class=\"p\"\u003e());\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"kd\"\u003easync\u003c/span\u003e \u003cspan class=\"n\"\u003eTask\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eList\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eIssue\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003eValidateAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003eissues\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eList\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eIssue\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"k\"\u003eforeach\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003esub\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"n\"\u003e_client\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetSubscriptions\u003c/span\u003e\u003cspan class=\"p\"\u003e())\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"c1\"\u003e// Storage: HTTPS only? Encryption on?\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"k\"\u003eforeach\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003estorage\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"n\"\u003esub\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetStorageAccountsAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e())\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"p\"\u003e(!\u003c/span\u003e\u003cspan class=\"n\"\u003estorage\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eData\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eEnableHttpsTrafficOnly\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetValueOrDefault\u003c/span\u003e\u003cspan class=\"p\"\u003e())\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                    \u003cspan class=\"n\"\u003eissues\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAdd\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003estorage\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eData\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eName\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;Storage allows HTTP\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e));\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003estorage\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eData\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eEncryption\u003c/span\u003e\u003cspan class=\"p\"\u003e?.\u003c/span\u003e\u003cspan class=\"n\"\u003eServices\u003c/span\u003e\u003cspan class=\"p\"\u003e?.\u003c/span\u003e\u003cspan class=\"n\"\u003eBlob\u003c/span\u003e\u003cspan class=\"p\"\u003e?.\u003c/span\u003e\u003cspan class=\"n\"\u003eEnabled\u003c/span\u003e \u003cspan class=\"p\"\u003e!=\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                    \u003cspan class=\"n\"\u003eissues\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAdd\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003estorage\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eData\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eName\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;Blob encryption disabled\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e));\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"c1\"\u003e// Key Vault: Soft delete? Purge protection?\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"k\"\u003eforeach\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003evault\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"n\"\u003esub\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetKeyVaultsAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e())\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"p\"\u003e(!\u003c/span\u003e\u003cspan class=\"n\"\u003evault\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eData\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eProperties\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eEnableSoftDelete\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eGetValueOrDefault\u003c/span\u003e\u003cspan class=\"p\"\u003e())\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                    \u003cspan class=\"n\"\u003eissues\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eAdd\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003evault\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eData\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eName\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;No soft delete - secrets can vanish\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e));\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e            \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"k\"\u003ereturn\u003c/span\u003e \u003cspan class=\"n\"\u003eissues\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eRun this the first time and prepare for uncomfortable conversations about \u0026ldquo;temporary\u0026rdquo; configuration changes from 2019.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"dependency-nightmares\"\u003e\u003ca href=\"/posts/compliance-verification-dotnet-cli/#dependency-nightmares\" title=\"Dependency Nightmares\"\u003eDependency Nightmares\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eGood news: .NET already has this built in. Bad news: nobody\u0026rsquo;s running it.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-csharp\" data-lang=\"csharp\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003epublic\u003c/span\u003e \u003cspan class=\"kd\"\u003estatic\u003c/span\u003e \u003cspan class=\"kd\"\u003easync\u003c/span\u003e \u003cspan class=\"n\"\u003eTask\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eList\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026lt;\u003c/span\u003e\u003cspan class=\"n\"\u003eViolation\u003c/span\u003e\u003cspan class=\"p\"\u003e\u0026gt;\u0026gt;\u003c/span\u003e \u003cspan class=\"n\"\u003eScanVulnerabilitiesAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"kt\"\u003estring\u003c/span\u003e \u003cspan class=\"n\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003eprocess\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eProcess\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eStart\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"k\"\u003enew\u003c/span\u003e \u003cspan class=\"n\"\u003eProcessStartInfo\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003eFileName\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;dotnet\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003eArguments\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;list package --vulnerable --include-transitive --format json\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003eWorkingDirectory\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003eRedirectStandardOutput\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003eUseShellExecute\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e});\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"kt\"\u003evar\u003c/span\u003e \u003cspan class=\"n\"\u003eoutput\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003eprocess\u003c/span\u003e\u003cspan class=\"p\"\u003e!.\u003c/span\u003e\u003cspan class=\"n\"\u003eStandardOutput\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eReadToEndAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003eawait\u003c/span\u003e \u003cspan class=\"n\"\u003eprocess\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eWaitForExitAsync\u003c/span\u003e\u003cspan class=\"p\"\u003e();\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"c1\"\u003e// Parse JSON, extract vulnerabilities, return violations\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"c1\"\u003e// The JSON structure is well-documented and straightforward\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe \u003ccode\u003e--include-transitive\u003c/code\u003e flag is crucial. That\u0026rsquo;s where the real horrors live: three levels deep in dependencies nobody knew existed.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"wiring-it-into-your-pipeline\"\u003e\u003ca href=\"/posts/compliance-verification-dotnet-cli/#wiring-it-into-your-pipeline\" title=\"Wiring It Into Your Pipeline\"\u003eWiring It Into Your Pipeline\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eHere\u0026rsquo;s the GitHub Actions workflow that makes compliance a gate, not a suggestion:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eCompliance Gate\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eon\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"l\"\u003epull_request, push]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ejobs\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ecompliance\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eruns-on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eubuntu-latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esteps\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/checkout@v6\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/setup-dotnet@v5\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ewith\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003edotnet-version\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;10.0.x\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eInstall Scanner\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edotnet tool install --global ComplianceScanner\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eScan Code\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ecompliance-scanner scan --path .\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eCheck Vulnerabilities  \u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edotnet list package --vulnerable --include-transitive\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eValidate Infrastructure\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ecompliance-scanner validate-infra\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eenv\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003eAZURE_CLIENT_ID\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e${{ secrets.AZURE_CLIENT_ID }}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003eAZURE_TENANT_ID\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e${{ secrets.AZURE_TENANT_ID }}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eMake this a required status check. No exceptions. \u0026ldquo;But we need to ship!\u0026rdquo; is exactly when you need it most.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"reports-for-the-auditors\"\u003e\u003ca href=\"/posts/compliance-verification-dotnet-cli/#reports-for-the-auditors\" title=\"Reports for the Auditors\"\u003eReports for the Auditors\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eGenerate JSON, HTML, or Markdown reports from your scan results. Auditors love artifacts they can file. More importantly, \u003cem\u003eyou\u003c/em\u003e love having evidence when someone asks \u0026ldquo;but did you actually check?\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eThe report generator is straightforward: serialize your violations to the format of choice, timestamp it, and archive it. The structure matters less than the fact that it exists and is generated automatically.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"for-the-auditors-in-the-room\"\u003e\u003ca href=\"/posts/compliance-verification-dotnet-cli/#for-the-auditors-in-the-room\" title=\"For the Auditors in the Room\"\u003eFor the Auditors in the Room\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eYes, this satisfies the standard. Specifically:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eA.5.36 (Regular compliance review):\u003c/strong\u003e Running on every PR and deployment is \u0026ldquo;regular.\u0026rdquo; Quarterly manual checks are not.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eA.8.8 (Vulnerability management):\u003c/strong\u003e Automated scanning that blocks deployment beats \u0026ldquo;we\u0026rsquo;ll check NuGet.org when we remember.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eA.5.37 (Documented procedures):\u003c/strong\u003e The code \u003cem\u003eis\u003c/em\u003e the documentation. It runs the same way every time. Unlike Bob\u0026rsquo;s interpretation of the checklist.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eA.8.24 (Cryptography):\u003c/strong\u003e Automated validation that encryption is actually enabled, not just assumed to be.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"rolling-this-out-without-a-mutiny\"\u003e\u003ca href=\"/posts/compliance-verification-dotnet-cli/#rolling-this-out-without-a-mutiny\" title=\"Rolling This Out (Without a Mutiny)\"\u003eRolling This Out (Without a Mutiny)\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eDon\u0026rsquo;t try to implement everything at once. That\u0026rsquo;s how you get ignored.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWeek 1: Secret scanning.\u003c/strong\u003e Start with warnings. Let the team see what\u0026rsquo;s been lurking. Move to failures once the initial panic subsides.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWeek 2: Vulnerability checks.\u003c/strong\u003e Add \u003ccode\u003edotnet list package --vulnerable\u003c/code\u003e to the pipeline. Fail on CRITICAL. Watch the dependency update PRs roll in.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWeek 3-4: Auth verification.\u003c/strong\u003e Audit existing endpoints first. You\u0026rsquo;ll find things. Fix them before enforcing.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWeek 5-6: Infrastructure validation.\u003c/strong\u003e Run manually first. Discover the \u0026ldquo;temporary\u0026rdquo; configs. Remediate. Then automate.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWeek 7: Reporting.\u003c/strong\u003e Generate artifacts. Distribute to stakeholders. Watch compliance become boring (which is exactly what you want).\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-bottom-line\"\u003e\u003ca href=\"/posts/compliance-verification-dotnet-cli/#the-bottom-line\" title=\"The Bottom Line\"\u003eThe Bottom Line\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eManual compliance is a lie everyone agrees to tell. \u0026ldquo;We checked\u0026rdquo; means \u0026ldquo;someone signed something.\u0026rdquo; It doesn\u0026rsquo;t mean the systems are actually secure.\u003c/p\u003e\n\u003cp\u003eAutomated CLI tools change the equation. They run every time. They check the same things. They generate evidence. They fail the build when something\u0026rsquo;s wrong.\u003c/p\u003e\n\u003cp\u003e.NET gives you \u003ccode\u003eSystem.CommandLine\u003c/code\u003e (finally at GA!), Roslyn, and the Azure SDK. The tooling exists. The patterns are straightforward. The only thing standing between your organization and actual compliance is the decision to stop pretending the Word doc is enough.\u003c/p\u003e\n\u003cp\u003eBuild the scanner. Wire it into the pipeline. Make it a required check.\u003c/p\u003e\n\u003cp\u003eOr keep signing the checklist. Your call.\u003c/p\u003e\n","date_modified":"2026-05-26T10:22:03+02:00","date_published":"2026-04-23T17:00:00+02:00","id":"https://daily-devops.net/posts/compliance-verification-dotnet-cli/","language":"en","summary":"Consultants paid. Docs filed. Then compliance becomes a Word doc ritual until an audit exposes the drift. CLI tools fix what checklists never could.","tags":["iso-standards","cli","dotnet","automation","compliance","security","devops","cicd","azure","codequality"],"title":"Certified, Filed, Forgotten: The Compliance Trainwreck","url":"https://daily-devops.net/posts/compliance-verification-dotnet-cli/"},{"authors":[{"name":"Jendrik Brack","url":"https://daily-devops.net/authors/jendrik/"}],"content_html":"\u003cp\u003eNobody warns you about the point where Kubernetes stops behaving like Kubernetes. At 100 nodes the platform feels manageable: logs are searchable, deployments finish quickly, and most incidents resolve with a kubectl command and some patience. Cross 500 nodes and small architectural assumptions start cracking. Cross 1,000 nodes and those cracks become structural.\u003c/p\u003e\n\u003cp\u003eThe problems described here are not hypothetical. etcd database sizes that stretched backup windows into hours. Observability stacks consuming more cluster resources than the workloads they were supposed to monitor. Network overlays running fine at 200 nodes that started dropping packets at 800. If you\u0026rsquo;re planning to push past 500 nodes, or already running infrastructure at that scale and things feel increasingly fragile, read on.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-scale-cliff-why-1000-nodes-changes-everything\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#the-scale-cliff-why-1000-nodes-changes-everything\" title=\"The Scale Cliff: Why 1,000 Nodes Changes Everything\"\u003eThe Scale Cliff: Why 1,000 Nodes Changes Everything\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eAt 100 nodes, Kubernetes feels manageable. Monitoring works. Logs are searchable. Network patterns make sense. Deployments complete in minutes. Then you cross 500 nodes and small cracks appear. By 1,000 nodes, those cracks become structural failures.\u003c/p\u003e\n\u003cp\u003eThe problem: Kubernetes components designed for graceful degradation hit hard limits at scale. etcd performance degrades non-linearly with keyspace size. Network overlay solutions that worked fine at 200 nodes saturate at 800. Observability stacks consuming 3% of cluster resources at 100 nodes consume 25% at 1,000. Cost-per-node stays flat but operational overhead per node increases exponentially.\u003c/p\u003e\n\u003cp\u003eThese aren\u0026rsquo;t bugs. They\u0026rsquo;re architectural realities. Understanding where the cliffs are lets you plan around them instead of discovering them in production outages.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"etcd-the-hidden-scaling-bottleneck\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#etcd-the-hidden-scaling-bottleneck\" title=\"etcd: The Hidden Scaling Bottleneck\"\u003eetcd: The Hidden Scaling Bottleneck\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eetcd is the single most critical component in your cluster and the first to hit scaling limits. It stores all cluster state: every pod, service, config map, secret, and custom resource. At 1,000 nodes with 200 pods per node, you\u0026rsquo;re managing 200,000+ objects. etcd wasn\u0026rsquo;t designed for that scale without careful tuning.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"performance-degradation-patterns\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#performance-degradation-patterns\" title=\"Performance Degradation Patterns\"\u003ePerformance Degradation Patterns\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eetcd performance degrades based on keyspace size, transaction rate, and storage backend latency. At small scale, these factors don\u0026rsquo;t matter. At mega-cluster scale, they dominate operational behavior.\u003c/p\u003e\n\u003cp\u003eSymptoms you\u0026rsquo;ll see:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAPI server latency spikes during deployments\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003ekubectl\u003c/code\u003e commands timing out intermittently\u003c/li\u003e\n\u003cli\u003eController reconciliation loops falling behind\u003c/li\u003e\n\u003cli\u003eScheduler making suboptimal placement decisions due to stale state\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe root cause is usually one of three things: etcd database size exceeding memory capacity, insufficient IOPS on the storage backend, or transaction rate overwhelming the commit pipeline.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"backup-size-and-recovery-time\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#backup-size-and-recovery-time\" title=\"Backup Size and Recovery Time\"\u003eBackup Size and Recovery Time\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eetcd backup size scales with keyspace. A 100-node cluster might produce 500MB backups. A 1,000-node cluster produces 8GB+ backups. That size creates operational problems:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eBackup windows extend from minutes to hours\u003c/li\u003e\n\u003cli\u003eNetwork transfer costs increase linearly\u003c/li\u003e\n\u003cli\u003eRecovery time objectives (RTO) slip from \u0026ldquo;15 minutes\u0026rdquo; to \u0026ldquo;2+ hours\u0026rdquo;\u003c/li\u003e\n\u003cli\u003eStorage costs for retention policies multiply unexpectedly\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eWorse: most backup solutions for etcd aren\u0026rsquo;t tested at mega-cluster scale. The tooling that works reliably at 100 nodes silently fails or creates corrupted snapshots at 1,000 nodes.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"practical-mitigation\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#practical-mitigation\" title=\"Practical Mitigation\"\u003ePractical Mitigation\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003e\u003ca href=\"https://learn.microsoft.com/en-us/azure/aks/concepts-clusters-workloads\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eAKS manages etcd for you\u003c/a\u003e, but you still need to monitor and validate its health. Here\u0026rsquo;s a Terraform configuration that sets up Azure Monitor alerts for etcd-related API server latency:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-hcl\" data-lang=\"hcl\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_monitor_metric_alert\u0026#34; \u0026#34;etcd_latency\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;aks-etcd-high-latency\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003emain\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  scopes\u003c/span\u003e              \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"k\"\u003eazurerm_kubernetes_cluster\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003emain\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  description\u003c/span\u003e         \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Alert when API server latency exceeds 200ms (etcd saturation signal)\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  severity\u003c/span\u003e            \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"m\"\u003e2\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  frequency\u003c/span\u003e           \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;PT1M\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  window_size\u003c/span\u003e         \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;PT5M\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003ecriteria\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    metric_namespace\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Microsoft.ContainerService/managedClusters\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    metric_name\u003c/span\u003e      \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;apiserver_request_duration_seconds\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    aggregation\u003c/span\u003e      \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Average\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    operator\u003c/span\u003e         \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;GreaterThan\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    threshold\u003c/span\u003e        \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"m\"\u003e0\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"m\"\u003e2\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003edimension\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e      name\u003c/span\u003e     \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;verb\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e      operator\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Include\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e      values\u003c/span\u003e   \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;GET\u0026#34;, \u0026#34;LIST\u0026#34;, \u0026#34;PATCH\u0026#34;, \u0026#34;UPDATE\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003eaction\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    action_group_id\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_monitor_action_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eplatform\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_monitor_metric_alert\u0026#34; \u0026#34;etcd_database_size\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;aks-etcd-database-size-warning\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003emain\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  scopes\u003c/span\u003e              \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"k\"\u003eazurerm_kubernetes_cluster\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003emain\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  description\u003c/span\u003e         \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Alert when etcd database approaches size limits\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  severity\u003c/span\u003e            \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"m\"\u003e3\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  frequency\u003c/span\u003e           \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;PT5M\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  window_size\u003c/span\u003e         \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;PT15M\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003ecriteria\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    metric_namespace\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Microsoft.ContainerService/managedClusters\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    metric_name\u003c/span\u003e      \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;etcd_db_total_size_in_bytes\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    aggregation\u003c/span\u003e      \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Average\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    operator\u003c/span\u003e         \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;GreaterThan\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    threshold\u003c/span\u003e        \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"m\"\u003e6442450944\u003c/span\u003e\u003cspan class=\"c1\"\u003e  # 6GB (warning threshold)\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003eaction\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    action_group_id\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_monitor_action_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eplatform\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThese alerts won\u0026rsquo;t prevent etcd saturation, but they\u0026rsquo;ll give you advance warning before cascading failures occur. At scale, that early warning is the difference between a controlled maintenance window and an all-hands incident.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"network-performance-when-overlay-solutions-hit-limits\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#network-performance-when-overlay-solutions-hit-limits\" title=\"Network Performance: When Overlay Solutions Hit Limits\"\u003eNetwork Performance: When Overlay Solutions Hit Limits\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eNetwork overlay performance is invisible at small scale and catastrophic at large scale. Container Network Interface (CNI) plugins that handle 50,000 pods without issue can saturate CPU and drop packets at 200,000 pods. There is no single right answer for which CNI to use. For a full breakdown of the tradeoffs between kubenet, Azure CNI, and Azure CNI Overlay, see \u003ca href=\"/posts/aks-networking-clash/\"\u003eAKS Networking Clash: kubenet vs. CNI vs. CNI Overlay\u003c/a\u003e.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"pod-density-and-node-saturation\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#pod-density-and-node-saturation\" title=\"Pod Density and Node Saturation\"\u003ePod Density and Node Saturation\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003e\u003ca href=\"https://learn.microsoft.com/en-us/azure/aks/azure-cni-overlay\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eAzure CNI Overlay\u003c/a\u003e supports up to 250 pods per node. That\u0026rsquo;s a theoretical maximum. Practical limits depend on network I/O patterns, pod churn rate, and service mesh overhead.\u003c/p\u003e\n\u003cp\u003eSignals that you\u0026rsquo;re approaching saturation:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eNodes showing high system CPU (kernel networking overhead)\u003c/li\u003e\n\u003cli\u003eIntermittent packet loss between pods on the same node\u003c/li\u003e\n\u003cli\u003eService discovery latency increasing over time\u003c/li\u003e\n\u003cli\u003eDNS resolution failures under load\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe underlying issue: network namespace creation, iptables rule updates, and conntrack table management all scale poorly. At 200 pods per node, these operations consume negligible resources. At 250 pods per node, they dominate system CPU.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"cross-node-latency-patterns\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#cross-node-latency-patterns\" title=\"Cross-Node Latency Patterns\"\u003eCross-Node Latency Patterns\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eOverlay networks add encapsulation overhead. Azure CNI Overlay typically adds 100-200 microseconds per hop. At small scale, that\u0026rsquo;s noise. At mega-cluster scale, it compounds across multi-tier applications.\u003c/p\u003e\n\u003cp\u003eExample: a request traversing frontend → API gateway → backend service → database proxy touches 4 pods. If those pods span nodes, you\u0026rsquo;ve added 400-800 microseconds of latency from network overhead alone. Multiply that by 10,000 requests per second and the impact becomes measurable in user-facing metrics.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"mitigation-strategy\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#mitigation-strategy\" title=\"Mitigation Strategy\"\u003eMitigation Strategy\u003c/a\u003e\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003ePin latency-sensitive workloads to the same node using pod affinity\u003c/li\u003e\n\u003cli\u003eUse host networking for data-plane components (with appropriate security controls)\u003c/li\u003e\n\u003cli\u003eMonitor conntrack table utilization: \u003ccode\u003esysctl net.netfilter.nf_conntrack_count\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003eSet conservative pod density limits (180-200 pods/node instead of 250)\u003c/li\u003e\n\u003cli\u003eImplement service mesh with extended Berkeley Packet Filter (eBPF) dataplane (\u003ca href=\"https://cilium.io/\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eCilium\u003c/a\u003e) to reduce iptables overhead\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThese aren\u0026rsquo;t performance optimizations. They\u0026rsquo;re operational requirements at scale.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"observability-overhead-when-monitoring-becomes-the-problem\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#observability-overhead-when-monitoring-becomes-the-problem\" title=\"Observability Overhead: When Monitoring Becomes the Problem\"\u003eObservability Overhead: When Monitoring Becomes the Problem\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eObservability at scale creates a paradox: the systems you need to diagnose problems become the source of resource exhaustion.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"logging-cost-explosion\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#logging-cost-explosion\" title=\"Logging Cost Explosion\"\u003eLogging Cost Explosion\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eA single pod generating 100KB/day of logs costs nothing. 200,000 pods generating the same logs produce 20GB/day. Over a month, that\u0026rsquo;s 600GB. With 3x replication and 90-day retention, you\u0026rsquo;re storing 162TB of log data.\u003c/p\u003e\n\u003cp\u003eStorage costs for that volume run into thousands of dollars monthly. Query performance degrades. Log ingestion pipelines fall behind. The tooling designed to help you debug problems becomes unusable during incidents.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"metric-cardinality-problems\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#metric-cardinality-problems\" title=\"Metric Cardinality Problems\"\u003eMetric Cardinality Problems\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003ePrometheus-based monitoring hits cardinality limits around 10 million active time series. A 1,000-node cluster with moderate instrumentation easily exceeds that threshold:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e200,000 pods × 20 metrics per pod = 4M series\u003c/li\u003e\n\u003cli\u003e1,000 nodes × 100 metrics per node = 100K series\u003c/li\u003e\n\u003cli\u003e50 services × 10K instances × 5 metrics = 2.5M series\u003c/li\u003e\n\u003cli\u003eCustom application metrics add another 3M+ series\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eWhen you exceed cardinality limits, Prometheus becomes unstable. Queries time out. Dashboards fail to render. Alerting rules stop evaluating. You lose observability exactly when you need it most.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"practical-approaches\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#practical-approaches\" title=\"Practical Approaches\"\u003ePractical Approaches\u003c/a\u003e\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eImplement aggressive log sampling: 1% sampling still gives 2GB/day of logs\u003c/li\u003e\n\u003cli\u003eUse structured logging with consistent field names to enable efficient compression\u003c/li\u003e\n\u003cli\u003eArchive cold logs to blob storage (pennies per GB vs. dollars per GB in hot storage)\u003c/li\u003e\n\u003cli\u003eDeploy federated Prometheus with careful metric filtering at scrape time\u003c/li\u003e\n\u003cli\u003eUse recording rules to pre-aggregate high-cardinality metrics\u003c/li\u003e\n\u003cli\u003eConsider managed observability services (Azure Monitor, Datadog) that handle scale for you\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe honest assessment: if your observability stack consumes more than 10% of cluster resources, it\u0026rsquo;s time to rethink your approach. At mega-cluster scale, that threshold is easy to exceed.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"cost-spirals-small-decisions-with-exponential-consequences\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#cost-spirals-small-decisions-with-exponential-consequences\" title=\"Cost Spirals: Small Decisions with Exponential Consequences\"\u003eCost Spirals: Small Decisions with Exponential Consequences\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eCost optimization at 100 nodes is optional. At 1,000 nodes, it\u0026rsquo;s mandatory. Small inefficiencies compound brutally.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"resource-overprovisioning\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#resource-overprovisioning\" title=\"Resource Overprovisioning\"\u003eResource Overprovisioning\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eTeams typically request 2x actual resource needs for safety margin. At 100 nodes, that\u0026rsquo;s wasteful but affordable. At 1,000 nodes with 250 pods per node, you\u0026rsquo;re paying for 125,000 unutilized CPU cores.\u003c/p\u003e\n\u003cp\u003eWith Azure D8s_v5 nodes at ~$0.40/hour, a 1,000-node cluster costs ~$288,000/year in compute alone. 50% overprovisioning adds $144,000 annually. That\u0026rsquo;s real budget impact.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"storage-cost-patterns\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#storage-cost-patterns\" title=\"Storage Cost Patterns\"\u003eStorage Cost Patterns\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eEvery pod gets ephemeral storage. Most clusters also provision persistent volumes. At scale, storage costs exceed compute costs.\u003c/p\u003e\n\u003cp\u003eExample: 200,000 pods with 10GB ephemeral storage each = 2PB of ephemeral storage. Persistent volume claims add another 500TB+. Azure Premium SSD costs $0.135/GB/month. You\u0026rsquo;re paying $300K+ monthly for storage alone.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"network-egress-surprises\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#network-egress-surprises\" title=\"Network Egress Surprises\"\u003eNetwork Egress Surprises\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eCross-region and internet egress costs scale linearly with traffic volume. A 1,000-node cluster handling 10TB/day of egress traffic incurs $1,500/day in bandwidth costs ($45,000/month).\u003c/p\u003e\n\u003cp\u003eTeams typically discover these costs 60 days into a scale-up when the first full billing cycle completes. By then, architectural changes are expensive and disruptive.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"cost-control-strategy\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#cost-control-strategy\" title=\"Cost Control Strategy\"\u003eCost Control Strategy\u003c/a\u003e\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eImplement cluster autoscaling with aggressive scale-down policies\u003c/li\u003e\n\u003cli\u003eUse spot instances for fault-tolerant workloads (70% cost reduction)\u003c/li\u003e\n\u003cli\u003eRight-size pod resource requests using \u003ca href=\"https://learn.microsoft.com/en-us/azure/aks/vertical-pod-autoscaler\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eVPA (Vertical Pod Autoscaler)\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003eEnable Azure Hybrid Benefit for Windows nodes\u003c/li\u003e\n\u003cli\u003eDeploy regional caching layers to reduce cross-region egress\u003c/li\u003e\n\u003cli\u003eMonitor and alert on cost metrics, not just resource metrics\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eTeams defer cost optimization in favor of operational simplicity, and early on that is usually the right call. At mega-cluster scale, that priority reverses. Cost efficiency becomes a constraint you cannot ignore. \u003ca href=\"/posts/cost-optimization-resource-governance-aks/\"\u003eAKS Cost Optimization: Resource Governance That Actually Works\u003c/a\u003e goes deeper on VPA configuration and autoscaling policies if you want the practical implementation details.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"debugging-at-scale-finding-needles-in-exponentially-larger-haystacks\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#debugging-at-scale-finding-needles-in-exponentially-larger-haystacks\" title=\"Debugging at Scale: Finding Needles in Exponentially Larger Haystacks\"\u003eDebugging at Scale: Finding Needles in Exponentially Larger Haystacks\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eDebugging a 100-node cluster means checking logs from a few thousand pods. Debugging a 1,000-node cluster means isolating the problem from millions of log lines across 200,000+ pods.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"correlation-and-isolation\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#correlation-and-isolation\" title=\"Correlation and Isolation\"\u003eCorrelation and Isolation\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eWhen a user reports an error, your troubleshooting workflow looks like this:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eIdentify the service handling the request (1 of 50+ services)\u003c/li\u003e\n\u003cli\u003eFind the pod instance that processed the request (1 of 5,000+ pod instances)\u003c/li\u003e\n\u003cli\u003eLocate the relevant log lines (1 of 10M+ log events in the time window)\u003c/li\u003e\n\u003cli\u003eCorrelate with upstream/downstream service calls\u003c/li\u003e\n\u003cli\u003eReproduce the issue in a controlled environment\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eAt small scale, steps 2-3 take minutes. At mega-cluster scale, they take hours, assuming correlation IDs exist and work correctly. Without proper instrumentation, they\u0026rsquo;re impossible.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"reproduction-challenges\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#reproduction-challenges\" title=\"Reproduction Challenges\"\u003eReproduction Challenges\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eIssues that reproduce reliably at scale rarely reproduce in test environments. A race condition that triggers once per 100,000 requests never manifests in pre-production. Network congestion patterns that emerge at 1,000 nodes don\u0026rsquo;t exist at 10 nodes.\u003c/p\u003e\n\u003cp\u003eThis creates a diagnostic blind spot. You can observe the failure in production but can\u0026rsquo;t reproduce it for root cause analysis.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"large-scale-troubleshooting-checklist\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#large-scale-troubleshooting-checklist\" title=\"Large-Scale Troubleshooting Checklist\"\u003eLarge-Scale Troubleshooting Checklist\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eHere\u0026rsquo;s a diagnostic script I use for investigating performance degradation at scale:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"cp\"\u003e#!/bin/bash\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Large-scale AKS cluster diagnostic script\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Run this when experiencing unexplained performance issues\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eset\u003c/span\u003e -euo pipefail\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eCLUSTER_NAME\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003e1\u003c/span\u003e\u003cspan class=\"p\"\u003e:?Cluster name required\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eRESOURCE_GROUP\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003e2\u003c/span\u003e\u003cspan class=\"p\"\u003e:?Resource group required\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eOUTPUT_DIR\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;./diagnostics-\u003c/span\u003e\u003cspan class=\"k\"\u003e$(\u003c/span\u003edate +%Y%m%d-%H%M%S\u003cspan class=\"k\"\u003e)\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Running diagnostics for cluster: \u003c/span\u003e\u003cspan class=\"nv\"\u003e$CLUSTER_NAME\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003emkdir -p \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$OUTPUT_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Get cluster credentials\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz aks get-credentials --resource-group \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$RESOURCE_GROUP\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e --name \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$CLUSTER_NAME\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e --overwrite-existing\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Node health check\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Checking node health...\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl get nodes -o wide \u0026gt; \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$OUTPUT_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e/nodes.txt\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl top nodes \u0026gt; \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$OUTPUT_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e/node-resources.txt\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e||\u003c/span\u003e \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Metrics server unavailable\u0026#34;\u003c/span\u003e \u0026gt; \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$OUTPUT_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e/node-resources.txt\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# API server latency check\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Checking API server latency...\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003efor\u003c/span\u003e i in \u003cspan class=\"o\"\u003e{\u003c/span\u003e1..5\u003cspan class=\"o\"\u003e}\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003edo\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nb\"\u003etime\u003c/span\u003e kubectl get nodes \u0026gt; /dev/null 2\u0026gt;\u003cspan class=\"p\"\u003e\u0026amp;\u003c/span\u003e\u003cspan class=\"m\"\u003e1\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003edone\u003c/span\u003e 2\u0026gt;\u003cspan class=\"p\"\u003e\u0026amp;\u003c/span\u003e\u003cspan class=\"m\"\u003e1\u003c/span\u003e \u003cspan class=\"p\"\u003e|\u003c/span\u003e grep real \u0026gt; \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$OUTPUT_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e/api-latency.txt\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# etcd health indicators\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Checking etcd health signals...\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl get --raw /metrics \u003cspan class=\"p\"\u003e|\u003c/span\u003e grep -E \u003cspan class=\"s2\"\u003e\u0026#34;apiserver_request_duration|etcd_request_duration\u0026#34;\u003c/span\u003e \u0026gt; \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$OUTPUT_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e/etcd-metrics.txt\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e||\u003c/span\u003e \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Metrics unavailable\u0026#34;\u003c/span\u003e \u0026gt; \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$OUTPUT_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e/etcd-metrics.txt\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Pod distribution analysis\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Analyzing pod distribution...\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl get pods -A -o json \u003cspan class=\"p\"\u003e|\u003c/span\u003e jq -r \u003cspan class=\"s1\"\u003e\u0026#39;.items[] | \u0026#34;\\(.spec.nodeName)\u0026#34;\u0026#39;\u003c/span\u003e \u003cspan class=\"p\"\u003e|\u003c/span\u003e sort \u003cspan class=\"p\"\u003e|\u003c/span\u003e uniq -c \u003cspan class=\"p\"\u003e|\u003c/span\u003e sort -rn \u0026gt; \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$OUTPUT_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e/pod-distribution.txt\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Network policy count (can cause iptables overhead)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Checking network policy count...\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl get networkpolicies -A --no-headers \u003cspan class=\"p\"\u003e|\u003c/span\u003e wc -l \u0026gt; \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$OUTPUT_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e/netpol-count.txt\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Service endpoint count (affects kube-proxy performance)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Checking service endpoint count...\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl get endpoints -A -o json \u003cspan class=\"p\"\u003e|\u003c/span\u003e jq \u003cspan class=\"s1\"\u003e\u0026#39;[.items[].subsets[].addresses] | flatten | length\u0026#39;\u003c/span\u003e \u0026gt; \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$OUTPUT_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e/endpoint-count.txt\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Resource pressure signals\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Identifying pods with resource pressure...\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl get pods -A -o json \u003cspan class=\"p\"\u003e|\u003c/span\u003e jq -r \u003cspan class=\"s1\"\u003e\u0026#39;.items[] | select(.status.conditions[]? | select(.type==\u0026#34;Ready\u0026#34; and .status==\u0026#34;False\u0026#34;)) | \u0026#34;\\(.metadata.namespace)/\\(.metadata.name)\u0026#34;\u0026#39;\u003c/span\u003e \u0026gt; \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$OUTPUT_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e/not-ready-pods.txt\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Recent events (truncated for performance)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Capturing recent cluster events...\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl get events -A --sort-by\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;.lastTimestamp\u0026#39;\u003c/span\u003e \u003cspan class=\"p\"\u003e|\u003c/span\u003e tail -1000 \u0026gt; \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$OUTPUT_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e/recent-events.txt\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Node condition checks\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Checking for node pressure conditions...\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl get nodes -o json \u003cspan class=\"p\"\u003e|\u003c/span\u003e jq -r \u003cspan class=\"s1\"\u003e\u0026#39;.items[] | select(.status.conditions[]? | select(.type==\u0026#34;MemoryPressure\u0026#34; or .type==\u0026#34;DiskPressure\u0026#34; or .type==\u0026#34;PIDPressure\u0026#34;) | select(.status==\u0026#34;True\u0026#34;)) | .metadata.name\u0026#39;\u003c/span\u003e \u0026gt; \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$OUTPUT_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e/nodes-under-pressure.txt\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# ConfigMap and Secret count (affects etcd size)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Counting ConfigMaps and Secrets...\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;ConfigMaps: \u003c/span\u003e\u003cspan class=\"k\"\u003e$(\u003c/span\u003ekubectl get configmaps -A --no-headers \u003cspan class=\"p\"\u003e|\u003c/span\u003e wc -l\u003cspan class=\"k\"\u003e)\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u0026gt; \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$OUTPUT_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e/object-counts.txt\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Secrets: \u003c/span\u003e\u003cspan class=\"k\"\u003e$(\u003c/span\u003ekubectl get secrets -A --no-headers \u003cspan class=\"p\"\u003e|\u003c/span\u003e wc -l\u003cspan class=\"k\"\u003e)\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u0026gt;\u0026gt; \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$OUTPUT_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e/object-counts.txt\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Total Pods: \u003c/span\u003e\u003cspan class=\"k\"\u003e$(\u003c/span\u003ekubectl get pods -A --no-headers \u003cspan class=\"p\"\u003e|\u003c/span\u003e wc -l\u003cspan class=\"k\"\u003e)\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u0026gt;\u0026gt; \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$OUTPUT_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e/object-counts.txt\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# DNS performance check\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Testing DNS resolution performance...\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl run dns-test --image\u003cspan class=\"o\"\u003e=\u003c/span\u003ebusybox:1.36 --restart\u003cspan class=\"o\"\u003e=\u003c/span\u003eNever --rm -i --command -- sh -c \u003cspan class=\"s2\"\u003e\u0026#34;time nslookup kubernetes.default\u0026#34;\u003c/span\u003e \u0026gt; \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$OUTPUT_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e/dns-test.txt\u0026#34;\u003c/span\u003e 2\u0026gt;\u003cspan class=\"p\"\u003e\u0026amp;\u003c/span\u003e\u003cspan class=\"m\"\u003e1\u003c/span\u003e \u003cspan class=\"o\"\u003e||\u003c/span\u003e \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;DNS test failed\u0026#34;\u003c/span\u003e \u0026gt; \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$OUTPUT_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e/dns-test.txt\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Diagnostics complete. Results in: \u003c/span\u003e\u003cspan class=\"nv\"\u003e$OUTPUT_DIR\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Quick analysis:\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Nodes: \u003c/span\u003e\u003cspan class=\"k\"\u003e$(\u003c/span\u003ekubectl get nodes --no-headers \u003cspan class=\"p\"\u003e|\u003c/span\u003e wc -l\u003cspan class=\"k\"\u003e)\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Pods: \u003c/span\u003e\u003cspan class=\"k\"\u003e$(\u003c/span\u003ekubectl get pods -A --no-headers \u003cspan class=\"p\"\u003e|\u003c/span\u003e wc -l\u003cspan class=\"k\"\u003e)\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Not Ready Pods: \u003c/span\u003e\u003cspan class=\"k\"\u003e$(\u003c/span\u003ecat \u003cspan class=\"nv\"\u003e$OUTPUT_DIR\u003c/span\u003e/not-ready-pods.txt \u003cspan class=\"p\"\u003e|\u003c/span\u003e wc -l\u003cspan class=\"k\"\u003e)\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Nodes Under Pressure: \u003c/span\u003e\u003cspan class=\"k\"\u003e$(\u003c/span\u003ecat \u003cspan class=\"nv\"\u003e$OUTPUT_DIR\u003c/span\u003e/nodes-under-pressure.txt \u003cspan class=\"p\"\u003e|\u003c/span\u003e wc -l\u003cspan class=\"k\"\u003e)\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Review the output files for detailed diagnostics.\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis script collects the signals that matter at scale: API latency, pod distribution skew, resource pressure indicators, and object count metrics. It doesn\u0026rsquo;t solve problems, but it eliminates 90% of the noise.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"patterns-that-prevent-catastrophe\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#patterns-that-prevent-catastrophe\" title=\"Patterns That Prevent Catastrophe\"\u003ePatterns That Prevent Catastrophe\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eAfter running mega-clusters through multiple incident cycles, a few patterns consistently prevent the worst outcomes:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eProgressive rollouts\u003c/strong\u003e: Never deploy to 1,000 nodes simultaneously. Deploy to 1 node, then 10, then 100, then all. Automate rollback triggers. This pattern catches 95% of scale-dependent bugs before they impact production.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eBlast radius isolation\u003c/strong\u003e: Segment your cluster into failure domains using node pools, namespaces, and network policies. When something fails (and it will), contain the damage. \u003ca href=\"/posts/aks-network-policies-zero-trust/\"\u003eAKS Network Policies: The Security Layer Your Cluster Is Missing\u003c/a\u003e covers practical policy configuration if you are starting from scratch.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCapacity reservation\u003c/strong\u003e: Reserve 15-20% headroom for burst traffic and incident response. Running at 90%+ utilization saves money until you need to scale during an outage and can\u0026rsquo;t.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eImmutable infrastructure\u003c/strong\u003e: Treat nodes as cattle, not pets. Automate node replacement on a fixed schedule (weekly or monthly). This prevents subtle configuration drift that compounds into unreproducible failures.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eOperational runbooks\u003c/strong\u003e: Document every common failure mode. When API server latency spikes at 2 AM, you don\u0026rsquo;t want to be reading Kubernetes source code to understand etcd compaction behavior.\u003c/p\u003e\n\u003cp\u003eThese patterns aren\u0026rsquo;t revolutionary. They\u0026rsquo;re boring, defensive engineering. At mega-cluster scale, boring wins.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"honest-takeaways\"\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/#honest-takeaways\" title=\"Honest Takeaways\"\u003eHonest Takeaways\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eRunning AKS at 1,000+ nodes isn\u0026rsquo;t fundamentally different from running it at 100 nodes. It\u0026rsquo;s exponentially different. Problems that self-heal at small scale cascade catastrophically at large scale. Architectural decisions that feel premature at 50 nodes become load-bearing at 500 nodes.\u003c/p\u003e\n\u003cp\u003eIf you\u0026rsquo;re planning to scale past 500 nodes: budget significant engineering time for operational tooling. Plan your observability strategy before your first node boots. Understand your cost model in detail. Test failure scenarios at scale before they happen in production.\u003c/p\u003e\n\u003cp\u003eIf you\u0026rsquo;re already running at scale: you know everything in this article because you\u0026rsquo;ve lived it. The value isn\u0026rsquo;t the advice. It\u0026rsquo;s knowing you\u0026rsquo;re not alone in discovering these lessons the hard way.\u003c/p\u003e\n\u003cp\u003eScale is honest. Every shortcut taken for velocity will surface eventually, usually at the worst possible moment. Budget engineering time to address that reality before you hit 500 nodes, not after. Fixing structural problems under production pressure costs significantly more than building them correctly from the start.\u003c/p\u003e\n","date_modified":"2026-05-26T10:22:03+02:00","date_published":"2026-04-01T17:00:00+01:00","id":"https://daily-devops.net/posts/aks-at-scale-mega-cluster-lessons/","language":"en","summary":"Real-world lessons from operating 1000+ node AKS clusters: etcd limits, network saturation, observability overhead, and cost spirals you need to know.","tags":["kubernetes","azure","cloud","devops","operations","infrastructure"],"title":"AKS at Scale: Hard-Won Lessons from 1000+ Node Clusters","url":"https://daily-devops.net/posts/aks-at-scale-mega-cluster-lessons/"},{"authors":[{"name":"Martin Stühmer","url":"https://daily-devops.net/authors/martin/"}],"content_html":"\u003cp\u003eIn the cloud era, infrastructure configuration has become code review material. Yet I\u0026rsquo;ve watched teams spend months preparing for ISO 27017 audits, frantically documenting Azure resources that were manually configured through the portal six months ago. When the auditor asks, \u0026ldquo;Show me the change control process for this storage account\u0026rsquo;s network rules,\u0026rdquo; the response is often an uncomfortable silence followed by scrambling through Azure Activity Logs.\u003c/p\u003e\n\u003cp\u003eISO/IEC 27017:2015 provides cloud-specific security controls extending ISO 27002. Control CLD 6.3.1 (Shared roles and responsibilities for cloud services) demands clear documentation of who changed what, when, and why. Manual configuration through the Azure Portal fundamentally violates this requirement. There\u0026rsquo;s no code review, no approval process, no version history explaining the reasoning behind configuration decisions.\u003c/p\u003e\n\u003cp\u003eInfrastructure as Code (IaC) using Azure Bicep isn\u0026rsquo;t just a DevOps best practice—it\u0026rsquo;s the compliance foundation that makes ISO 27017 audits survivable. When your infrastructure is code, your Git history becomes your audit trail, your pull requests become your change control process, and your CI/CD pipeline becomes your enforcement mechanism.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-fatal-pattern-click-ops-configuration-management\"\u003e\u003ca href=\"/posts/infrastructure-as-code-compliance-bicep/#the-fatal-pattern-click-ops-configuration-management\" title=\"The Fatal Pattern: Click-Ops Configuration Management\"\u003eThe Fatal Pattern: Click-Ops Configuration Management\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eLet me show you what non-compliance looks like in practice. This isn\u0026rsquo;t a code example—it\u0026rsquo;s a process anti-pattern I\u0026rsquo;ve encountered in dozens of organizations:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDay 1:\u003c/strong\u003e DevOps engineer creates Azure SQL Database through the portal. Clicks \u0026ldquo;Review + Create\u0026rdquo; without documenting firewall rules or network isolation decisions.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDay 30:\u003c/strong\u003e Security team requests network lockdown. Same engineer adds VNet integration and private endpoints through the portal. No documentation of which IP ranges need access or why.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDay 60:\u003c/strong\u003e Application breaks in production. Emergency change: someone adds a firewall rule allowing all Azure services. Incident ticket mentions \u0026ldquo;temporary fix\u0026rdquo; but the rule stays forever.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDay 180:\u003c/strong\u003e Audit preparation begins. Team discovers:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eSeven different people modified the database configuration\u003c/li\u003e\n\u003cli\u003eNo documentation exists explaining current network rules\u003c/li\u003e\n\u003cli\u003eActivity Logs show changes but not the reasoning\u003c/li\u003e\n\u003cli\u003eConfiguration drift exists between dev, staging, and production\u003c/li\u003e\n\u003cli\u003eNo evidence of security review or approval process\u003c/li\u003e\n\u003cli\u003eCannot recreate current state reliably\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch3 id=\"what-an-audit-finding-actually-reads-like\"\u003e\u003ca href=\"/posts/infrastructure-as-code-compliance-bicep/#what-an-audit-finding-actually-reads-like\" title=\"What An Audit Finding Actually Reads Like\"\u003eWhat An Audit Finding Actually Reads Like\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eThis is what ISO 27017 auditors flag as non-compliant:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-plaintext\" data-lang=\"plaintext\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eAudit Finding: CLD 6.3.1 Violation\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eSeverity: High\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eEvidence of manual configuration changes to production Azure SQL Database\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ewithout documented change control process. Network security rules modified\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eby multiple individuals without approval workflow. Unable to demonstrate\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eseparation of duties or security review process. Configuration baseline\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ecannot be established for ongoing compliance monitoring.\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eRecommendation: Implement Infrastructure as Code with version control,\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ecode review, and automated deployment pipeline.\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe fundamental problem isn\u0026rsquo;t the tools—it\u0026rsquo;s the lack of process enforcement. The Azure Portal is a GUI that bypasses every control you\u0026rsquo;d apply to application code: no peer review, no automated testing, no approval gates, no rollback capability, no audit trail beyond basic activity logs.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"manual-configuration-violates-multiple-iso-controls\"\u003e\u003ca href=\"/posts/infrastructure-as-code-compliance-bicep/#manual-configuration-violates-multiple-iso-controls\" title=\"Manual Configuration Violates Multiple ISO Controls\"\u003eManual Configuration Violates Multiple ISO Controls\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eBeyond CLD 6.3.1, click-ops infrastructure management violates:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eA.12.1.2 (Change Management):\u003c/strong\u003e \u0026ldquo;Changes to the organization, business processes, information processing facilities and systems that affect information security shall be controlled.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eManual portal changes are uncontrolled. There\u0026rsquo;s no systematic process ensuring security requirements are evaluated before infrastructure modifications.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eA.14.2.2 (System Change Control Procedures):\u003c/strong\u003e \u0026ldquo;Changes to systems within the development lifecycle shall be controlled by the use of formal change control procedures.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eClicking buttons in the Azure Portal doesn\u0026rsquo;t constitute formal change control. There\u0026rsquo;s no structured approval workflow, no testing in lower environments with identical infrastructure definitions, no rollback procedure beyond manual reversal.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eA.18.2.3 (Technical Compliance Review):\u003c/strong\u003e \u0026ldquo;Information systems shall be regularly reviewed for compliance with the organization\u0026rsquo;s information security policies and standards.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eHow do you review compliance when your infrastructure exists as portal clicks rather than declarative code? You can\u0026rsquo;t diff two portal sessions. You can\u0026rsquo;t run policy checks against mouse movements.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"why-you-cannot-diff-portal-sessions\"\u003e\u003ca href=\"/posts/infrastructure-as-code-compliance-bicep/#why-you-cannot-diff-portal-sessions\" title=\"Why You Cannot Diff Portal Sessions\"\u003eWhy You Cannot Diff Portal Sessions\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eThe moment you accept manual infrastructure changes, you\u0026rsquo;ve conceded that infrastructure security is less important than application security. No one would allow developers to deploy application code without peer review—why accept the same for infrastructure that controls network access, encryption, and identity management?\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-compliant-pattern-infrastructure-as-auditable-code\"\u003e\u003ca href=\"/posts/infrastructure-as-code-compliance-bicep/#the-compliant-pattern-infrastructure-as-auditable-code\" title=\"The Compliant Pattern: Infrastructure as Auditable Code\"\u003eThe Compliant Pattern: Infrastructure as Auditable Code\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eEnough about what\u0026rsquo;s broken. Let\u0026rsquo;s look at what works.\u003c/p\u003e\n\u003cp\u003eInfrastructure as Code with Azure Bicep transforms compliance from documentation theater into automated enforcement. Here\u0026rsquo;s what a compliant Azure SQL Database deployment looks like—notice how every security decision becomes self-documenting:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bicep\" data-lang=\"bicep\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e// main.bicep - Compliant SQL Database configuration\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e@\u003c/span\u003e\u003cspan class=\"nf\"\u003eallowed\u003c/span\u003e\u003cspan class=\"p\"\u003e([\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;dev\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;staging\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;production\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e])\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eparam\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eenvironment\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003estring\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eparam\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003elocation\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003estring\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003eresourceGroup\u003c/span\u003e\u003cspan class=\"p\"\u003e().\u003c/span\u003e\u003cspan class=\"nv\"\u003elocation\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e@\u003c/span\u003e\u003cspan class=\"nf\"\u003esecure\u003c/span\u003e\u003cspan class=\"p\"\u003e()\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eparam\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003esqlAdminPassword\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003estring\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003evar\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003esqlServerName\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;sql-\u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eenvironment\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s\"\u003e-\u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nf\"\u003euniqueString\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nf\"\u003eresourceGroup\u003c/span\u003e\u003cspan class=\"p\"\u003e().\u003c/span\u003e\u003cspan class=\"nv\"\u003eid\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eresource\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003esqlServer\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Microsoft.Sql/servers@2023-05-01-preview\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003esqlServerName\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003elocation\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003elocation\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003eproperties\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003eadministratorLogin\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;sqladmin\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003eadministratorLoginPassword\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003esqlAdminPassword\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"c1\"\u003e// Security baseline: No public access in production\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003epublicNetworkAccess\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eenvironment\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e==\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;production\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e?\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Disabled\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Enabled\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003eminimalTlsVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;1.2\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"c1\"\u003e// Entra-only authentication enforced\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003eadministrators\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nv\"\u003eadministratorType\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;ActiveDirectory\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nv\"\u003eazureADOnlyAuthentication\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"c1\"\u003e// ... Entra ID configuration\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"kd\"\u003eresource\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eauditingSettings\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;auditingSettings\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;default\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003eproperties\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nv\"\u003estate\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Enabled\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nv\"\u003eretentionDays\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003e90\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"c1\"\u003e// Audit authentication attempts and queries\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e// Private endpoint for production - no portal clicks allowed\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003emodule\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eprivateEndpoint\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;modules/private-endpoint.bicep\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kd\"\u003eif\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nv\"\u003eenvironment\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"o\"\u003e==\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;production\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;sqlPrivateEndpoint\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003eparams\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003eprivateLinkServiceId\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003esqlServer\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nv\"\u003eid\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"c1\"\u003e// ... network configuration\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis template embeds compliance directly into code. The Git history for this file becomes your audit trail—every commit message explains why something changed, every pull request documents who approved it.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"modular-bicep-structure-for-reusability\"\u003e\u003ca href=\"/posts/infrastructure-as-code-compliance-bicep/#modular-bicep-structure-for-reusability\" title=\"Modular Bicep Structure for Reusability\"\u003eModular Bicep Structure for Reusability\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eBut one template isn\u0026rsquo;t enough. Real compliance requires consistent patterns across your entire Azure estate.\u003c/p\u003e\n\u003cp\u003eBicep modules let you define security patterns once and reuse them everywhere:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bicep\" data-lang=\"bicep\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e// modules/private-endpoint.bicep - Reusable security pattern\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eparam\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eprivateEndpointName\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003estring\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eparam\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eprivateLinkServiceId\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003estring\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eparam\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003esubnetId\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003estring\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eparam\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003elocation\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003estring\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eparam\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003egroupIds\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003earray\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eresource\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eprivateEndpoint\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Microsoft.Network/privateEndpoints@2023-06-01\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eprivateEndpointName\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003elocation\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003elocation\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003eproperties\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003esubnet\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eid\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003esubnetId\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003eprivateLinkServiceConnections\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eprivateEndpointName\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nv\"\u003eproperties\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nv\"\u003eprivateLinkServiceId\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eprivateLinkServiceId\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nv\"\u003egroupIds\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003egroupIds\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e}]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e// Private DNS integration - name resolution without public exposure\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eresource\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eprivateDnsZoneGroup\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Microsoft.Network/privateEndpoints/privateDnsZoneGroups@2023-06-01\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003eparent\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eprivateEndpoint\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;default\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003eproperties\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003eprivateDnsZoneConfigs\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;sqlConfig\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nv\"\u003eproperties\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nv\"\u003eprivateDnsZoneId\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;/subscriptions/.../privatelink.database.windows.net\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e}]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eWhen your security team updates this pattern—say, adding a new DNS configuration requirement—every resource using this module inherits the improvement automatically. One change, organization-wide compliance.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"policy-as-code-automated-compliance-enforcement\"\u003e\u003ca href=\"/posts/infrastructure-as-code-compliance-bicep/#policy-as-code-automated-compliance-enforcement\" title=\"Policy-as-Code: Automated Compliance Enforcement\"\u003ePolicy-as-Code: Automated Compliance Enforcement\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eModules enforce patterns for teams that use them. But what about teams that don\u0026rsquo;t?\u003c/p\u003e\n\u003cp\u003eAzure Policy catches non-compliant configurations before they reach production:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bicep\" data-lang=\"bicep\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e// policy/sql-security-baseline.bicep\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003etargetScope\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;subscription\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e// Enforce TLS 1.2 minimum - no exceptions\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eresource\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003epolicyTlsVersion\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Microsoft.Authorization/policyDefinitions@2021-06-01\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;enforce-sql-tls-12\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003eproperties\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003edisplayName\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;SQL Servers must use TLS 1.2 or higher\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003epolicyType\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Custom\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003emode\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;All\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003epolicyRule\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"kd\"\u003eif\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nv\"\u003eallOf\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003efield\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;type\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eequals\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Microsoft.Sql/servers\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003efield\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Microsoft.Sql/servers/minimalTlsVersion\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003enotEquals\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;1.2\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nv\"\u003ethen\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eeffect\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Deny\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e// Deny public network access in production\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eresource\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003epolicyPublicAccess\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Microsoft.Authorization/policyDefinitions@2021-06-01\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;deny-sql-public-network-production\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003eproperties\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003edisplayName\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Production SQL Servers must disable public network access\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003epolicyType\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Custom\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003emode\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;All\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003epolicyRule\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"kd\"\u003eif\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nv\"\u003eallOf\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003efield\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;type\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eequals\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Microsoft.Sql/servers\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003efield\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;tags[environment]\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eequals\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;production\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003efield\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Microsoft.Sql/servers/publicNetworkAccess\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003enotEquals\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Disabled\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nv\"\u003ethen\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eeffect\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Deny\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e// Assign policies to subscription\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eresource\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003epolicyAssignmentTls\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Microsoft.Authorization/policyAssignments@2022-06-01\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;sql-tls-baseline\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003eproperties\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003edisplayName\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;SQL TLS Version Baseline\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003epolicyDefinitionId\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003epolicyTlsVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nv\"\u003eid\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003eenforcementMode\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Default\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThese policies act as guardrails. Non-compliant Bicep templates fail validation before any Azure API calls are made. Your security baseline becomes enforceable code, not a PDF checklist gathering dust.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"github-actions-workflow-deployment-with-compliance-gates\"\u003e\u003ca href=\"/posts/infrastructure-as-code-compliance-bicep/#github-actions-workflow-deployment-with-compliance-gates\" title=\"GitHub Actions Workflow: Deployment with Compliance Gates\"\u003eGitHub Actions Workflow: Deployment with Compliance Gates\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eWith templates and policies in place, the deployment pipeline ties everything together:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# .github/workflows/deploy-infrastructure.yml\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eDeploy Infrastructure\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eon\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003epull_request\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003epaths\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;infrastructure/**\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003epush\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ebranches\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"l\"\u003emain]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003epaths\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;infrastructure/**\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003epermissions\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eid-token\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ewrite\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003econtents\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eread\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003epull-requests\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ewrite\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ejobs\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003evalidate\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eValidate Bicep Templates\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eruns-on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eubuntu-latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esteps\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/checkout@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eazure/login@v2\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ewith\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003eclient-id\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e${{ secrets.AZURE_CLIENT_ID }}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003etenant-id\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e${{ secrets.AZURE_TENANT_ID }}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003esubscription-id\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e${{ secrets.AZURE_SUBSCRIPTION_ID }}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eBicep Lint \u0026amp; Build\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eaz bicep build --file infrastructure/main.bicep\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eWhat-If Analysis\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          az deployment group what-if \\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            --resource-group rg-production \\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            --template-file infrastructure/main.bicep \\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            --parameters @infrastructure/parameters.production.json\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ePolicy Compliance Check\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          az deployment group validate \\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            --resource-group rg-production \\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            --template-file infrastructure/main.bicep \\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            --parameters @infrastructure/parameters.production.json\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003edeploy\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eDeploy to Azure\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eneeds\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003evalidate\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eif\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003egithub.ref == \u0026#39;refs/heads/main\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eenvironment\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eproduction\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eruns-on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eubuntu-latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esteps\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/checkout@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eazure/login@v2\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ewith\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003eclient-id\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e${{ secrets.AZURE_CLIENT_ID }}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003etenant-id\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e${{ secrets.AZURE_TENANT_ID }}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003esubscription-id\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e${{ secrets.AZURE_SUBSCRIPTION_ID }}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eDeploy Bicep Template\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          az deployment group create \\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            --resource-group rg-production \\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            --template-file infrastructure/main.bicep \\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            --parameters @infrastructure/parameters.production.json\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eArchive Deployment Evidence\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/upload-artifact@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ewith\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edeployment-evidence\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003epath\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edeployment-output.json\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003eretention-days\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e2555\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"c\"\u003e# 7 years for compliance\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003edrift-detection\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eConfiguration Drift Detection\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eneeds\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edeploy\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eruns-on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eubuntu-latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esteps\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/checkout@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eazure/login@v2\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ewith\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003eclient-id\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e${{ secrets.AZURE_CLIENT_ID }}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003etenant-id\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e${{ secrets.AZURE_TENANT_ID }}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003esubscription-id\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e${{ secrets.AZURE_SUBSCRIPTION_ID }}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eDetect Configuration Drift\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          az deployment group what-if \\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            --resource-group rg-production \\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            --template-file infrastructure/main.bicep \\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            --parameters @infrastructure/parameters.production.json \\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            --mode Complete \u0026gt; drift-check.txt\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          if grep -q \u0026#34;Resource changes:\u0026#34; drift-check.txt; then\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            echo \u0026#34;::error::Configuration drift detected!\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            exit 1\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          fi\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\n\n\n\n\u003ch3 id=\"seven-years-of-deployment-evidence\"\u003e\u003ca href=\"/posts/infrastructure-as-code-compliance-bicep/#seven-years-of-deployment-evidence\" title=\"Seven Years Of Deployment Evidence\"\u003eSeven Years Of Deployment Evidence\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eThis workflow enforces change control automatically. Every infrastructure change requires code review before merge. What-if analysis shows exactly what will change. Policy compliance verifies security baselines. And deployment artifacts get retained for 7 years—because auditors love documentation they can actually find.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"drift-detection-as-continuous-compliance-monitoring\"\u003e\u003ca href=\"/posts/infrastructure-as-code-compliance-bicep/#drift-detection-as-continuous-compliance-monitoring\" title=\"Drift Detection as Continuous Compliance Monitoring\"\u003eDrift Detection as Continuous Compliance Monitoring\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eHere\u0026rsquo;s the uncomfortable truth: even with IaC in place, someone will eventually click something in the portal. \u0026ldquo;Just this once\u0026rdquo; to fix a production issue.\u003c/p\u003e\n\u003cp\u003eConfiguration drift detection catches these changes before they become audit findings:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bicep\" data-lang=\"bicep\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e// modules/drift-detection.bicep - Azure Function for periodic compliance checks\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eparam\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003efunctionAppName\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003estring\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eparam\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003elocation\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003estring\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003eresourceGroup\u003c/span\u003e\u003cspan class=\"p\"\u003e().\u003c/span\u003e\u003cspan class=\"nv\"\u003elocation\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eparam\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003erepositoryUrl\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003estring\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eresource\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eappServicePlan\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Microsoft.Web/serverfarms@2023-01-01\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;\u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003efunctionAppName\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s\"\u003e-plan\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003elocation\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003elocation\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003esku\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Y1\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003etier\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Dynamic\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"kd\"\u003eresource\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003efunctionApp\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;Microsoft.Web/sites@2023-01-01\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003efunctionAppName\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003elocation\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003elocation\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;functionapp\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nv\"\u003eproperties\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003eserverFarmId\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003eappServicePlan\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"nv\"\u003eid\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nv\"\u003esiteConfig\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nv\"\u003eappSettings\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;FUNCTIONS_WORKER_RUNTIME\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003evalue\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;powershell\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;REPOSITORY_URL\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003evalue\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003erepositoryUrl\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s\"\u003e\u0026#39;RESOURCE_GROUP\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nv\"\u003evalue\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nf\"\u003eresourceGroup\u003c/span\u003e\u003cspan class=\"p\"\u003e().\u003c/span\u003e\u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe Azure Function runs hourly, comparing deployed resources against your Bicep definitions:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-powershell\" data-lang=\"powershell\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# DriftDetection/run.ps1\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eparam\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nv\"\u003e$Timer\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003e$ErrorActionPreference\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;Stop\u0026#39;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003e$tempDir\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"nb\"\u003eJoin-Path\u003c/span\u003e \u003cspan class=\"nv\"\u003e$env:TEMP\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;infra-\u003c/span\u003e\u003cspan class=\"p\"\u003e$(\u003c/span\u003e\u003cspan class=\"nb\"\u003eGet-Date\u003c/span\u003e \u003cspan class=\"n\"\u003e-Format\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;yyyyMMddHHmmss\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# Clone infrastructure repo and run what-if analysis\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003egit\u003c/span\u003e \u003cspan class=\"n\"\u003eclone\u003c/span\u003e \u003cspan class=\"p\"\u003e-\u003c/span\u003e\u003cspan class=\"n\"\u003e-depth\u003c/span\u003e \u003cspan class=\"mf\"\u003e1\u003c/span\u003e \u003cspan class=\"nv\"\u003e$env:REPOSITORY_URL\u003c/span\u003e \u003cspan class=\"nv\"\u003e$tempDir\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003e$whatIfResult\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eaz\u003c/span\u003e \u003cspan class=\"n\"\u003edeployment\u003c/span\u003e \u003cspan class=\"nb\"\u003egroup what-if\u003c/span\u003e \u003cspan class=\"p\"\u003e`\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e-\u003c/span\u003e\u003cspan class=\"n\"\u003e-resource-group\u003c/span\u003e \u003cspan class=\"nv\"\u003e$env:RESOURCE_GROUP\u003c/span\u003e \u003cspan class=\"p\"\u003e`\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e-\u003c/span\u003e\u003cspan class=\"n\"\u003e-template\u003c/span\u003e\u003cspan class=\"o\"\u003e-file\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$tempDir\u003c/span\u003e\u003cspan class=\"s2\"\u003e/infrastructure/main.bicep\u0026#34;\u003c/span\u003e \u003cspan class=\"p\"\u003e`\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e-\u003c/span\u003e\u003cspan class=\"n\"\u003e-parameters\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;@\u003c/span\u003e\u003cspan class=\"nv\"\u003e$tempDir\u003c/span\u003e\u003cspan class=\"s2\"\u003e/infrastructure/parameters.production.json\u0026#34;\u003c/span\u003e \u003cspan class=\"p\"\u003e`\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e-\u003c/span\u003e\u003cspan class=\"n\"\u003e-mode\u003c/span\u003e \u003cspan class=\"n\"\u003eComplete\u003c/span\u003e \u003cspan class=\"p\"\u003e-\u003c/span\u003e\u003cspan class=\"n\"\u003e-output\u003c/span\u003e \u003cspan class=\"n\"\u003ejson\u003c/span\u003e \u003cspan class=\"p\"\u003e|\u003c/span\u003e \u003cspan class=\"nb\"\u003eConvertFrom-Json\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# Check for unauthorized changes\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003e$driftChanges\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"nv\"\u003e$whatIfResult\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"py\"\u003echanges\u003c/span\u003e \u003cspan class=\"p\"\u003e|\u003c/span\u003e \u003cspan class=\"nb\"\u003eWhere-Object\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nv\"\u003e$_\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"py\"\u003echangeType\u003c/span\u003e \u003cspan class=\"n\"\u003e-in\u003c/span\u003e \u003cspan class=\"vm\"\u003e@\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;Modify\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;Delete\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;Create\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nv\"\u003e$driftChanges\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nb\"\u003eWrite-Warning\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Configuration drift detected!\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"c\"\u003e# Alert security team via webhook\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nv\"\u003e$incident\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"vm\"\u003e@\u003c/span\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003etitle\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Drift Detected in \u003c/span\u003e\u003cspan class=\"p\"\u003e$(\u003c/span\u003e\u003cspan class=\"nv\"\u003e$env:RESOURCE_GROUP\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003eseverity\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;High\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003echanges\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"nv\"\u003e$driftChanges\u003c/span\u003e \u003cspan class=\"p\"\u003e|\u003c/span\u003e \u003cspan class=\"nb\"\u003eConvertTo-Json\u003c/span\u003e \u003cspan class=\"n\"\u003e-Depth\u003c/span\u003e \u003cspan class=\"mf\"\u003e5\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nb\"\u003eInvoke-RestMethod\u003c/span\u003e \u003cspan class=\"n\"\u003e-Uri\u003c/span\u003e \u003cspan class=\"nv\"\u003e$env:INCIDENT_WEBHOOK_URL\u003c/span\u003e \u003cspan class=\"n\"\u003e-Method\u003c/span\u003e \u003cspan class=\"n\"\u003ePost\u003c/span\u003e \u003cspan class=\"p\"\u003e`\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"n\"\u003e-Body\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nv\"\u003e$incident\u003c/span\u003e \u003cspan class=\"p\"\u003e|\u003c/span\u003e \u003cspan class=\"nb\"\u003eConvertTo-Json\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"n\"\u003e-ContentType\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;application/json\u0026#39;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003ethrow\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Manual changes found in Azure infrastructure.\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eWrite-Host\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;No drift detected. Infrastructure matches Git.\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eRemove-Item\u003c/span\u003e \u003cspan class=\"nv\"\u003e$tempDir\u003c/span\u003e \u003cspan class=\"n\"\u003e-Recurse\u003c/span\u003e \u003cspan class=\"n\"\u003e-Force\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\n\n\n\n\u003ch3 id=\"catching-the-just-this-once-click\"\u003e\u003ca href=\"/posts/infrastructure-as-code-compliance-bicep/#catching-the-just-this-once-click\" title=\"Catching The \u0026ldquo;Just This Once\u0026rdquo; Click\"\u003eCatching The \u0026ldquo;Just This Once\u0026rdquo; Click\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eWith hourly drift detection, manual portal changes get caught within hours, not months. The security team receives immediate notification, and the audit trail documents when drift was detected and how it was remediated.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-compliance-transformation\"\u003e\u003ca href=\"/posts/infrastructure-as-code-compliance-bicep/#the-compliance-transformation\" title=\"The Compliance Transformation\"\u003eThe Compliance Transformation\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eInfrastructure as Code with Azure Bicep transforms ISO 27017 compliance from documentation burden to automated enforcement:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eBefore (Click-Ops):\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eConfiguration exists as portal clicks, not code\u003c/li\u003e\n\u003cli\u003eNo version history or change rationale\u003c/li\u003e\n\u003cli\u003eManual documentation prone to errors and drift\u003c/li\u003e\n\u003cli\u003eAudit preparation requires months of retroactive investigation\u003c/li\u003e\n\u003cli\u003eNon-compliance discovered during audit\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eAfter (IaC with Bicep):\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eInfrastructure defined as declarative code with inline compliance documentation\u003c/li\u003e\n\u003cli\u003eGit history provides complete audit trail\u003c/li\u003e\n\u003cli\u003ePull requests enforce code review and approval workflow\u003c/li\u003e\n\u003cli\u003eAzure Policy prevents non-compliant deployments\u003c/li\u003e\n\u003cli\u003eDrift detection catches unauthorized changes automatically\u003c/li\u003e\n\u003cli\u003eCompliance demonstrated continuously, not just during audits\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch3 id=\"from-three-months-to-two-weeks\"\u003e\u003ca href=\"/posts/infrastructure-as-code-compliance-bicep/#from-three-months-to-two-weeks\" title=\"From Three Months To Two Weeks\"\u003eFrom Three Months To Two Weeks\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eThe cost savings are measurable. One organization I worked with reduced ISO 27017 audit preparation from 3 months to 2 weeks by implementing Bicep-based IaC. Their auditor\u0026rsquo;s report literally stated: \u0026ldquo;Version-controlled infrastructure templates with inline compliance documentation exceed ISO 27017 requirements for change control.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eMore importantly, compliance became continuous rather than periodic. Security teams shifted from reactive documentation to proactive policy enforcement. When a new ISO control requirement emerged, they updated Bicep modules and Azure Policies once—every resource using those modules inherited compliance automatically.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"practical-implementation-recommendations\"\u003e\u003ca href=\"/posts/infrastructure-as-code-compliance-bicep/#practical-implementation-recommendations\" title=\"Practical Implementation Recommendations\"\u003ePractical Implementation Recommendations\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eIf you\u0026rsquo;re starting IaC implementation for ISO 27017 compliance:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eStart with new resources:\u003c/strong\u003e Don\u0026rsquo;t try to Bicep-ify your entire Azure estate overnight. New deployments go through IaC from day one.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eIncrementally import existing resources:\u003c/strong\u003e Use \u003ccode\u003eaz bicep decompile\u003c/code\u003e to generate Bicep from existing ARM templates or portal-created resources. Treat the output as a starting point, not production-ready code.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eEstablish modules early:\u003c/strong\u003e Create reusable Bicep modules for common patterns (private endpoints, managed identities, diagnostic settings). Consistency across resources simplifies compliance demonstration.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eImplement policy-as-code in parallel:\u003c/strong\u003e Azure Policy enforcement prevents regression to manual configuration. Deploy policies to audit mode first, then switch to deny mode after teams adapt.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eAutomate drift detection from the beginning:\u003c/strong\u003e Manual changes will happen during the transition period. Automated detection ensures they\u0026rsquo;re caught and documented.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eRetain deployment artifacts:\u003c/strong\u003e Store deployment outputs, what-if results, and approval records for the retention period required by your compliance framework (typically 7 years for ISO).\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eTrain teams on compliance rationale:\u003c/strong\u003e Developers need to understand \u003cem\u003ewhy\u003c/em\u003e IaC matters for compliance, not just \u003cem\u003ehow\u003c/em\u003e to write Bicep. When they understand the audit implications, they become compliance advocates.\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eInfrastructure as Code isn\u0026rsquo;t optional for compliance in cloud environments. Manual configuration through the Azure Portal fundamentally cannot satisfy change control requirements. The moment you accept click-ops, you\u0026rsquo;ve accepted non-compliance.\u003c/p\u003e\n\u003cp\u003eBicep makes compliance enforceable rather than aspirational. Your Git history becomes your audit trail. Your pull request reviews become your change approval process. Your CI/CD pipeline becomes your enforcement mechanism.\u003c/p\u003e\n\u003cp\u003eThe question isn\u0026rsquo;t whether to implement IaC for compliance—it\u0026rsquo;s whether you implement it proactively or after an audit finding forces your hand.\u003c/p\u003e\n","date_modified":"2026-05-26T10:22:03+02:00","date_published":"2026-03-31T17:00:00+01:00","id":"https://daily-devops.net/posts/infrastructure-as-code-compliance-bicep/","language":"en","summary":"Azure Portal clicks fail ISO 27017 Control CLD 6.3.1. Move to Bicep so Git history becomes the audit trail and pull requests the change control.","tags":["iso-standards","cloud","azure","infrastructure-as-code","devops","bicep"],"title":"Why Your Azure Portal Clicks Will Fail the Next Audit","url":"https://daily-devops.net/posts/infrastructure-as-code-compliance-bicep/"},{"authors":[{"name":"Martin Stühmer","url":"https://daily-devops.net/authors/martin/"}],"content_html":"\u003cp\u003eLast month I watched a deployment workflow push code to production with three critical vulnerabilities, a failing integration test, and an API key hardcoded in plain text. The team\u0026rsquo;s response? \u0026ldquo;We\u0026rsquo;ll fix it next sprint.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eTwo weeks later, that API key was on GitHub\u0026rsquo;s leaked secrets list. The vulnerabilities got exploited. And suddenly \u0026ldquo;next sprint\u0026rdquo; became \u0026ldquo;incident response war room.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eThis wasn\u0026rsquo;t some junior developer\u0026rsquo;s side project. This was an enterprise team at a company with security certifications, compliance officers, and a CISO who probably makes more than all of us combined.\u003c/p\u003e\n\u003cp\u003eHere\u0026rsquo;s what their pipeline—and probably yours—looks like:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eFailing tests treated as warnings, not blockers\u003c/li\u003e\n\u003cli\u003eVulnerability scans that run but never fail the build\u003c/li\u003e\n\u003cli\u003eSAST (Static Application Security Testing) findings suppressed \u0026ldquo;temporarily\u0026rdquo;\u0026hellip; for six months\u003c/li\u003e\n\u003cli\u003eSecrets committed to Git because \u0026ldquo;it\u0026rsquo;s a private repo\u0026rdquo;\u003c/li\u003e\n\u003cli\u003eZero approval gates between \u003ccode\u003egit push\u003c/code\u003e and production\u003c/li\u003e\n\u003cli\u003eNo rollback plan when everything catches fire\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eEvery single pattern on this list is a breach waiting to happen. And if you\u0026rsquo;re ISO 27001 certified? These patterns also violate Controls A.14.2 and A.18.2—which means your certification is essentially fraudulent.\u003c/p\u003e\n\u003cp\u003eLet me show you what secure deployment actually looks like.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-ship-it-mentality-and-why-its-destroying-you\"\u003e\u003ca href=\"/posts/continuous-deployment-security-gates/#the-ship-it-mentality-and-why-its-destroying-you\" title=\"The \u0026ldquo;Ship It\u0026rdquo; Mentality (And Why It\u0026rsquo;s Destroying You)\"\u003eThe \u0026ldquo;Ship It\u0026rdquo; Mentality (And Why It\u0026rsquo;s Destroying You)\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eHere\u0026rsquo;s what most deployment pipelines actually look like. I\u0026rsquo;ve seen this exact pattern—or worse—at companies you\u0026rsquo;ve definitely heard of:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eYOLO Deploy\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eon\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003epush\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ebranches\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"l\"\u003emain]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ejobs\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003edeploy\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eruns-on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eubuntu-latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esteps\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/checkout@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eRun Tests\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003enpm test || echo \u0026#34;Tests failed, deploying anyway\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"c\"\u003e# No vulnerability scanning. No SAST. No secrets detection.\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eDeploy to Production\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eenv\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003eAPI_KEY\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;sk-prod-abc123\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"c\"\u003e# Hardcoded secret in Git history forever\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003enpm run deploy\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003cstrong\u003eWhat\u0026rsquo;s catastrophically wrong here?\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTests don\u0026rsquo;t block anything.\u003c/strong\u003e That \u003ccode\u003e|| echo\u003c/code\u003e pattern means failures get logged but deployment continues. You might as well not have tests at all.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eNo security scanning whatsoever.\u003c/strong\u003e No vulnerability checks. No static analysis. No secrets detection. An attacker could commit a backdoor and your pipeline would cheerfully deploy it.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSecrets in plain text.\u003c/strong\u003e That API key is now in your Git history. Forever. Even if you delete the file, anyone with repo access can find it. And if your repo ever gets leaked, cloned, or accessed by a compromised developer machine? Game over.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eZero approval gates.\u003c/strong\u003e Push to main, deploy to production. No human verification. No \u0026ldquo;are we sure about this?\u0026rdquo; checkpoint. Just blind faith that the code works.\u003c/p\u003e\n\u003cp\u003eThis pipeline doesn\u0026rsquo;t just violate security best practices—it violates ISO 27001 Controls A.14.2 (Secure Development) and A.18.2 (Security Reviews). If you have that certification and this is your pipeline, you\u0026rsquo;re one audit away from losing it.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-excuses-and-why-theyre-all-wrong\"\u003e\u003ca href=\"/posts/continuous-deployment-security-gates/#the-excuses-and-why-theyre-all-wrong\" title=\"The Excuses (And Why They\u0026rsquo;re All Wrong)\"\u003eThe Excuses (And Why They\u0026rsquo;re All Wrong)\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eI\u0026rsquo;ve heard every rationalization for shipping without security gates. Let me save you the breath.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u0026ldquo;We\u0026rsquo;re a startup—we\u0026rsquo;ll add security later.\u0026rdquo;\u003c/strong\u003e\u003cbr\u003e\nNo, you won\u0026rsquo;t. Technical debt compounds. The \u0026ldquo;later\u0026rdquo; you\u0026rsquo;re imagining never arrives because you\u0026rsquo;re always shipping the next feature. Meanwhile, you\u0026rsquo;re building on a foundation of sand. When you finally try to add security, you\u0026rsquo;ll discover it requires rewriting half your infrastructure.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u0026ldquo;Security scanning slows us down.\u0026rdquo;\u003c/strong\u003e\u003cbr\u003e\nBy 2-5 minutes. That\u0026rsquo;s it. A proper security gate adds minutes to your pipeline. A breach costs weeks of incident response, customer notification, regulatory reporting, and reputation damage. You\u0026rsquo;re not saving time—you\u0026rsquo;re borrowing it at loan shark interest rates.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u0026ldquo;We trust our developers.\u0026rdquo;\u003c/strong\u003e\u003cbr\u003e\nCool. I trust my developers too. I also know that trusted developers make mistakes. They commit secrets by accident. They copy-paste vulnerable code from Stack Overflow. They forget to update dependencies. Trust is not a security control. Automated scanning is.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u0026ldquo;The deadline is more important.\u0026rdquo;\u003c/strong\u003e\u003cbr\u003e\nThan what, exactly? Than not getting breached? Than keeping your certification? Than not explaining to your CEO why customer data is on the dark web? There is no deadline important enough to justify deploying unverified code to production.\u003c/p\u003e\n\u003cp\u003eHere\u0026rsquo;s the uncomfortable truth: Teams that bypass security gates don\u0026rsquo;t have a velocity problem. They have a \u003cstrong\u003ediscipline problem\u003c/strong\u003e dressed up as agility.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"what-actual-security-looks-like\"\u003e\u003ca href=\"/posts/continuous-deployment-security-gates/#what-actual-security-looks-like\" title=\"What Actual Security Looks Like\"\u003eWhat Actual Security Looks Like\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eEnough about what\u0026rsquo;s broken. Here\u0026rsquo;s a GitHub Actions workflow that actually protects your systems. Copy it. Use it. Stop deploying garbage.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eSecure Deployment\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eon\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003epush\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ebranches\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"l\"\u003emain]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003epull_request\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ebranches\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"l\"\u003emain]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003epermissions\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003econtents\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eread\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003esecurity-events\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ewrite\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003epull-requests\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ewrite\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ejobs\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"c\"\u003e# Gate 1: Tests must pass\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003etests\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eruns-on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eubuntu-latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esteps\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/checkout@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/setup-dotnet@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ewith\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003edotnet-version\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;10.0.x\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edotnet test --configuration Release\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"c\"\u003e# Gate 2: No vulnerable dependencies\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003edependency-scan\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eruns-on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eubuntu-latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eneeds\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003etests\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esteps\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/checkout@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          dotnet list package --vulnerable --include-transitive 2\u0026gt;\u0026amp;1 | tee vuln.txt\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          grep -q \u0026#34;vulnerable packages\u0026#34; vuln.txt \u0026amp;\u0026amp; exit 1 || exit 0\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"c\"\u003e# Gate 3: SAST analysis\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ecode-scanning\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eruns-on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eubuntu-latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eneeds\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003etests\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esteps\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/checkout@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003egithub/codeql-action/init@v3\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ewith\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003elanguages\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;csharp\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003egithub/codeql-action/autobuild@v3\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003egithub/codeql-action/analyze@v3\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"c\"\u003e# Gate 4: No hardcoded secrets\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003esecrets-scan\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eruns-on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eubuntu-latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eneeds\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003etests\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esteps\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/checkout@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ewith\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003efetch-depth\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e0\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003etrufflesecurity/trufflehog@v3.82.13\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"c\"\u003e# Staging: All gates must pass\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003edeploy-staging\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eruns-on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eubuntu-latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eneeds\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"l\"\u003etests, dependency-scan, code-scanning, secrets-scan]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eenvironment\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003estaging\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esteps\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/checkout@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edotnet publish -c Release -o ./publish\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eecho \u0026#34;Deploy to staging\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"c\"\u003e# Production: Requires manual approval via GitHub Environment\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003edeploy-production\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eruns-on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eubuntu-latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eneeds\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edeploy-staging\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eenvironment\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eproduction \u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"c\"\u003e# This enforces manual approval\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esteps\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/checkout@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edotnet publish -c Release -o ./publish\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eecho \u0026#34;Deploy to production\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"c\"\u003e# Audit trail for compliance evidence\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          echo \u0026#34;Deployed by: ${{ github.actor }}\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          echo \u0026#34;Commit: ${{ github.sha }}\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          echo \u0026#34;Timestamp: $(date -Iseconds)\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003cstrong\u003eWhy this actually works:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEvery gate is mandatory.\u003c/strong\u003e The \u003ccode\u003eneeds\u003c/code\u003e chain means production deployment literally cannot happen unless tests pass, vulnerabilities are clean, SAST finds nothing critical, and no secrets are detected. There\u0026rsquo;s no \u003ccode\u003econtinue-on-error\u003c/code\u003e escape hatch.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSecrets stay secret.\u003c/strong\u003e No credentials in the workflow file. GitHub environments handle authentication. The \u003ccode\u003epermissions\u003c/code\u003e block enforces least privilege—the workflow can only do what it absolutely needs.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eHumans verify production deployments.\u003c/strong\u003e The \u003ccode\u003eenvironment: production\u003c/code\u003e line triggers GitHub\u0026rsquo;s environment protection rules. Someone has to explicitly approve before code hits production. That approval is logged forever.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAudit trail is automatic.\u003c/strong\u003e Every deployment records who approved it, what commit was deployed, and when. When auditors ask for evidence (and they will), you have it.\u003c/p\u003e\n\u003cp\u003eThis satisfies ISO 27001 Controls A.14.2 (Secure Development) and A.18.2 (Security Reviews). But more importantly, it prevents you from deploying exploitable code to production.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"setting-up-the-approval-gate\"\u003e\u003ca href=\"/posts/continuous-deployment-security-gates/#setting-up-the-approval-gate\" title=\"Setting Up the Approval Gate\"\u003eSetting Up the Approval Gate\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eThat \u003ccode\u003eenvironment: production\u003c/code\u003e line is where the magic happens. Here\u0026rsquo;s how to configure it:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eRepository Settings → Environments → Create \u0026ldquo;production\u0026rdquo;\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAdd protection rules:\u003c/strong\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eRequired reviewers\u003c/strong\u003e: 1-2 people who aren\u0026rsquo;t the deployer\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eDeployment branches\u003c/strong\u003e: \u003ccode\u003emain\u003c/code\u003e only (no deploying random feature branches)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003ePrevent self-review\u003c/strong\u003e: You can\u0026rsquo;t approve your own deployment\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eNow every production deployment requires a second pair of eyes. Someone has to look at the PR, verify the security gates passed, and explicitly click \u0026ldquo;Approve.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eIs this slower than YOLO deploying? Yes, by about 30 seconds. Is it worth avoiding breaches and audit failures? Obviously.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-evidence-trail-for-when-auditors-come-knocking\"\u003e\u003ca href=\"/posts/continuous-deployment-security-gates/#the-evidence-trail-for-when-auditors-come-knocking\" title=\"The Evidence Trail (For When Auditors Come Knocking)\"\u003eThe Evidence Trail (For When Auditors Come Knocking)\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eAuditors don\u0026rsquo;t care about your intentions. They want proof. With this setup, you automatically generate:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eTest execution logs\u003c/strong\u003e: Every workflow run shows what tests ran and whether they passed\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eVulnerability scan results\u003c/strong\u003e: CodeQL findings, dependency audit reports, Dependabot alerts\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eApproval records\u003c/strong\u003e: Who approved each production deployment, when, and for what commit\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eDeployment history\u003c/strong\u003e: Timestamped record of every production push\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eWhen an auditor asks \u0026ldquo;How do you ensure security validation before deployment?\u0026rdquo; you don\u0026rsquo;t explain your process. You show them the GitHub Actions logs. Evidence beats explanation.\u003c/p\u003e\n\u003cp\u003eExport these quarterly. Keep them for at least three years. You\u0026rsquo;ll need them.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"ways-teams-screw-this-up\"\u003e\u003ca href=\"/posts/continuous-deployment-security-gates/#ways-teams-screw-this-up\" title=\"Ways Teams Screw This Up\"\u003eWays Teams Screw This Up\u003c/a\u003e\u003c/h2\u003e\n\n\n\n\n\u003ch3 id=\"the-continue-on-error-trap\"\u003e\u003ca href=\"/posts/continuous-deployment-security-gates/#the-continue-on-error-trap\" title=\"The \u0026ldquo;continue-on-error\u0026rdquo; Trap\"\u003eThe \u0026ldquo;continue-on-error\u0026rdquo; Trap\u003c/a\u003e\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# WRONG: Continuing on error defeats the entire purpose\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eSecurity scan\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003enpm audit\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003econtinue-on-error\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003cstrong\u003eISO violation:\u003c/strong\u003e If the gate can be bypassed, it\u0026rsquo;s not a control. Control A.14.2 requires enforced security measures, not suggestions.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFix:\u003c/strong\u003e Remove \u003ccode\u003econtinue-on-error\u003c/code\u003e. If the scan finds issues, deployment \u003cstrong\u003emust\u003c/strong\u003e halt until remediated.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"accidentally-logging-secrets\"\u003e\u003ca href=\"/posts/continuous-deployment-security-gates/#accidentally-logging-secrets\" title=\"Accidentally Logging Secrets\"\u003eAccidentally Logging Secrets\u003c/a\u003e\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# WRONG: Secrets in environment variables are still visible in logs if misused\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eDeploy\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eenv\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eAPI_KEY\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e${{ secrets.API_KEY }}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eecho \u0026#34;Using key $API_KEY\u0026#34; \u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"c\"\u003e# Logs the secret!\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003cstrong\u003eISO violation:\u003c/strong\u003e Control A.9.4.5 requires protection of authentication credentials. Accidental logging exposes secrets.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFix:\u003c/strong\u003e Never reference secrets in commands that might log them. Use secrets only in secure contexts:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eDeploy\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eenv\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eAPI_KEY\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e${{ secrets.API_KEY }}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e./deploy.sh \u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"c\"\u003e# Script uses $API_KEY internally, never echoed\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\n\n\n\n\u003ch3 id=\"no-rollback-plan\"\u003e\u003ca href=\"/posts/continuous-deployment-security-gates/#no-rollback-plan\" title=\"No Rollback Plan\"\u003eNo Rollback Plan\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eYou deployed. It\u0026rsquo;s broken. Now what? If your answer is \u0026ldquo;panic,\u0026rdquo; you\u0026rsquo;ve already failed.\u003c/p\u003e\n\u003cp\u003eEvery production deployment needs a rollback strategy. Ideally automated—health check fails, previous version gets redeployed. At minimum, documented and tested.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"deploying-from-random-branches\"\u003e\u003ca href=\"/posts/continuous-deployment-security-gates/#deploying-from-random-branches\" title=\"Deploying from Random Branches\"\u003eDeploying from Random Branches\u003c/a\u003e\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# WRONG: Any branch can deploy to production\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eon\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003epush\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ebranches\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;**\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eIf feature branches can deploy to production, your approval gates are meaningless. Someone can push to \u003ccode\u003emy-feature-branch\u003c/code\u003e and bypass every review.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFix:\u003c/strong\u003e Lock production deployments to \u003ccode\u003emain\u003c/code\u003e only. Use GitHub environment branch restrictions to enforce it.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"going-further-because-compliance-is-the-floor-not-the-ceiling\"\u003e\u003ca href=\"/posts/continuous-deployment-security-gates/#going-further-because-compliance-is-the-floor-not-the-ceiling\" title=\"Going Further (Because Compliance Is the Floor, Not the Ceiling)\"\u003eGoing Further (Because Compliance Is the Floor, Not the Ceiling)\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eThe workflow above is the minimum. Here\u0026rsquo;s how to make it actually effective:\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"fail-on-real-thresholds\"\u003e\u003ca href=\"/posts/continuous-deployment-security-gates/#fail-on-real-thresholds\" title=\"Fail on Real Thresholds\"\u003eFail on Real Thresholds\u003c/a\u003e\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eCode coverage:\u003c/strong\u003e 70% minimum. Below that, you\u0026rsquo;re guessing whether code works.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eSAST severity:\u003c/strong\u003e Block on critical and high. Medium can be warnings.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eDependencies:\u003c/strong\u003e Zero critical CVEs (Common Vulnerabilities and Exposures). Period.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eSecrets:\u003c/strong\u003e Any detection = immediate failure. No exceptions.\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch3 id=\"catch-problems-earlier\"\u003e\u003ca href=\"/posts/continuous-deployment-security-gates/#catch-problems-earlier\" title=\"Catch Problems Earlier\"\u003eCatch Problems Earlier\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eSecurity gates at deployment time are good. Security gates at commit time are better:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003ePre-commit hooks for secrets detection (find them before they hit the repo)\u003c/li\u003e\n\u003cli\u003ePR checks for SAST and tests (fail fast, fix fast)\u003c/li\u003e\n\u003cli\u003eDependabot alerts enabled and actually triaged (not just ignored)\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch3 id=\"handle-exceptions-properly\"\u003e\u003ca href=\"/posts/continuous-deployment-security-gates/#handle-exceptions-properly\" title=\"Handle Exceptions Properly\"\u003eHandle Exceptions Properly\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eSometimes you genuinely need to deploy with a known medium-severity issue. That\u0026rsquo;s fine—but document it:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eWritten justification (why this can\u0026rsquo;t wait)\u003c/li\u003e\n\u003cli\u003eApproval from someone who isn\u0026rsquo;t the developer\u003c/li\u003e\n\u003cli\u003eDeadline for remediation (not \u0026ldquo;someday\u0026rdquo;)\u003c/li\u003e\n\u003cli\u003eTracked in your issue system\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe gate stays. You just document why you\u0026rsquo;re accepting the risk.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"stop-making-excuses\"\u003e\u003ca href=\"/posts/continuous-deployment-security-gates/#stop-making-excuses\" title=\"Stop Making Excuses\"\u003eStop Making Excuses\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eThe workflow I showed you isn\u0026rsquo;t complicated. It\u0026rsquo;s maybe 70 lines of YAML. Copy it, adapt it to your stack, enforce it.\u003c/p\u003e\n\u003cp\u003eWhat\u0026rsquo;s hard isn\u0026rsquo;t the technical implementation. What\u0026rsquo;s hard is the discipline to keep the gates closed when someone with a title says \u0026ldquo;we need to ship NOW.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eBut here\u0026rsquo;s what I\u0026rsquo;ve learned from 15 years of watching deployments go wrong: \u003cstrong\u003ethe shortcuts always cost more than the delay\u003c/strong\u003e. The security scan you skipped catches the vulnerability that gets exploited. The approval you bypassed would have noticed the breaking change. The rollback you didn\u0026rsquo;t plan for becomes a 3 AM incident.\u003c/p\u003e\n\u003cp\u003eSecurity gates aren\u0026rsquo;t bureaucracy. They\u0026rsquo;re the engineering discipline that separates professionals from gamblers.\u003c/p\u003e\n\u003cp\u003eImplement them. Enforce them. Stop deploying garbage to production.\u003c/p\u003e\n","date_modified":"2026-05-26T10:22:03+02:00","date_published":"2026-03-26T17:00:00+01:00","id":"https://daily-devops.net/posts/continuous-deployment-security-gates/","language":"en","summary":"Failing tests as warnings, secrets in Git, no approvals. Build GitHub Actions gates that enforce ISO 27001 A.14.2 and A.18.2 before production.\n","tags":["iso-standards","security","github-actions","devops","cicd"],"title":"Stop Deploying Garbage to Production\n","url":"https://daily-devops.net/posts/continuous-deployment-security-gates/"},{"authors":[{"name":"Jendrik Brack","url":"https://daily-devops.net/authors/jendrik/"}],"content_html":"\n\n\n\n\u003ch2 id=\"the-problem-cloud-and-on-prem-as-operational-silos\"\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/#the-problem-cloud-and-on-prem-as-operational-silos\" title=\"The Problem: Cloud and On-Prem as Operational Silos\"\u003eThe Problem: Cloud and On-Prem as Operational Silos\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eMost organizations don\u0026rsquo;t run purely in the cloud. Legacy systems, compliance requirements, data gravity, and latency concerns keep critical workloads on-premises indefinitely. Running AKS in Azure alongside on-prem Kubernetes clusters multiplies management overhead: two separate control planes to patch, two policy frameworks to keep in sync, two identity configurations to audit, and two observability stacks generating alerts nobody wants to correlate manually.\u003c/p\u003e\n\u003cp\u003eThe temptation is to build custom tooling that bridges the gap. That usually ends as a fragile script collection that only one person on the team understands. Azure Arc changes the equation: it extends Azure\u0026rsquo;s management plane to any Kubernetes cluster without migrating workloads.\u003c/p\u003e\n\u003cp\u003eThis article covers the practical pieces: network connectivity options, Azure Arc for unified management, DNS resolution across environment boundaries, policy enforcement, and identity federation.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"connectivity-models-getting-traffic-between-environments\"\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/#connectivity-models-getting-traffic-between-environments\" title=\"Connectivity Models: Getting Traffic Between Environments\"\u003eConnectivity Models: Getting Traffic Between Environments\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eBefore you can manage hybrid Kubernetes deployments, you need reliable network connectivity. Three primary patterns exist, each with distinct trade-offs.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"expressroute-dedicated-private-connectivity\"\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/#expressroute-dedicated-private-connectivity\" title=\"ExpressRoute: Dedicated Private Connectivity\"\u003eExpressRoute: Dedicated Private Connectivity\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003e\u003ca href=\"https://learn.microsoft.com/en-us/azure/expressroute/expressroute-introduction\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eExpressRoute\u003c/a\u003e provides a dedicated, private connection between on-premises and Azure, bypassing the public internet entirely. Latency is predictable, throughput is consistent, and the connection doesn\u0026rsquo;t compete with general internet traffic.\u003c/p\u003e\n\u003cp\u003eThe operational reality: provisioning takes weeks, requires coordination with a connectivity provider, and demands solid Border Gateway Protocol (BGP) knowledge from your network team. Cost is significant. For production workloads with compliance requirements or sustained high-bandwidth data transfer, those trade-offs are usually acceptable. For a dev/test environment or proof-of-concept, they aren\u0026rsquo;t.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"site-to-site-vpn-cost-effective-alternative\"\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/#site-to-site-vpn-cost-effective-alternative\" title=\"Site-to-Site VPN: Cost-Effective Alternative\"\u003eSite-to-Site VPN: Cost-Effective Alternative\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eSite-to-Site (S2S) VPN creates encrypted tunnels over the public internet. Setup takes hours rather than weeks, cost is a fraction of ExpressRoute, and it works without engaging a connectivity provider.\u003c/p\u003e\n\u003cp\u003eThe catch is performance variability. Throughput degrades under load, latency spikes during congestion periods, and encryption overhead adds up. For proof-of-concept environments, dev/test workloads, or bursty low-volume traffic, S2S VPN is the pragmatic choice. For production databases replicating continuously across the boundary, it usually isn\u0026rsquo;t enough.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"vnet-peering-cloud-only-hybrid\"\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/#vnet-peering-cloud-only-hybrid\" title=\"VNet Peering: Cloud-Only Hybrid\"\u003eVNet Peering: Cloud-Only Hybrid\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003e\u003ca href=\"https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-peering-overview\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eVNet peering\u003c/a\u003e connects Azure VNets across regions or subscription boundaries. If both sides run in Azure and you\u0026rsquo;re drawing a line between subscriptions rather than between cloud and datacenter, this is the simplest option: no gateways, no BGP, no provider contracts.\u003c/p\u003e\n\u003cp\u003eIt doesn\u0026rsquo;t solve the on-prem connectivity problem. Peering only works between Azure VNets.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"infrastructure-as-code-expressroute--aks\"\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/#infrastructure-as-code-expressroute--aks\" title=\"Infrastructure as Code: ExpressRoute \u0026#43; AKS\"\u003eInfrastructure as Code: ExpressRoute + AKS\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eWhatever connectivity model you choose, infrastructure repeatability matters from day one. Deploying gateways, subnets, route tables, and AKS clusters manually works once and creates problems on the second environment. The Terraform configuration below covers the full stack: ExpressRoute gateway, private DNS zone, and AKS with Azure CNI and private cluster enabled.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-hcl\" data-lang=\"hcl\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Terraform configuration for ExpressRoute + AKS hybrid connectivity\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Variables and provider configuration assumed\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_resource_group\u0026#34; \u0026#34;hybrid\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e     \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;hybrid-aks-rg\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  location\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;westeurope\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Virtual Network for AKS and hybrid connectivity\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_virtual_network\u0026#34; \u0026#34;aks_vnet\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;aks-vnet\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  address_space\u003c/span\u003e       \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;10.1.0.0/16\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  location\u003c/span\u003e            \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ehybrid\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003elocation\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ehybrid\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Subnet for AKS nodes\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_subnet\u0026#34; \u0026#34;aks_nodes\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                 \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;aks-nodes-subnet\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e  \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ehybrid\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  virtual_network_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_virtual_network\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eaks_vnet\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  address_prefixes\u003c/span\u003e     \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;10.1.1.0/24\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Gateway subnet for ExpressRoute\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_subnet\u0026#34; \u0026#34;gateway\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                 \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;GatewaySubnet\u0026#34;\u003c/span\u003e\u003cspan class=\"c1\"\u003e  # Name must be exactly this\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e  \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ehybrid\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  virtual_network_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_virtual_network\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eaks_vnet\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  address_prefixes\u003c/span\u003e     \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;10.1.255.0/27\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Public IP for ExpressRoute Gateway\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_public_ip\u0026#34; \u0026#34;er_gateway_ip\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;er-gateway-pip\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  location\u003c/span\u003e            \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ehybrid\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003elocation\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ehybrid\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  allocation_method\u003c/span\u003e   \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Static\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  sku\u003c/span\u003e                 \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Standard\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# ExpressRoute Gateway\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_virtual_network_gateway\u0026#34; \u0026#34;er_gateway\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;er-gateway\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  location\u003c/span\u003e            \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ehybrid\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003elocation\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ehybrid\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  type\u003c/span\u003e                \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;ExpressRoute\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  sku\u003c/span\u003e                 \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Standard\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003eip_configuration\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    name\u003c/span\u003e                          \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;gateway-ip-config\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    public_ip_address_id\u003c/span\u003e          \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_public_ip\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eer_gateway_ip\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    private_ip_address_allocation\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Dynamic\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    subnet_id\u003c/span\u003e                     \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_subnet\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003egateway\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Connection from ExpressRoute Gateway to the pre-provisioned circuit\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Configure var.expressroute_circuit_id with your existing circuit resource ID:\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# var.expressroute_circuit_id = \u0026#34;/subscriptions/.../expressRouteCircuits/...\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_virtual_network_gateway_connection\u0026#34; \u0026#34;onprem\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                       \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;er-onprem-connection\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  location\u003c/span\u003e                   \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ehybrid\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003elocation\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e        \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ehybrid\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  type\u003c/span\u003e                       \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;ExpressRoute\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  virtual_network_gateway_id\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_virtual_network_gateway\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eer_gateway\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  express_route_circuit_id\u003c/span\u003e   \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003evar\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eexpressroute_circuit_id\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Private DNS Zone for internal services\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_private_dns_zone\u0026#34; \u0026#34;internal\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;internal.azure.local\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ehybrid\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Link DNS zone to VNet\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_private_dns_zone_virtual_network_link\u0026#34; \u0026#34;aks\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                  \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;aks-vnet-link\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e   \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ehybrid\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  private_dns_zone_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_private_dns_zone\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003einternal\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  virtual_network_id\u003c/span\u003e    \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_virtual_network\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eaks_vnet\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  registration_enabled\u003c/span\u003e  \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kt\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# AKS cluster with ExpressRoute connectivity\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_kubernetes_cluster\u0026#34; \u0026#34;aks\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;hybrid-aks\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  location\u003c/span\u003e            \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ehybrid\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003elocation\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ehybrid\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  dns_prefix\u003c/span\u003e          \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;hybrid-aks\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  kubernetes_version\u003c/span\u003e  \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;1.31\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003edefault_node_pool\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    name\u003c/span\u003e                \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;system\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    node_count\u003c/span\u003e          \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"m\"\u003e3\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    vm_size\u003c/span\u003e             \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Standard_D4s_v5\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    vnet_subnet_id\u003c/span\u003e      \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_subnet\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eaks_nodes\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    auto_scaling_enabled\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kt\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    min_count\u003c/span\u003e           \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"m\"\u003e3\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    max_count\u003c/span\u003e           \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"m\"\u003e10\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003eidentity\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    type\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;SystemAssigned\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003enetwork_profile\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    network_plugin\u003c/span\u003e     \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azure\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    network_policy\u003c/span\u003e     \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;calico\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    service_cidr\u003c/span\u003e       \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;10.2.0.0/16\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    dns_service_ip\u003c/span\u003e     \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;10.2.0.10\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    load_balancer_sku\u003c/span\u003e  \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;standard\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  private_cluster_enabled\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kt\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  depends_on\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003eazurerm_virtual_network_gateway\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eer_gateway\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Route table for on-prem traffic via ExpressRoute\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_route_table\u0026#34; \u0026#34;onprem_routes\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;onprem-routes\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  location\u003c/span\u003e            \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ehybrid\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003elocation\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ehybrid\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003eroute\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    name\u003c/span\u003e                   \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;to-onprem-datacenter\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    address_prefix\u003c/span\u003e         \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;10.0.0.0/8\u0026#34;\u003c/span\u003e\u003cspan class=\"c1\"\u003e  # On-prem network range\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    next_hop_type\u003c/span\u003e          \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;VirtualNetworkGateway\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Associate route table with AKS subnet\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_subnet_route_table_association\u0026#34; \u0026#34;aks_routes\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  subnet_id\u003c/span\u003e      \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_subnet\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eaks_nodes\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  route_table_id\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_route_table\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eonprem_routes\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis Terraform configuration establishes the foundation for hybrid connectivity: ExpressRoute gateway, private DNS, and AKS with network policies. Customize address ranges, SKUs, and routing rules for your environment.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"azure-arc-unified-kubernetes-management\"\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/#azure-arc-unified-kubernetes-management\" title=\"Azure Arc: Unified Kubernetes Management\"\u003eAzure Arc: Unified Kubernetes Management\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eAzure Arc extends Azure management to any Kubernetes cluster: on-prem, edge locations, or other clouds. It registers external clusters as Azure resources, enabling centralized management without forcing workload migration.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"what-arc-provides\"\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/#what-arc-provides\" title=\"What Arc Provides\"\u003eWhat Arc Provides\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eArc-enabled Kubernetes clusters gain:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eUnified inventory\u003c/strong\u003e: View all clusters in Azure Resource Manager\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003ePolicy enforcement\u003c/strong\u003e: Azure Policy extends to Arc clusters\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eGitOps deployment\u003c/strong\u003e: Flux configurations apply consistently\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eMonitoring integration\u003c/strong\u003e: Azure Monitor collects metrics and logs\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eRBAC integration\u003c/strong\u003e: Azure AD for cluster authentication\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eArc doesn\u0026rsquo;t move workloads to Azure. It extends Azure\u0026rsquo;s control plane to wherever your clusters run.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"onboarding-an-on-prem-cluster\"\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/#onboarding-an-on-prem-cluster\" title=\"Onboarding an On-Prem Cluster\"\u003eOnboarding an On-Prem Cluster\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eConnecting an existing Kubernetes cluster to Arc requires cluster admin access and network connectivity to Azure endpoints.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"cp\"\u003e#!/bin/bash\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Azure Arc onboarding script\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Requires: Azure CLI, kubectl, cluster admin kubeconfig\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eRESOURCE_GROUP\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;hybrid-infra-rg\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eCLUSTER_NAME\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;onprem-k8s-01\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eLOCATION\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;westeurope\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Login and set subscription\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz login\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz account \u003cspan class=\"nb\"\u003eset\u003c/span\u003e --subscription \u003cspan class=\"s2\"\u003e\u0026#34;your-subscription-id\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Create resource group if needed\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz group create --name \u003cspan class=\"nv\"\u003e$RESOURCE_GROUP\u003c/span\u003e --location \u003cspan class=\"nv\"\u003e$LOCATION\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Register Arc providers\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz provider register --namespace Microsoft.Kubernetes\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz provider register --namespace Microsoft.KubernetesConfiguration\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz provider register --namespace Microsoft.ExtendedLocation\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Wait for registration (can take several minutes)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz provider show -n Microsoft.Kubernetes -o table\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz provider show -n Microsoft.KubernetesConfiguration -o table\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Install Arc extensions\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz extension add --name connectedk8s\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz extension add --name k8s-configuration\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Connect cluster to Arc\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz connectedk8s connect \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --name \u003cspan class=\"nv\"\u003e$CLUSTER_NAME\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --resource-group \u003cspan class=\"nv\"\u003e$RESOURCE_GROUP\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --location \u003cspan class=\"nv\"\u003e$LOCATION\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --tags \u003cspan class=\"nv\"\u003eenvironment\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003eproduction \u003cspan class=\"nv\"\u003edatacenter\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003eonprem\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Verify connection\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz connectedk8s show \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --name \u003cspan class=\"nv\"\u003e$CLUSTER_NAME\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --resource-group \u003cspan class=\"nv\"\u003e$RESOURCE_GROUP\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --query \u003cspan class=\"s2\"\u003e\u0026#34;connectivityStatus\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eOnce connected, the cluster appears in the Azure portal alongside AKS clusters. Management operations (viewing workloads, applying policies, deploying via GitOps) work identically.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"policy-enforcement-across-environments\"\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/#policy-enforcement-across-environments\" title=\"Policy Enforcement Across Environments\"\u003ePolicy Enforcement Across Environments\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eAzure Policy for Kubernetes applies consistent governance rules across AKS and Arc clusters. Define policies once, enforce everywhere.\u003c/p\u003e\n\u003cp\u003eExample policy: require resource limits on all pods.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# pod-resource-limits-policy.yaml\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003econstraints.gatekeeper.sh/v1beta1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eK8sRequiredResources\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003erequire-pod-resource-limits\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ematch\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ekinds\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003eapiGroups\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ekinds\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Pod\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003enamespaces\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"l\"\u003eproduction\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"l\"\u003estaging\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eparameters\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003elimits\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"l\"\u003ecpu\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"l\"\u003ememory\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003erequests\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"l\"\u003ecpu\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"l\"\u003ememory\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eApply this policy through Azure Policy, and it enforces on both AKS and Arc-connected on-prem clusters. No duplicated configuration, no drift between environments.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"gitops-single-source-of-truth\"\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/#gitops-single-source-of-truth\" title=\"GitOps: Single Source of Truth\"\u003eGitOps: Single Source of Truth\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eArc supports Flux-based GitOps configurations. Define cluster state in Git, and Arc ensures compliance across environments. The \u003ccode\u003eaz k8s-configuration flux create\u003c/code\u003e command links your Git repository to both AKS and Arc clusters. Changes sync automatically. Configuration drift gets corrected within minutes.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"dns-and-service-discovery-hybrid-resolution-without-complexity\"\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/#dns-and-service-discovery-hybrid-resolution-without-complexity\" title=\"DNS and Service Discovery: Hybrid Resolution Without Complexity\"\u003eDNS and Service Discovery: Hybrid Resolution Without Complexity\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eHybrid deployments need service discovery across boundaries. Pods in AKS must resolve on-prem services, and vice versa.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"approach-1-azure-private-dns-with-conditional-forwarding\"\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/#approach-1-azure-private-dns-with-conditional-forwarding\" title=\"Approach 1: Azure Private DNS with Conditional Forwarding\"\u003eApproach 1: Azure Private DNS with Conditional Forwarding\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eCreate a Private DNS zone in Azure, link it to your VNet, and configure on-prem DNS servers to forward queries for Azure domains to Azure\u0026rsquo;s DNS resolver at 168.63.129.16. AKS clusters inherit VNet DNS configuration automatically. On-prem services get custom DNS entries pointing to ExpressRoute or VPN endpoints.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"approach-2-coredns-custom-forwarding\"\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/#approach-2-coredns-custom-forwarding\" title=\"Approach 2: CoreDNS Custom Forwarding\"\u003eApproach 2: CoreDNS Custom Forwarding\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eFor cluster-level control, patch the CoreDNS ConfigMap to forward specific domain queries to on-prem DNS servers. This is the right approach when on-prem services use a domain suffix that doesn\u0026rsquo;t overlap with Azure Private DNS zones, or when you need different forwarding behavior per cluster.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# CoreDNS custom configmap - forward internal corporate domain to on-prem resolver\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ev1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eConfigMap\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ecoredns-custom\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003enamespace\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ekube-system\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003edata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ecorp.server\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    corp.example.com:53 {\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e        errors\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e        cache 30\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e        forward . 10.0.0.53 {\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            prefer_udp\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e        }\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    }\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eApply with \u003ccode\u003ekubectl apply -f coredns-custom.yaml\u003c/code\u003e. AKS detects the \u003ccode\u003ecoredns-custom\u003c/code\u003e ConfigMap automatically. For the reverse path, configure on-prem DNS to forward \u003ccode\u003e*.privatelink.blob.core.windows.net\u003c/code\u003e and similar zones to Azure\u0026rsquo;s virtual resolver at \u003ccode\u003e168.63.129.16\u003c/code\u003e.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthor note:\u003c/strong\u003e DNS is usually where hybrid setups produce the most subtle and hardest-to-debug failures. A pod resolves a name correctly in testing, then silently times out in production because the CoreDNS cache held a stale entry across a VPN reconnect. Keep TTLs short for cross-boundary records and verify the full resolver chain with \u003ccode\u003enslookup\u003c/code\u003e from inside the cluster, not just from a workstation.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eKey principle:\u003c/strong\u003e Avoid split-horizon DNS designs where the same name resolves differently depending on source location. Use Azure Private DNS as the primary zone authority where possible, and fall back to conditional forwarding only for domains you don\u0026rsquo;t control.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"identity-across-boundaries-federation-without-duplication\"\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/#identity-across-boundaries-federation-without-duplication\" title=\"Identity Across Boundaries: Federation Without Duplication\"\u003eIdentity Across Boundaries: Federation Without Duplication\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eHybrid deployments shouldn\u0026rsquo;t duplicate identity systems. Azure AD (now Microsoft Entra ID) integration extends to Arc clusters, providing centralized authentication and significantly reducing the number of credential systems to maintain.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"service-principals-for-cross-environment-access\"\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/#service-principals-for-cross-environment-access\" title=\"Service Principals for Cross-Environment Access\"\u003eService Principals for Cross-Environment Access\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eApplications running on-prem that need access to Azure services (Key Vault, storage accounts, managed databases) can use Azure AD service principals with certificate-based authentication. Create a service principal, assign the appropriate role, and mount the certificate as a Kubernetes secret in the on-prem pod.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Create service principal and assign Key Vault access\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz ad sp create-for-rbac \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --name \u003cspan class=\"s2\"\u003e\u0026#34;onprem-app-sp\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --role \u003cspan class=\"s2\"\u003e\u0026#34;Key Vault Secrets User\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --scopes \u003cspan class=\"s2\"\u003e\u0026#34;/subscriptions/\u0026lt;sub-id\u0026gt;/resourceGroups/\u0026lt;rg\u0026gt;/providers/Microsoft.KeyVault/vaults/\u0026lt;vault\u0026gt;\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis works reliably, but carries ongoing maintenance: certificate rotation, secret distribution across on-prem clusters, and audit trails that span two systems. For new workloads, federated credentials are worth the initial setup complexity.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"federated-credentials-for-workload-identity\"\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/#federated-credentials-for-workload-identity\" title=\"Federated Credentials for Workload Identity\"\u003eFederated Credentials for Workload Identity\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003e\u003ca href=\"https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eWorkload identity federation\u003c/a\u003e allows on-prem Kubernetes service accounts to authenticate as Azure AD identities without long-lived secrets. The on-prem cluster\u0026rsquo;s OIDC issuer endpoint issues tokens for service accounts; Azure AD trusts that issuer and exchanges the projected token for an Azure AD access token.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Register the on-prem cluster\u0026#39;s OIDC issuer with an Azure AD app registration\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz ad app federated-credential create \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --id \u0026lt;app-registration-id\u0026gt; \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --parameters \u003cspan class=\"s1\"\u003e\u0026#39;{\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s1\"\u003e    \u0026#34;name\u0026#34;: \u0026#34;onprem-k8s-workload\u0026#34;,\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s1\"\u003e    \u0026#34;issuer\u0026#34;: \u0026#34;https://\u0026lt;your-onprem-oidc-issuer\u0026gt;\u0026#34;,\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s1\"\u003e    \u0026#34;subject\u0026#34;: \u0026#34;system:serviceaccount:production:my-app\u0026#34;,\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s1\"\u003e    \u0026#34;audiences\u0026#34;: [\u0026#34;api://AzureADTokenExchange\u0026#34;]\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s1\"\u003e  }\u0026#39;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe on-prem cluster needs to expose its OIDC discovery document at a publicly reachable (or Azure-reachable) endpoint. That\u0026rsquo;s the step that most commonly blocks initial setup. Verify the discovery document is accessible before spending time debugging token exchange errors.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthor note:\u003c/strong\u003e Migrating workloads from service principal secrets to federated credentials removes certificate rotation as a recurring task entirely. Secret sprawl across on-prem clusters was one of the more uncomfortable findings in the security reviews I\u0026rsquo;ve participated in. Federated credentials make the problem structurally impossible rather than just less likely.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"operational-consistency-making-hybrid-work-long-term\"\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/#operational-consistency-making-hybrid-work-long-term\" title=\"Operational Consistency: Making Hybrid Work Long-Term\"\u003eOperational Consistency: Making Hybrid Work Long-Term\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eHybrid deployments fail when operational practices diverge between environments. Consistency requires deliberate effort.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"monitoring-and-observability\"\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/#monitoring-and-observability\" title=\"Monitoring and Observability\"\u003eMonitoring and Observability\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eUse \u003ca href=\"https://learn.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-overview\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eAzure Monitor Container Insights\u003c/a\u003e for both AKS and Arc clusters. Install the extension on Arc-connected clusters explicitly (AKS picks it up automatically with the add-on flag):\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz k8s-extension create \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --name azuremonitor-containers \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --cluster-type connectedClusters \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --cluster-name onprem-k8s-01 \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --resource-group hybrid-infra-rg \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --extension-type Microsoft.AzureMonitor.Containers \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --configuration-settings \u003cspan class=\"nv\"\u003elogAnalyticsWorkspaceResourceID\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u0026lt;workspace-resource-id\u0026gt;\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eMetrics, logs, and cluster health flow to a single Log Analytics workspace regardless of where the cluster runs. A simple Kusto Query Language (KQL) query surfaces pod restart counts across all environments at once:\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode class=\"language-kql\" data-lang=\"kql\"\u003eKubePodInventory\n| where TimeGenerated \u0026gt; ago(24h)\n| summarize Restarts=sum(ContainerRestartCount) by ClusterName, Namespace\n| order by Restarts desc\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eHaving AKS and on-prem clusters reporting to the same workspace makes cross-environment incident correlation significantly faster.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"update-management\"\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/#update-management\" title=\"Update Management\"\u003eUpdate Management\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003e\u003ca href=\"https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/agent-upgrade\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eAzure Arc cluster autoupgrade\u003c/a\u003e reduces the operational gap between AKS (where upgrades are automated and well-understood) and self-managed on-prem clusters (where upgrades have historically been postponed due to complexity). You can define upgrade channels, schedule maintenance windows, and receive notifications through the same Azure portal used for AKS fleet management.\u003c/p\u003e\n\u003cp\u003eThis doesn\u0026rsquo;t eliminate the need for upgrade validation in staging environments. But it removes the operational friction that leads to on-prem clusters running three minor versions behind production AKS.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"cost-and-resource-tracking\"\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/#cost-and-resource-tracking\" title=\"Cost and Resource Tracking\"\u003eCost and Resource Tracking\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eArc-enabled clusters report resource utilization to Azure. Tag clusters consistently with environment, cost-center, and region labels using \u003ccode\u003eaz connectedk8s update\u003c/code\u003e. Use Azure Cost Management to track total Kubernetes spend across cloud and on-prem, enabling accurate chargeback and budget planning.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"key-takeaways\"\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/#key-takeaways\" title=\"Key Takeaways\"\u003eKey Takeaways\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eHybrid AKS deployments succeed when you:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eChoose the right connectivity\u003c/strong\u003e: ExpressRoute for production, S2S VPN for dev/test, VNet peering for Azure-only scenarios\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eUse Azure Arc for unified management\u003c/strong\u003e: Extend Azure\u0026rsquo;s control plane rather than building parallel tooling\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eEnforce policies consistently\u003c/strong\u003e: Azure Policy + GitOps eliminate configuration drift\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eSimplify DNS\u003c/strong\u003e: Azure Private DNS with conditional forwarding avoids complexity\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eFederate identity\u003c/strong\u003e: Azure AD integration reduces secret sprawl and management overhead\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eMonitor everything in one place\u003c/strong\u003e: Azure Monitor provides visibility across environments\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eHybrid infrastructure doesn\u0026rsquo;t have to mean duplicated effort. Arc, proper networking, and consistent operational practices make multi-environment Kubernetes manageable.\u003c/p\u003e\n\u003cp\u003eThe goal isn\u0026rsquo;t cloud purity. It\u0026rsquo;s operational efficiency wherever your workloads run.\u003c/p\u003e\n","date_modified":"2026-05-26T10:22:03+02:00","date_published":"2026-03-25T17:00:00+01:00","id":"https://daily-devops.net/posts/hybrid-aks-on-prem-azure-arc/","language":"en","summary":"Practical patterns for connecting AKS to on-prem: ExpressRoute, VPN connectivity, Azure Arc management, DNS resolution, and identity federation.","tags":["hybrid","azure","kubernetes","cloud","devops","onprem","infrastructure"],"title":"Hybrid AKS: Bridging Cloud and On-Prem with Azure Arc","url":"https://daily-devops.net/posts/hybrid-aks-on-prem-azure-arc/"},{"authors":[{"name":"Jendrik Brack","url":"https://daily-devops.net/authors/jendrik/"}],"content_html":"\u003cp\u003eYour cluster will fail. The question is not if, but when, and whether you can recover before customers notice. Most organizations discover their backup strategy does not work during an actual outage, when recovery time matters most and manual heroics cannot save you.\u003c/p\u003e\n\u003cp\u003eIf you run Azure Kubernetes Service (AKS) in production, you need a recovery plan that engineers can execute half asleep at 2 AM. We will go through what to back up, how Velero works in day-to-day operations, when Azure Backup for AKS is enough, and how to design realistic failover with measurable Recovery Time Objective (RTO) and Recovery Point Objective (RPO).\u003c/p\u003e\n\u003cp\u003eThe goal is simple: repeatable recovery procedures you have already tested, not a document that looks good in Confluence but fails during an incident.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-problem-untested-recovery-fails-when-it-matters\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#the-problem-untested-recovery-fails-when-it-matters\" title=\"The problem: Untested recovery fails when it matters\"\u003eThe problem: Untested recovery fails when it matters\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eEvery Kubernetes cluster accumulates state that must survive failures. Application data lives in persistent volumes. Cluster configuration exists in custom resource definitions. Workload definitions sit in YAML manifests scattered across repositories. Identity mappings, secrets, network policies, and RBAC rules define how services authenticate and communicate. Losing any of these components means downtime, data loss, and manual reconstruction under time pressure.\u003c/p\u003e\n\u003cp\u003eThe real risk is not having a backup strategy. The real risk is discovering your backup strategy does not work during an actual incident, when recovery time directly determines customer impact and business cost.\u003c/p\u003e\n\u003cp\u003eOperational reality: Most teams test backup creation but never test restoration. A backup you have never restored is a backup that will fail when you need it. Recovery procedures that require manual steps will fail during high-pressure incidents when engineers make mistakes and documentation is incomplete.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"what-needs-backup-understanding-cluster-state\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#what-needs-backup-understanding-cluster-state\" title=\"What needs backup: Understanding cluster state\"\u003eWhat needs backup: Understanding cluster state\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eKubernetes clusters contain multiple layers of state that require different backup approaches.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"application-data-persistent-volumes\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#application-data-persistent-volumes\" title=\"Application data: Persistent volumes\"\u003eApplication data: Persistent volumes\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003ePersistent volumes hold databases, file storage, configuration data, and application state. Losing persistent volume data typically means permanent data loss unless you maintain application-level replication or external backups. Azure Disks and Azure Files both support snapshot-based backup, but snapshots alone do not capture the Kubernetes metadata required to restore volumes to the correct pods in the correct namespaces.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"cluster-configuration-custom-resources-and-crds\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#cluster-configuration-custom-resources-and-crds\" title=\"Cluster configuration: Custom resources and CRDs\"\u003eCluster configuration: Custom resources and CRDs\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eCustom Resource Definitions extend Kubernetes with domain-specific objects. Operators, service meshes, monitoring stacks, and policy engines all define Custom Resource Definitions (CRDs) that control cluster behavior. Losing CRDs means losing the schema and logic that your cluster depends on. Restoring CRDs without the corresponding custom resource objects leaves your cluster in an inconsistent state.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"application-definitions-workload-manifests\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#application-definitions-workload-manifests\" title=\"Application definitions: Workload manifests\"\u003eApplication definitions: Workload manifests\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eDeployments, StatefulSets, Services, ConfigMaps, and Secrets define what runs in your cluster. Most teams store these manifests in Git, but cluster state drifts from Git over time due to manual changes, automated rollouts, and operator modifications. Restoring from Git alone may not reflect actual production state.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"identity-and-access-rbac-and-service-accounts\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#identity-and-access-rbac-and-service-accounts\" title=\"Identity and access: RBAC and service accounts\"\u003eIdentity and access: RBAC and service accounts\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eRole-based access control, ServiceAccounts, and Azure AD integration define who can access what resources. Losing role-based access control (RBAC) configuration means losing security boundaries and breaking automated workflows that depend on specific service account permissions.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"network-configuration-policies-and-ingress-rules\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#network-configuration-policies-and-ingress-rules\" title=\"Network configuration: Policies and ingress rules\"\u003eNetwork configuration: Policies and ingress rules\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eNetwork policies, ingress controllers, and DNS mappings control how traffic flows into and within your cluster. Restoring workloads without restoring network configuration results in unreachable services and broken traffic routing.\u003c/p\u003e\n\u003cp\u003eA complete backup strategy captures all of these layers and validates that restoration procedures actually work.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"velero-production-backup-workflows\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#velero-production-backup-workflows\" title=\"Velero: Production backup workflows\"\u003eVelero: Production backup workflows\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eVelero is the de facto standard for Kubernetes backup and restore. It runs as a controller inside your cluster, captures cluster state and persistent volume snapshots, and stores backups in object storage.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"how-velero-works\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#how-velero-works\" title=\"How Velero works\"\u003eHow Velero works\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eVelero operates in two phases: backup and restore. During backup, Velero queries the Kubernetes API for resources matching your backup selectors, serializes those resources to JSON, and uploads the result to cloud object storage (Azure Blob Storage for AKS). For persistent volumes, Velero triggers volume snapshots using Azure Disk snapshots or uses Restic to perform file-level backups.\u003c/p\u003e\n\u003cp\u003eDuring restore, Velero downloads the backup manifest, applies resources to the target cluster, and restores persistent volume data from snapshots or Restic archives. Velero handles dependency ordering and namespace mapping automatically.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"backup-scheduling-and-retention\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#backup-scheduling-and-retention\" title=\"Backup scheduling and retention\"\u003eBackup scheduling and retention\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eProduction backup strategies require automated scheduling and retention policies. Velero supports cron-based schedules and configurable retention windows.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# Velero backup schedule - Helm values\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003evelero\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eschedules\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003edaily\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"c\"\u003e# Run full backup daily at 2 AM UTC\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003eschedule\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;0 2 * * *\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003etemplate\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ettl\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;720h\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"c\"\u003e# Retain backups for 30 days\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eincludedNamespaces\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e- \u003cspan class=\"l\"\u003eproduction\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e- \u003cspan class=\"l\"\u003estaging\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003esnapshotVolumes\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ehourly-critical\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"c\"\u003e# Run hourly backup for critical namespaces\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003eschedule\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;0 * * * *\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003etemplate\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ettl\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;168h\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"c\"\u003e# Retain backups for 7 days\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eincludedNamespaces\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e- \u003cspan class=\"l\"\u003eproduction\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003elabelSelector\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003ematchLabels\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"nt\"\u003ebackup-frequency\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ehourly\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003esnapshotVolumes\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eFor many teams, this minimal Terraform baseline is easier to maintain than a large, custom module. It creates the storage account and container Velero needs.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-hcl\" data-lang=\"hcl\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_storage_account\u0026#34; \u0026#34;velero\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                     \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;velerobackup${var.environment}\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e      \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003evar\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eresource_group_name\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  location\u003c/span\u003e                 \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003evar\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003elocation\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  account_tier\u003c/span\u003e             \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Standard\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  account_replication_type\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;GRS\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_storage_container\u0026#34; \u0026#34;velero\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                  \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;velero\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  storage_account_name\u003c/span\u003e  \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_storage_account\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003evelero\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  container_access_type\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;private\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThen install Velero with Helm and pass only four required values: provider (\u003ccode\u003eazure\u003c/code\u003e), storage account name, blob container name, and resource group. Keep advanced tuning for later once backups and restores are stable.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"testing-restore-procedures\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#testing-restore-procedures\" title=\"Testing restore procedures\"\u003eTesting restore procedures\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eBackup creation means nothing without verified restore capability. Production-grade DR requires regular restore testing in isolated environments.\u003c/p\u003e\n\u003cp\u003eRestore testing workflow:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eCreate a test AKS cluster in a separate resource group\u003c/li\u003e\n\u003cli\u003eInstall Velero with access to production backup storage\u003c/li\u003e\n\u003cli\u003eExecute restore operation for a representative namespace\u003c/li\u003e\n\u003cli\u003eValidate application functionality and data integrity\u003c/li\u003e\n\u003cli\u003eDocument restoration time and any issues encountered\u003c/li\u003e\n\u003cli\u003eDestroy test cluster\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eRun this workflow monthly at minimum. Quarterly is too infrequent because configuration drift and Velero version updates will cause surprises. Teams that skip restore testing discover broken procedures during actual outages.\u003c/p\u003e\n\u003cp\u003eCommon restore failures: Missing CRDs (restore CRDs before custom resources), incorrect namespace mappings (use Velero namespace mapping features), persistent volume availability zones (Azure Disks are zone-locked), and missing secrets (external secret management requires separate backup).\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"azure-native-backup-when-to-use-it\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#azure-native-backup-when-to-use-it\" title=\"Azure native backup: When to use it\"\u003eAzure native backup: When to use it\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eAzure Backup for AKS launched in 2023 and provides Azure-native cluster backup without deploying Velero. It integrates with Azure Backup vaults and uses the same portal experience as VM and database backups.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"azure-backup-vs-velero\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#azure-backup-vs-velero\" title=\"Azure Backup vs Velero\"\u003eAzure Backup vs Velero\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eAzure Backup works well for organizations heavily invested in Azure tooling who want unified backup management across all Azure resources. It handles backup scheduling, retention, and monitoring through familiar Azure interfaces.\u003c/p\u003e\n\u003cp\u003eLimitations compared to Velero: Less flexibility in backup selectors and namespace filtering, fewer options for cross-region backup replication, and vendor lock-in to Azure. Velero supports multi-cloud scenarios and offers more granular control over what gets backed up.\u003c/p\u003e\n\u003cp\u003eRecommendation: Use Azure Backup if your organization already standardizes on Azure Backup for other resources and you do not require multi-cloud portability. Use Velero if you need maximum flexibility, cross-region replication control, or multi-cloud backup capability.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"multi-region-failover-designing-for-actual-recovery\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#multi-region-failover-designing-for-actual-recovery\" title=\"Multi-region failover: Designing for actual recovery\"\u003eMulti-region failover: Designing for actual recovery\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eSingle-region deployments create single points of failure. Multi-region architectures provide genuine disaster recovery capability but introduce complexity in state synchronization, traffic routing, and recovery orchestration.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"failover-architecture-patterns\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#failover-architecture-patterns\" title=\"Failover architecture patterns\"\u003eFailover architecture patterns\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eActive-passive:\u003c/strong\u003e Primary region handles all traffic. Secondary region remains idle but receives regular backup replication. During failover, you restore backups to the secondary cluster and redirect traffic. Recovery time depends on backup restore speed and DNS propagation.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eActive-active:\u003c/strong\u003e Both regions handle production traffic simultaneously. Application state synchronizes continuously (database replication, event streaming, or shared storage). During regional failure, traffic shifts to the remaining region. Recovery time depends on health check detection and DNS/load balancer failover speed.\u003c/p\u003e\n\u003cp\u003eActive-passive costs less but requires longer recovery time. Active-active provides faster failover but doubles infrastructure cost and requires application-level state synchronization.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"dns-failover-automation\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#dns-failover-automation\" title=\"DNS failover automation\"\u003eDNS failover automation\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eDNS-based failover redirects traffic between regions by updating DNS records to point at healthy endpoints. Azure Traffic Manager and Azure Front Door both provide automatic failover based on health probes.\u003c/p\u003e\n\u003cp\u003eUse a small script first, then expand it over time. This keeps incident handling understandable for on-call engineers.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"cp\"\u003e#!/usr/bin/env bash\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eset\u003c/span\u003e -euo pipefail\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eSECONDARY_RG\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;rg-aks-westus\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eSECONDARY_CLUSTER\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;aks-dr-westus\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eTM_RG\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;rg-networking\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eTM_PROFILE\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;tm-aks-prod\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;1) Connect to secondary cluster\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz aks get-credentials -g \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$SECONDARY_RG\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e -n \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$SECONDARY_CLUSTER\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e --overwrite-existing\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl cluster-info\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;2) Trigger restore from latest Velero backup\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003evelero restore create dr-\u003cspan class=\"k\"\u003e$(\u003c/span\u003edate +%Y%m%d-%H%M\u003cspan class=\"k\"\u003e)\u003c/span\u003e --from-backup \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"k\"\u003e$(\u003c/span\u003evelero backup get -o name \u003cspan class=\"p\"\u003e|\u003c/span\u003e tail -n1 \u003cspan class=\"p\"\u003e|\u003c/span\u003e cut -d/ -f2\u003cspan class=\"k\"\u003e)\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;3) Switch Traffic Manager endpoint\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz network traffic-manager endpoint update --resource-group \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$TM_RG\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e --profile-name \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$TM_PROFILE\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e --name endpoint-eastus --type azureEndpoints --endpoint-status Disabled\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz network traffic-manager endpoint update --resource-group \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$TM_RG\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e --profile-name \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$TM_PROFILE\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e --name endpoint-westus --type azureEndpoints --endpoint-status Enabled\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis script is intentionally small. Add pre-checks and post-checks later, but start with a version every engineer can understand quickly during an outage.\u003c/p\u003e\n\u003cp\u003eThis script automates critical failover steps but requires human verification at each stage. Fully automated failover without human approval risks unnecessary region switches during transient failures.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"state-synchronization-strategies\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#state-synchronization-strategies\" title=\"State synchronization strategies\"\u003eState synchronization strategies\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eMulti-region architectures require careful state management. Databases need replication (Azure SQL geo-replication, Cosmos DB multi-region writes). Object storage needs cross-region replication (Azure Blob Storage GRS). Message queues require either regional isolation or cross-region synchronization (Azure Service Bus premium tier supports geo-replication).\u003c/p\u003e\n\u003cp\u003eStateless services fail over easily. Stateful services require replication strategy planning during design phase, not during incident response.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"rto-and-rpo-calculating-realistic-targets\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#rto-and-rpo-calculating-realistic-targets\" title=\"RTO and RPO: Calculating realistic targets\"\u003eRTO and RPO: Calculating realistic targets\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eRecovery Time Objective (RTO) measures how long systems can be down before business impact becomes unacceptable. Recovery Point Objective (RPO) measures how much data loss is acceptable.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"calculating-rto\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#calculating-rto\" title=\"Calculating RTO\"\u003eCalculating RTO\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eRTO includes: detection time (how long until you know there is a problem), decision time (how long to decide failover is necessary), restore time (how long to restore from backup or switch regions), and validation time (how long to confirm restoration worked).\u003c/p\u003e\n\u003cp\u003eExample calculation:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eDetection: 5 minutes (health check interval)\u003c/li\u003e\n\u003cli\u003eDecision: 10 minutes (incident escalation and approval)\u003c/li\u003e\n\u003cli\u003eRestore: 45 minutes (Velero restore for 500GB cluster)\u003c/li\u003e\n\u003cli\u003eValidation: 15 minutes (smoke tests and traffic verification)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eTotal RTO: 75 minutes\u003c/strong\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eIf business requirements demand 30-minute RTO, your current backup-based approach will not meet SLOs. You need active-active architecture or pre-warmed standby clusters.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"calculating-rpo\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#calculating-rpo\" title=\"Calculating RPO\"\u003eCalculating RPO\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eRPO depends on backup frequency. Hourly backups mean up to 60 minutes of data loss. If your application cannot tolerate 60 minutes of data loss, you need more frequent backups or continuous replication.\u003c/p\u003e\n\u003cp\u003eExample calculation:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eBackup frequency: Every 4 hours\u003c/li\u003e\n\u003cli\u003eLast backup: 2 hours ago\u003c/li\u003e\n\u003cli\u003eRegional failure occurs now\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eData loss: 2 hours\u003c/strong\u003e (time since last backup)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eIf business requirements demand 15-minute RPO, 4-hour backup intervals will not meet SLOs. You need hourly backups, application-level replication, or continuous event streaming to secondary region.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"designing-for-slos-without-over-engineering\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#designing-for-slos-without-over-engineering\" title=\"Designing for SLOs without over-engineering\"\u003eDesigning for SLOs without over-engineering\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eMany teams over-engineer DR solutions trying to achieve zero data loss and instant failover without understanding actual business requirements. A 4-hour RTO may be acceptable for internal tooling but catastrophic for customer-facing APIs.\u003c/p\u003e\n\u003cp\u003ePractical use case:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eInternal reporting API: 2-hour RTO and 1-hour RPO can be enough, active-passive is usually fine.\u003c/li\u003e\n\u003cli\u003eCustomer checkout API: 15-minute RTO and near-zero RPO usually require active-active plus database replication.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe recurring theme is business impact, not architecture fashion.\u003c/p\u003e\n\u003cp\u003eStart by identifying actual business impact:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eWhat revenue is lost per hour of downtime?\u003c/li\u003e\n\u003cli\u003eWhat customer commitments exist in SLAs?\u003c/li\u003e\n\u003cli\u003eWhat regulatory requirements mandate specific recovery times?\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThen design the minimum viable DR solution that meets those requirements. Do not build active-active multi-region architecture with continuous replication if business requirements allow 2-hour RTO and 1-hour RPO. That level of complexity costs significant engineering time and operational overhead.\u003c/p\u003e\n\u003cp\u003eConversely, do not assume daily backups suffice for production systems without validating business tolerance for 24-hour data loss.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"best-practices-what-actually-works\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#best-practices-what-actually-works\" title=\"Best practices: What actually works\"\u003eBest practices: What actually works\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eTest restore procedures regularly.\u003c/strong\u003e Monthly restore testing in isolated environments catches broken procedures before actual incidents. Quarterly testing is too infrequent.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAutomate backup verification.\u003c/strong\u003e Run automated restore tests that verify backup integrity and measure restoration time. Manual testing does not scale and gets skipped under time pressure.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDocument recovery procedures.\u003c/strong\u003e Runbooks that sit in Confluence do not get updated and will be wrong during incidents. Store recovery procedures as executable scripts in version control and test them regularly.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSeparate backup storage from cluster infrastructure.\u003c/strong\u003e Do not store backups in the same region or subscription as the cluster. Regional Azure outages impact all resources in that region including backup storage.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePlan for partial failures.\u003c/strong\u003e Not every incident requires full cluster restore. Design procedures for restoring individual namespaces, specific workloads, or single persistent volumes.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eUse infrastructure as code for cluster rebuild.\u003c/strong\u003e Terraform or Bicep definitions for cluster creation enable rapid cluster recreation when restoration is not the best recovery path.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMonitor backup jobs.\u003c/strong\u003e Failed backups are worthless. Alert on backup failures and missing backup runs. Do not discover backup gaps during recovery.\u003c/p\u003e\n\u003cp\u003eIf you are defining a monthly DR game day, include three quick checks every time:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eCan we restore one namespace end to end in a clean test cluster?\u003c/li\u003e\n\u003cli\u003eCan we switch traffic and run smoke tests in less than our RTO?\u003c/li\u003e\n\u003cli\u003eCan we prove data freshness is inside the RPO window?\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eIf one answer is no, your DR posture is weaker than your dashboard suggests.\u003c/p\u003e\n\u003cp\u003eCommon mistakes: Storing backups in same region as cluster (regional failure loses backups and cluster), never testing restore procedures (broken backups discovered during incidents), manual recovery procedures (humans make mistakes under pressure), and no RTO/RPO measurement (cannot tell if recovery meets business requirements).\u003c/p\u003e\n\u003cp\u003eAuthor note: I have participated in exactly two real disaster recovery situations involving Kubernetes clusters. In the first incident, backup restoration worked but took 3 hours longer than documented because volume snapshot region restrictions were not tested. In the second incident, backups existed but CRD restoration failed because CRD versions changed between backup and restore. Both incidents would have been prevented by regular restore testing. Do not learn this lesson during a production outage.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"conclusion\"\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/#conclusion\" title=\"Conclusion\"\u003eConclusion\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eDisaster recovery for AKS requires deliberate planning, regular testing, and honest assessment of recovery capabilities. Velero provides proven backup and restore workflows. Azure native backup offers simplified management for Azure-focused organizations. Multi-region architectures enable faster recovery but increase complexity and cost.\u003c/p\u003e\n\u003cp\u003eThe real test is not having a backup strategy documented in Confluence. The real test is whether you can restore your cluster from backup in under 60 minutes during an actual regional outage at 2 AM when half your team is asleep and the incident commander is asking for status updates.\u003c/p\u003e\n\u003cp\u003eBuild repeatable procedures. Test them monthly. Automate everything you can. Measure actual RTO and RPO. Add one more rule: if a step cannot be executed from version-controlled scripts, it is probably not ready for production incidents.\u003c/p\u003e\n\u003cp\u003eRelated reading for AKS operations maturity: \u003ca href=\"/posts/cluster-upgrades-zero-downtime-aks/\"\u003eAKS Cluster Upgrades Without Downtime\u003c/a\u003e.\u003c/p\u003e","date_modified":"2026-05-26T10:22:03+02:00","date_published":"2026-03-11T17:00:00+01:00","id":"https://daily-devops.net/posts/disaster-recovery-business-continuity-aks/","language":"en","summary":"AKS outages happen. Build a tested DR plan with Velero, realistic RTO/RPO targets, and multi-region failover steps your team can run under pressure.","tags":["disaster-recovery","azure","kubernetes","cloud","devops","reliability","compliance"],"title":"AKS Disaster Recovery: Why Your Untested Backup Will Fail","url":"https://daily-devops.net/posts/disaster-recovery-business-continuity-aks/"},{"authors":[{"name":"Jendrik Brack","url":"https://daily-devops.net/authors/jendrik/"}],"content_html":"\u003cp\u003eYour Azure Kubernetes Service (AKS) cluster is running smoothly. Deployments are automated. Teams ship features daily. Everything looks secure, until you discover that a container image pulled from your Azure Container Registry contains a critical vulnerability that\u0026rsquo;s been actively exploited in the wild for weeks.\u003c/p\u003e\n\u003cp\u003eThis isn\u0026rsquo;t hypothetical. Supply chain attacks targeting container registries have become a primary attack vector. An unvetted image in production can expose sensitive data, allow lateral movement within your cluster, or provide an entry point for ransomware. The worst part: the vulnerability might not be in your code at all. It could be in a base image dependency you didn\u0026rsquo;t even know you were using.\u003c/p\u003e\n\u003cp\u003eContainer registry security isn\u0026rsquo;t optional. It\u0026rsquo;s foundational to your entire Kubernetes security posture. And ACR (Azure Container Registry) provides the tools you need to enforce it, if you configure them correctly.\u003c/p\u003e\n\u003cp\u003eAuthor note: I have seen teams invest heavily in runtime controls while treating the registry as a passive artifact store. That gap usually shows up during incident response, not during happy-path deployments.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"image-scanning--vulnerability-management\"\u003e\u003ca href=\"/posts/container-registry-image-security-aks/#image-scanning--vulnerability-management\" title=\"Image Scanning \u0026amp; Vulnerability Management\"\u003eImage Scanning \u0026amp; Vulnerability Management\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eThe first line of defense is knowing what\u0026rsquo;s actually in your images before they reach production. Image scanning tools like Trivy, Microsoft Defender for Containers (formerly Azure Defender), and Anchore analyze container layers for known vulnerabilities (CVEs), malware, and configuration issues.\u003c/p\u003e\n\u003cp\u003eBut scanning alone isn\u0026rsquo;t enough. You need a policy-based approach that blocks vulnerable images from being deployed.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"microsoft-defender-for-containers\"\u003e\u003ca href=\"/posts/container-registry-image-security-aks/#microsoft-defender-for-containers\" title=\"Microsoft Defender for Containers\"\u003eMicrosoft Defender for Containers\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eMicrosoft Defender for Containers integrates with ACR and provides agentless vulnerability assessment for container images when the plan and required extension are enabled. In practice, you get push-triggered assessment plus recurring reassessment over time. Findings are surfaced as recommendations in Microsoft Defender for Cloud, with severity and remediation guidance.\u003c/p\u003e\n\u003cp\u003eThe critical configuration: set up alerting and response workflows. A scan report that nobody reads is worthless. Configure alerts to notify your DevOps team when high or critical vulnerabilities are detected. Better yet, integrate with your CI/CD pipeline to fail builds that introduce new vulnerabilities above a defined threshold.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"trivy-integration\"\u003e\u003ca href=\"/posts/container-registry-image-security-aks/#trivy-integration\" title=\"Trivy Integration\"\u003eTrivy Integration\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eTrivy is an open-source vulnerability scanner that\u0026rsquo;s lightweight, fast, and highly accurate. It scans container images, filesystem artifacts, and even Infrastructure as Code templates for vulnerabilities and misconfigurations.\u003c/p\u003e\n\u003cp\u003eIntegrate Trivy into your CI/CD pipeline as a gate:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Scan image before pushing to ACR\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003etrivy image --severity HIGH,CRITICAL --exit-code \u003cspan class=\"m\"\u003e1\u003c/span\u003e myapp:latest\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# If vulnerabilities are found, build fails\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Otherwise, push to ACR\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz acr login --name myregistry\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003edocker tag myapp:latest myregistry.azurecr.io/myapp:latest\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003edocker push myregistry.azurecr.io/myapp:latest\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis approach ensures that only images meeting your security threshold reach your registry in the first place.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"image-signing--verification\"\u003e\u003ca href=\"/posts/container-registry-image-security-aks/#image-signing--verification\" title=\"Image Signing \u0026amp; Verification\"\u003eImage Signing \u0026amp; Verification\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eVulnerability scanning tells you what\u0026rsquo;s in an image. Image signing tells you who built it and whether it\u0026rsquo;s been tampered with. This is supply chain security at its core.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"notation-and-notary-v2\"\u003e\u003ca href=\"/posts/container-registry-image-security-aks/#notation-and-notary-v2\" title=\"Notation and Notary v2\"\u003eNotation and Notary v2\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eAzure Container Registry supports Notary v2 (via the Notation CLI), which implements the CNCF Notary specification for signing and verifying container artifacts. When you sign an image, you\u0026rsquo;re cryptographically attesting that:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eThe image was built by a trusted party (your CI/CD system)\u003c/li\u003e\n\u003cli\u003eThe image hasn\u0026rsquo;t been modified since it was signed\u003c/li\u003e\n\u003cli\u003eThe image meets specific criteria (e.g., passed security scans)\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eHere\u0026rsquo;s a practical workflow:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eIn CI/CD:\u003c/strong\u003e After building and scanning an image, sign it using Notation\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eIn ACR:\u003c/strong\u003e Store signatures alongside the image\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eIn AKS:\u003c/strong\u003e Use Azure Policy or admission controllers (OPA Gatekeeper, Kyverno) to verify signatures before allowing pod creation\u003c/li\u003e\n\u003c/ol\u003e\n\n\n\n\n\u003ch3 id=\"cosign-alternative\"\u003e\u003ca href=\"/posts/container-registry-image-security-aks/#cosign-alternative\" title=\"Cosign Alternative\"\u003eCosign Alternative\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eCosign, part of the Sigstore project, is another popular option for image signing. It\u0026rsquo;s simpler to set up than Notary v2 and integrates well with Kubernetes admission controllers. The choice between Notation and Cosign often comes down to your broader toolchain: Notation if you\u0026rsquo;re heavily invested in Azure, Cosign if you prefer open-source portability.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"rbac-for-registry-access\"\u003e\u003ca href=\"/posts/container-registry-image-security-aks/#rbac-for-registry-access\" title=\"RBAC for Registry Access\"\u003eRBAC for Registry Access\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eWho can push images to your registry? Who can pull them? These questions matter more than you might think.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"azure-rbac-for-acr\"\u003e\u003ca href=\"/posts/container-registry-image-security-aks/#azure-rbac-for-acr\" title=\"Azure RBAC for ACR\"\u003eAzure RBAC for ACR\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eACR supports Azure role-based access control (RBAC) with granular permissions:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eAcrPull:\u003c/strong\u003e Read-only access to pull images\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAcrPush:\u003c/strong\u003e Ability to push and pull images\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAcrDelete:\u003c/strong\u003e Permission to delete images\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eOwner/Contributor:\u003c/strong\u003e Full management rights\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eIn practice, your setup should look like this:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eCI/CD service principals:\u003c/strong\u003e AcrPush role (can build and push images)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAKS node pools:\u003c/strong\u003e AcrPull role via managed identity (can pull images for workloads)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eDevelopers:\u003c/strong\u003e No direct registry access (deployments go through CI/CD)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eSecurity team:\u003c/strong\u003e Reader role for auditing\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThis model ensures that production images flow through controlled pipelines, not from developer laptops.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"aks-managed-identity-for-acr-access\"\u003e\u003ca href=\"/posts/container-registry-image-security-aks/#aks-managed-identity-for-acr-access\" title=\"AKS Managed Identity for ACR Access\"\u003eAKS Managed Identity for ACR Access\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eInstead of storing registry credentials in Kubernetes secrets, use AKS managed identity to grant pull access. This eliminates credential management overhead and reduces the risk of credential leakage.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Attach ACR to AKS cluster using managed identity\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz aks update \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --name myakscluster \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --resource-group myresourcegroup \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --attach-acr myregistry\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eNow your AKS nodes can pull images from ACR without any credentials stored in the cluster.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"private-endpoints-network-isolation\"\u003e\u003ca href=\"/posts/container-registry-image-security-aks/#private-endpoints-network-isolation\" title=\"Private Endpoints: Network Isolation\"\u003ePrivate Endpoints: Network Isolation\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eBy default, Azure Container Registry is accessible over the public internet. Even with RBAC, this creates an unnecessary attack surface. Private endpoints solve this by routing registry traffic through your Azure virtual network.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"terraform-example-acr-with-private-endpoint\"\u003e\u003ca href=\"/posts/container-registry-image-security-aks/#terraform-example-acr-with-private-endpoint\" title=\"Terraform Example: ACR with Private Endpoint\"\u003eTerraform Example: ACR with Private Endpoint\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eHere\u0026rsquo;s a practical Terraform configuration for deploying ACR with a private endpoint:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-hcl\" data-lang=\"hcl\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Azure Container Registry with Premium SKU (required for private endpoints)\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_container_registry\u0026#34; \u0026#34;acr\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;myacrregistry\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003erg\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  location\u003c/span\u003e            \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003erg\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003elocation\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  sku\u003c/span\u003e                 \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Premium\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  admin_enabled\u003c/span\u003e       \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kt\"\u003efalse\u003c/span\u003e\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e  \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e  # Disable public network access\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  public_network_access_enabled\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kt\"\u003efalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Private endpoint for ACR\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_private_endpoint\u0026#34; \u0026#34;acr_pe\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;acr-private-endpoint\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003erg\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  location\u003c/span\u003e            \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003erg\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003elocation\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  subnet_id\u003c/span\u003e           \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_subnet\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eprivate_endpoints\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003eprivate_service_connection\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    name\u003c/span\u003e                           \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;acr-connection\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    private_connection_resource_id\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_container_registry\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eacr\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    is_manual_connection\u003c/span\u003e           \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kt\"\u003efalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    subresource_names\u003c/span\u003e              \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;registry\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003eprivate_dns_zone_group\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    name\u003c/span\u003e                 \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;acr-dns-zone-group\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    private_dns_zone_ids\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"k\"\u003eazurerm_private_dns_zone\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eacr\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Private DNS zone for ACR\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_private_dns_zone\u0026#34; \u0026#34;acr\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;privatelink.azurecr.io\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003erg\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Link DNS zone to VNet\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_private_dns_zone_virtual_network_link\u0026#34; \u0026#34;acr\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                  \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;acr-dns-link\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e   \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003erg\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  private_dns_zone_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_private_dns_zone\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eacr\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  virtual_network_id\u003c/span\u003e    \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_virtual_network\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003evnet\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eWith this configuration:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eACR is only accessible from within your VNet (or peered VNets)\u003c/li\u003e\n\u003cli\u003eDNS resolution automatically routes registry traffic through the private endpoint\u003c/li\u003e\n\u003cli\u003ePublic internet access to your registry is completely disabled\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eThis is especially critical if your AKS cluster handles sensitive workloads or regulated data.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"policy-enforcement-with-gatekeeper\"\u003e\u003ca href=\"/posts/container-registry-image-security-aks/#policy-enforcement-with-gatekeeper\" title=\"Policy Enforcement with Gatekeeper\"\u003ePolicy Enforcement with Gatekeeper\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eScanning and signing only matter if you enforce them. Kubernetes admission controllers intercept pod creation requests and enforce policies before workloads are admitted to the cluster.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"opa-gatekeeper-for-image-source-enforcement\"\u003e\u003ca href=\"/posts/container-registry-image-security-aks/#opa-gatekeeper-for-image-source-enforcement\" title=\"OPA Gatekeeper for Image Source Enforcement\"\u003eOPA Gatekeeper for Image Source Enforcement\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eOpen Policy Agent (OPA) Gatekeeper is a common admission controller for policy enforcement in Kubernetes. The following example enforces that workloads only pull images from approved registries. It does not verify cryptographic signatures by itself:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# Gatekeeper ConstraintTemplate for image signature verification\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003etemplates.gatekeeper.sh/v1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eConstraintTemplate\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eacrverifiedimages\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ecrd\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003enames\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eAcrVerifiedImages\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003evalidation\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eopenAPIV3Schema\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003etype\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eobject\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003eproperties\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"nt\"\u003eallowedRegistries\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e              \u003c/span\u003e\u003cspan class=\"nt\"\u003etype\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003earray\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e              \u003c/span\u003e\u003cspan class=\"nt\"\u003eitems\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e                \u003c/span\u003e\u003cspan class=\"nt\"\u003etype\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003estring\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003etargets\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e- \u003cspan class=\"nt\"\u003etarget\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eadmission.k8s.gatekeeper.sh\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003erego\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e        package acrverifiedimages\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e        violation[{\u0026#34;msg\u0026#34;: msg}] {\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          container := input.review.object.spec.containers[_]\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          not registry_allowed(container.image)\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          msg := sprintf(\u0026#34;Container image \u0026#39;%v\u0026#39; is not from an allowed registry\u0026#34;, [container.image])\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e        }\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e        registry_allowed(image) {\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          allowed := input.parameters.allowedRegistries[_]\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          startswith(image, allowed)\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e        }\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# Constraint to enforce ACR-only images\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003econstraints.gatekeeper.sh/v1beta1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eAcrVerifiedImages\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003erequire-acr-images\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ematch\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ekinds\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003eapiGroups\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ekinds\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Pod\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003enamespaces\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"s2\"\u003e\u0026#34;production\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eparameters\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eallowedRegistries\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"s2\"\u003e\u0026#34;myacrregistry.azurecr.io/\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis policy ensures that pods in the \u003ccode\u003eproduction\u003c/code\u003e namespace can only use images from your approved ACR instance. Any attempt to deploy an image from Docker Hub, a public registry, or an unknown source will be rejected at admission time.\u003c/p\u003e\n\u003cp\u003eFor signature verification, extend this pattern with Ratify and policy enforcement so signatures and trust policies are validated before admission. AKS Image Integrity also exists, but it is currently a preview feature with notable production limitations.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"common-mistakes-with-policy-enforcement\"\u003e\u003ca href=\"/posts/container-registry-image-security-aks/#common-mistakes-with-policy-enforcement\" title=\"Common Mistakes with Policy Enforcement\"\u003eCommon Mistakes with Policy Enforcement\u003c/a\u003e\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003eNaming a registry-allowlist policy as \u0026ldquo;image verification\u0026rdquo; even though no signature verification happens\u003c/li\u003e\n\u003cli\u003eEnforcing policies only in \u003ccode\u003eproduction\u003c/code\u003e and leaving staging unrestricted\u003c/li\u003e\n\u003cli\u003eEnabling admission control without a break-glass process for incident response\u003c/li\u003e\n\u003cli\u003eForgetting to version and test policy changes like application code\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch2 id=\"multi-region-replication-distribution-strategy\"\u003e\u003ca href=\"/posts/container-registry-image-security-aks/#multi-region-replication-distribution-strategy\" title=\"Multi-Region Replication: Distribution Strategy\"\u003eMulti-Region Replication: Distribution Strategy\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eIf your AKS workloads span multiple Azure regions, you need a registry replication strategy that balances availability, performance, and cost.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"geo-replication-in-acr\"\u003e\u003ca href=\"/posts/container-registry-image-security-aks/#geo-replication-in-acr\" title=\"Geo-Replication in ACR\"\u003eGeo-Replication in ACR\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eACR Premium SKU supports geo-replication, allowing you to maintain a single registry name while automatically replicating images to multiple Azure regions. This reduces latency for image pulls and provides failover capabilities.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Enable geo-replication to multiple regions\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz acr replication create \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --registry myregistry \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --location westeurope\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz acr replication create \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --registry myregistry \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --location eastus\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eNow when an AKS cluster in West Europe pulls an image, it\u0026rsquo;s served from the local replica. If that replica becomes unavailable, ACR automatically fails over to another region.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"replication-patterns\"\u003e\u003ca href=\"/posts/container-registry-image-security-aks/#replication-patterns\" title=\"Replication Patterns\"\u003eReplication Patterns\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eSingle-region workloads:\u003c/strong\u003e No replication needed. Keep it simple.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMulti-region with low traffic:\u003c/strong\u003e Geo-replication provides good balance between availability and cost.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMulti-region with high traffic or strict latency requirements:\u003c/strong\u003e Consider dedicated ACR instances per region with automated image promotion pipelines. This gives you more control over what images are available in each region and when they\u0026rsquo;re promoted.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDisaster recovery:\u003c/strong\u003e Geo-replication is not a backup strategy. If an image is accidentally deleted, it\u0026rsquo;s deleted from all replicas. Implement immutability policies (supported in ACR Premium) to prevent accidental deletion of critical images.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"practical-implementation-checklist\"\u003e\u003ca href=\"/posts/container-registry-image-security-aks/#practical-implementation-checklist\" title=\"Practical Implementation Checklist\"\u003ePractical Implementation Checklist\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eIf you\u0026rsquo;re implementing ACR security for an existing AKS deployment, here\u0026rsquo;s the order of operations I\u0026rsquo;d recommend:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eEnable Microsoft Defender for Containers\u003c/strong\u003e on your ACR instance (quick win, no code changes)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eSet up RBAC\u003c/strong\u003e to limit who can push/pull images (reduces blast radius)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eIntegrate Trivy or equivalent scanning\u003c/strong\u003e into your CI/CD pipeline (prevents new vulnerabilities from entering the registry)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eConfigure private endpoints\u003c/strong\u003e if your workloads are in a VNet (reduces attack surface)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eImplement image signing\u003c/strong\u003e with Notation or Cosign (establishes trust boundary)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eDeploy Gatekeeper or Kyverno\u003c/strong\u003e to enforce policies at admission time (prevents policy violations from reaching runtime)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eEnable geo-replication\u003c/strong\u003e if needed (improves availability and performance)\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eThis sequence minimizes risk while keeping deployments flowing. Don\u0026rsquo;t try to implement everything at once. Layered security is iterative.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"what-this-actually-prevents\"\u003e\u003ca href=\"/posts/container-registry-image-security-aks/#what-this-actually-prevents\" title=\"What This Actually Prevents\"\u003eWhat This Actually Prevents\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eLet\u0026rsquo;s ground this in real scenarios:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eScenario 1: Compromised base image\u003c/strong\u003e\u003cbr\u003e\nYour application uses a popular Node.js base image. A critical vulnerability is discovered (e.g., log4shell equivalent). With vulnerability scanning enabled, you\u0026rsquo;re alerted within hours. With policy enforcement, existing vulnerable images can\u0026rsquo;t be deployed until patched.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eScenario 2: Rogue developer\u003c/strong\u003e\u003cbr\u003e\nA developer with push access to ACR tries to deploy an unsigned image from their laptop. With signature verification enforced via Gatekeeper, the deployment is rejected at admission time. Your cluster never runs unverified code.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eScenario 3: Supply chain attack\u003c/strong\u003e\u003cbr\u003e\nAn attacker compromises your CI/CD pipeline and attempts to push a backdoored image to ACR. With RBAC properly configured, the service principal has limited scope. With private endpoints enabled, the attacker can\u0026rsquo;t even access your registry from outside your network.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eScenario 4: Accidental public exposure\u003c/strong\u003e\u003cbr\u003e\nA misconfiguration exposes your ACR to the public internet. With public network access disabled and private endpoints enforced, there\u0026rsquo;s no route to your registry from outside your VNet, so configuration mistakes don\u0026rsquo;t result in exposure.\u003c/p\u003e\n\u003cp\u003eThese aren\u0026rsquo;t theoretical. They\u0026rsquo;re patterns I\u0026rsquo;ve seen in production environments that failed to implement registry security correctly.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"conclusion-security-without-friction\"\u003e\u003ca href=\"/posts/container-registry-image-security-aks/#conclusion-security-without-friction\" title=\"Conclusion: Security Without Friction\"\u003eConclusion: Security Without Friction\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eThe goal isn\u0026rsquo;t to lock down everything so tightly that deployments become painful. The goal is to build security into your workflow so seamlessly that it doesn\u0026rsquo;t slow down your teams.\u003c/p\u003e\n\u003cp\u003eImage scanning, signing, RBAC, private endpoints, and policy enforcement work together to create a defense-in-depth strategy. No single control is perfect. But layered together, they make successful attacks exponentially harder while keeping legitimate deployments fast.\u003c/p\u003e\n\u003cp\u003eStart with the quick wins: enable Defender for Containers, configure RBAC, integrate scanning into CI/CD. Then progressively layer on signing, private endpoints, and policy enforcement as your security maturity grows.\u003c/p\u003e\n\u003cp\u003eYour AKS cluster is only as secure as the images running inside it. Treat your container registry as the trust boundary it actually is.\u003c/p\u003e\n","date_modified":"2026-05-26T10:22:03+02:00","date_published":"2026-03-04T09:00:00+01:00","id":"https://daily-devops.net/posts/container-registry-image-security-aks/","language":"en","summary":"ACR security is foundational. Learn practical hardening: image scanning, signing, RBAC, private endpoints, and policy enforcement for AKS clusters.","tags":["kubernetes","azure","cloud","devops","security"],"title":"Container Registry \u0026 Image Security in AKS Deployments","url":"https://daily-devops.net/posts/container-registry-image-security-aks/"},{"authors":[{"name":"Martin Stühmer","url":"https://daily-devops.net/authors/martin/"}],"content_html":"\u003cp\u003eEvery compliance audit eventually arrives at the same uncomfortable question: \u0026ldquo;Show me how you control changes to production systems.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eIf your answer involves developers pushing directly to the main branch, running deployment scripts from their laptops, or \u0026ldquo;we have a process document somewhere,\u0026rdquo; you\u0026rsquo;re not demonstrating compliance—you\u0026rsquo;re demonstrating risk. ISO/IEC 27001 doesn\u0026rsquo;t accept \u0026ldquo;we trust our developers\u0026rdquo; as a control mechanism. It requires demonstrable, auditable, technically enforced change management.\u003c/p\u003e\n\u003cp\u003eHere\u0026rsquo;s what I\u0026rsquo;ve learned from teams navigating ISO 27001 certification: the ones who struggle aren\u0026rsquo;t lacking documentation. They\u0026rsquo;re lacking technical controls that make compliance violations impossible. The ones who succeed embed compliance directly into their development workflow using tools they already have—specifically, GitHub\u0026rsquo;s branch protection and pull request mechanisms.\u003c/p\u003e\n\u003cp\u003eThis article examines how \u003ca href=\"https://www.iso.org/standard/27001\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eISO 27001\u003c/a\u003e Controls A.12.1 (Operational procedures and responsibilities) and A.14.2 (Security in development and support processes) translate into concrete GitHub configurations. Not theoretical compliance theater. Not checkbox exercises for auditors. Real technical controls that simultaneously improve code quality and satisfy audit requirements.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-compliance-gap-what-iso-27001-actually-requires\"\u003e\u003ca href=\"/posts/change-control-github-branch-protection/#the-compliance-gap-what-iso-27001-actually-requires\" title=\"The Compliance Gap: What ISO 27001 Actually Requires\"\u003eThe Compliance Gap: What ISO 27001 Actually Requires\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eISO 27001 Control A.12.1.2 (Change management) states clearly: \u0026ldquo;Changes to the organization, business processes, information processing facilities and systems that affect information security shall be controlled.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eControl A.14.2.2 (System change control procedures) gets more specific: \u0026ldquo;Changes to systems within the development lifecycle shall be controlled by the use of formal change control procedures.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eWhat does \u0026ldquo;controlled\u0026rdquo; mean in practical terms? It means you can demonstrate:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eAuthorization\u003c/strong\u003e: Who approved this change for production deployment?\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eReview\u003c/strong\u003e: Who verified this change meets security and quality standards?\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eTesting\u003c/strong\u003e: What evidence exists that this change was tested before production?\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eTraceability\u003c/strong\u003e: Can you trace any production code back to the specific change request that introduced it?\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eSegregation\u003c/strong\u003e: Are developers prevented from unilaterally deploying their own changes to production?\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAuditability\u003c/strong\u003e: Can you reconstruct exactly what changed, when, and by whom?\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eIf you can\u0026rsquo;t answer these questions with concrete evidence—not process documents, but actual system logs and access controls—you don\u0026rsquo;t have change control. You have change chaos with documentation.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-fatal-pattern-uncontrolled-change-management\"\u003e\u003ca href=\"/posts/change-control-github-branch-protection/#the-fatal-pattern-uncontrolled-change-management\" title=\"The Fatal Pattern: Uncontrolled Change Management\"\u003eThe Fatal Pattern: Uncontrolled Change Management\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eLet me show you what compliance violation looks like in practice. This is the pattern I\u0026rsquo;ve seen in countless pre-certification assessments:\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"fatal-example-the-we-trust-our-developers-approach\"\u003e\u003ca href=\"/posts/change-control-github-branch-protection/#fatal-example-the-we-trust-our-developers-approach\" title=\"Fatal Example: The \u0026ldquo;We Trust Our Developers\u0026rdquo; Approach\"\u003eFatal Example: The \u0026ldquo;We Trust Our Developers\u0026rdquo; Approach\u003c/a\u003e\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Developer on their laptop, any Tuesday morning\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003egit clone https://github.com/company/production-api\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003ecd\u003c/span\u003e production-api\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Make changes directly to main\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003egit checkout main\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# ... make changes ...\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003egit add .\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003egit commit -m \u003cspan class=\"s2\"\u003e\u0026#34;fix bug\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003egit push origin main\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Deployment happens automatically from main branch\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Or worse: developer runs deployment script manually\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e./deploy-to-prod.sh\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003cstrong\u003eWhat\u0026rsquo;s wrong here?\u003c/strong\u003e Everything. Let\u0026rsquo;s enumerate the compliance failures:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eNo authorization control\u003c/strong\u003e: Any developer with write access can push to main\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eNo mandatory review\u003c/strong\u003e: Changes reach production without peer review\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eNo testing gate\u003c/strong\u003e: No required status checks, builds, or tests\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eNo separation of duties\u003c/strong\u003e: Same person writes code and deploys to production\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eWeak audit trail\u003c/strong\u003e: Git log shows commits but not approval workflow\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eNo rollback procedure\u003c/strong\u003e: No systematic way to revert changes safely\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eEmergency bypass\u003c/strong\u003e: \u0026ldquo;Hotfix\u0026rdquo; culture encourages skipping all controls \u0026ldquo;because urgent\u0026rdquo;\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eThis pattern violates ISO 27001 fundamentally. You cannot demonstrate control. You cannot prove testing occurred. You cannot trace authorization. You cannot show segregation of duties.\u003c/p\u003e\n\u003cp\u003eWhen the auditor asks \u0026ldquo;show me how you prevent untested code from reaching production,\u0026rdquo; you have no technical answer. Only organizational promises that humans will follow the right process. That\u0026rsquo;s not a control. That\u0026rsquo;s hope.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"the-hidden-risk-cultural-normalization\"\u003e\u003ca href=\"/posts/change-control-github-branch-protection/#the-hidden-risk-cultural-normalization\" title=\"The Hidden Risk: Cultural Normalization\"\u003eThe Hidden Risk: Cultural Normalization\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eThe insidious aspect of this pattern is how normal it feels. Developers are trusted professionals. They know the codebase. They understand the risks. Surely they won\u0026rsquo;t deploy broken code to production?\u003c/p\u003e\n\u003cp\u003eExcept they do. Not maliciously—accidentally. Under pressure. At 3 AM during an incident. When the customer is screaming. When the CEO is watching. Human processes fail under stress. That\u0026rsquo;s why ISO 27001 requires technical controls, not just process documentation.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-compliant-pattern-github-branch-protection\"\u003e\u003ca href=\"/posts/change-control-github-branch-protection/#the-compliant-pattern-github-branch-protection\" title=\"The Compliant Pattern: GitHub Branch Protection\"\u003eThe Compliant Pattern: GitHub Branch Protection\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eNow let me show you what compliance looks like when embedded in technical controls:\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"correct-example-branch-protection--pull-request-workflow\"\u003e\u003ca href=\"/posts/change-control-github-branch-protection/#correct-example-branch-protection--pull-request-workflow\" title=\"Correct Example: Branch Protection \u0026#43; Pull Request Workflow\"\u003eCorrect Example: Branch Protection + Pull Request Workflow\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eStep 1: \u003ca href=\"https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eBranch Protection\u003c/a\u003e Configuration\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# Branch protection rules for \u0026#39;main\u0026#39; branch\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# (Configured in GitHub repository Settings → Branches)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eProtection Rules\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"nt\"\u003eRequire pull request before merging\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e✓\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"nt\"\u003eRequire approvals\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e2\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"nt\"\u003eDismiss stale reviews when new commits are pushed\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e✓\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"nt\"\u003eRequire review from code owners\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e✓\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"nt\"\u003eRequire status checks to pass before merging\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e✓\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"nt\"\u003eRequire signed commits\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e✓\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"nt\"\u003eInclude administrators\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e✓\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"nt\"\u003eAllow force pushes\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e✗\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"nt\"\u003eAllow deletions\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e✗\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis configuration ensures:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eAuthorization\u003c/strong\u003e: Pull requests require explicit approval from designated reviewers\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eReview\u003c/strong\u003e: Minimum two approvals enforced technically, not by policy\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eTesting\u003c/strong\u003e: Status checks must pass—builds, tests, security scans—before merge is possible\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eSegregation\u003c/strong\u003e: No direct pushes to main, even for administrators\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eTraceability\u003c/strong\u003e: Every change linked to a pull request with full discussion history\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAuditability\u003c/strong\u003e: Complete record in GitHub audit log\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eStep 2: \u003ca href=\"https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eCODEOWNERS\u003c/a\u003e File for Domain Expert Review\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-plaintext\" data-lang=\"plaintext\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e# .github/CODEOWNERS\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e# Default owners for everything\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e* @team-leads\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e# Database migrations require DBA (Database Administrator) approval\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e/database/migrations/** @database-team\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e# Infrastructure as code requires ops review\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e/infrastructure/** @ops-team @security-team\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e# Payment processing requires specialized review\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e/src/payments/** @payments-team @compliance-officer\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe result:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eMandatory expert review\u003c/strong\u003e: Security-sensitive areas automatically require security team approval\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eDistributed responsibility\u003c/strong\u003e: Domain experts own their areas explicitly\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eCompliance verification\u003c/strong\u003e: Specific reviewers (like compliance officers) for regulated components\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAudit evidence\u003c/strong\u003e: Pull request history shows which experts reviewed each change\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eStep 3: Required Status Checks\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# .github/workflows/pr-checks.yml\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ePull Request Checks\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eon\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003epull_request\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ebranches\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003emain ]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ejobs\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ebuild-and-test\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eruns-on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eubuntu-latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esteps\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/checkout@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eBuild and test\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          dotnet build --configuration Release\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          dotnet test --configuration Release\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003esecurity-scan\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eruns-on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eubuntu-latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esteps\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/checkout@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eRun security analysis\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003egithub/codeql-action/analyze@v3\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eBenefits:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eAutomated testing gate\u003c/strong\u003e: Code must build, pass tests, and meet quality standards\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eSecurity verification\u003c/strong\u003e: Automated security scanning catches vulnerabilities before merge\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eQuality enforcement\u003c/strong\u003e: Code formatting and static analysis prevent quality degradation\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eCompliance evidence\u003c/strong\u003e: Workflow run logs prove testing occurred\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eStep 4: Deployment Workflow with \u003ca href=\"https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eEnvironment Protection\u003c/a\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# .github/workflows/deploy-production.yml\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eDeploy to Production\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eon\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003epush\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ebranches\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003emain ]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ejobs\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003edeploy-production\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eruns-on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eubuntu-latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eenvironment\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eproduction\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esteps\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/checkout@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eDeploy to production\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eaz webapp deploy --resource-group prod-rg --name api --src-path ./release.zip\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eVerify deployment\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ecurl -f https://api.company.com/health\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003cstrong\u003eEnvironment Protection Configuration:\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# Settings → Environments → production\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eEnvironment Protection Rules\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"nt\"\u003eRequired reviewers\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e@\u003cspan class=\"l\"\u003eops-lead, @security-lead\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"nt\"\u003eWait timer\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e5\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eminutes\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"nt\"\u003eDeployment branches\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003emain only\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis delivers:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eDeployment authorization\u003c/strong\u003e: Production deployments require explicit approval from designated leads\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eSegregation of duties\u003c/strong\u003e: Developers cannot trigger production deployments directly\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eCooling-off period\u003c/strong\u003e: 5-minute wait prevents panic deployments\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAutomated verification\u003c/strong\u003e: Smoke tests confirm deployment succeeded\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eComplete audit trail\u003c/strong\u003e: GitHub deployment history records who approved, when, and what was deployed\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch2 id=\"how-this-satisfies-iso-27001-audit-requirements\"\u003e\u003ca href=\"/posts/change-control-github-branch-protection/#how-this-satisfies-iso-27001-audit-requirements\" title=\"How This Satisfies ISO 27001 Audit Requirements\"\u003eHow This Satisfies ISO 27001 Audit Requirements\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eLet\u0026rsquo;s map these technical controls directly to ISO 27001 requirements:\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"control-a1212-change-management\"\u003e\u003ca href=\"/posts/change-control-github-branch-protection/#control-a1212-change-management\" title=\"Control A.12.1.2 (Change Management)\"\u003eControl A.12.1.2 (Change Management)\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eRequirement\u003c/strong\u003e: \u0026ldquo;Changes to the organization, business processes, information processing facilities and systems that affect information security shall be controlled.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTechnical Evidence\u003c/strong\u003e:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003ePull request workflow demonstrates formal change control procedure\u003c/li\u003e\n\u003cli\u003eRequired approvals prove authorization before implementation\u003c/li\u003e\n\u003cli\u003eStatus checks demonstrate testing and security validation\u003c/li\u003e\n\u003cli\u003eGitHub audit log provides complete change history\u003c/li\u003e\n\u003cli\u003eCODEOWNERS ensures security team reviews security-sensitive changes\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch3 id=\"control-a1422-system-change-control-procedures\"\u003e\u003ca href=\"/posts/change-control-github-branch-protection/#control-a1422-system-change-control-procedures\" title=\"Control A.14.2.2 (System Change Control Procedures)\"\u003eControl A.14.2.2 (System Change Control Procedures)\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eRequirement\u003c/strong\u003e: \u0026ldquo;Changes to systems within the development lifecycle shall be controlled by the use of formal change control procedures.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTechnical Evidence\u003c/strong\u003e:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eBranch protection prevents uncontrolled changes\u003c/li\u003e\n\u003cli\u003eRequired status checks enforce testing in development lifecycle\u003c/li\u003e\n\u003cli\u003ePull request reviews document technical review process\u003c/li\u003e\n\u003cli\u003eDeployment workflows enforce controlled release procedures\u003c/li\u003e\n\u003cli\u003eEnvironment protection gates ensure authorization before production deployment\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch3 id=\"control-a1429-system-acceptance-testing\"\u003e\u003ca href=\"/posts/change-control-github-branch-protection/#control-a1429-system-acceptance-testing\" title=\"Control A.14.2.9 (System Acceptance Testing)\"\u003eControl A.14.2.9 (System Acceptance Testing)\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eRequirement\u003c/strong\u003e: \u0026ldquo;Acceptance testing programs and related criteria shall be established for new information systems, upgrades and new versions.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTechnical Evidence\u003c/strong\u003e:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eRequired status checks include automated test execution\u003c/li\u003e\n\u003cli\u003eIntegration tests validate system behavior\u003c/li\u003e\n\u003cli\u003eSecurity scans verify security requirements\u003c/li\u003e\n\u003cli\u003eDeployment workflows include smoke tests for acceptance validation\u003c/li\u003e\n\u003cli\u003eTest execution logs prove testing occurred before production deployment\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch2 id=\"the-audit-evidence-package\"\u003e\u003ca href=\"/posts/change-control-github-branch-protection/#the-audit-evidence-package\" title=\"The Audit Evidence Package\"\u003eThe Audit Evidence Package\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eWhen the auditor asks \u0026ldquo;show me your change control process,\u0026rdquo; you present:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eBranch Protection Configuration\u003c/strong\u003e: Screenshot or exported settings showing technical controls\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003ePull Request History\u003c/strong\u003e: Demonstrating approval workflow, reviewer identity, timestamp\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eStatus Check Logs\u003c/strong\u003e: Showing builds, tests, security scans passed before merge\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eCODEOWNERS File\u003c/strong\u003e: Proving security team mandatory review for security-sensitive code\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eDeployment History\u003c/strong\u003e: GitHub deployments page showing who approved, when deployed\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAudit Log Export\u003c/strong\u003e: Complete GitHub audit log showing access, approvals, merges\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eRollback Procedure\u003c/strong\u003e: Documented process using Git tags and revert workflows\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eThis is concrete, technical, auditable evidence. Not process documents. Not promises. System-enforced controls that make compliance violations technically impossible.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"rollback-procedures-the-missing-control\"\u003e\u003ca href=\"/posts/change-control-github-branch-protection/#rollback-procedures-the-missing-control\" title=\"Rollback Procedures: The Missing Control\"\u003eRollback Procedures: The Missing Control\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eOne frequently overlooked aspect of ISO 27001 change control: demonstrable rollback capability. When a change causes a production incident, you need a systematic, tested procedure to revert to the last known good state.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"rollback-workflow\"\u003e\u003ca href=\"/posts/change-control-github-branch-protection/#rollback-workflow\" title=\"Rollback Workflow\"\u003eRollback Workflow\u003c/a\u003e\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# .github/workflows/rollback-production.yml\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eRollback Production\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eon\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eworkflow_dispatch\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003einputs\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003etarget_tag\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003edescription\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;Git tag to rollback to (e.g., v2.1.4)\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erequired\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003etrue\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ejobs\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003erollback\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eruns-on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eubuntu-latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eenvironment\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eproduction\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esteps\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/checkout@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ewith\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003eref\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e${{ inputs.target_tag }}\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eDeploy rollback\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eaz webapp deploy --resource-group prod-rg --name api --src-path ./release.zip\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eVerify rollback\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ecurl -f https://api.company.com/health\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eKey outcomes:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eControlled rollback\u003c/strong\u003e: Rollback requires same environment protection as deployment\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eValidation\u003c/strong\u003e: Ensures rollback target is a valid, previously deployed version\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAudit trail\u003c/strong\u003e: Records who executed rollback, to which version, and when\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eTesting\u003c/strong\u003e: Automated smoke tests verify rollback succeeded\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch3 id=\"tagging-strategy-for-rollback\"\u003e\u003ca href=\"/posts/change-control-github-branch-protection/#tagging-strategy-for-rollback\" title=\"Tagging Strategy for Rollback\"\u003eTagging Strategy for Rollback\u003c/a\u003e\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Automated tagging in deployment workflow\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Every successful production deployment creates a Git tag\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e- name: Tag successful deployment\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  run: \u003cspan class=\"p\"\u003e|\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nv\"\u003eVERSION\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"k\"\u003e$(\u003c/span\u003ecat version.txt\u003cspan class=\"k\"\u003e)\u003c/span\u003e  \u003cspan class=\"c1\"\u003e# e.g., 2.1.5\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    git tag -a \u003cspan class=\"s2\"\u003e\u0026#34;v\u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eVERSION\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e -m \u003cspan class=\"s2\"\u003e\u0026#34;Production deployment \u003c/span\u003e\u003cspan class=\"k\"\u003e$(\u003c/span\u003edate -u\u003cspan class=\"k\"\u003e)\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    git push origin \u003cspan class=\"s2\"\u003e\u0026#34;v\u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eVERSION\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis creates a complete deployment history. Each tag represents a known-good production state. Rollback becomes straightforward: redeploy from a previous tag.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"common-pitfalls-where-teams-fail-compliance\"\u003e\u003ca href=\"/posts/change-control-github-branch-protection/#common-pitfalls-where-teams-fail-compliance\" title=\"Common Pitfalls: Where Teams Fail Compliance\"\u003eCommon Pitfalls: Where Teams Fail Compliance\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eEven with these controls configured, I\u0026rsquo;ve seen teams accidentally violate their own compliance by:\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"pitfall-1-emergency-bypass-mechanisms\"\u003e\u003ca href=\"/posts/change-control-github-branch-protection/#pitfall-1-emergency-bypass-mechanisms\" title=\"Pitfall 1: Emergency Bypass Mechanisms\"\u003ePitfall 1: Emergency Bypass Mechanisms\u003c/a\u003e\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# DON\u0026#39;T: \u0026#34;Break glass\u0026#34; override that defeats all controls\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# Some teams create a separate branch or emergency workflow\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# that bypasses required reviews \u0026#34;for emergencies\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# This violates ISO 27001 immediately\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# Emergency changes must ALSO follow change control procedures\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eISO 27001 doesn\u0026rsquo;t have an \u0026ldquo;unless it\u0026rsquo;s urgent\u0026rdquo; exception. Emergency changes require expedited procedures with the same controls—faster approvals, not bypassed approvals.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"pitfall-2-administrator-exemptions\"\u003e\u003ca href=\"/posts/change-control-github-branch-protection/#pitfall-2-administrator-exemptions\" title=\"Pitfall 2: Administrator Exemptions\"\u003ePitfall 2: Administrator Exemptions\u003c/a\u003e\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# DON\u0026#39;T: Exclude administrators from branch protection\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eProtection Rules\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"nt\"\u003eInclude administrators\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e✗ \u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"c\"\u003e# WRONG - creates compliance gap\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eIf administrators can bypass controls, auditors will note the gap. Include administrators in protection rules. If emergencies require override, use GitHub\u0026rsquo;s temporary bypass mechanism with full audit logging.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"pitfall-3-manual-deployment-scripts\"\u003e\u003ca href=\"/posts/change-control-github-branch-protection/#pitfall-3-manual-deployment-scripts\" title=\"Pitfall 3: Manual Deployment Scripts\"\u003ePitfall 3: Manual Deployment Scripts\u003c/a\u003e\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# DON\u0026#39;T: Deployment scripts runnable from developer laptops\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Even if main branch is protected, if deployments happen manually\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# you\u0026#39;ve lost segregation of duties\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e./deploy-prod.sh  \u003cspan class=\"c1\"\u003e# This should not exist\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eAll production deployments must flow through GitHub Actions with environment protection. No deployment scripts on developer machines. No SSH access to production servers. No \u0026ldquo;just this once because urgent.\u0026rdquo;\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"pitfall-4-incomplete-audit-logging\"\u003e\u003ca href=\"/posts/change-control-github-branch-protection/#pitfall-4-incomplete-audit-logging\" title=\"Pitfall 4: Incomplete Audit Logging\"\u003ePitfall 4: Incomplete Audit Logging\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eGitHub\u0026rsquo;s built-in \u003ca href=\"https://docs.github.com/en/organizations/keeping-your-organization-secure/managing-security-settings-for-your-organization/reviewing-the-audit-log-for-your-organization\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eaudit log\u003c/a\u003e is extensive, but it has retention limits (90-180 days depending on plan). For long-term compliance, you need:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# .github/workflows/export-audit-logs.yml\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eExport Audit Logs\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eon\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eschedule\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e- \u003cspan class=\"nt\"\u003ecron\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;0 2 * * 0\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"c\"\u003e# Weekly\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ejobs\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eexport\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eruns-on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eubuntu-latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esteps\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eExport and archive audit logs\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          gh api /orgs/{org}/audit-log --paginate \u0026gt; audit-log-$(date +%Y%m%d).json\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          az storage blob upload --account-name complianceaudit --container-name logs --file audit-log-*.json\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eISO 27001 expects audit logs retained for the full audit period (typically 3-7 years). GitHub\u0026rsquo;s retention isn\u0026rsquo;t sufficient. Export and archive.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"beyond-compliance-the-quality-benefits\"\u003e\u003ca href=\"/posts/change-control-github-branch-protection/#beyond-compliance-the-quality-benefits\" title=\"Beyond Compliance: The Quality Benefits\"\u003eBeyond Compliance: The Quality Benefits\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eHere\u0026rsquo;s the pragmatic reality: these controls don\u0026rsquo;t just satisfy auditors. They improve code quality, reduce production incidents, and make teams more productive.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMandatory peer review catches bugs.\u003c/strong\u003e I\u0026rsquo;ve seen data from teams that introduced required reviews: production incident rates dropped 40-60% within three months. Not because developers suddenly got better at coding, but because a second set of eyes spotted issues before merge.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAutomated testing prevents regressions.\u003c/strong\u003e Required status checks mean broken code never reaches main branch. Test failures block merge. You catch problems in pull requests, not production.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSecurity scanning reduces vulnerabilities.\u003c/strong\u003e Automated CodeQL analysis and dependency scanning catch security issues before they ship. You fix them in development, not after exploitation.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDeployment gates prevent panic changes.\u003c/strong\u003e The 5-minute cooling-off period and required approval for production deployments stops teams from making panicked changes during incidents. You think first, deploy second.\u003c/p\u003e\n\u003cp\u003eISO 27001 compliance doesn\u0026rsquo;t fight against development velocity. It reinforces the practices that high-performing teams already use. The controls aren\u0026rsquo;t bureaucracy—they\u0026rsquo;re engineering discipline codified.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"practical-implementation-path\"\u003e\u003ca href=\"/posts/change-control-github-branch-protection/#practical-implementation-path\" title=\"Practical Implementation Path\"\u003ePractical Implementation Path\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eIf you\u0026rsquo;re approaching ISO 27001 certification, here\u0026rsquo;s the pragmatic implementation sequence:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWeek 1: Enable Branch Protection\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eConfigure main branch protection with pull request requirement\u003c/li\u003e\n\u003cli\u003eAdd minimum two required approvals\u003c/li\u003e\n\u003cli\u003ePrevent direct pushes to main\u003c/li\u003e\n\u003cli\u003eInclude administrators in restrictions\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eWeek 2: Implement Status Checks\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eCreate GitHub Actions workflow for build + test\u003c/li\u003e\n\u003cli\u003eAdd security scanning (CodeQL)\u003c/li\u003e\n\u003cli\u003eAdd code quality checks\u003c/li\u003e\n\u003cli\u003eConfigure these as required status checks\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eWeek 3: Create CODEOWNERS\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eIdentify security-sensitive code paths\u003c/li\u003e\n\u003cli\u003eAssign domain experts as code owners\u003c/li\u003e\n\u003cli\u003eConfigure required review from code owners\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eWeek 4: Deploy Environment Protection\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eCreate production environment in GitHub\u003c/li\u003e\n\u003cli\u003eConfigure required approvers (ops + security leads)\u003c/li\u003e\n\u003cli\u003eMigrate deployment workflows to use environment protection\u003c/li\u003e\n\u003cli\u003eRemove manual deployment scripts\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eWeek 5: Implement Rollback Procedures\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eCreate rollback workflow with environment protection\u003c/li\u003e\n\u003cli\u003eImplement automated tagging for deployments\u003c/li\u003e\n\u003cli\u003eTest rollback procedure with non-production environment\u003c/li\u003e\n\u003cli\u003eDocument rollback process for operations team\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eWeek 6: Audit Log Archival\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eSet up automated audit log export\u003c/li\u003e\n\u003cli\u003eConfigure long-term archival storage\u003c/li\u003e\n\u003cli\u003eVerify export and retrieval procedures\u003c/li\u003e\n\u003cli\u003eDocument retention policy\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThis gives you six weeks to implement compliant change control. Not six months. Not a year-long initiative. Six weeks of focused engineering work.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"conclusion-compliance-through-technical-control\"\u003e\u003ca href=\"/posts/change-control-github-branch-protection/#conclusion-compliance-through-technical-control\" title=\"Conclusion: Compliance Through Technical Control\"\u003eConclusion: Compliance Through Technical Control\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eISO 27001 change control isn\u0026rsquo;t about documentation. It\u0026rsquo;s not about process manuals. It\u0026rsquo;s not about trusting humans to follow procedures under pressure.\u003c/p\u003e\n\u003cp\u003eIt\u0026rsquo;s about technical controls that make compliance violations impossible. Branch protection that prevents unreviewed code from reaching production. Required status checks that block merges when tests fail. Environment protection that demands explicit authorization before deployment. Audit logs that record every action, every approval, every change.\u003c/p\u003e\n\u003cp\u003eGitHub provides these controls natively. You don\u0026rsquo;t need expensive compliance platforms. You don\u0026rsquo;t need complex workflows. You don\u0026rsquo;t need bureaucratic overhead. You need to configure the tools you already have to enforce the controls ISO 27001 requires.\u003c/p\u003e\n\u003cp\u003eThe teams that struggle with compliance are the ones trying to prove they \u003cem\u003efollow\u003c/em\u003e processes. The teams that succeed are the ones who \u003cem\u003eenforce\u003c/em\u003e processes through technical controls. When the auditor asks \u0026ldquo;show me your change control,\u0026rdquo; you don\u0026rsquo;t present a process document. You show them your branch protection configuration, your pull request history, your deployment logs, your audit trail.\u003c/p\u003e\n\u003cp\u003eThat\u0026rsquo;s evidence. That\u0026rsquo;s compliance. That\u0026rsquo;s how modern software teams satisfy ISO 27001 without sacrificing development velocity or burying engineers in compliance theater.\u003c/p\u003e\n\u003cp\u003eConfigure these controls. Enforce them technically. Document the evidence. Pass the audit. Then get back to building software—knowing that your change control process is working in the background, preventing the catastrophic mistakes that destroy certifications and damage customer trust.\u003c/p\u003e\n","date_modified":"2026-05-26T10:22:03+02:00","date_published":"2026-02-26T17:00:00+01:00","id":"https://daily-devops.net/posts/change-control-github-branch-protection/","language":"en","summary":"\"We trust our developers\" fails audits. GitHub branch protection makes ISO 27001 change control technically enforceable, not just documented.","tags":["iso-standards","security","github","devops","governance"],"title":"Trust Is Not a Control: ISO 27001 Compliance via GitHub","url":"https://daily-devops.net/posts/change-control-github-branch-protection/"},{"authors":[{"name":"Jendrik Brack","url":"https://daily-devops.net/authors/jendrik/"}],"content_html":"\u003cp\u003eYour first production Azure Kubernetes Service (AKS) cluster often feels manageable for months, sometimes for years. Then demand grows and a second cluster appears. Regional resiliency might require it. Team isolation might require it. Compliance boundaries might require it.\u003c/p\u003e\n\u003cp\u003eThe hard part is not creating cluster number two. The hard part is networking between clusters in a way your team can operate at 2 a.m.\u003c/p\u003e\n\u003cp\u003eThis guide focuses on practical multi-cluster AKS networking: connectivity models, DNS (Domain Name System), ingress patterns, and the trade-offs that matter in production.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"why-single-clusters-hit-their-limits\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#why-single-clusters-hit-their-limits\" title=\"Why Single Clusters Hit Their Limits\"\u003eWhy Single Clusters Hit Their Limits\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eSingle-cluster architectures work until they stop being a sensible risk boundary. Three constraints usually force the move:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eScale ceilings.\u003c/strong\u003e Azure CNI Overlay supports large cluster sizes, including documented scale targets up to 5,000 nodes per cluster in current AKS guidance. Verify current limits before architecture decisions because limits evolve over time (\u003ca href=\"https://learn.microsoft.com/azure/aks/quotas-skus-regions\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eAKS scale limits\u003c/a\u003e).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFailure domain isolation.\u003c/strong\u003e Control plane failures are uncommon, but when they happen the impact is serious. Multi-cluster design contains incidents. A failure in cluster A should not automatically break cluster B.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTeam and workload separation.\u003c/strong\u003e Different compliance requirements, service level objectives, and release cadence often require separate clusters. Shared clusters can become an organizational bottleneck.\u003c/p\u003e\n\u003cp\u003eOnce you commit to multiple clusters, networking becomes the core design problem. Services in cluster A need controlled access to cluster B. Shared infrastructure such as DNS, observability, and data platforms must stay reachable. This must still be simple enough to run day to day.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"connectivity-models-vnet-peering-vs-private-link\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#connectivity-models-vnet-peering-vs-private-link\" title=\"Connectivity Models: VNet Peering vs Private Link\"\u003eConnectivity Models: VNet Peering vs Private Link\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eTwo patterns handle most Azure multi-cluster scenarios: Virtual Network (VNet) peering and Private Link. Both are valid, but they solve different problems.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"vnet-peering-direct-layer-3-connectivity\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#vnet-peering-direct-layer-3-connectivity\" title=\"VNet Peering: Direct Layer 3 Connectivity\"\u003eVNet Peering: Direct Layer 3 Connectivity\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eVNet peering creates bidirectional connectivity between virtual networks over the Azure backbone. Traffic stays private, latency is low, and throughput is high (\u003ca href=\"https://learn.microsoft.com/azure/virtual-network/virtual-network-peering-overview\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eVirtual network peering overview\u003c/a\u003e).\u003c/p\u003e\n\u003cp\u003eFor multi-cluster AKS, peering allows direct IP connectivity between pods and services, assuming routing and policies allow it.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eUse peering when:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eClusters are in the same region or a paired region\u003c/li\u003e\n\u003cli\u003eYou need low latency between workloads\u003c/li\u003e\n\u003cli\u003eYou move significant data volume between clusters\u003c/li\u003e\n\u003cli\u003eYou want simple routing with minimal translation overhead\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003ePeering limitations:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAddress spaces cannot overlap\u003c/li\u003e\n\u003cli\u003ePeering is not transitive\u003c/li\u003e\n\u003cli\u003eSecurity controls must be correct on both sides\u003c/li\u003e\n\u003cli\u003eCross-region transfer costs can become noticeable\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003ePeering is still the default starting point for most environments because it is predictable and easy to reason about.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"private-link-service-endpoint-connectivity\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#private-link-service-endpoint-connectivity\" title=\"Private Link: Service Endpoint Connectivity\"\u003ePrivate Link: Service Endpoint Connectivity\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003ePrivate Link exposes selected services through private endpoints. Instead of full network reachability, consumers connect only to what you explicitly publish (\u003ca href=\"https://learn.microsoft.com/azure/private-link/private-link-overview\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eWhat is Azure Private Link\u003c/a\u003e).\u003c/p\u003e\n\u003cp\u003eIn AKS, this is commonly used to expose internal services through an internal load balancer and Private Link Service. Consumer networks do not need full peering to the provider VNet.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eUse Private Link when:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eYou need strict service-level exposure across boundaries\u003c/li\u003e\n\u003cli\u003eYou cannot avoid overlapping IP ranges\u003c/li\u003e\n\u003cli\u003eYou want narrow, auditable connectivity contracts\u003c/li\u003e\n\u003cli\u003eYou want to reduce broad peering relationships\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003ePrivate Link trade-offs:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eSlightly higher latency than direct peering\u003c/li\u003e\n\u003cli\u003eMore setup and lifecycle management\u003c/li\u003e\n\u003cli\u003eService-specific by design, not full network connectivity\u003c/li\u003e\n\u003cli\u003eEndpoint cost accumulates as service count grows\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eIf your goal is broad cluster-to-cluster communication, peering is simpler. If your goal is controlled service publishing, Private Link is often the better boundary.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"hub-spoke-topology-centralized-connectivity\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#hub-spoke-topology-centralized-connectivity\" title=\"Hub-Spoke Topology: Centralized Connectivity\"\u003eHub-Spoke Topology: Centralized Connectivity\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eHub-spoke is the topology that usually wins once cluster count grows. Instead of a full mesh, each cluster VNet connects to a central hub.\u003c/p\u003e\n\u003cdiv class=\"mermaid\"\u003egraph TB\n  Hub[\"Hub VNet\u003cbr/\u003e(Shared)\"]\n  SpokeA[\"Spoke A\u003cbr/\u003e(Prod)\"]\n  SpokeB[\"Spoke B\u003cbr/\u003e(Dev)\"]\n  SpokeC[\"Spoke C\u003cbr/\u003e(Stage)\"]\n\n  Hub --\u003e SpokeA\n  Hub --\u003e SpokeB\n  Hub --\u003e SpokeC\n\u003c/div\u003e\n\n\u003cp\u003eEach spoke VNet hosts one AKS cluster. The hub carries shared services such as firewalling, gateway connectivity, DNS forwarding, and centralized observability.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"why-hub-spoke-works\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#why-hub-spoke-works\" title=\"Why Hub-Spoke Works\"\u003eWhy Hub-Spoke Works\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eSimplified management.\u003c/strong\u003e A full mesh requires $N\\times(N-1)/2$ peerings. Hub-spoke usually needs one peering per spoke.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCentralized policy enforcement.\u003c/strong\u003e Spoke egress can pass through hub security controls. Policy, logging, and compliance become easier to govern.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCost allocation clarity.\u003c/strong\u003e Shared services stay in the hub. Team-owned workload costs stay in spokes. Chargeback becomes easier.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFailure domain separation.\u003c/strong\u003e Spoke incidents are usually isolated. Hub incidents affect connectivity and must be treated as critical.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"practical-implementation-with-terraform\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#practical-implementation-with-terraform\" title=\"Practical Implementation with Terraform\"\u003ePractical Implementation with Terraform\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eThis Terraform excerpt shows the core peering pattern:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-hcl\" data-lang=\"hcl\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Hub VNet with shared services\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003emodule\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;hub_vnet\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  source\u003c/span\u003e              \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;./modules/vnet\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;hub-vnet\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  address_space\u003c/span\u003e       \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;10.0.0.0/16\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ehub\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  location\u003c/span\u003e            \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003evar\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003elocation\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  subnets\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    firewall\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e      address_prefixes\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;10.0.1.0/24\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    gateway\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e      address_prefixes\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;10.0.2.0/24\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    shared-services\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e      address_prefixes\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;10.0.10.0/24\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Spoke VNet for production AKS cluster\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003emodule\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;spoke_prod_vnet\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  source\u003c/span\u003e              \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;./modules/vnet\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;spoke-prod-vnet\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  address_space\u003c/span\u003e       \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;10.1.0.0/16\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003espoke_prod\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  location\u003c/span\u003e            \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003evar\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003elocation\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  subnets\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    aks-nodes\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e      address_prefixes\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;10.1.0.0/19\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Peering: Spoke to Hub\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_virtual_network_peering\u0026#34; \u0026#34;spoke_prod_to_hub\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                         \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;spoke-prod-to-hub\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e          \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003espoke_prod\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  virtual_network_name\u003c/span\u003e         \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003emodule\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003espoke_prod_vnet\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  remote_virtual_network_id\u003c/span\u003e    \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003emodule\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ehub_vnet\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  allow_virtual_network_access\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kt\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  allow_forwarded_traffic\u003c/span\u003e      \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kt\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  allow_gateway_transit\u003c/span\u003e        \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kt\"\u003efalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  use_remote_gateways\u003c/span\u003e          \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kt\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Peering: Hub to Spoke\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_virtual_network_peering\u0026#34; \u0026#34;hub_to_spoke_prod\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                         \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;hub-to-spoke-prod\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e          \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ehub\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  virtual_network_name\u003c/span\u003e         \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003emodule\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ehub_vnet\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  remote_virtual_network_id\u003c/span\u003e    \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003emodule\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003espoke_prod_vnet\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  allow_virtual_network_access\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kt\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  allow_forwarded_traffic\u003c/span\u003e      \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kt\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  allow_gateway_transit\u003c/span\u003e        \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kt\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  use_remote_gateways\u003c/span\u003e          \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kt\"\u003efalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003cstrong\u003eKey configuration points:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003eallow_forwarded_traffic = true\u003c/code\u003e permits routing through the hub for spoke-to-spoke communication if needed\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eallow_gateway_transit = true\u003c/code\u003e (hub side) allows spokes to use hub\u0026rsquo;s VPN or ExpressRoute gateway\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003euse_remote_gateways = true\u003c/code\u003e (spoke side) leverages hub gateway for on-premises connectivity\u003c/li\u003e\n\u003cli\u003eAddress spaces must not overlap; plan your CIDR ranges before deployment\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch3 id=\"hub-spoke-trade-offs\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#hub-spoke-trade-offs\" title=\"Hub-Spoke Trade-Offs\"\u003eHub-Spoke Trade-Offs\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eLatency.\u003c/strong\u003e Spoke-to-spoke paths include an extra hop through the hub. Usually this is acceptable, but very latency-sensitive paths should be measured.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eHub as a critical dependency.\u003c/strong\u003e If core hub components fail, cross-spoke and on-premises connectivity can fail with them. Critical environments should plan for redundancy.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAdded infrastructure complexity.\u003c/strong\u003e You now own central routing, firewalling, and gateway operations. For two or three clusters, direct peering may still be simpler.\u003c/p\u003e\n\u003cp\u003eUse hub-spoke when you have several clusters, need central governance, or depend on shared network services.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"dns-resolution-across-clusters\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#dns-resolution-across-clusters\" title=\"DNS Resolution Across Clusters\"\u003eDNS Resolution Across Clusters\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eDNS is where many multi-cluster designs fail quietly. Connectivity may exist while name resolution does not.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"the-dns-challenge\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#the-dns-challenge\" title=\"The DNS Challenge\"\u003eThe DNS Challenge\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eEach AKS cluster runs its own CoreDNS service. By default, it resolves cluster-local names such as \u003ccode\u003e.svc.cluster.local\u003c/code\u003e. Cross-cluster discovery needs explicit design.\u003c/p\u003e\n\u003cp\u003eYou need answers to two questions:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eHow does cluster A resolve service names from cluster B?\u003c/li\u003e\n\u003cli\u003eHow does this remain accurate as services change over time?\u003c/li\u003e\n\u003c/ol\u003e\n\n\n\n\n\u003ch3 id=\"approach-1-dns-forwarding-with-custom-coredns-configuration\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#approach-1-dns-forwarding-with-custom-coredns-configuration\" title=\"Approach 1: DNS Forwarding with Custom CoreDNS Configuration\"\u003eApproach 1: DNS Forwarding with Custom CoreDNS Configuration\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eYou can extend CoreDNS to forward specific zones to resolvers in another cluster.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ev1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eConfigMap\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ecoredns-custom\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003enamespace\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ekube-system\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003edata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eclusterb.server\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    clusterb.local:53 {\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e        errors\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e        cache 30\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e        forward . 10.2.0.10\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    }\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis forwards queries for \u003ccode\u003eclusterb.local\u003c/code\u003e to the resolver in cluster B. Services become reachable by names such as \u003ccode\u003eservice-name.namespace.svc.clusterb.local\u003c/code\u003e.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eLimitations:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eManual configuration in each cluster\u003c/li\u003e\n\u003cli\u003eResolver endpoints must stay reachable\u003c/li\u003e\n\u003cli\u003eFragile if upstream DNS endpoints change\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch3 id=\"approach-2-external-dns-with-shared-zone\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#approach-2-external-dns-with-shared-zone\" title=\"Approach 2: External DNS with Shared Zone\"\u003eApproach 2: External DNS with Shared Zone\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eA more scalable pattern is running ExternalDNS in each cluster and writing records into a shared Azure Private DNS zone.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ev1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eService\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eapi-service\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003enamespace\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eproduction\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eannotations\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eexternal-dns.alpha.kubernetes.io/hostname\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eapi.shared.internal\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003etype\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eLoadBalancer\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eloadBalancerIP\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e10.1.5.100\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eports\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e- \u003cspan class=\"nt\"\u003eport\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e443\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003etargetPort\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e8443\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eExternalDNS creates records such as \u003ccode\u003eapi.shared.internal\u003c/code\u003e and updates them as service endpoints change.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eBenefits:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAutomatic DNS management\u003c/li\u003e\n\u003cli\u003eCentralized control through Azure DNS\u003c/li\u003e\n\u003cli\u003eWorks across clusters without manual forwarding rules\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eTrade-offs:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eRequires ExternalDNS operations in every cluster\u003c/li\u003e\n\u003cli\u003eAdds a small DNS zone cost\u003c/li\u003e\n\u003cli\u003eNaming conventions are required to avoid collisions\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eFor most production teams, this is the pragmatic default because it scales and removes manual DNS drift. In AKS, this model aligns well with Private DNS zone integration and standard cluster DNS behavior (\u003ca href=\"https://learn.microsoft.com/azure/aks/concepts-network\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eAKS networking concepts\u003c/a\u003e).\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"shared-ingress-architectures\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#shared-ingress-architectures\" title=\"Shared Ingress Architectures\"\u003eShared Ingress Architectures\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eYou can expose multi-cluster services in two common ways: centralized ingress in a hub, or distributed ingress behind a global load balancer.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"centralized-ingress-in-hub-vnet\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#centralized-ingress-in-hub-vnet\" title=\"Centralized Ingress in Hub VNet\"\u003eCentralized Ingress in Hub VNet\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eRun ingress in the hub VNet, for example with NGINX, Azure Application Gateway, or Envoy. External traffic enters once and is routed to spokes.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAdvantages:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eSingle public IP for all clusters\u003c/li\u003e\n\u003cli\u003eCentralized TLS termination and certificate management\u003c/li\u003e\n\u003cli\u003eSimplified firewall rules (only hub ingress needs public exposure)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eLimitations:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eHub becomes a bottleneck for all ingress traffic\u003c/li\u003e\n\u003cli\u003eAdditional latency (traffic routes hub → spoke)\u003c/li\u003e\n\u003cli\u003eHub failure impacts all clusters\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eUse centralized hub ingress when operational simplicity and unified policy enforcement outweigh performance concerns.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"distributed-ingress-with-azure-front-door\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#distributed-ingress-with-azure-front-door\" title=\"Distributed Ingress with Azure Front Door\"\u003eDistributed Ingress with Azure Front Door\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eRun ingress in each spoke and front it with Azure Front Door or Traffic Manager. Routing decisions can use health, latency, and geographic criteria (\u003ca href=\"https://learn.microsoft.com/azure/frontdoor/front-door-overview\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eAzure Front Door overview\u003c/a\u003e).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAdvantages:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eHigh availability (cluster failures don\u0026rsquo;t take down all ingress)\u003c/li\u003e\n\u003cli\u003eLower latency (traffic routes directly to closest cluster)\u003c/li\u003e\n\u003cli\u003eScalable ingress capacity (not bottlenecked on hub)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eLimitations:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eMultiple public IPs to manage\u003c/li\u003e\n\u003cli\u003eDistributed certificate management (mitigated with cert-manager and Let\u0026rsquo;s Encrypt)\u003c/li\u003e\n\u003cli\u003eRequires global load balancer (Azure Front Door, Traffic Manager)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eFor high availability and regional resilience, distributed ingress is often the better long-term model.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"service-mesh-considerations-when-complexity-is-worth-it\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#service-mesh-considerations-when-complexity-is-worth-it\" title=\"Service Mesh Considerations: When Complexity Is Worth It\"\u003eService Mesh Considerations: When Complexity Is Worth It\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eService meshes such as Istio, Linkerd, or Consul can solve real problems, but they also add a major operational layer.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"what-service-mesh-solves\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#what-service-mesh-solves\" title=\"What Service Mesh Solves\"\u003eWhat Service Mesh Solves\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eCross-cluster service discovery.\u003c/strong\u003e Meshes can federate service catalogs, letting cluster A discover and route to services in cluster B without manual DNS configuration.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTraffic shifting and canary deployments.\u003c/strong\u003e Route a percentage of traffic from cluster A to a new version in cluster B for testing before full cutover.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMutual TLS and zero-trust networking.\u003c/strong\u003e Encrypt all inter-service traffic and enforce identity-based policies across cluster boundaries.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eObservability.\u003c/strong\u003e Centralized metrics, tracing, and logging for requests flowing between clusters.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"when-service-mesh-is-not-worth-it\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#when-service-mesh-is-not-worth-it\" title=\"When Service Mesh Is Not Worth It\"\u003eWhen Service Mesh Is Not Worth It\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eMost multi-cluster environments do not need a mesh on day one. Managing control planes, sidecar upgrades, and mesh debugging is expensive in terms of engineering time.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsider service mesh only when:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eYou\u0026rsquo;re running 5+ clusters with complex inter-cluster traffic patterns\u003c/li\u003e\n\u003cli\u003eZero-trust networking with mTLS is a hard requirement\u003c/li\u003e\n\u003cli\u003eAdvanced traffic management (gradual rollouts, A/B testing across clusters) is core to your deployment strategy\u003c/li\u003e\n\u003cli\u003eYour team has service mesh expertise or dedicated platform engineering resources\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eAuthor note: in most organizations I have worked with, peering plus ExternalDNS plus standard ingress handled the majority of real requirements with far less cognitive load.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"the-pragmatic-alternative-keep-the-baseline-simple\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#the-pragmatic-alternative-keep-the-baseline-simple\" title=\"The Pragmatic Alternative: Keep the Baseline Simple\"\u003eThe Pragmatic Alternative: Keep the Baseline Simple\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eBefore adding a mesh, validate whether baseline Kubernetes networking already meets your goals. Start with clean CIDR planning, network policies, ExternalDNS, and a proven ingress setup.\u003c/p\u003e\n\u003cp\u003eThis baseline is proven and easier to run. Add mesh capabilities only when a measurable requirement demands them.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"cost-and-operational-simplicity\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#cost-and-operational-simplicity\" title=\"Cost and Operational Simplicity\"\u003eCost and Operational Simplicity\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eMulti-cluster architecture increases both spend and operational load. Design intentionally so cost and complexity stay proportional to business value.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"cost-drivers\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#cost-drivers\" title=\"Cost Drivers\"\u003eCost Drivers\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eData transfer between regions.\u003c/strong\u003e Cross-region peering incurs egress charges. High-volume replication paths can become a significant monthly cost. Validate current pricing in the Azure bandwidth and networking pricing pages before committing to traffic-heavy topologies (\u003ca href=\"https://azure.microsoft.com/pricing/details/bandwidth/\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eAzure bandwidth pricing\u003c/a\u003e).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eShared infrastructure.\u003c/strong\u003e Hub-spoke designs require gateway, firewall, and DNS components. These costs usually scale with hub count, not spoke count.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDuplicated platform components.\u003c/strong\u003e More clusters often mean duplicated logging, metrics, and ingress layers. Consolidate where this does not weaken isolation.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"operational-overhead\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#operational-overhead\" title=\"Operational Overhead\"\u003eOperational Overhead\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eConfiguration drift.\u003c/strong\u003e More clusters create more drift opportunities. GitOps tools such as Flux or Argo CD help enforce consistency.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eUpgrade coordination.\u003c/strong\u003e Upgrading many clusters is not linear work. Standardize upgrade pipelines and validate in staging first.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eIncident response.\u003c/strong\u003e Cross-cluster incidents are harder to debug. Centralized logs and tracing are mandatory, not optional.\u003c/p\u003e\n\u003cp\u003eBalance isolation against complexity. Extra clusters without clear boundaries usually become operational debt.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"conclusion-start-simple-scale-deliberately\"\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/#conclusion-start-simple-scale-deliberately\" title=\"Conclusion: Start Simple, Scale Deliberately\"\u003eConclusion: Start Simple, Scale Deliberately\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eMulti-cluster AKS solves real problems: scale boundaries, failure isolation, and team autonomy. It also introduces networking complexity that is easy to underestimate.\u003c/p\u003e\n\u003cp\u003eFor most teams, this sequence works well:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eStart with peering and clean IP planning\u003c/li\u003e\n\u003cli\u003eMove to hub-spoke when cluster count or governance requirements grow\u003c/li\u003e\n\u003cli\u003eUse ExternalDNS for shared service discovery\u003c/li\u003e\n\u003cli\u003eChoose centralized or distributed ingress based on availability and latency goals\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eService mesh can be valuable, but only when its capabilities are tied to concrete requirements that justify the overhead.\u003c/p\u003e\n\u003cp\u003eDesign with the fewest moving parts that satisfy your constraints. Every extra layer raises troubleshooting effort and incident duration.\u003c/p\u003e\n\u003cp\u003eBuild for your current scale, then add components when measurable pain proves the need. That is the operationally honest path.\u003c/p\u003e\n","date_modified":"2026-05-26T10:22:03+02:00","date_published":"2026-02-25T18:30:00+01:00","id":"https://daily-devops.net/posts/multi-aks-cluster-networking-hub-spoke/","language":"en","summary":"Practical multi-cluster AKS networking with VNet peering, hub-spoke routing, DNS, shared ingress, and clear criteria to keep mesh complexity in check.","tags":["networking","azure","kubernetes","cloud","devops","architecture"],"title":"Multi-AKS Cluster Networking \u0026 Hub-Spoke Topology","url":"https://daily-devops.net/posts/multi-aks-cluster-networking-hub-spoke/"},{"authors":[{"name":"Jendrik Brack","url":"https://daily-devops.net/authors/jendrik/"}],"content_html":"\u003cp\u003eCNI Overlay solves IP exhaustion by keeping pod IPs in an internal overlay network. Excellent for resource efficiency. The problem? Your observability stack just lost visibility into half your traffic. Pod IPs get masked behind node IPs through SNAT, and debugging network issues becomes a puzzle where half the pieces are missing.\u003c/p\u003e\n\u003cp\u003eWhen a pod makes an outbound connection to an Azure service, NSG logs show the node IP as the source. Try correlating that with application logs to identify which specific pod initiated the connection, and you\u0026rsquo;ll discover your traditional tooling is useless. The pod IP exists only inside the cluster. From outside, it\u0026rsquo;s invisible.\u003c/p\u003e\n\u003cp\u003eIf you run CNI Overlay in production, you need observability patterns that work with this reality: Container Insights for metadata enrichment, network flow correlation via KQL queries, SNAT port tracking, and distributed tracing.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-root-cause-snat-changes-everything\"\u003e\u003ca href=\"/posts/observability-logging-aks-cni-overlay/#the-root-cause-snat-changes-everything\" title=\"The Root Cause: SNAT Changes Everything\"\u003eThe Root Cause: SNAT Changes Everything\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eIn traditional Azure CNI, each pod receives a VNet-routable IP address. Network flows are straightforward to track. Correlation is direct.\u003c/p\u003e\n\u003cp\u003eCNI Overlay changes this. Pods receive IPs from an internal overlay network (typically \u003ccode\u003e10.244.0.0/16\u003c/code\u003e) that exist only within the cluster. When a pod communicates with anything outside the cluster, the traffic undergoes Source Network Address Translation (SNAT). The pod\u0026rsquo;s internal IP gets replaced with the node\u0026rsquo;s IP before leaving the cluster.\u003c/p\u003e\n\u003cp\u003eFrom the perspective of Azure Network Watcher or NSG Flow Logs, all outbound traffic from pods on a node appears to originate from that single node IP. You lose pod-level granularity. This isn\u0026rsquo;t a bug. It\u0026rsquo;s how overlay networking works. But it breaks every observability pattern you\u0026rsquo;ve built for traditional CNI.\u003c/p\u003e\n\u003cp\u003eThe challenge is correlation. Application logs contain pod IPs. Network logs contain node IPs. Connecting these requires additional context that standard tooling doesn\u0026rsquo;t provide. Microsoft\u0026rsquo;s documentation glosses over this. They\u0026rsquo;ll tell you Container Insights \u0026ldquo;solves observability,\u0026rdquo; but won\u0026rsquo;t mention you\u0026rsquo;re about to spend weeks building KQL queries to answer \u0026ldquo;which pod is talking to this IP?\u0026rdquo;\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"container-insights-your-first-layer-of-defense\"\u003e\u003ca href=\"/posts/observability-logging-aks-cni-overlay/#container-insights-your-first-layer-of-defense\" title=\"Container Insights: Your First Layer of Defense\"\u003eContainer Insights: Your First Layer of Defense\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eContainer Insights is the Azure-native solution for AKS observability. For CNI Overlay clusters, it\u0026rsquo;s mandatory if you want to maintain sanity during production incidents. It\u0026rsquo;s the only thing that maintains the pod-to-node relationship that network logs lose.\u003c/p\u003e\n\u003cp\u003eContainer Insights deploys a DaemonSet (\u003ccode\u003eama-logs\u003c/code\u003e) on every node that scrapes metrics from kubelet and collects stdout/stderr logs. Crucially, it enriches data with Kubernetes metadata: pod name, namespace, node name, labels, annotations. This enables correlation between application logs and network flows.\u003c/p\u003e\n\u003cp\u003eWhen you query Container Insights logs, you can join pod identity with node identity, bridging the gap between application-level events and network-level events. Without this enrichment, you\u0026rsquo;re stuck running kubectl commands during incidents while your cluster burns.\u003c/p\u003e\n\u003cp\u003eHere\u0026rsquo;s a practical Terraform configuration for enabling Container Insights on an AKS cluster with CNI Overlay:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-hcl\" data-lang=\"hcl\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_log_analytics_workspace\u0026#34; \u0026#34;aks_monitoring\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;aks-logs-${var.environment}\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  location\u003c/span\u003e            \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eaks\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003elocation\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eaks\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  sku\u003c/span\u003e                 \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;PerGB2018\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  retention_in_days\u003c/span\u003e   \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"m\"\u003e30\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_kubernetes_cluster\u0026#34; \u0026#34;aks\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;aks-cluster-${var.environment}\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  location\u003c/span\u003e            \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eaks\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003elocation\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eaks\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  dns_prefix\u003c/span\u003e          \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;aks-${var.environment}\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003enetwork_profile\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    network_plugin\u003c/span\u003e      \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azure\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    network_plugin_mode\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;overlay\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    pod_cidr\u003c/span\u003e            \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;10.244.0.0/16\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003eoms_agent\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    log_analytics_workspace_id\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_log_analytics_workspace\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eaks_monitoring\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003emonitor_metrics\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    annotations_allowed\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enull\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    labels_allowed\u003c/span\u003e      \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003enull\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Optional: Data Collection Rule for cost control\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_monitor_data_collection_rule\u0026#34; \u0026#34;aks_container_insights\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;MSCI-${azurerm_kubernetes_cluster.aks.name}\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  location\u003c/span\u003e            \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eaks\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003elocation\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eaks\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003edestinations\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003elog_analytics\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e      workspace_resource_id\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_log_analytics_workspace\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eaks_monitoring\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e      name\u003c/span\u003e                  \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;ciworkspace\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003edata_flow\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    streams\u003c/span\u003e      \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Microsoft-ContainerInsights-Group-Default\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    destinations\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;ciworkspace\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003edata_sources\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003eextension\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e      streams\u003c/span\u003e        \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Microsoft-ContainerInsights-Group-Default\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e      extension_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;ContainerInsights\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e      name\u003c/span\u003e           \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;ContainerInsightsExtension\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\n\n\n\n\u003ch3 id=\"budgeting-for-ingestion-cost\"\u003e\u003ca href=\"/posts/observability-logging-aks-cni-overlay/#budgeting-for-ingestion-cost\" title=\"Budgeting For Ingestion Cost\"\u003eBudgeting For Ingestion Cost\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eA 100-node cluster generates roughly 50-100 GB of logs per month. That\u0026rsquo;s $10-20/month in ingestion costs alone. A 200-node cluster with verbose logging can push $500-1000/month in Log Analytics costs. The optional Data Collection Rule provides granular control to filter out noisy namespaces (kube-system) or low-value metrics before the bill surprises you.\u003c/p\u003e\n\u003cp\u003eCommon mistakes:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eEnabling Container Insights without DCRs on large clusters, then discovering a $2000 Azure Monitor bill\u003c/li\u003e\n\u003cli\u003eSetting retention to 365 days without calculating cost ($0.10/GB/month beyond 31 days)\u003c/li\u003e\n\u003cli\u003eCollecting metrics at 15-second intervals when 60-second suffices for 95% of use cases\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eOnce deployed, Container Insights starts populating several key tables in Log Analytics:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003eContainerLog\u003c/code\u003e: Application logs (stdout/stderr)\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003ePerf\u003c/code\u003e: Performance metrics and resource usage\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eKubePodInventory\u003c/code\u003e: Pod metadata and lifecycle events\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eKubeNodeInventory\u003c/code\u003e: Node metadata and capacity information\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThese tables are your foundation for correlation queries.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"network-observability-flow-logs-and-nsg-logs\"\u003e\u003ca href=\"/posts/observability-logging-aks-cni-overlay/#network-observability-flow-logs-and-nsg-logs\" title=\"Network Observability: Flow Logs and NSG Logs\"\u003eNetwork Observability: Flow Logs and NSG Logs\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eContainer Insights gives you pod-level visibility inside the cluster. But what about traffic leaving the cluster? This is where NSG Flow Logs come into play, and where your observability problems begin in earnest.\u003c/p\u003e\n\u003cp\u003eFlow Logs capture network traffic metadata: source IP, destination IP, port, protocol, allow/deny. For CNI Overlay, Flow Logs show node IPs as the source for all outbound pod traffic. You\u0026rsquo;ve lost pod-level attribution the moment traffic leaves the cluster.\u003c/p\u003e\n\u003cp\u003eThe correlation happens through timestamps and Log Analytics queries. When a pod generates outbound traffic, Container Insights logs the event with the pod\u0026rsquo;s identity and node. Flow Logs capture the same event with the node\u0026rsquo;s IP and destination. Join these datasets on node name and timestamp to reconstruct which pod initiated which connection.\u003c/p\u003e\n\u003cp\u003eAuthor note: This works in theory. In practice, timestamp-based correlation is fragile. Flow Logs have variable latency (5-10 minutes), Container Insights has ingestion delays, and timestamp precision issues mean you\u0026rsquo;ll occasionally join the wrong events. For critical debugging, correlation IDs in application logs are more reliable.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"joining-pod-identity-to-flow-logs-with-kql\"\u003e\u003ca href=\"/posts/observability-logging-aks-cni-overlay/#joining-pod-identity-to-flow-logs-with-kql\" title=\"Joining Pod Identity To Flow Logs With KQL\"\u003eJoining Pod Identity To Flow Logs With KQL\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eHere\u0026rsquo;s a practical KQL query that demonstrates this correlation for debugging outbound connectivity issues:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-csharp\" data-lang=\"csharp\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003elet\u003c/span\u003e \u003cspan class=\"n\"\u003epodName\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"s\"\u003e\u0026#34;my-application-pod-xyz\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003elet\u003c/span\u003e \u003cspan class=\"n\"\u003etimeRange\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eago\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"m\"\u003e1\u003c/span\u003e\u003cspan class=\"n\"\u003eh\u003c/span\u003e\u003cspan class=\"p\"\u003e);\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003elet\u003c/span\u003e \u003cspan class=\"n\"\u003epodNode\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eKubePodInventory\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e \u003cspan class=\"k\"\u003ewhere\u003c/span\u003e \u003cspan class=\"n\"\u003eTimeGenerated\u003c/span\u003e \u003cspan class=\"p\"\u003e\u0026gt;=\u003c/span\u003e \u003cspan class=\"n\"\u003etimeRange\u003c/span\u003e \u003cspan class=\"n\"\u003eand\u003c/span\u003e \u003cspan class=\"n\"\u003eName\u003c/span\u003e \u003cspan class=\"p\"\u003e==\u003c/span\u003e \u003cspan class=\"n\"\u003epodName\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e \u003cspan class=\"n\"\u003eproject\u003c/span\u003e \u003cspan class=\"n\"\u003eTimeGenerated\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003ePodName\u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"n\"\u003eName\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eNamespace\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eNodeName\u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"n\"\u003eComputer\u003c/span\u003e \u003cspan class=\"p\"\u003e|\u003c/span\u003e \u003cspan class=\"n\"\u003etake\u003c/span\u003e \u003cspan class=\"m\"\u003e1\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003elet\u003c/span\u003e \u003cspan class=\"n\"\u003enodeIP\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003eKubeNodeInventory\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e \u003cspan class=\"k\"\u003ewhere\u003c/span\u003e \u003cspan class=\"n\"\u003eTimeGenerated\u003c/span\u003e \u003cspan class=\"p\"\u003e\u0026gt;=\u003c/span\u003e \u003cspan class=\"n\"\u003etimeRange\u003c/span\u003e \u003cspan class=\"n\"\u003eand\u003c/span\u003e \u003cspan class=\"n\"\u003eComputer\u003c/span\u003e \u003cspan class=\"k\"\u003ein\u003c/span\u003e \u003cspan class=\"p\"\u003e((\u003c/span\u003e\u003cspan class=\"n\"\u003epodNode\u003c/span\u003e \u003cspan class=\"p\"\u003e|\u003c/span\u003e \u003cspan class=\"n\"\u003eproject\u003c/span\u003e \u003cspan class=\"n\"\u003eNodeName\u003c/span\u003e\u003cspan class=\"p\"\u003e))\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e \u003cspan class=\"n\"\u003eextend\u003c/span\u003e \u003cspan class=\"n\"\u003eNodeIP\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003etostring\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eparse_json\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"n\"\u003eStatus\u003c/span\u003e\u003cspan class=\"p\"\u003e).\u003c/span\u003e\u003cspan class=\"n\"\u003eaddresses\u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"m\"\u003e0\u003c/span\u003e\u003cspan class=\"p\"\u003e].\u003c/span\u003e\u003cspan class=\"n\"\u003eaddress\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e \u003cspan class=\"n\"\u003eproject\u003c/span\u003e \u003cspan class=\"n\"\u003eNodeName\u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"n\"\u003eComputer\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eNodeIP\u003c/span\u003e \u003cspan class=\"p\"\u003e|\u003c/span\u003e \u003cspan class=\"n\"\u003etake\u003c/span\u003e \u003cspan class=\"m\"\u003e1\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003elet\u003c/span\u003e \u003cspan class=\"n\"\u003epodNodeInfo\u003c/span\u003e \u003cspan class=\"p\"\u003e=\u003c/span\u003e \u003cspan class=\"n\"\u003epodNode\u003c/span\u003e \u003cspan class=\"p\"\u003e|\u003c/span\u003e \u003cspan class=\"k\"\u003ejoin\u003c/span\u003e \u003cspan class=\"n\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"n\"\u003einner\u003c/span\u003e \u003cspan class=\"n\"\u003enodeIP\u003c/span\u003e \u003cspan class=\"k\"\u003eon\u003c/span\u003e \u003cspan class=\"n\"\u003eNodeName\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003epodNodeInfo\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e \u003cspan class=\"k\"\u003ejoin\u003c/span\u003e \u003cspan class=\"n\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"n\"\u003einner\u003c/span\u003e \u003cspan class=\"p\"\u003e(\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"n\"\u003eAzureNetworkAnalytics_CL\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e|\u003c/span\u003e \u003cspan class=\"k\"\u003ewhere\u003c/span\u003e \u003cspan class=\"n\"\u003eTimeGenerated\u003c/span\u003e \u003cspan class=\"p\"\u003e\u0026gt;=\u003c/span\u003e \u003cspan class=\"n\"\u003etimeRange\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e|\u003c/span\u003e \u003cspan class=\"n\"\u003eproject\u003c/span\u003e \u003cspan class=\"n\"\u003eFlowTime\u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"n\"\u003eTimeGenerated\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eSourceIP\u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"n\"\u003eSrcIP_s\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eDestinationIP\u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"n\"\u003eDestIP_s\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eDestinationPort\u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"n\"\u003eDestPort_d\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eProtocol\u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"n\"\u003eL7Protocol_s\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eFlowDirection\u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"n\"\u003eFlowDirection_s\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eDecision\u003c/span\u003e\u003cspan class=\"p\"\u003e=\u003c/span\u003e\u003cspan class=\"n\"\u003eFlowStatus_s\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e \u003cspan class=\"k\"\u003eon\u003c/span\u003e \u003cspan class=\"err\"\u003e$\u003c/span\u003e\u003cspan class=\"n\"\u003eleft\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eNodeIP\u003c/span\u003e \u003cspan class=\"p\"\u003e==\u003c/span\u003e \u003cspan class=\"err\"\u003e$\u003c/span\u003e\u003cspan class=\"n\"\u003eright\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"n\"\u003eSourceIP\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e \u003cspan class=\"n\"\u003eproject\u003c/span\u003e \u003cspan class=\"n\"\u003ePodName\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eNamespace\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eNodeName\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eNodeIP\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eFlowTime\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eDestinationIP\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eDestinationPort\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eProtocol\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eFlowDirection\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"n\"\u003eDecision\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe limitation: all pods on the same node appear in the results because they share the node IP after SNAT. If you have 50 pods on that node, you get 50 potential sources. To narrow this down, correlate timestamps with application-level logs. Without correlation IDs, you\u0026rsquo;re guessing based on timing.\u003c/p\u003e\n\u003cp\u003eCommon mistakes:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAssuming timestamp correlation is accurate to the second (Flow Logs can be off by minutes)\u003c/li\u003e\n\u003cli\u003eNot accounting for pod restarts that change pod-to-node mapping mid-incident\u003c/li\u003e\n\u003cli\u003eForgetting that pods on the same node share the same source IP\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch2 id=\"snat-tracking-and-port-exhaustion\"\u003e\u003ca href=\"/posts/observability-logging-aks-cni-overlay/#snat-tracking-and-port-exhaustion\" title=\"SNAT Tracking and Port Exhaustion\"\u003eSNAT Tracking and Port Exhaustion\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eSNAT doesn\u0026rsquo;t just mask IPs. It introduces a finite resource constraint: SNAT ports. Each node has 64,000 ephemeral ports. In CNI Overlay, all pods on a node share this pool. Under heavy load, you exhaust SNAT ports, causing intermittent connection failures that are nearly impossible to diagnose.\u003c/p\u003e\n\u003cp\u003eThe real risk: SNAT port exhaustion looks exactly like network instability, DNS issues, or backend degradation. You\u0026rsquo;ll spend hours troubleshooting the wrong layer while your SNAT ports silently hit 100%.\u003c/p\u003e\n\u003cp\u003eAzure Monitor provides \u003ccode\u003eAllocatedSnatPorts\u003c/code\u003e and \u003ccode\u003eUsedSnatPorts\u003c/code\u003e metrics at the Load Balancer level, but you need to enable them explicitly. Microsoft\u0026rsquo;s quickstart documentation conveniently omits this.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"what-snat-exhaustion-actually-looks-like\"\u003e\u003ca href=\"/posts/observability-logging-aks-cni-overlay/#what-snat-exhaustion-actually-looks-like\" title=\"What SNAT Exhaustion Actually Looks Like\"\u003eWhat SNAT Exhaustion Actually Looks Like\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eSymptoms when approaching exhaustion:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eConnections timing out during peak traffic, but only for some pods\u003c/li\u003e\n\u003cli\u003eSporadic DNS resolution failures\u003c/li\u003e\n\u003cli\u003eError logs showing \u0026ldquo;unable to bind to port\u0026rdquo; or \u0026ldquo;connection refused\u0026rdquo;\u003c/li\u003e\n\u003cli\u003eRetries succeeding randomly (because a SNAT port became available)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eAuthor note: I\u0026rsquo;ve seen teams spend 6+ hours investigating \u0026ldquo;intermittent Azure Storage failures\u0026rdquo; before someone checked SNAT port metrics and found 99% utilization. The fix took 10 minutes (deploy NAT Gateway). The diagnosis took half a day because SNAT exhaustion wasn\u0026rsquo;t on their radar.\u003c/p\u003e\n\u003cp\u003eMitigation strategies:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eConnection pooling\u003c/strong\u003e: Reuse connections in your application code. Critical in CNI Overlay.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eHTTP keep-alive\u003c/strong\u003e: Reuse TCP connections. A single pod making 1000 requests/second without keep-alive exhausts SNAT ports in under a minute.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eNAT Gateway\u003c/strong\u003e: Deploy NAT Gateway for outbound connectivity. Provides 64,000 SNAT ports per public IP (multiple IPs supported). Not optional for high-throughput clusters.\u003c/li\u003e\n\u003c/ol\u003e\n\n\n\n\n\u003ch3 id=\"when-to-reach-for-nat-gateway\"\u003e\u003ca href=\"/posts/observability-logging-aks-cni-overlay/#when-to-reach-for-nat-gateway\" title=\"When To Reach For NAT Gateway\"\u003eWhen To Reach For NAT Gateway\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eNAT Gateway is the most effective solution. Load Balancer SNAT works for dev clusters, but production clusters handling thousands of outbound requests per second will exhaust ports.\u003c/p\u003e\n\u003cp\u003eUse for: Production clusters with 20+ nodes, clusters making frequent outbound API calls, workloads with poor connection reuse.\u003c/p\u003e\n\u003cp\u003eTrade-offs: $35/month plus $0.045/GB egress (cheaper than debugging SNAT exhaustion at 2 AM). Doesn\u0026rsquo;t solve observability, only port exhaustion.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"debugging-at-scale-correlation-ids-and-distributed-tracing\"\u003e\u003ca href=\"/posts/observability-logging-aks-cni-overlay/#debugging-at-scale-correlation-ids-and-distributed-tracing\" title=\"Debugging at Scale: Correlation IDs and Distributed Tracing\"\u003eDebugging at Scale: Correlation IDs and Distributed Tracing\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eWhen you operate a cluster with hundreds of pods across dozens of nodes, manual correlation becomes impossible. You need structured logging and distributed tracing.\u003c/p\u003e\n\u003cp\u003eCorrelation IDs are the simplest effective pattern. Generate a unique ID at request entry point and propagate it through your entire call chain. When debugging, filter all logs by correlation ID to see the complete request flow across pods and services.\u003c/p\u003e\n\u003cp\u003eDistributed tracing models requests as traces with parent-child relationships between spans. OpenTelemetry is the current standard. Instrument your applications to emit traces to Azure Monitor Application Insights, Jaeger, or Tempo.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"why-traces-survive-snat\"\u003e\u003ca href=\"/posts/observability-logging-aks-cni-overlay/#why-traces-survive-snat\" title=\"Why Traces Survive SNAT\"\u003eWhy Traces Survive SNAT\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eThe value in CNI Overlay: traces maintain pod identity regardless of SNAT. A trace records which pod initiated a call, which pod received it, and operation duration. Network-level SNAT becomes irrelevant because the trace exists at the application layer.\u003c/p\u003e\n\u003cp\u003eThe recurring theme: correlation IDs and distributed tracing aren\u0026rsquo;t \u0026ldquo;nice to have\u0026rdquo; in CNI Overlay. They\u0026rsquo;re operational requirements. Without them, you\u0026rsquo;re correlating timestamps across data sources with different ingestion latencies and hoping you got the right pod. That\u0026rsquo;s not observability. That\u0026rsquo;s guessing.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"practical-recommendations-for-production\"\u003e\u003ca href=\"/posts/observability-logging-aks-cni-overlay/#practical-recommendations-for-production\" title=\"Practical Recommendations for Production\"\u003ePractical Recommendations for Production\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eEnable Container Insights from day one.\u003c/strong\u003e In CNI Overlay, pod-to-node mapping is invisible from network logs. You\u0026rsquo;re flying blind without it. Budget $50-200/month for monitoring, or budget significantly more for postmortems explaining why you couldn\u0026rsquo;t identify which pod caused the outage.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConfigure log retention based on compliance, not convenience.\u003c/strong\u003e Thirty days is reasonable for most use cases. A 100-node cluster at 365-day retention can cost $500-1000/month just for storage.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eImplement structured logging with correlation IDs across all applications.\u003c/strong\u003e This is not optional. JSON logging with consistent field names makes querying possible. Include: timestamp, log level, correlation ID, message. Container Insights adds pod metadata automatically, don\u0026rsquo;t duplicate it.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSet up alerts for SNAT port usage before production.\u003c/strong\u003e Monitor \u003ccode\u003eUsedSnatPorts\u003c/code\u003e and alert at 80% capacity. Better yet, deploy NAT Gateway proactively. The cost ($35/month) is trivial compared to a production outage from SNAT exhaustion.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eUse distributed tracing for multi-service architectures.\u003c/strong\u003e Overhead is low (1-5% CPU), debugging value is high. Start with critical paths. Without tracing, debugging cascade failures in CNI Overlay is nearly impossible.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDocument your correlation queries.\u003c/strong\u003e Keep these queries in version control alongside your infrastructure code. Tribal knowledge doesn\u0026rsquo;t scale.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-hard-truth-about-cni-overlay-observability\"\u003e\u003ca href=\"/posts/observability-logging-aks-cni-overlay/#the-hard-truth-about-cni-overlay-observability\" title=\"The Hard Truth About CNI Overlay Observability\"\u003eThe Hard Truth About CNI Overlay Observability\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eCNI Overlay makes AKS operationally better by solving IP exhaustion. But it makes observability harder by hiding pod IPs behind node IPs. This isn\u0026rsquo;t a flaw. It\u0026rsquo;s a tradeoff. The solution isn\u0026rsquo;t to avoid CNI Overlay (it\u0026rsquo;s the right choice for most clusters) but to build your observability stack with this reality in mind before production.\u003c/p\u003e\n\u003cp\u003eContainer Insights provides the metadata layer that network logs lack. Flow Logs give network-level visibility even though pod IPs are masked. Distributed tracing maintains request context regardless of SNAT. Correlation IDs make manual debugging feasible when automated tools fall short.\u003c/p\u003e\n\u003cp\u003eNone of this is automatic. You have to configure it deliberately, budget for it appropriately, and train your team to use it. Microsoft\u0026rsquo;s CNI Overlay documentation presents it as \u0026ldquo;simpler\u0026rdquo; than traditional CNI. What they don\u0026rsquo;t mention is that you\u0026rsquo;re trading networking simplicity for observability complexity. That\u0026rsquo;s a good trade for large clusters, but it\u0026rsquo;s still a trade.\u003c/p\u003e\n\u003cp\u003eThe operational reality: I\u0026rsquo;ve debugged production incidents in CNI Overlay clusters where the initial response was \u0026ldquo;we can\u0026rsquo;t see which pod is causing this.\u0026rdquo; That\u0026rsquo;s only true if you haven\u0026rsquo;t built the correlation infrastructure upfront. With Container Insights, structured logging, and distributed tracing in place, CNI Overlay observability is no harder than traditional CNI. It\u0026rsquo;s just different, and it requires deliberate tooling investment before the first incident, not during it.\u003c/p\u003e\n\u003cp\u003eThe honest assessment: CNI Overlay is the right choice for most production AKS clusters. The IP efficiency gains are significant. But if your organization isn\u0026rsquo;t prepared to invest in proper observability tooling (Container Insights, distributed tracing, structured logging with correlation IDs), you\u0026rsquo;ll regret choosing CNI Overlay the first time you debug an outbound connectivity issue at 3 AM. Plan accordingly.\u003c/p\u003e\n","date_modified":"2026-05-26T10:22:03+02:00","date_published":"2026-02-18T17:00:00+01:00","id":"https://daily-devops.net/posts/observability-logging-aks-cni-overlay/","language":"en","summary":"CNI Overlay hides pod IPs behind nodes, breaking observability. Practical patterns for log aggregation, network flows, and debugging at scale.","tags":["observability","azure","kubernetes","cloud","devops","monitoring"],"title":"Observability in AKS CNI Overlay: When Pod IPs Hide Behind Nodes","url":"https://daily-devops.net/posts/observability-logging-aks-cni-overlay/"},{"authors":[{"name":"Martin Stühmer","url":"https://daily-devops.net/authors/martin/"}],"content_html":"\u003cp\u003eYou discover on a Tuesday morning that an authentication anomaly in your production API has been happening since last Thursday. Five days of potential unauthorized access. Your team learns about it from a customer complaint, not from your monitoring. Your \u0026ldquo;incident response plan\u0026rdquo; is a Word document last updated 18 months ago. The on-call engineer is unavailable. Nobody remembers where the rollback runbooks are stored.\u003c/p\u003e\n\u003cp\u003eThis isn\u0026rsquo;t a hypothetical nightmare scenario—this is what happens when incident response is a manual process instead of an automated system.\u003c/p\u003e\n\u003cp\u003eISO/IEC 27001 Control A.16 (Information security incident management) and A.17.1 (Information security continuity) don\u0026rsquo;t merely suggest having incident response procedures. They mandate documented, tested, and continuously improved processes for detecting, responding to, and learning from security incidents. The standard recognizes a fundamental truth: incidents aren\u0026rsquo;t just data breaches or server compromises. They include failed deployments that expose sensitive data, configuration drift that opens attack vectors, dependency vulnerabilities that attackers exploit, and authentication anomalies that signal reconnaissance attempts.\u003c/p\u003e\n\u003cp\u003eThe challenge isn\u0026rsquo;t writing an incident response plan. Every organization has one gathering digital dust. The challenge is making incident response actually work when the production environment is on fire, the security team is in different time zones, and every minute of undetected compromise expands the blast radius.\u003c/p\u003e\n\u003cp\u003eGitHub Actions transforms incident response from a theoretical document into an executable workflow. When properly configured, it detects security events in real-time, automatically creates structured incident tickets with severity classification, executes rollback procedures without human intervention, notifies on-call rotations through the channels they actually monitor, and generates immutable audit logs that satisfy both ISO 27001 evidence requirements and post-incident analysis.\u003c/p\u003e\n\u003cp\u003eThis isn\u0026rsquo;t about replacing security teams with automation. It\u0026rsquo;s about ensuring that when incidents occur—and they will occur—the initial detection, triage, and containment happen in seconds instead of days.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-fatal-pattern-when-incident-response-is-a-manual-afterthought\"\u003e\u003ca href=\"/posts/incident-response-github-actions/#the-fatal-pattern-when-incident-response-is-a-manual-afterthought\" title=\"The Fatal Pattern: When Incident Response Is a Manual Afterthought\"\u003eThe Fatal Pattern: When Incident Response Is a Manual Afterthought\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eLet\u0026rsquo;s examine what \u0026ldquo;incident response\u0026rdquo; looks like in organizations that haven\u0026rsquo;t automated their processes. This is the baseline against which ISO 27001 auditors measure your security maturity.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"the-manual-incident-response-anti-pattern\"\u003e\u003ca href=\"/posts/incident-response-github-actions/#the-manual-incident-response-anti-pattern\" title=\"The Manual Incident Response Anti-Pattern\"\u003eThe Manual Incident Response Anti-Pattern\u003c/a\u003e\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# ❌ Manual Incident Response Reality\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# Detection: Quarterly audits (if someone remembers)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# Notification: Email to security-team@company.com (4 recipients left months ago)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# Response: Find someone with prod access. Hope they\u0026#39;re awake. Pray.\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# Patching: \u0026#34;When someone has time\u0026#34; (never)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# Postmortem: Meeting notes nobody reads\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# Results:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# Time to Detection: Days to weeks\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# Time to Response: Hours to days\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# Recurrence Rate: High\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis isn\u0026rsquo;t an exaggeration. This is the documented reality in organizations where incident response remains a manual process. The incident response plan exists as a compliance artifact, not as an operational system. When actual incidents occur, teams improvise under pressure, miss critical steps, and create gaps that both attackers and auditors exploit.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"the-real-cost-hidden-until-it-isnt\"\u003e\u003ca href=\"/posts/incident-response-github-actions/#the-real-cost-hidden-until-it-isnt\" title=\"The Real Cost: Hidden Until It Isn\u0026rsquo;t\"\u003eThe Real Cost: Hidden Until It Isn\u0026rsquo;t\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eThe fatal flaw of manual incident response isn\u0026rsquo;t that it never works—it\u0026rsquo;s that it fails unpredictably and catastrophically:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDiscovery Latency\u003c/strong\u003e: Security incidents discovered manually weeks after occurrence give attackers extended dwell time to escalate privileges, exfiltrate data, and establish persistence. The 2023 Verizon Data Breach Investigations Report found that 68% of breaches took months to discover. Manual processes don\u0026rsquo;t just delay response—they extend the window of vulnerability.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eKnowledge Silos\u003c/strong\u003e: When incident response procedures live in people\u0026rsquo;s heads instead of executable workflows, your response capability depends on who\u0026rsquo;s available when the incident occurs. The engineer who knows the rollback procedure is on vacation. The security analyst who understands the authentication system left the company three months ago. Tribal knowledge doesn\u0026rsquo;t scale, doesn\u0026rsquo;t survive turnover, and creates single points of failure.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAlert Fatigue and Desensitization\u003c/strong\u003e: Email distribution lists for security alerts create a tragedy of the commons. When everyone is responsible, nobody is responsible. Critical alerts drown in noise. Teams develop learned helplessness—\u0026ldquo;the alerts are always firing, they\u0026rsquo;re probably false positives\u0026rdquo;—until the one real incident gets ignored alongside the noise.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompliance Theater vs. Operational Reality\u003c/strong\u003e: The incident response plan that satisfies auditors during annual reviews bears no resemblance to what actually happens during incidents. Teams improvise, skip documented steps that don\u0026rsquo;t work in practice, and create shadow processes that aren\u0026rsquo;t captured in compliance documentation. This gap creates audit risk and operational brittleness.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompounding Delays\u003c/strong\u003e: Manual processes create cascading delays. Detection takes days. Notification takes hours. Triage takes hours more. Finding the right person with the right access takes more time. Each handoff introduces latency, miscommunication, and opportunities for mistakes. By the time response actions execute, the incident has evolved into a crisis.\u003c/p\u003e\n\u003cp\u003eISO 27001 A.16.1 requires \u0026ldquo;management responsibilities and procedures shall be established to ensure a quick, effective and orderly response to information security incidents.\u0026rdquo; The standard doesn\u0026rsquo;t specify \u003cem\u003ehow\u003c/em\u003e to achieve this, but it does mandate measurable effectiveness. Manual processes can\u0026rsquo;t deliver the speed, consistency, or evidence trail that both operational excellence and compliance require.\u003c/p\u003e\n\u003cp\u003eThe alternative isn\u0026rsquo;t theoretical. It\u0026rsquo;s automatable, testable, and already implemented in organizations that treat incident response as a system engineering problem rather than a documentation exercise.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-solution-automated-incident-response-with-github-actions\"\u003e\u003ca href=\"/posts/incident-response-github-actions/#the-solution-automated-incident-response-with-github-actions\" title=\"The Solution: Automated Incident Response with GitHub Actions\"\u003eThe Solution: Automated Incident Response with GitHub Actions\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eEffective incident response requires four capabilities: rapid detection, structured notification, automated containment, and systematic learning. GitHub Actions provides the automation substrate to deliver all four with auditability that satisfies ISO 27001 evidence requirements.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"architecture-event-driven-incident-detection\"\u003e\u003ca href=\"/posts/incident-response-github-actions/#architecture-event-driven-incident-detection\" title=\"Architecture: Event-Driven Incident Detection\"\u003eArchitecture: Event-Driven Incident Detection\u003c/a\u003e\u003c/h3\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# ✅ Automated incident detection and response\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eSecurity Incident Detection and Response\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eon\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003erepository_vulnerability_alert\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003etypes\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"l\"\u003ecreate]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eworkflow_run\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eworkflows\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Production Deploy\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003etypes\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"l\"\u003ecompleted]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eschedule\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e- \u003cspan class=\"nt\"\u003ecron\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;0 */6 * * *\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eworkflow_dispatch\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003einputs\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003eincident_type\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003etype\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003echoice\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eoptions\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"l\"\u003eauthentication_anomaly, failed_deployment, dependency_vulnerability]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ejobs\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003edetect-auth-anomalies\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eruns-on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eubuntu-latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esteps\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eQuery Application Insights\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eid\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003equery\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eazure/CLI@v2\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ewith\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003einlineScript\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            QUERY=\u0026#39;requests \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            | where timestamp \u0026gt; ago(5m) and name contains \u0026#34;auth\u0026#34; and success == false\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            | summarize FailureCount=count() by client_IP\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            | where FailureCount \u0026gt; 100\u0026#39;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            RESULT=$(az monitor app-insights query --app ${{ secrets.APPINSIGHTS_ID }} --analytics-query \u0026#34;$QUERY\u0026#34; -o json)\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            echo \u0026#34;detected=$([ $(echo $RESULT | jq \u0026#39;.tables[0].rows | length\u0026#39;) -gt 0 ] \u0026amp;\u0026amp; echo true || echo false)\u0026#34; \u0026gt;\u0026gt; $GITHUB_OUTPUT\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eCreate Incident Issue\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eif\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003esteps.query.outputs.detected == \u0026#39;true\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/github-script@v7\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ewith\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003escript\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            await github.rest.issues.create({\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e              owner: context.repo.owner,\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e              repo: context.repo.repo,\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e              title: `[CRITICAL] Auth Anomaly - ${new Date().toISOString()}`,\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e              body: `## Security Incident\\n\\n**ISO 27001 Control**: A.16.1\\n\\n### Actions\\n- [ ] Investigate IPs\\n- [ ] Apply rate limiting`,\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e              labels: [\u0026#39;security-incident\u0026#39;, \u0026#39;critical\u0026#39;]\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            });\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eauto-rollback\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eif\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003egithub.event.workflow_run.conclusion == \u0026#39;failure\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eruns-on\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eubuntu-latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003esteps\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/checkout@v4\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ewith\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003efetch-depth\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e0\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eRollback to Last Good Commit\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          LAST_GOOD=$(git log --format=\u0026#34;%H\u0026#34; --grep=\u0026#34;deploy: success\u0026#34; -1)\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          git revert --no-commit $LAST_GOOD..${{ github.sha }}\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          git commit -m \u0026#34;chore: auto-rollback [skip ci]\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          git push origin HEAD:main\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eCreate Incident Issue\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/github-script@v7\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ewith\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003escript\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            await github.rest.issues.create({\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e              owner: context.repo.owner,\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e              repo: context.repo.repo,\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e              title: `[HIGH] Auto-Rollback - ${new Date().toISOString()}`,\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e              body: `## Rollback Executed\\n\\n**Failed**: \\`${{ github.sha }}\\`\\n**ISO 27001**: A.17.1\\n\\n### Postmortem\\n- [ ] Root cause?\\n- [ ] Prevention?`,\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e              labels: [\u0026#39;security-incident\u0026#39;, \u0026#39;rollback\u0026#39;]\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e            });\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis workflow architecture delivers several capabilities that manual processes cannot:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eReal-Time Detection\u003c/strong\u003e: Security events trigger workflows immediately, not when someone checks a dashboard. Authentication anomalies, deployment failures, and dependency vulnerabilities generate incidents within seconds of occurrence. Detection latency measured in seconds, not days.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eStructured Incident Classification\u003c/strong\u003e: Every incident automatically receives severity classification, ISO 27001 control mapping, unique incident ID, and consistent metadata. No more ambiguous \u0026ldquo;urgent\u0026rdquo; emails that might mean anything from minor configuration drift to active breach.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAutomated Containment\u003c/strong\u003e: Critical incidents trigger automated response actions—rollbacks execute without human intervention, security patches apply automatically for high-severity CVEs, rate limiting engages for authentication anomalies. Containment happens in minutes, not hours.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eImmutable Audit Trail\u003c/strong\u003e: Every incident generates a GitHub Issue with complete timeline, automated actions taken, manual actions required, and ISO 27001 control references. Auditors can trace detection→response→resolution for every incident with Git-backed evidence.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"integration-with-monitoring-application-insights-as-security-sensor\"\u003e\u003ca href=\"/posts/incident-response-github-actions/#integration-with-monitoring-application-insights-as-security-sensor\" title=\"Integration with Monitoring: Application Insights as Security Sensor\"\u003eIntegration with Monitoring: Application Insights as Security Sensor\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eThe authentication anomaly detection demonstrates a pattern applicable to any monitoring platform. Application Insights becomes a security sensor, not just a performance dashboard:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Query pattern for authentication anomalies\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003erequests \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e where timestamp \u0026gt; ago\u003cspan class=\"o\"\u003e(\u003c/span\u003e5m\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e where name contains \u003cspan class=\"s2\"\u003e\u0026#34;login\u0026#34;\u003c/span\u003e or name contains \u003cspan class=\"s2\"\u003e\u0026#34;auth\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e where \u003cspan class=\"nv\"\u003esuccess\u003c/span\u003e \u003cspan class=\"o\"\u003e==\u003c/span\u003e \u003cspan class=\"nb\"\u003efalse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e summarize \u003cspan class=\"nv\"\u003eFailureCount\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003ecount\u003cspan class=\"o\"\u003e()\u003c/span\u003e by client_IP, user_Id\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e|\u003c/span\u003e where FailureCount \u0026gt; \u003cspan class=\"m\"\u003e100\u003c/span\u003e or \u003cspan class=\"o\"\u003e(\u003c/span\u003eisnotempty\u003cspan class=\"o\"\u003e(\u003c/span\u003euser_Id\u003cspan class=\"o\"\u003e)\u003c/span\u003e and FailureCount \u0026gt; 10\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis query identifies two distinct attack patterns: credential stuffing (many failed attempts from single IP) and targeted account compromise (many failed attempts for single user). The workflow executes this query every 6 hours via scheduled trigger, or immediately after authentication-related code deployments.\u003c/p\u003e\n\u003cp\u003eWhen anomalies surface, the automated response includes:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eImmediate notification\u003c/strong\u003e to on-call security team via Slack webhook that they actually monitor\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eStructured incident ticket\u003c/strong\u003e with actionable investigation steps, not vague \u0026ldquo;something\u0026rsquo;s wrong\u0026rdquo; alerts\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eSeverity classification\u003c/strong\u003e that distinguishes between rate-limiting evasion attempts (medium) and coordinated breach attempts (critical)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eISO 27001 control mapping\u003c/strong\u003e that satisfies auditor evidence requirements without manual documentation\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch3 id=\"automated-rollback-when-code-becomes-incident\"\u003e\u003ca href=\"/posts/incident-response-github-actions/#automated-rollback-when-code-becomes-incident\" title=\"Automated Rollback: When Code Becomes Incident\"\u003eAutomated Rollback: When Code Becomes Incident\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eFailed deployments aren\u0026rsquo;t just DevOps problems—they\u0026rsquo;re security incidents when they expose sensitive data, disable authentication checks, or create configuration vulnerabilities. The rollback workflow treats deployment failures as incidents requiring the same rigor as authentication breaches:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# Rollback procedure: automated, tested, auditable\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003egit log --format=\u0026#34;%H\u0026#34; --grep=\u0026#34;deploy: success\u0026#34; -1 \u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"c\"\u003e# Find last good commit\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003egit revert --no-commit $LAST_GOOD..$CURRENT       \u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"c\"\u003e# Create revert commit\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003egit push origin HEAD:main                         \u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"c\"\u003e# Restore production immediately\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis isn\u0026rsquo;t novel version control usage—it\u0026rsquo;s applying Git semantics to incident response. The last known good state is always recoverable. The rollback action is reversible. The entire incident timeline exists in commit history. Auditors can verify rollback procedures work because they\u0026rsquo;re executed automatically on every deployment failure, not just tested during annual disaster recovery drills.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"the-postmortem-requirement-learning-as-compliance\"\u003e\u003ca href=\"/posts/incident-response-github-actions/#the-postmortem-requirement-learning-as-compliance\" title=\"The Postmortem Requirement: Learning as Compliance\"\u003eThe Postmortem Requirement: Learning as Compliance\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eISO 27001 A.16.1 requires \u0026ldquo;information security incidents shall be assessed and it shall be decided if they are to be classified as information security incidents.\u0026rdquo; This assessment isn\u0026rsquo;t optional—it\u0026rsquo;s how organizations demonstrate continuous improvement.\u003c/p\u003e\n\u003cp\u003eThe incident ticket template includes a postmortem structure:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-markdown\" data-lang=\"markdown\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"gu\"\u003e### Postmortem Template\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003e- [ ]\u003c/span\u003e What was the intended change?\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003e- [ ]\u003c/span\u003e What actually happened?  \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003e- [ ]\u003c/span\u003e What was the root cause?\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003e- [ ]\u003c/span\u003e How was it detected?\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003e- [ ]\u003c/span\u003e How was it resolved?\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003e- [ ]\u003c/span\u003e What prevented earlier detection?\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003e- [ ]\u003c/span\u003e What will prevent recurrence?\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis isn\u0026rsquo;t bureaucratic box-checking. These questions systematically capture knowledge that prevents incident recurrence. The answers inform:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eDetection improvements\u003c/strong\u003e: \u0026ldquo;What prevented earlier detection?\u0026rdquo; identifies monitoring gaps\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eResponse improvements\u003c/strong\u003e: \u0026ldquo;How was it resolved?\u0026rdquo; documents effective response patterns\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003ePrevention improvements\u003c/strong\u003e: \u0026ldquo;What will prevent recurrence?\u0026rdquo; drives architectural changes\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eWhen postmortems are structured GitHub Issues instead of meeting notes, they become searchable, linkable, and trackable. Teams can reference previous incidents when designing new features. Patterns emerge across incidents that single events don\u0026rsquo;t reveal. Knowledge persists beyond employee tenure.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"iso-27001-compliance-evidence-audit-trail-without-theater\"\u003e\u003ca href=\"/posts/incident-response-github-actions/#iso-27001-compliance-evidence-audit-trail-without-theater\" title=\"ISO 27001 Compliance Evidence: Audit Trail Without Theater\"\u003eISO 27001 Compliance Evidence: Audit Trail Without Theater\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eAuditors evaluating ISO 27001 A.16 compliance ask three questions:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eHow do you detect incidents?\u003c/strong\u003e Manual monitoring vs. automated detection with defined thresholds\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eHow do you respond to incidents?\u003c/strong\u003e Ad-hoc improvisation vs. documented, tested procedures\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eHow do you improve after incidents?\u003c/strong\u003e Verbal discussions vs. systematic postmortem analysis\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eGitHub Actions provides auditable evidence for all three:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDetection Evidence\u003c/strong\u003e: Workflow run history shows detection workflows executing on schedule, incident tickets created with timestamps and triggering events, queries executed against monitoring platforms. Auditors can verify detection capabilities are tested continuously, not just demonstrated during audits.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResponse Evidence\u003c/strong\u003e: GitHub Issues contain complete incident timeline from detection through resolution, automated actions taken (rollback commits, security patches, notifications), manual actions performed (investigation notes, coordination with external teams), and closure criteria (how was resolution verified?).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eImprovement Evidence\u003c/strong\u003e: Postmortem issues linked to infrastructure changes, new detection workflows added after incident gaps identified, and rollback procedures refined based on actual incident experience. The Git history of workflow files shows incident response procedures evolving over time.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"github-audit-log-the-compliance-requirement-you-already-satisfy\"\u003e\u003ca href=\"/posts/incident-response-github-actions/#github-audit-log-the-compliance-requirement-you-already-satisfy\" title=\"GitHub Audit Log: The Compliance Requirement You Already Satisfy\"\u003eGitHub Audit Log: The Compliance Requirement You Already Satisfy\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eISO 27001 A.16.1.7 requires \u0026ldquo;organizations shall establish procedures for the collection and preservation of evidence.\u0026rdquo; GitHub\u0026rsquo;s audit log automatically provides:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eWho\u003c/strong\u003e triggered security workflows (user attribution)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eWhat\u003c/strong\u003e actions were taken (workflow execution logs)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eWhen\u003c/strong\u003e incidents occurred (immutable timestamps)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eWhy\u003c/strong\u003e actions were automated vs. manual (workflow trigger events)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThis audit trail is tamper-evident, continuously available, and doesn\u0026rsquo;t require additional infrastructure. Organizations already using GitHub for source control satisfy evidence requirements without additional compliance tooling—if they architect incident response as code instead of documentation.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"practical-implementation-from-theory-to-production\"\u003e\u003ca href=\"/posts/incident-response-github-actions/#practical-implementation-from-theory-to-production\" title=\"Practical Implementation: From Theory to Production\"\u003ePractical Implementation: From Theory to Production\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eThe workflow examples demonstrate capabilities, not prescriptive templates. Actual implementation requires adapting these patterns to your specific infrastructure, monitoring platforms, and organizational constraints.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"start-with-one-incident-type\"\u003e\u003ca href=\"/posts/incident-response-github-actions/#start-with-one-incident-type\" title=\"Start with One Incident Type\"\u003eStart with One Incident Type\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eDon\u0026rsquo;t attempt to automate all incident response simultaneously. Start with the incident type that\u0026rsquo;s both frequent enough to test regularly and impactful enough to justify automation effort:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFrequent incidents\u003c/strong\u003e: Failed deployments, dependency vulnerabilities, configuration drift\n\u003cstrong\u003eImpactful incidents\u003c/strong\u003e: Authentication anomalies, data exposure, privilege escalation\u003c/p\u003e\n\u003cp\u003eA production-ready starting point: automated rollback on deployment failure. This incident type occurs naturally during development, doesn\u0026rsquo;t require complex security instrumentation, and provides immediate value (reduced downtime) while building compliance evidence.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"test-incident-response-before-incidents-occur\"\u003e\u003ca href=\"/posts/incident-response-github-actions/#test-incident-response-before-incidents-occur\" title=\"Test Incident Response Before Incidents Occur\"\u003eTest Incident Response Before Incidents Occur\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eThe \u003ccode\u003eworkflow_dispatch\u003c/code\u003e trigger enables incident simulation:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eworkflow_dispatch\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003einputs\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eincident_type\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003edescription\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;Type of incident to simulate\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003etype\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003echoice\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003eoptions\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e- \u003cspan class=\"l\"\u003eauthentication_anomaly\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e- \u003cspan class=\"l\"\u003econfiguration_drift  \u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e- \u003cspan class=\"l\"\u003efailed_deployment\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eQuarterly incident response testing becomes: trigger the workflow manually, verify incident ticket creation, validate notification delivery, confirm automated containment actions execute correctly, and review postmortem template completeness.\u003c/p\u003e\n\u003cp\u003eThis simulation satisfies ISO 27001\u0026rsquo;s requirement for testing incident response procedures while building team familiarity with automated workflows before actual incidents occur.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"integrate-gradually-with-existing-tools\"\u003e\u003ca href=\"/posts/incident-response-github-actions/#integrate-gradually-with-existing-tools\" title=\"Integrate Gradually with Existing Tools\"\u003eIntegrate Gradually with Existing Tools\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eOrganizations already have monitoring platforms (Datadog, New Relic, Azure Monitor), ticketing systems (Jira, ServiceNow), and notification channels (Slack, Teams, PagerDuty). GitHub Actions integrates with all of them:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eMonitoring integration\u003c/strong\u003e: Azure CLI action, Datadog API, custom API queries via \u003ccode\u003ecurl\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eTicketing integration\u003c/strong\u003e: REST APIs, webhook notifications, email gateways\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eNotification integration\u003c/strong\u003e: Slack actions, Teams webhooks, PagerDuty events\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe authentication anomaly workflow uses Application Insights, but the same pattern applies to any monitoring platform that exposes query APIs. Replace the Azure CLI step with your monitoring platform\u0026rsquo;s equivalent.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-honest-assessment-what-automation-cannot-do\"\u003e\u003ca href=\"/posts/incident-response-github-actions/#the-honest-assessment-what-automation-cannot-do\" title=\"The Honest Assessment: What Automation Cannot Do\"\u003eThe Honest Assessment: What Automation Cannot Do\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eAutomated incident response isn\u0026rsquo;t a replacement for security expertise. It\u0026rsquo;s a force multiplier that ensures the initial detection, triage, and containment happen reliably when experts aren\u0026rsquo;t immediately available.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAutomation cannot\u003c/strong\u003e: Determine if an authentication anomaly is a credential stuffing attack vs. legitimate user behavior (requires context human analysts provide). Decide if a failed deployment justifies rollback vs. forward-fix (depends on business risk assessment). Coordinate incident response across organizational boundaries (legal, PR, customer support). Perform root cause analysis beyond correlation (distinguishing causation from coincidence requires domain knowledge).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAutomation excels at\u003c/strong\u003e: Detecting predefined patterns in monitoring data at scale. Executing documented procedures consistently under pressure. Creating structured evidence trails for compliance and analysis. Reducing time-to-containment for known incident types.\u003c/p\u003e\n\u003cp\u003eThe combination—automated detection and containment plus human expertise for investigation and improvement—delivers both operational resilience and compliance evidence. Neither component alone suffices.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"conclusion-incident-response-as-engineering-not-theater\"\u003e\u003ca href=\"/posts/incident-response-github-actions/#conclusion-incident-response-as-engineering-not-theater\" title=\"Conclusion: Incident Response as Engineering, Not Theater\"\u003eConclusion: Incident Response as Engineering, Not Theater\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eISO 27001 A.16 doesn\u0026rsquo;t mandate specific tools or technologies. It mandates effective incident management with evidence of continuous improvement. Organizations can satisfy this requirement with manual processes and documentation—but manual processes fail unpredictably when incidents occur outside business hours, involve unfamiliar attack vectors, or overwhelm available staff.\u003c/p\u003e\n\u003cp\u003eGitHub Actions transforms incident response from a compliance document into an operational system. Detection happens automatically. Notifications reach on-call teams through channels they monitor. Containment actions execute without requiring production access credentials shared across teams. Audit trails generate automatically without manual documentation overhead.\u003c/p\u003e\n\u003cp\u003eThis isn\u0026rsquo;t about replacing security teams with YAML files. It\u0026rsquo;s about ensuring that when Tuesday\u0026rsquo;s authentication anomaly occurs, your organization detects it on Tuesday—not the following Monday when a customer complains. The difference between seconds-to-detection and days-to-detection determines whether an incident is a containable event or a reportable breach.\u003c/p\u003e\n\u003cp\u003eThe incident response plan that satisfies auditors but fails operationally provides neither security nor compliance. Automated incident response provides both, with Git-backed evidence that demonstrates effectiveness rather than intent.\u003c/p\u003e\n\u003cp\u003eISO 27001 requires incident response procedures that actually work when incidents actually occur. GitHub Actions delivers executable procedures with immutable audit trails. The choice between compliant documentation and compliant operations determines which incidents become learning opportunities and which become crisis escalations.\u003c/p\u003e\n","date_modified":"2026-05-26T10:22:03+02:00","date_published":"2026-02-12T17:00:00+01:00","id":"https://daily-devops.net/posts/incident-response-github-actions/","language":"en","summary":"ISO 27001 demands effective incident response. GitHub Actions transforms your dusty Word doc into automated workflows that actually work at 3 AM.\n","tags":["security","github-actions","devops","automation","bestpractices","codequality"],"title":"Your Incident Response Plan Is a Lie. Here's How to Fix It.\n","url":"https://daily-devops.net/posts/incident-response-github-actions/"},{"authors":[{"name":"Jendrik Brack","url":"https://daily-devops.net/authors/jendrik/"}],"content_html":"\u003cp\u003eAKS costs are brutally simple: node sizing, pod density, workload sprawl, and reserved capacity. If you don\u0026rsquo;t have visibility and governance, your cloud bill will punch you in the face—usually when it\u0026rsquo;s too late to react without pain. I\u0026rsquo;ve watched teams scramble to cut costs after the invoice lands, breaking production in the process. This guide is for practitioners who want to avoid that mess. No theory, no vendor fluff: just what actually works to keep AKS costs under control without sacrificing reliability.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-problem-visibility-prevents-shocks\"\u003e\u003ca href=\"/posts/cost-optimization-resource-governance-aks/#the-problem-visibility-prevents-shocks\" title=\"The problem: visibility prevents shocks\"\u003eThe problem: visibility prevents shocks\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eThe real risk is flying blind on cost. Most organizations throw infrastructure at problems, then act shocked when the bill arrives. This delay creates a vicious cycle: teams over-provision to avoid risk, costs spiral, finance panics, and engineers are told to \u0026ldquo;just cut spend\u0026rdquo;—usually with zero context. If you don\u0026rsquo;t know what your resources will cost before you deploy, you\u0026rsquo;re setting yourself up for failure. In AKS, node cost is king. If you don\u0026rsquo;t understand node sizing, pod density, and workload distribution, you\u0026rsquo;re not in control: you\u0026rsquo;re just hoping for the best.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"pod-density-vs-node-size\"\u003e\u003ca href=\"/posts/cost-optimization-resource-governance-aks/#pod-density-vs-node-size\" title=\"Pod density vs. node size\"\u003ePod density vs. node size\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003ePod density: the number of pods per node. Higher density slashes per-pod cost, but if you push it too far, a single node failure can wipe out half your workloads. Lower density means more nodes, more overhead, and more Azure spend for the same business value. Most teams get this wrong by guessing or copying defaults.\u003c/p\u003e\n\u003cp\u003eHere\u0026rsquo;s the honest assessment: Small nodes with low pod density are easy to replace, but you pay for the privilege. Large nodes with high pod density are efficient, but when they go down, you feel it. For most real-world workloads, start with medium-sized nodes (4-8 vCPUs, 16-32 GB RAM) and target 20-30 pods per node. Don\u0026rsquo;t trust vendor sizing calculators: watch your actual pod usage and adjust. Memory hogs need lower density. Stateless microservices? Push the density, but monitor for pain.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"the-blast-radius-of-a-single-node\"\u003e\u003ca href=\"/posts/cost-optimization-resource-governance-aks/#the-blast-radius-of-a-single-node\" title=\"The Blast Radius Of A Single Node\"\u003eThe Blast Radius Of A Single Node\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eNode size hits your Azure bill directly. A Standard_D4s_v5 (4 vCPU, 16 GB) is about $140/month in West Europe. A D8s_v5 (8 vCPU, 32 GB) is $280/month. If you run 40 pods at 0.5 vCPU and 2 GB RAM each, you can fit them on two D4s_v5 or one D8s_v5. The cost is identical, but the blast radius is not. Lose the big node, lose everything on it. The real cost difference? System overhead. Kubernetes always takes a cut for kubelet, container runtime, and system pods. Small nodes waste more on overhead. Big nodes are more efficient, but you pay in risk.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"node-pool-stratification\"\u003e\u003ca href=\"/posts/cost-optimization-resource-governance-aks/#node-pool-stratification\" title=\"Node-pool stratification\"\u003eNode-pool stratification\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eMixing workloads in a single node pool is a rookie mistake. Batch jobs, web services, databases, and background workers all have different needs. If you lump them together, you guarantee wasted money and operational headaches. Stratify your node pools. Optimize each for cost, performance, and reliability. No exceptions.\u003c/p\u003e\n\u003cp\u003eCreate separate node pools for:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eSystem workloads:\u003c/strong\u003e Small, reliable nodes for kube-system and monitoring\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eProduction services:\u003c/strong\u003e Standard nodes for user-facing apps\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eBatch/background jobs:\u003c/strong\u003e Spot or big nodes for interruptible stuff\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eStateful services:\u003c/strong\u003e Nodes with local SSD for low-latency storage\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eBut here\u0026rsquo;s the catch: Node-pool stratification is useless if you don\u0026rsquo;t enforce it. Use node affinity and taints/tolerations. If you skip this, your pods will land wherever they want, and your cost model is toast.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"enforcing-pool-separation-with-taints\"\u003e\u003ca href=\"/posts/cost-optimization-resource-governance-aks/#enforcing-pool-separation-with-taints\" title=\"Enforcing Pool Separation With Taints\"\u003eEnforcing Pool Separation With Taints\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eExample configuration for a production node pool with cost labels in Terraform:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-hcl\" data-lang=\"hcl\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_kubernetes_cluster_node_pool\u0026#34; \u0026#34;production\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                  \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;production\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  kubernetes_cluster_id\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_kubernetes_cluster\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eaks\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  vm_size\u003c/span\u003e              \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Standard_D4s_v5\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  node_count\u003c/span\u003e           \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"m\"\u003e3\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"n\"\u003etags\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    environment\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;production\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    workload\u003c/span\u003e    \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;web-services\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    cost-center\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;engineering\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    managed-by\u003c/span\u003e  \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;terraform\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"n\"\u003enode_labels\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    \u0026#34;workload-type\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;production\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    \u0026#34;node-pool\u0026#34;\u003c/span\u003e     \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;production\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"n\"\u003enode_taints\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    \u0026#34;workload\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"k\"\u003eproduction\u003c/span\u003e\u003cspan class=\"err\"\u003e:\u003c/span\u003e\u003cspan class=\"k\"\u003eNoSchedule\u003c/span\u003e\u003cspan class=\"err\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_kubernetes_cluster_node_pool\u0026#34; \u0026#34;batch\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                  \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;batch\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  kubernetes_cluster_id\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_kubernetes_cluster\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eaks\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  vm_size\u003c/span\u003e              \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Standard_D8s_v5\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  priority\u003c/span\u003e             \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Spot\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  eviction_policy\u003c/span\u003e      \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Delete\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  spot_max_price\u003c/span\u003e       \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"err\"\u003e-\u003c/span\u003e\u003cspan class=\"m\"\u003e1\u003c/span\u003e\u003cspan class=\"c1\"\u003e  # Pay up to regular price\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  node_count\u003c/span\u003e           \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"m\"\u003e1\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  enable_auto_scaling\u003c/span\u003e  \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"kt\"\u003etrue\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  min_count\u003c/span\u003e            \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"m\"\u003e1\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  max_count\u003c/span\u003e            \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"m\"\u003e5\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"n\"\u003etags\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    environment\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;production\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    workload\u003c/span\u003e    \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;batch-processing\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    cost-center\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;data-engineering\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    managed-by\u003c/span\u003e  \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;terraform\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"n\"\u003enode_labels\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    \u0026#34;workload-type\u0026#34;\u003c/span\u003e        \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;batch\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    \u0026#34;node-pool\u0026#34;\u003c/span\u003e            \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;batch\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    \u0026#34;kubernetes.azure.com/scalesetpriority\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;spot\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"n\"\u003enode_taints\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    \u0026#34;workload\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"k\"\u003ebatch\u003c/span\u003e\u003cspan class=\"err\"\u003e:\u003c/span\u003e\u003cspan class=\"k\"\u003eNoSchedule\u003c/span\u003e\u003cspan class=\"err\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e    \u0026#34;kubernetes.azure.com/scalesetpriority\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"k\"\u003espot\u003c/span\u003e\u003cspan class=\"err\"\u003e:\u003c/span\u003e\u003cspan class=\"k\"\u003eNoSchedule\u003c/span\u003e\u003cspan class=\"err\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eTag your node pools with cost-center and workload. If you don\u0026rsquo;t, finance will treat your AKS bill as a black box, and you\u0026rsquo;ll lose every argument about budget. Azure Cost Management can only help if you give it the data.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"spot-vms-and-reserved-instances\"\u003e\u003ca href=\"/posts/cost-optimization-resource-governance-aks/#spot-vms-and-reserved-instances\" title=\"Spot VMs and reserved instances\"\u003eSpot VMs and reserved instances\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eSpot VMs are the cloud\u0026rsquo;s version of a fire sale: up to 90% off, but you can get kicked out with 30 seconds notice. Use them for workloads that can be interrupted—batch jobs, CI/CD runners, dev environments, stateless background stuff. If you put production or stateful workloads on spot VMs without bulletproof redundancy, you\u0026rsquo;re asking for trouble. The 30-second eviction is real, and Azure doesn\u0026rsquo;t care about your SLAs.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"when-to-commit-to-reserved-instances\"\u003e\u003ca href=\"/posts/cost-optimization-resource-governance-aks/#when-to-commit-to-reserved-instances\" title=\"When To Commit To Reserved Instances\"\u003eWhen To Commit To Reserved Instances\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eReserved instances are the opposite: predictable discounts (up to 72% for 3-year commitments), but you pay whether you use them or not. Only buy what you know you\u0026rsquo;ll need for the next 1-3 years. Overcommit, and you\u0026rsquo;re stuck.\u003c/p\u003e\n\u003cp\u003eCost model comparison for a Standard_D4s_v5 node in West Europe:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003ePay-as-you-go: ~$140/month\u003c/li\u003e\n\u003cli\u003e1-year reserved: ~$100/month (29% discount)\u003c/li\u003e\n\u003cli\u003e3-year reserved: ~$65/month (54% discount)\u003c/li\u003e\n\u003cli\u003eSpot VM (typical): ~$20-40/month (70-85% discount, variable)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eRisk models for spot vs. reserved instances:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eSpot VMs: high discount, high operational risk, suitable for interruptible workloads\u003c/li\u003e\n\u003cli\u003e1-year reserved: moderate discount, low risk, suitable for proven steady-state capacity\u003c/li\u003e\n\u003cli\u003e3-year reserved: high discount, moderate risk (commitment lock-in), suitable for long-term stable workloads\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eAuthor tip: Spot VMs for batch, dev, and test. 1-year reserved for production baseline, but only after 3-6 months of real usage data. Avoid 3-year commitments unless you\u0026rsquo;re absolutely sure. Cloud changes fast: don\u0026rsquo;t lock yourself in.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"resource-requests-and-limits\"\u003e\u003ca href=\"/posts/cost-optimization-resource-governance-aks/#resource-requests-and-limits\" title=\"Resource requests and limits\"\u003eResource requests and limits\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eResource requests and limits are where most teams burn money. Over-requesting is rampant: people set requests based on worst-case guesses, not real data. The result? Nodes look full, but are actually running at 20-30% utilization. Kubernetes blocks new pods, you scale out, and your CFO wonders why the bill keeps climbing.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"under-requesting-hurts-just-as-much\"\u003e\u003ca href=\"/posts/cost-optimization-resource-governance-aks/#under-requesting-hurts-just-as-much\" title=\"Under-Requesting Hurts Just As Much\"\u003eUnder-Requesting Hurts Just As Much\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eUnder-requesting is just as bad: pods get scheduled, but then fight for resources, get throttled, or OOMKilled. Kubernetes thinks there\u0026rsquo;s room, but your workloads suffer.\u003c/p\u003e\n\u003cp\u003eThe fix is not rocket science, but it requires discipline. Monitor actual usage with Azure Monitor or Prometheus. Set requests to the 95th percentile of real usage, not what you wish was true. Example: If a pod averages 200m CPU and 500Mi memory, but spikes to 400m/1Gi, set requests to 250m/600Mi, limits to 500m/1.5Gi. Review and adjust regularly. If you don\u0026rsquo;t, you\u0026rsquo;re just guessing—and paying for it.\u003c/p\u003e\n\u003cp\u003eIncorrect resource settings create a cascading waste problem. Over-requested resources kill node packing, so you run more nodes than you need. More nodes mean more cost, more operational pain, and more complexity. Don\u0026rsquo;t be lazy: measure, set, and review. It\u0026rsquo;s the only way.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"cost-attribution-and-chargeback\"\u003e\u003ca href=\"/posts/cost-optimization-resource-governance-aks/#cost-attribution-and-chargeback\" title=\"Cost attribution and chargeback\"\u003eCost attribution and chargeback\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eCost attribution is not optional. If you can\u0026rsquo;t answer \u0026ldquo;who owns this cost?\u0026rdquo;, you\u0026rsquo;re going to lose every budget discussion. Finance hates black boxes. Tag your node pools, disks, and load balancers with cost-center, team, and workload. If you don\u0026rsquo;t, your AKS bill is just a giant question mark.\u003c/p\u003e\n\u003cp\u003ePractical implementation steps:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eTag all AKS resources (node pools, disks, load balancers) with cost-center, workload, environment, and managed-by tags\u003c/li\u003e\n\u003cli\u003eConfigure Azure Cost Management to group costs by these tags\u003c/li\u003e\n\u003cli\u003eExport cost data monthly and distribute reports to team leads\u003c/li\u003e\n\u003cli\u003eReview high-cost workloads and investigate optimization opportunities\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eKubernetes-native attribution is possible. Use Kubecost or OpenCost to estimate per-pod cost based on real resource requests and node pricing. Aggregate by namespace, label, or deployment. If you don\u0026rsquo;t have this visibility, you\u0026rsquo;re just hoping your spend is \u0026ldquo;reasonable\u0026rdquo;. See the \u003ca href=\"https://docs.kubecost.com/\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eKubecost documentation\u003c/a\u003e and \u003ca href=\"https://www.opencost.io/\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eOpenCost project\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eExample Azure CLI query for cost by tag:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Query AKS costs grouped by cost-center tag for the last 30 days\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz costmanagement query \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --type Usage \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --timeframe MonthToDate \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --scope \u003cspan class=\"s2\"\u003e\u0026#34;/subscriptions/\u0026lt;subscription-id\u0026gt;\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --dataset-aggregation \u003cspan class=\"s1\"\u003e\u0026#39;{\u0026#34;totalCost\u0026#34;:{\u0026#34;name\u0026#34;:\u0026#34;Cost\u0026#34;,\u0026#34;function\u0026#34;:\u0026#34;Sum\u0026#34;}}\u0026#39;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --dataset-grouping \u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Tags\u0026#34;\u003c/span\u003e \u003cspan class=\"nv\"\u003etype\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Tag\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --query \u003cspan class=\"s1\"\u003e\u0026#39;properties.rows[]\u0026#39;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --output table\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Export detailed cost breakdown to CSV\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz costmanagement query \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --type Usage \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --timeframe MonthToDate \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --scope \u003cspan class=\"s2\"\u003e\u0026#34;/subscriptions/\u0026lt;subscription-id\u0026gt;\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --dataset-aggregation \u003cspan class=\"s1\"\u003e\u0026#39;{\u0026#34;totalCost\u0026#34;:{\u0026#34;name\u0026#34;:\u0026#34;Cost\u0026#34;,\u0026#34;function\u0026#34;:\u0026#34;Sum\u0026#34;}}\u0026#39;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --dataset-grouping \u003cspan class=\"nv\"\u003ename\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Tags\u0026#34;\u003c/span\u003e \u003cspan class=\"nv\"\u003etype\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Tag\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --output json \u003cspan class=\"p\"\u003e|\u003c/span\u003e jq -r \u003cspan class=\"s1\"\u003e\u0026#39;.properties.rows[] | @csv\u0026#39;\u003c/span\u003e \u0026gt; aks-costs.csv\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\n\n\n\n\u003ch3 id=\"chargeback-or-showback-first\"\u003e\u003ca href=\"/posts/cost-optimization-resource-governance-aks/#chargeback-or-showback-first\" title=\"Chargeback Or Showback First\"\u003eChargeback Or Showback First\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eChargeback or showback? Chargeback means teams pay for what they use. Showback just shows them the numbers. Start with showback—it\u0026rsquo;s less political. But if you want real accountability, move to chargeback once teams have seen the data and had a chance to optimize. Don\u0026rsquo;t try to force chargeback on day one unless you like chaos.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"practical-checklist\"\u003e\u003ca href=\"/posts/cost-optimization-resource-governance-aks/#practical-checklist\" title=\"Practical checklist\"\u003ePractical checklist\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eChecklist for teams who want to stop wasting money:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eMeasure before optimizing:\u003c/strong\u003e 30 days of real pod usage before you touch requests/limits.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eTag everything:\u003c/strong\u003e Cost-center, workload, environment—no exceptions.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eStratify node pools:\u003c/strong\u003e Production, batch, system—separate or pay the price.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eRight-size nodes:\u003c/strong\u003e 20-30 pods per node for most. Adjust for reality, not theory.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eUse spot VMs for batch:\u003c/strong\u003e 70-85% discounts, but only for workloads that can die anytime.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eReserve baseline capacity:\u003c/strong\u003e 1-year reserved, but only after you know your steady state.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eSet accurate requests:\u003c/strong\u003e 95th percentile of real usage, not guesses.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eEnable autoscaling:\u003c/strong\u003e Let the cluster scale up/down based on pending pods.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eReview monthly:\u003c/strong\u003e Export cost data, find the top spenders, and dig in.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAutomate reporting:\u003c/strong\u003e Monthly cost breakdowns by team, sent automatically. No excuses.\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eCost optimization never ends. Workload patterns shift, prices change, new VM types appear. Review every month. If you don\u0026rsquo;t, your bill will creep up and nobody will notice until it\u0026rsquo;s a crisis.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"conclusion\"\u003e\u003ca href=\"/posts/cost-optimization-resource-governance-aks/#conclusion\" title=\"Conclusion\"\u003eConclusion\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eAKS cost optimization is not about slashing spend after the fact. It\u0026rsquo;s about ruthless visibility, honest governance, and making the right calls up front. If you don\u0026rsquo;t understand pod density, node-pool design, spot VM risk, and resource requests, you\u0026rsquo;re just hoping for a good outcome. Hope is not a strategy.\u003c/p\u003e\n\u003cp\u003eTag everything, export cost data, and make sure the people who control resources see the numbers. Technical optimization plus organizational accountability is the only way to keep your cloud bill from spiraling out of control. Ignore this, and you\u0026rsquo;ll be back here next quarter, wondering where all the money went.\u003c/p\u003e","date_modified":"2026-05-26T10:22:03+02:00","date_published":"2026-02-11T17:00:00+01:00","id":"https://daily-devops.net/posts/cost-optimization-resource-governance-aks/","language":"en","summary":"How to control AKS costs with pod density, node-pool design, spot VMs, and FinOps tagging—without sacrificing reliability or operational control.","tags":["kubernetes","azure","cloud","devops"],"title":"AKS Cost Optimization: Resource Governance That Actually Works","url":"https://daily-devops.net/posts/cost-optimization-resource-governance-aks/"},{"authors":[{"name":"Jendrik Brack","url":"https://daily-devops.net/authors/jendrik/"}],"content_html":"\u003cp\u003eAKS cluster upgrades are routine maintenance, but executing them without dropping traffic or losing state is the operational challenge that separates theory from production reality. Every Kubernetes version upgrade involves replacing nodes, which means evicting pods, draining workloads, and hoping your assumptions about resilience hold true under pressure.\u003c/p\u003e\n\u003cp\u003eI have participated in dozens of AKS upgrades across production clusters ranging from 10 to 500+ nodes. The pattern is consistent: teams that treat upgrades as a checkbox operation eventually experience an outage. Teams that understand the underlying mechanics and configure explicit constraints rarely do.\u003c/p\u003e\n\u003cp\u003eThis article covers the real mechanics: how cordon and drain actually work, why Pod Disruption Budgets exist, and how to orchestrate multi-node-pool rollouts with automation that survives contact with production.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-problem-uncontrolled-node-drains-cause-cascading-failures\"\u003e\u003ca href=\"/posts/cluster-upgrades-zero-downtime-aks/#the-problem-uncontrolled-node-drains-cause-cascading-failures\" title=\"The problem: uncontrolled node drains cause cascading failures\"\u003eThe problem: uncontrolled node drains cause cascading failures\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eWhen you upgrade an AKS cluster, Azure replaces nodes with new VMs running the updated Kubernetes version. That replacement process triggers pod eviction. Without proper controls, evictions happen simultaneously across multiple nodes, stateful workloads lose quorum, and traffic drops because there are no healthy replicas left to serve requests.\u003c/p\u003e\n\u003cp\u003eThe default behavior is optimistic: Kubernetes assumes your workloads are designed for failure. But production workloads are rarely that resilient. Databases need time to transfer leadership, message queues need to flush buffers, and stateless apps still need at least one replica running to handle incoming connections.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003e\u003cstrong\u003eAuthor note:\u003c/strong\u003e\u003c/em\u003e The \u003ca href=\"https://learn.microsoft.com/en-us/azure/aks/upgrade-aks-cluster\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eofficial AKS upgrade documentation\u003c/a\u003e covers the mechanics, but it does not emphasize how quickly things go wrong without proper constraints. I have seen a three-minute upgrade window turn into a two-hour incident because nobody configured Pod Disruption Budgets.\u003c/p\u003e\n\u003cp\u003eUncontrolled drains create several failure modes:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eData loss:\u003c/strong\u003e Stateful workloads evicted before flushing state to disk or replicating to peers.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eService interruption:\u003c/strong\u003e All replicas terminated before new ones become ready.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eCascading failures:\u003c/strong\u003e Dependent services timeout waiting for unavailable backends, triggering retries that amplify load.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe solution is not to avoid upgrades. The solution is to control the eviction process with explicit constraints that match your workload requirements.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"cordon-and-drain-mechanics-what-actually-happens\"\u003e\u003ca href=\"/posts/cluster-upgrades-zero-downtime-aks/#cordon-and-drain-mechanics-what-actually-happens\" title=\"Cordon and drain mechanics: what actually happens\"\u003eCordon and drain mechanics: what actually happens\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eThe Kubernetes eviction API follows a three-step process when draining a node:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eCordon:\u003c/strong\u003e Mark the node as unschedulable. New pods will not be placed on this node, but existing pods continue running.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eEvict:\u003c/strong\u003e Send termination signals to all pods on the node, respecting grace periods and Pod Disruption Budgets (PDBs).\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eWait:\u003c/strong\u003e Block until all pods have terminated or the drain timeout expires.\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eAKS automates this process during upgrades, but you can trigger it manually using kubectl for maintenance or troubleshooting:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl cordon \u0026lt;node-name\u0026gt;\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl drain \u0026lt;node-name\u0026gt; --ignore-daemonsets --delete-emptydir-data\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe \u003ccode\u003e--ignore-daemonsets\u003c/code\u003e flag prevents drain from failing on DaemonSet pods, which are designed to run on every node and will be recreated automatically. The \u003ccode\u003e--delete-emptydir-data\u003c/code\u003e flag allows drain to proceed even if pods use emptyDir volumes, which are ephemeral and will be lost.\u003c/p\u003e\n\u003cp\u003eFor AKS automated upgrades, you can configure the drain behavior per node pool using \u003ca href=\"https://learn.microsoft.com/en-us/azure/aks/upgrade-aks-node-pools-rolling\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003erolling upgrade settings\u003c/a\u003e:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz aks nodepool update \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --resource-group myResourceGroup \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --cluster-name myAKSCluster \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --name myNodePool \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --max-surge 33% \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --drain-timeout \u003cspan class=\"m\"\u003e45\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --node-soak-duration \u003cspan class=\"m\"\u003e5\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe \u003ccode\u003e--drain-timeout\u003c/code\u003e parameter (in minutes) controls how long AKS waits for pods to terminate before force-killing them. The \u003ccode\u003e--node-soak-duration\u003c/code\u003e (in minutes) adds a stabilization period after each node upgrade before proceeding to the next. Microsoft recommends \u003ccode\u003e--max-surge 33%\u003c/code\u003e for production workloads.\u003c/p\u003e\n\u003cp\u003eManual drain remains useful for pre-maintenance validation, testing PDB configurations, or debugging eviction failures before committing to a full cluster upgrade.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"pod-disruption-budgets-the-safety-mechanism-you-should-always-configure\"\u003e\u003ca href=\"/posts/cluster-upgrades-zero-downtime-aks/#pod-disruption-budgets-the-safety-mechanism-you-should-always-configure\" title=\"Pod Disruption Budgets: the safety mechanism you should always configure\"\u003ePod Disruption Budgets: the safety mechanism you should always configure\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eA \u003ca href=\"https://kubernetes.io/docs/tasks/run-application/configure-pdb/\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003ePod Disruption Budget\u003c/a\u003e (PDB) defines the minimum number of pods that must remain available during voluntary disruptions like node drains. PDBs do not prevent involuntary disruptions like node crashes or resource exhaustion, but they block evictions that would violate availability constraints.\u003c/p\u003e\n\u003cp\u003ePDBs are defined using either \u003ccode\u003eminAvailable\u003c/code\u003e or \u003ccode\u003emaxUnavailable\u003c/code\u003e:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eminAvailable:\u003c/strong\u003e The minimum number of pods (or percentage) that must remain running during a disruption.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003emaxUnavailable:\u003c/strong\u003e The maximum number of pods (or percentage) that can be unavailable during a disruption.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eExample PDB for a three-replica deployment that must keep at least two replicas running:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003epolicy/v1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ePodDisruptionBudget\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003emyapp-pdb\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003enamespace\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eproduction\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eminAvailable\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e2\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eselector\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ematchLabels\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003emyapp\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eWith this PDB in place, drain will evict only one pod at a time, waiting for a replacement to become ready before proceeding to the next eviction. If no replacement becomes ready (for example, due to resource constraints or image pull failures), the drain blocks until the timeout expires.\u003c/p\u003e\n\u003cp\u003ePDBs are particularly critical for:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eStateful workloads:\u003c/strong\u003e Databases, message queues, and distributed systems that require quorum.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eLow-replica deployments:\u003c/strong\u003e Services with two or three replicas where losing one pod reduces capacity significantly.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eLong startup times:\u003c/strong\u003e Workloads that take minutes to initialize and become ready.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003ePractical PDB configuration advice:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eSet \u003ccode\u003eminAvailable: 1\u003c/code\u003e for stateless services with two replicas.\u003c/li\u003e\n\u003cli\u003eSet \u003ccode\u003eminAvailable: N-1\u003c/code\u003e for N-replica stateful services that tolerate one failure (for example, three-node etcd allows \u003ccode\u003eminAvailable: 2\u003c/code\u003e).\u003c/li\u003e\n\u003cli\u003eAvoid \u003ccode\u003eminAvailable: N\u003c/code\u003e (all replicas), which blocks drain indefinitely and prevents upgrades.\u003c/li\u003e\n\u003cli\u003eUse percentages for large replica counts: \u003ccode\u003eminAvailable: 75%\u003c/code\u003e for a 10-replica deployment allows up to 2-3 pods to be evicted simultaneously.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eAuthor tip: Before any upgrade, run \u003ccode\u003ekubectl get pdb -A\u003c/code\u003e and verify that no PDB has \u003ccode\u003eALLOWED DISRUPTIONS\u003c/code\u003e showing zero. A PDB with zero allowed disruptions will block node drain indefinitely, and your upgrade will hang until the drain timeout expires or you manually intervene.\u003c/p\u003e\n\u003cp\u003ePDBs only apply to voluntary disruptions. Node failures ignore PDBs and evict all pods immediately.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"workload-categories-stateless-stateful-daemonsets\"\u003e\u003ca href=\"/posts/cluster-upgrades-zero-downtime-aks/#workload-categories-stateless-stateful-daemonsets\" title=\"Workload categories: stateless, stateful, DaemonSets\"\u003eWorkload categories: stateless, stateful, DaemonSets\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eDifferent workload types require different upgrade strategies. A one-size-fits-all approach causes either unnecessary downtime (overly conservative) or unexpected failures (overly aggressive).\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"stateless-workloads\"\u003e\u003ca href=\"/posts/cluster-upgrades-zero-downtime-aks/#stateless-workloads\" title=\"Stateless workloads\"\u003eStateless workloads\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eStateless services like web frontends, API gateways, and workers can tolerate rapid eviction as long as at least one replica remains available. Configure PDBs with \u003ccode\u003eminAvailable: 1\u003c/code\u003e or \u003ccode\u003emaxUnavailable: N-1\u003c/code\u003e to allow fast rollouts while maintaining service availability.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"stateful-workloads\"\u003e\u003ca href=\"/posts/cluster-upgrades-zero-downtime-aks/#stateful-workloads\" title=\"Stateful workloads\"\u003eStateful workloads\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eDatabases, message queues, and distributed storage systems require careful sequencing. Evicting multiple replicas simultaneously can cause quorum loss, split-brain scenarios, or data corruption.\u003c/p\u003e\n\u003cp\u003eBest practices for stateful workloads:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eSet conservative PDBs that preserve quorum (for example, \u003ccode\u003eminAvailable: 2\u003c/code\u003e for a three-node cluster).\u003c/li\u003e\n\u003cli\u003eConfigure long grace periods (60+ seconds) to allow state transfer and leadership handoff.\u003c/li\u003e\n\u003cli\u003eUse StatefulSets with proper readiness probes to ensure new replicas are fully initialized before old ones are terminated.\u003c/li\u003e\n\u003cli\u003eTest upgrade scenarios in staging with realistic data volumes and latency.\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch3 id=\"daemonsets\"\u003e\u003ca href=\"/posts/cluster-upgrades-zero-downtime-aks/#daemonsets\" title=\"DaemonSets\"\u003eDaemonSets\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eDaemonSets run exactly one pod per node (or per matching node). Examples include logging agents, monitoring exporters, and network plugins. Draining a node automatically terminates the DaemonSet pod, and the pod is recreated on the new node after upgrade.\u003c/p\u003e\n\u003cp\u003eDaemonSets do not require PDBs because they are designed to tolerate single-node failures. Use the \u003ccode\u003e--ignore-daemonsets\u003c/code\u003e flag during manual drain to skip these pods.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"multi-node-pool-rollout-strategies-graduated-risk-management\"\u003e\u003ca href=\"/posts/cluster-upgrades-zero-downtime-aks/#multi-node-pool-rollout-strategies-graduated-risk-management\" title=\"Multi-node-pool rollout strategies: graduated risk management\"\u003eMulti-node-pool rollout strategies: graduated risk management\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eAKS supports multiple node pools within a single cluster. Each pool can have different VM sizes, availability zones, and upgrade schedules. Multi-node-pool architectures enable graduated rollouts that reduce risk by upgrading non-critical workloads first.\u003c/p\u003e\n\u003cp\u003eRecommended upgrade sequence:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eDev/test pools first:\u003c/strong\u003e Upgrade node pools running non-production workloads to validate the new Kubernetes version and catch compatibility issues early.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eStateless application pools:\u003c/strong\u003e Upgrade pools running stateless services that can tolerate brief capacity reductions.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eStateful application pools last:\u003c/strong\u003e Upgrade pools running databases and stateful services only after validating the rollout on stateless workloads.\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eExample multi-pool upgrade using Azure CLI:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"cp\"\u003e#!/bin/bash\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eset\u003c/span\u003e -euo pipefail\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eRESOURCE_GROUP\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;myResourceGroup\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eCLUSTER_NAME\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;myAKSCluster\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eTARGET_VERSION\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;1.29.2\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Configure rolling upgrade settings for production safety\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eMAX_SURGE\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;33%\u0026#34;\u003c/span\u003e        \u003cspan class=\"c1\"\u003e# Microsoft recommended for production\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eDRAIN_TIMEOUT\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;45\u0026#34;\u003c/span\u003e     \u003cspan class=\"c1\"\u003e# Minutes to wait for pod eviction\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eNODE_SOAK\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;5\u0026#34;\u003c/span\u003e          \u003cspan class=\"c1\"\u003e# Minutes to stabilize after each node\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Upgrade control plane first (does not affect workloads)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Upgrading control plane to \u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eTARGET_VERSION\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e...\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz aks upgrade \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --resource-group \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$RESOURCE_GROUP\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --name \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$CLUSTER_NAME\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --kubernetes-version \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$TARGET_VERSION\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --control-plane-only \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --yes\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Upgrade node pools in sequence: system -\u0026gt; stateless -\u0026gt; stateful\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eNODE_POOLS\u003c/span\u003e\u003cspan class=\"o\"\u003e=(\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;system\u0026#34;\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;stateless\u0026#34;\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;stateful\u0026#34;\u003c/span\u003e\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003efor\u003c/span\u003e POOL in \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eNODE_POOLS\u003c/span\u003e\u003cspan class=\"p\"\u003e[@]\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003edo\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Upgrading node pool: \u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003ePOOL\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e...\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"c1\"\u003e# Verify current node count and health\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nv\"\u003eCURRENT_COUNT\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"k\"\u003e$(\u003c/span\u003eaz aks nodepool show \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --resource-group \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$RESOURCE_GROUP\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --cluster-name \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$CLUSTER_NAME\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --name \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$POOL\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --query count -o tsv\u003cspan class=\"k\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Current node count for \u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003ePOOL\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e: \u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eCURRENT_COUNT\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"c1\"\u003e# Configure rolling upgrade settings before upgrade\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  az aks nodepool update \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --resource-group \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$RESOURCE_GROUP\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --cluster-name \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$CLUSTER_NAME\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --name \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$POOL\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --max-surge \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$MAX_SURGE\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --drain-timeout \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$DRAIN_TIMEOUT\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --node-soak-duration \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$NODE_SOAK\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"c1\"\u003e# Upgrade node pool\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  az aks nodepool upgrade \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --resource-group \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$RESOURCE_GROUP\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --cluster-name \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$CLUSTER_NAME\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --name \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$POOL\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --kubernetes-version \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$TARGET_VERSION\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --yes\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"c1\"\u003e# Wait for upgrade to complete\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Waiting for \u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003ePOOL\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e upgrade to complete...\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  az aks nodepool \u003cspan class=\"nb\"\u003ewait\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --resource-group \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$RESOURCE_GROUP\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --cluster-name \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$CLUSTER_NAME\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --name \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$POOL\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --updated\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"c1\"\u003e# Verify upgraded node count matches original\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nv\"\u003eUPGRADED_COUNT\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"k\"\u003e$(\u003c/span\u003eaz aks nodepool show \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --resource-group \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$RESOURCE_GROUP\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --cluster-name \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$CLUSTER_NAME\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --name \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$POOL\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    --query count -o tsv\u003cspan class=\"k\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"o\"\u003e[\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$CURRENT_COUNT\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e !\u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$UPGRADED_COUNT\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003ethen\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;ERROR: Node count mismatch for \u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003ePOOL\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e. Expected \u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eCURRENT_COUNT\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e, got \u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eUPGRADED_COUNT\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nb\"\u003eexit\u003c/span\u003e \u003cspan class=\"m\"\u003e1\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003efi\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Pool \u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003ePOOL\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e upgraded successfully.\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;---\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003edone\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;All node pools upgraded to \u003c/span\u003e\u003cspan class=\"si\"\u003e${\u003c/span\u003e\u003cspan class=\"nv\"\u003eTARGET_VERSION\u003c/span\u003e\u003cspan class=\"si\"\u003e}\u003c/span\u003e\u003cspan class=\"s2\"\u003e.\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis script upgrades the control plane first (which is a non-disruptive operation), then upgrades each node pool sequentially, validating node count before and after each upgrade to detect unexpected node losses.\u003c/p\u003e\n\u003cp\u003eKey operational notes:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eControl plane upgrades are non-disruptive:\u003c/strong\u003e The control plane upgrade updates the Kubernetes API server and controllers but does not affect running workloads. Only node pool upgrades trigger pod evictions.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eOne node pool at a time:\u003c/strong\u003e Upgrading multiple pools simultaneously multiplies risk. Sequential upgrades allow you to catch issues early and halt the rollout before affecting critical workloads.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eValidate before proceeding:\u003c/strong\u003e Check pod health, replica counts, and application metrics after each pool upgrade. Use kubectl, Azure Monitor, or Prometheus to verify that workloads are stable before moving to the next pool.\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch2 id=\"planned-maintenance-windows-scheduling-upgrades-safely\"\u003e\u003ca href=\"/posts/cluster-upgrades-zero-downtime-aks/#planned-maintenance-windows-scheduling-upgrades-safely\" title=\"Planned maintenance windows: scheduling upgrades safely\"\u003ePlanned maintenance windows: scheduling upgrades safely\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eFor clusters with \u003ca href=\"https://learn.microsoft.com/en-us/azure/aks/auto-upgrade-cluster\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eautomatic upgrades\u003c/a\u003e enabled, AKS supports \u003ca href=\"https://learn.microsoft.com/en-us/azure/aks/planned-maintenance\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eplanned maintenance windows\u003c/a\u003e to control when upgrades occur. This prevents upgrades from starting during peak traffic periods.\u003c/p\u003e\n\u003cp\u003eConfigure a weekly maintenance window using Azure CLI:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz aks maintenanceconfiguration add \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --resource-group myResourceGroup \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --cluster-name myAKSCluster \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --name aksManagedAutoUpgradeSchedule \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --schedule-type Weekly \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --day-of-week Saturday \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --start-time 02:00 \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --duration \u003cspan class=\"m\"\u003e4\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eMicrosoft recommends a minimum four-hour maintenance window to ensure upgrades complete without interruption. Combine this with the \u003ccode\u003estable\u003c/code\u003e auto-upgrade channel, which targets the previous minor version with latest patches, for a balance between staying current and avoiding bleeding-edge issues.\u003c/p\u003e\n\u003cp\u003eFor production clusters, I prefer manual upgrades with planned maintenance windows as a safety net. The automation handles the scheduling, but I control when the actual upgrade starts.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"automation-and-rollback-scripting-safe-upgrades\"\u003e\u003ca href=\"/posts/cluster-upgrades-zero-downtime-aks/#automation-and-rollback-scripting-safe-upgrades\" title=\"Automation and rollback: scripting safe upgrades\"\u003eAutomation and rollback: scripting safe upgrades\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eAutomation reduces human error during upgrades, but only if the automation includes validation and rollback capabilities. A fully automated upgrade script should:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eValidate current cluster state (replica counts, PDB configurations, node health).\u003c/li\u003e\n\u003cli\u003eUpgrade in stages with validation checkpoints.\u003c/li\u003e\n\u003cli\u003eDetect failures and halt or rollback automatically.\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003ePractical validation checks before upgrade:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"cp\"\u003e#!/bin/bash\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eset\u003c/span\u003e -euo pipefail\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;=== Pre-Upgrade Validation ===\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Check available Kubernetes versions\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Available upgrades:\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz aks get-upgrades \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --resource-group myResourceGroup \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --name myAKSCluster \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --output table\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Verify all nodes are ready\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eNOTREADY\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"k\"\u003e$(\u003c/span\u003ekubectl get nodes --no-headers \u003cspan class=\"p\"\u003e|\u003c/span\u003e grep -v \u003cspan class=\"s2\"\u003e\u0026#34; Ready \u0026#34;\u003c/span\u003e \u003cspan class=\"p\"\u003e|\u003c/span\u003e wc -l\u003cspan class=\"k\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"o\"\u003e[\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$NOTREADY\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e -gt \u003cspan class=\"m\"\u003e0\u003c/span\u003e \u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003ethen\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;ERROR: \u003c/span\u003e\u003cspan class=\"nv\"\u003e$NOTREADY\u003c/span\u003e\u003cspan class=\"s2\"\u003e nodes are not ready. Aborting upgrade.\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  kubectl get nodes \u003cspan class=\"p\"\u003e|\u003c/span\u003e grep -v \u003cspan class=\"s2\"\u003e\u0026#34; Ready \u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nb\"\u003eexit\u003c/span\u003e \u003cspan class=\"m\"\u003e1\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003efi\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;✓ All nodes ready\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Check for PDBs that would block drain\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eBLOCKED\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"k\"\u003e$(\u003c/span\u003ekubectl get pdb -A -o \u003cspan class=\"nv\"\u003ejsonpath\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;{range .items[?(@.status.disruptionsAllowed==0)]}{.metadata.namespace}/{.metadata.name}{\u0026#34;\\n\u0026#34;}{end}\u0026#39;\u003c/span\u003e\u003cspan class=\"k\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"o\"\u003e[\u003c/span\u003e -n \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$BLOCKED\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003ethen\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;WARNING: The following PDBs have zero allowed disruptions:\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$BLOCKED\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;These will block node drain. Verify this is intentional.\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003efi\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Verify PDBs exist for critical namespaces\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003efor\u003c/span\u003e NS in production\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003edo\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nv\"\u003ePDBS\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"k\"\u003e$(\u003c/span\u003ekubectl get pdb -n \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$NS\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e --no-headers 2\u0026gt;/dev/null \u003cspan class=\"p\"\u003e|\u003c/span\u003e wc -l\u003cspan class=\"k\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"o\"\u003e[\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$PDBS\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e -eq \u003cspan class=\"m\"\u003e0\u003c/span\u003e \u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003ethen\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;WARNING: No PDBs configured in namespace \u003c/span\u003e\u003cspan class=\"nv\"\u003e$NS\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003eelse\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;✓ \u003c/span\u003e\u003cspan class=\"nv\"\u003e$PDBS\u003c/span\u003e\u003cspan class=\"s2\"\u003e PDBs configured in \u003c/span\u003e\u003cspan class=\"nv\"\u003e$NS\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003efi\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003edone\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Verify critical deployments have sufficient replicas\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Checking critical deployments...\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003efor\u003c/span\u003e DEPLOYMENT in myapp-frontend myapp-backend\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003edo\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nv\"\u003eREPLICAS\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"k\"\u003e$(\u003c/span\u003ekubectl get deployment \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$DEPLOYMENT\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e -n production -o \u003cspan class=\"nv\"\u003ejsonpath\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;{.status.readyReplicas}\u0026#39;\u003c/span\u003e 2\u0026gt;/dev/null \u003cspan class=\"o\"\u003e||\u003c/span\u003e \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;0\u0026#34;\u003c/span\u003e\u003cspan class=\"k\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"o\"\u003e[\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$REPLICAS\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e -lt \u003cspan class=\"m\"\u003e2\u003c/span\u003e \u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003ethen\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;ERROR: \u003c/span\u003e\u003cspan class=\"nv\"\u003e$DEPLOYMENT\u003c/span\u003e\u003cspan class=\"s2\"\u003e has fewer than 2 ready replicas (\u003c/span\u003e\u003cspan class=\"nv\"\u003e$REPLICAS\u003c/span\u003e\u003cspan class=\"s2\"\u003e). Aborting upgrade.\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nb\"\u003eexit\u003c/span\u003e \u003cspan class=\"m\"\u003e1\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"k\"\u003efi\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;✓ \u003c/span\u003e\u003cspan class=\"nv\"\u003e$DEPLOYMENT\u003c/span\u003e\u003cspan class=\"s2\"\u003e: \u003c/span\u003e\u003cspan class=\"nv\"\u003e$REPLICAS\u003c/span\u003e\u003cspan class=\"s2\"\u003e replicas ready\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003edone\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;=== Validation Complete ===\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eRollback is more complex. AKS does not support in-place downgrades. If an upgrade introduces breaking changes, the rollback path involves:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eRestoring from a snapshot or backup (for stateful workloads).\u003c/li\u003e\n\u003cli\u003eDeploying a new node pool with the previous Kubernetes version.\u003c/li\u003e\n\u003cli\u003eMigrating workloads to the new pool.\u003c/li\u003e\n\u003cli\u003eDeleting the upgraded pool.\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eThis process is slow and disruptive, which is why validation before upgrade is critical. Test upgrades in staging, validate application compatibility with the new Kubernetes version, and maintain rollback procedures even if you hope never to use them.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"practical-recommendations\"\u003e\u003ca href=\"/posts/cluster-upgrades-zero-downtime-aks/#practical-recommendations\" title=\"Practical recommendations\"\u003ePractical recommendations\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eBased on production experience, the following practices reduce upgrade-related failures:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eAlways configure PDBs for production workloads.\u003c/strong\u003e Even stateless services benefit from \u003ccode\u003eminAvailable: 1\u003c/code\u003e to prevent simultaneous eviction of all replicas.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eTest upgrades in staging first.\u003c/strong\u003e Validate application compatibility, verify PDB behavior, and measure upgrade duration under realistic load.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eUpgrade during low-traffic windows.\u003c/strong\u003e Even with proper PDBs, upgrades reduce available capacity. Schedule upgrades when traffic is lowest to minimize user impact.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eMonitor during upgrades.\u003c/strong\u003e Track pod eviction events, replica counts, and application error rates. Use Azure Monitor, Prometheus, or your existing observability stack to detect issues early.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAutomate validation, not just execution.\u003c/strong\u003e Scripts that upgrade without validation are worse than manual upgrades because they fail faster and more completely.\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch2 id=\"conclusion\"\u003e\u003ca href=\"/posts/cluster-upgrades-zero-downtime-aks/#conclusion\" title=\"Conclusion\"\u003eConclusion\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eAKS cluster upgrades are unavoidable, but service disruption is not. Cordon and drain mechanics provide the foundation, Pod Disruption Budgets enforce availability constraints, and multi-node-pool rollouts allow graduated risk management. Combine these tools with validation-driven automation, and zero-downtime upgrades become reliable rather than aspirational.\u003c/p\u003e\n\u003cp\u003eThe key insight: upgrades succeed when the automation respects the constraints of your workloads, not when the automation assumes resilience that does not exist.\u003c/p\u003e\n\u003cp\u003eStart with the basics: configure PDBs for every production workload, set \u003ccode\u003e--max-surge 33%\u003c/code\u003e on your node pools, and always upgrade control plane before node pools. Test in staging first. Monitor during the upgrade. These practices are not optional for production clusters.\u003c/p\u003e\n","date_modified":"2026-05-26T10:22:03+02:00","date_published":"2026-01-28T17:00:00+01:00","id":"https://daily-devops.net/posts/cluster-upgrades-zero-downtime-aks/","language":"en","summary":"Master AKS upgrades with cordon/drain mechanics, Pod Disruption Budgets, multi-node-pool rollouts, and automation for zero-downtime operations.\n","tags":["aks","azure","kubernetes","cloud","devops","operations","reliability"],"title":"AKS Cluster Upgrades: Zero-Downtime Operations That Actually Work\n","url":"https://daily-devops.net/posts/cluster-upgrades-zero-downtime-aks/"},{"authors":[{"name":"Jendrik Brack","url":"https://daily-devops.net/authors/jendrik/"}],"content_html":"\u003cp\u003eYour pod authenticates successfully in staging. Production fails with a cryptic 401. The service account exists, the managed identity is configured, Azure RBAC looks correct. Three hours later, you discover the federated credential subject doesn\u0026rsquo;t match the namespace you deployed to.\u003c/p\u003e\n\u003cp\u003eThis is the new reality of AKS authentication. Workload Identity Federation eliminates the credential lifecycle nightmares we dealt with for years: secrets expiring at 2 AM, credentials leaking into logs, service principals with subscription-wide access because someone took a shortcut during initial setup. But it replaces those problems with configuration complexity that spans three separate RBAC systems.\u003c/p\u003e\n\u003cp\u003eThis article covers what actually breaks: where credentials still leak despite federation, how Kubernetes RBAC, Azure RBAC, and Azure AD permissions interact (and fail), and the validation patterns that catch misconfigurations before they become production incidents.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-problem-with-pod-level-credentials\"\u003e\u003ca href=\"/posts/pod-identity-access-control-aks/#the-problem-with-pod-level-credentials\" title=\"The problem with pod-level credentials\"\u003eThe problem with pod-level credentials\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eTraditional approaches to AKS pod authentication relied on passing Azure service principal credentials directly to workloads. Teams stored client secrets in Kubernetes secrets, mounted them as environment variables, and hoped developers wouldn\u0026rsquo;t log them accidentally. This pattern had obvious weaknesses:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCredential lifecycle management:\u003c/strong\u003e Secrets expire. When they do, workloads fail unpredictably. Rotation requires redeploying pods or restarting containers, creating operational overhead and deployment windows for what should be a background task.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eBlast radius:\u003c/strong\u003e A compromised pod credential grants full access to whatever Azure resources the service principal can reach. There\u0026rsquo;s no inherent scoping to the pod, namespace, or even cluster. The credential works from anywhere—your laptop, an attacker\u0026rsquo;s server, a developer\u0026rsquo;s local environment.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eObservability gaps:\u003c/strong\u003e When authentication fails, you get a generic 401. Was the secret wrong? Expired? Never properly mounted? The pod doesn\u0026rsquo;t know, and your logs won\u0026rsquo;t tell you until you start instrumenting credential fetching yourself.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAudit trails:\u003c/strong\u003e Service principal credentials obscure which workload actually made an Azure API call. All requests appear to come from the same identity, making it impossible to trace blast radius during incidents or satisfy compliance requirements for request attribution.\u003c/p\u003e\n\u003cp\u003eWorkload Identity Federation addresses these architectural issues, but introduces new operational complexity.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"workload-identity-vs-managed-identity-vs-service-accounts\"\u003e\u003ca href=\"/posts/pod-identity-access-control-aks/#workload-identity-vs-managed-identity-vs-service-accounts\" title=\"Workload Identity vs. Managed Identity vs. Service Accounts\"\u003eWorkload Identity vs. Managed Identity vs. Service Accounts\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eUnderstanding when to use each identity type prevents misconfiguration and operational failures.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"workload-identity-federation\"\u003e\u003ca href=\"/posts/pod-identity-access-control-aks/#workload-identity-federation\" title=\"Workload Identity Federation\"\u003eWorkload Identity Federation\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eWorkload Identity Federation maps Kubernetes service accounts to Azure AD identities through OpenID Connect (OIDC). The AKS cluster acts as an OIDC issuer, pods authenticate using their service account tokens, and Azure AD validates those tokens to grant Azure resource access.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWhen to use it:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003ePods need access to Azure resources (Storage, Key Vault, Cosmos DB, etc.)\u003c/li\u003e\n\u003cli\u003eYou want credential-free authentication without managing secrets\u003c/li\u003e\n\u003cli\u003eYou need per-workload identity isolation within the same cluster\u003c/li\u003e\n\u003cli\u003eCompliance requires audit trails showing which pod made which Azure API call\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eWhen not to use it:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003ePods only communicate within Kubernetes—use standard Kubernetes service accounts\u003c/li\u003e\n\u003cli\u003eYou\u0026rsquo;re running on non-AKS infrastructure—Managed Identity or service principals may be better fits\u003c/li\u003e\n\u003cli\u003eYour workload runs outside of Azure AD tenant boundaries\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch3 id=\"managed-identity\"\u003e\u003ca href=\"/posts/pod-identity-access-control-aks/#managed-identity\" title=\"Managed Identity\"\u003eManaged Identity\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eManaged Identities work at the node or cluster level. The Azure platform manages credentials automatically, and workloads running on those resources inherit the identity.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWhen to use it:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eNode-level access patterns (monitoring agents, logging daemons, backup solutions)\u003c/li\u003e\n\u003cli\u003eCluster-wide operations (DNS, ingress controllers, cluster autoscaler)\u003c/li\u003e\n\u003cli\u003eWorkloads where per-pod identity isolation isn\u0026rsquo;t required\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eWhen not to use it:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eMultiple workloads on the same node need different Azure permissions\u003c/li\u003e\n\u003cli\u003eYou need audit trails distinguishing between pod-level actions\u003c/li\u003e\n\u003cli\u003eYou\u0026rsquo;re implementing least privilege at the workload level, not the node level\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch3 id=\"kubernetes-service-accounts\"\u003e\u003ca href=\"/posts/pod-identity-access-control-aks/#kubernetes-service-accounts\" title=\"Kubernetes Service Accounts\"\u003eKubernetes Service Accounts\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eService accounts provide identity within Kubernetes. They control access to Kubernetes API resources through RBAC, but have no inherent Azure permissions.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWhen to use them:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eWorkloads that only interact with Kubernetes APIs\u003c/li\u003e\n\u003cli\u003eRBAC policies scoped to namespaces, pods, or specific Kubernetes resources\u003c/li\u003e\n\u003cli\u003eAs the foundation for Workload Identity Federation (every federated identity maps to a service account)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eWhen not to use them:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eWorkloads need Azure resource access—layer Workload Identity Federation on top\u003c/li\u003e\n\u003cli\u003eCross-cluster identity is required—service accounts are cluster-scoped\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch2 id=\"rbac-layering-where-permissions-actually-fail\"\u003e\u003ca href=\"/posts/pod-identity-access-control-aks/#rbac-layering-where-permissions-actually-fail\" title=\"RBAC layering: Where permissions actually fail\"\u003eRBAC layering: Where permissions actually fail\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eAKS identity and access control spans three separate RBAC systems. Each layer has different failure modes, and misalignment between layers causes the majority of production authentication failures.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"layer-1-kubernetes-rbac\"\u003e\u003ca href=\"/posts/pod-identity-access-control-aks/#layer-1-kubernetes-rbac\" title=\"Layer 1: Kubernetes RBAC\"\u003eLayer 1: Kubernetes RBAC\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eKubernetes RBAC controls access to Kubernetes API resources. This includes pods, services, deployments, config maps, and secrets. Permissions are scoped to namespaces or cluster-wide, defined through roles and role bindings.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCommon failures:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eService account lacks permission to read secrets it needs to mount\u003c/li\u003e\n\u003cli\u003eDeployment controller can\u0026rsquo;t create pods because the service account is missing \u003ccode\u003epods/create\u003c/code\u003e permissions\u003c/li\u003e\n\u003cli\u003eMonitoring workload can\u0026rsquo;t list nodes because it\u0026rsquo;s assigned a namespace-scoped role instead of a cluster role\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eValidation:\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Check what a service account can do\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl auth can-i --list --as\u003cspan class=\"o\"\u003e=\u003c/span\u003esystem:serviceaccount:NAMESPACE:SERVICE_ACCOUNT_NAME\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Check specific permission\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl auth can-i get secrets --as\u003cspan class=\"o\"\u003e=\u003c/span\u003esystem:serviceaccount:production:my-workload\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\n\n\n\n\u003ch3 id=\"layer-2-azure-rbac\"\u003e\u003ca href=\"/posts/pod-identity-access-control-aks/#layer-2-azure-rbac\" title=\"Layer 2: Azure RBAC\"\u003eLayer 2: Azure RBAC\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eAzure RBAC controls access to Azure resources. Even with Workload Identity properly configured, pods fail to access Azure resources if the federated identity lacks appropriate Azure role assignments.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCommon failures:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eWorkload Identity is configured correctly, but the Azure identity has no role assignments—pod can\u0026rsquo;t read from Storage\u003c/li\u003e\n\u003cli\u003eIdentity has \u003ccode\u003eReader\u003c/code\u003e role when it needs \u003ccode\u003eStorage Blob Data Reader\u003c/code\u003e—Azure API returns 403\u003c/li\u003e\n\u003cli\u003eRole assigned at wrong scope (subscription vs resource group vs specific resource)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eValidation:\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# List role assignments for a managed identity\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz role assignment list --assignee \u0026lt;managed-identity-client-id\u0026gt; --output table\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Verify specific permission\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz role assignment list --assignee \u0026lt;managed-identity-client-id\u0026gt; \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --scope /subscriptions/\u0026lt;sub-id\u0026gt;/resourceGroups/\u0026lt;rg\u0026gt;/providers/Microsoft.Storage/storageAccounts/\u0026lt;account\u0026gt;\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\n\n\n\n\u003ch3 id=\"layer-3-azure-ad-permissions\"\u003e\u003ca href=\"/posts/pod-identity-access-control-aks/#layer-3-azure-ad-permissions\" title=\"Layer 3: Azure AD permissions\"\u003eLayer 3: Azure AD permissions\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eSome Azure services require Azure AD directory permissions in addition to Azure RBAC. Microsoft Graph API calls, reading Azure AD groups, and certain Key Vault operations require directory-level permissions that aren\u0026rsquo;t managed through RBAC.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCommon failures:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eWorkload can authenticate to Azure AD but can\u0026rsquo;t call Graph API—missing \u003ccode\u003eUser.Read.All\u003c/code\u003e directory permission\u003c/li\u003e\n\u003cli\u003eKey Vault access configured with access policies instead of RBAC, but identity isn\u0026rsquo;t in the access policy list\u003c/li\u003e\n\u003cli\u003eCross-tenant scenarios where the identity exists in a different Azure AD tenant than the resource\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eValidation:\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Check Azure AD application permissions (if using app registration)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz ad app permission list --id \u0026lt;app-id\u0026gt;\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# For Key Vault access policies\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaz keyvault show --name \u0026lt;vault-name\u0026gt; --query properties.accessPolicies\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\n\n\n\n\u003ch2 id=\"common-misconfigurations-that-lead-to-security-breaches\"\u003e\u003ca href=\"/posts/pod-identity-access-control-aks/#common-misconfigurations-that-lead-to-security-breaches\" title=\"Common misconfigurations that lead to security breaches\"\u003eCommon misconfigurations that lead to security breaches\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eWorkload Identity Federation reduces credential exposure, but doesn\u0026rsquo;t eliminate configuration mistakes that create security vulnerabilities.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"over-permissioned-service-principals\"\u003e\u003ca href=\"/posts/pod-identity-access-control-aks/#over-permissioned-service-principals\" title=\"Over-permissioned service principals\"\u003eOver-permissioned service principals\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eTeams often grant broad permissions to simplify initial setup, then never revisit those permissions. A workload that only needs to read from one storage container ends up with \u003ccode\u003eContributor\u003c/code\u003e on the entire subscription.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMitigation:\u003c/strong\u003e Start with minimal permissions. Grant access to specific resources, not resource groups or subscriptions. Use managed identities with RBAC roles scoped to individual blobs, queues, or Key Vault secrets rather than blanket \u003ccode\u003eContributor\u003c/code\u003e or \u003ccode\u003eOwner\u003c/code\u003e roles.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"credential-exposure-in-logs-and-traces\"\u003e\u003ca href=\"/posts/pod-identity-access-control-aks/#credential-exposure-in-logs-and-traces\" title=\"Credential exposure in logs and traces\"\u003eCredential exposure in logs and traces\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eEven with Workload Identity, tokens can leak. Application logging frameworks sometimes log HTTP headers, distributed tracing may capture authorization headers, and crash dumps may contain in-memory tokens.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMitigation:\u003c/strong\u003e Configure logging libraries to redact authorization headers. Review telemetry configurations to ensure tokens aren\u0026rsquo;t captured in traces. Use structured logging with explicit field filtering rather than logging entire request objects.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"identity-drift-between-environments\"\u003e\u003ca href=\"/posts/pod-identity-access-control-aks/#identity-drift-between-environments\" title=\"Identity drift between environments\"\u003eIdentity drift between environments\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eDevelopment clusters use one set of identities, staging uses another, production uses a third. Workloads behave differently across environments because the underlying identities have different permissions.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMitigation:\u003c/strong\u003e Use infrastructure as code (Terraform, Bicep, ARM) to define identities and role assignments consistently. Version control your identity configurations alongside application deployments. Validate permissions in CI/CD pipelines before deploying to production.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"missing-federation-trust-relationships\"\u003e\u003ca href=\"/posts/pod-identity-access-control-aks/#missing-federation-trust-relationships\" title=\"Missing federation trust relationships\"\u003eMissing federation trust relationships\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eWorkload Identity requires a trust relationship between the Kubernetes service account and the Azure managed identity. If the federated credential isn\u0026rsquo;t configured, authentication fails silently—the pod gets a valid Kubernetes token that Azure AD rejects.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMitigation:\u003c/strong\u003e Automate federated credential creation as part of your cluster provisioning process. Validate that service account annotations match the correct Azure identity. Use admission controllers to enforce annotation standards and prevent deployment of workloads with missing or incorrect identity configurations.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"validation-patterns-how-to-audit-identity-configurations-safely\"\u003e\u003ca href=\"/posts/pod-identity-access-control-aks/#validation-patterns-how-to-audit-identity-configurations-safely\" title=\"Validation patterns: How to audit identity configurations safely\"\u003eValidation patterns: How to audit identity configurations safely\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eProactive validation catches misconfigurations before they cause production failures.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"pre-deployment-validation\"\u003e\u003ca href=\"/posts/pod-identity-access-control-aks/#pre-deployment-validation\" title=\"Pre-deployment validation\"\u003ePre-deployment validation\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eBefore deploying a workload, validate that all three RBAC layers are correctly configured:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eKubernetes service account exists and has necessary Kubernetes RBAC permissions\u003c/li\u003e\n\u003cli\u003eAzure managed identity exists and has federated credential linking to the service account\u003c/li\u003e\n\u003cli\u003eAzure managed identity has required Azure RBAC role assignments on target resources\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003e\u003cstrong\u003eExample validation script (Bash):\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"cp\"\u003e#!/bin/bash\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eset\u003c/span\u003e -e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eNAMESPACE\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;production\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eSERVICE_ACCOUNT\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;my-workload\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eMANAGED_IDENTITY_CLIENT_ID\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;00000000-0000-0000-0000-000000000000\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eSTORAGE_ACCOUNT_ID\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;/subscriptions/\u0026lt;sub-id\u0026gt;/resourceGroups/\u0026lt;rg\u0026gt;/providers/Microsoft.Storage/storageAccounts/\u0026lt;account\u0026gt;\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# 1. Verify service account exists\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl get serviceaccount \u003cspan class=\"nv\"\u003e$SERVICE_ACCOUNT\u003c/span\u003e -n \u003cspan class=\"nv\"\u003e$NAMESPACE\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# 2. Verify service account has Workload Identity annotation\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eANNOTATION\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"k\"\u003e$(\u003c/span\u003ekubectl get serviceaccount \u003cspan class=\"nv\"\u003e$SERVICE_ACCOUNT\u003c/span\u003e -n \u003cspan class=\"nv\"\u003e$NAMESPACE\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  -o \u003cspan class=\"nv\"\u003ejsonpath\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;{.metadata.annotations.azure\\.workload\\.identity/client-id}\u0026#39;\u003c/span\u003e\u003cspan class=\"k\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"o\"\u003e[\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$ANNOTATION\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e !\u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$MANAGED_IDENTITY_CLIENT_ID\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003ethen\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;ERROR: Service account annotation mismatch\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nb\"\u003eexit\u003c/span\u003e \u003cspan class=\"m\"\u003e1\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003efi\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# 3. Verify Azure role assignment\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eROLE_COUNT\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"k\"\u003e$(\u003c/span\u003eaz role assignment list \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --assignee \u003cspan class=\"nv\"\u003e$MANAGED_IDENTITY_CLIENT_ID\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --scope \u003cspan class=\"nv\"\u003e$STORAGE_ACCOUNT_ID\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --query \u003cspan class=\"s2\"\u003e\u0026#34;length([?roleDefinitionName==\u0026#39;Storage Blob Data Reader\u0026#39;])\u0026#34;\u003c/span\u003e \u003cspan class=\"se\"\u003e\\\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  --output tsv\u003cspan class=\"k\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"o\"\u003e[\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$ROLE_COUNT\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e -eq \u003cspan class=\"s2\"\u003e\u0026#34;0\u0026#34;\u003c/span\u003e \u003cspan class=\"o\"\u003e]\u003c/span\u003e\u003cspan class=\"p\"\u003e;\u003c/span\u003e \u003cspan class=\"k\"\u003ethen\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;ERROR: Missing Storage Blob Data Reader role assignment\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nb\"\u003eexit\u003c/span\u003e \u003cspan class=\"m\"\u003e1\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003efi\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Validation passed\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\n\n\n\n\u003ch3 id=\"runtime-verification\"\u003e\u003ca href=\"/posts/pod-identity-access-control-aks/#runtime-verification\" title=\"Runtime verification\"\u003eRuntime verification\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eOnce deployed, monitor workloads for authentication failures. Azure Monitor, Application Insights, and Kubernetes events provide signals when identity issues occur.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eKey metrics to track:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAzure AD token acquisition failures (4xx responses from Azure AD endpoints)\u003c/li\u003e\n\u003cli\u003eAzure RBAC authorization failures (403 responses from Azure resource APIs)\u003c/li\u003e\n\u003cli\u003eKubernetes RBAC denials (audit log events with \u003ccode\u003eForbidden\u003c/code\u003e responses)\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch3 id=\"periodic-audits\"\u003e\u003ca href=\"/posts/pod-identity-access-control-aks/#periodic-audits\" title=\"Periodic audits\"\u003ePeriodic audits\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eIdentity configurations drift over time. Regular audits catch permissions that have grown beyond initial requirements or identities that no longer align with current workload needs.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAudit checklist:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eList all managed identities and their role assignments—remove unused identities\u003c/li\u003e\n\u003cli\u003eReview role assignments for over-privileged access—scope down to specific resources\u003c/li\u003e\n\u003cli\u003eValidate federated credentials still match deployed service accounts—remove orphaned federations\u003c/li\u003e\n\u003cli\u003eCheck for service accounts with Workload Identity annotations but no corresponding Azure identity\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch2 id=\"practical-configuration-minimal-working-example\"\u003e\u003ca href=\"/posts/pod-identity-access-control-aks/#practical-configuration-minimal-working-example\" title=\"Practical configuration: Minimal working example\"\u003ePractical configuration: Minimal working example\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eHere\u0026rsquo;s a complete Workload Identity configuration showing the Kubernetes and Azure components required for a pod to access Azure Storage.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eKubernetes manifest (pod with Workload Identity):\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ev1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eServiceAccount\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003estorage-reader\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003enamespace\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eproduction\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eannotations\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eazure.workload.identity/client-id\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;00000000-0000-0000-0000-000000000000\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nn\"\u003e---\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ev1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ePod\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003estorage-reader-pod\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003enamespace\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eproduction\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003elabels\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eazure.workload.identity/use\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;true\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eserviceAccountName\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003estorage-reader\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003econtainers\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eapp\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eimage\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003emyregistry.azurecr.io/storage-app:latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eenv\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eAZURE_CLIENT_ID\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003evalue\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;00000000-0000-0000-0000-000000000000\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eAZURE_TENANT_ID\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003evalue\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;00000000-0000-0000-0000-000000000000\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003cstrong\u003eKey configuration points:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eService account must have \u003ccode\u003eazure.workload.identity/client-id\u003c/code\u003e annotation matching the Azure managed identity\u003c/li\u003e\n\u003cli\u003ePod must have \u003ccode\u003eazure.workload.identity/use: \u0026quot;true\u0026quot;\u003c/code\u003e label\u003c/li\u003e\n\u003cli\u003ePod must reference the service account via \u003ccode\u003eserviceAccountName\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003eContainer environment variables provide Azure SDK with identity information\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eAzure RBAC assignment (Terraform):\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-hcl\" data-lang=\"hcl\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Managed identity for the workload\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_user_assigned_identity\u0026#34; \u0026#34;storage_reader\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;storage-reader-identity\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eaks\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  location\u003c/span\u003e            \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eaks\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003elocation\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Federated credential linking Kubernetes SA to Azure identity\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_federated_identity_credential\u0026#34; \u0026#34;storage_reader\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  name\u003c/span\u003e                \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;storage-reader-federation\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  resource_group_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_resource_group\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eaks\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003ename\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  parent_id\u003c/span\u003e           \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_user_assigned_identity\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003estorage_reader\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  audience\u003c/span\u003e            \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;api://AzureADTokenExchange\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  issuer\u003c/span\u003e              \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_kubernetes_cluster\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eaks\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eoidc_issuer_url\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  subject\u003c/span\u003e             \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;system:serviceaccount:production:storage-reader\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Grant Storage Blob Data Reader to the identity\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eresource\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;azurerm_role_assignment\u0026#34; \u0026#34;storage_reader\u0026#34;\u003c/span\u003e {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  scope\u003c/span\u003e                \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_storage_account\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003edata\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eid\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  role_definition_name\u003c/span\u003e \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Storage Blob Data Reader\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003e  principal_id\u003c/span\u003e         \u003cspan class=\"o\"\u003e=\u003c/span\u003e \u003cspan class=\"k\"\u003eazurerm_user_assigned_identity\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003estorage_reader\u003c/span\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e\u003cspan class=\"k\"\u003eprincipal_id\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003cstrong\u003eCritical details:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003eaudience\u003c/code\u003e must be \u003ccode\u003e[\u0026quot;api://AzureADTokenExchange\u0026quot;]\u003c/code\u003e for Workload Identity\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eissuer\u003c/code\u003e must match the AKS cluster\u0026rsquo;s OIDC issuer URL exactly\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003esubject\u003c/code\u003e format is \u003ccode\u003esystem:serviceaccount:NAMESPACE:SERVICE_ACCOUNT_NAME\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003eRole assignment scope should be as narrow as possible—specific storage account, not resource group\u003c/li\u003e\n\u003c/ul\u003e\n\n\n\n\n\u003ch2 id=\"final-thoughts\"\u003e\u003ca href=\"/posts/pod-identity-access-control-aks/#final-thoughts\" title=\"Final thoughts\"\u003eFinal thoughts\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eWorkload Identity Federation solves credential lifecycle and audit trail problems that plagued earlier AKS authentication patterns. It doesn\u0026rsquo;t eliminate configuration complexity or RBAC layering challenges. Understanding how Kubernetes RBAC, Azure RBAC, and Azure AD permissions interact is essential. Knowing where credentials still leak despite federation, what misconfigurations create security vulnerabilities, and how to validate configurations before they fail in production separates functioning workloads from 3 AM incidents.\u003c/p\u003e\n\u003cp\u003eStart with minimal permissions. Automate identity provisioning and role assignments through infrastructure as code. Validate configurations before deployment. Monitor for authentication failures and audit identity drift over time. These patterns prevent the majority of identity-related failures in production AKS environments.\u003c/p\u003e\n","date_modified":"2026-05-26T10:22:03+02:00","date_published":"2026-01-21T17:00:00+01:00","id":"https://daily-devops.net/posts/pod-identity-access-control-aks/","language":"en","summary":"Workload Identity Federation changed how AKS handles authentication. Credential leaks, RBAC failures, identity drift: what breaks and how to fix it.","tags":["identity","azure","kubernetes","cloud","devops","rbac","security"],"title":"Pod Identity \u0026 Access Control in AKS: What Actually Breaks","url":"https://daily-devops.net/posts/pod-identity-access-control-aks/"},{"authors":[{"name":"Jendrik Brack","url":"https://daily-devops.net/authors/jendrik/"}],"content_html":"\u003cp\u003eAKS documentation will get you to a running cluster. It won\u0026rsquo;t tell you why your pod authenticated in staging and gets a 401 in production. It won\u0026rsquo;t explain why upgrading a 50-node cluster at 2 AM felt fine but a 300-node upgrade at noon caused cascading evictions. It won\u0026rsquo;t show you which storage class to avoid when your database needs to survive node pool replacements.\u003c/p\u003e\n\u003cp\u003eThis series covers the operational reality — the decisions that distinguish AKS clusters that run quietly in production from clusters that generate 3 AM alerts. Nine articles, each examining a specific architectural domain with the specificity that matters when something breaks.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"why-aks-operations-is-different\"\u003e\u003ca href=\"/posts/aks-architecture-operations/#why-aks-operations-is-different\" title=\"Why AKS Operations Is Different\"\u003eWhy AKS Operations Is Different\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eMicrosoft manages the AKS control plane. That sounds like less work, and in some ways it is — you don\u0026rsquo;t patch etcd, you don\u0026rsquo;t replace failed control plane VMs, you don\u0026rsquo;t worry about API server certificate rotation. What it doesn\u0026rsquo;t mean is that running AKS in production is simple or that managed Kubernetes hands you a reliable platform and steps aside.\u003c/p\u003e\n\u003cp\u003eEvery node pool configuration decision is yours. Every storage class binding, every PVC lifecycle policy, every decision about which node pool hosts which workload — that\u0026rsquo;s on you. RBAC spans three separate systems simultaneously: Kubernetes RBAC, Azure RBAC, and Azure AD. A misconfiguration in any one of them produces an access failure that looks identical from the application\u0026rsquo;s perspective. The documentation will show you how to configure each system in isolation. It will not show you why they interact in non-obvious ways under specific conditions, or what the failure mode looks like when you get the federation configuration slightly wrong.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"where-networking-stops-being-managed\"\u003e\u003ca href=\"/posts/aks-architecture-operations/#where-networking-stops-being-managed\" title=\"Where Networking Stops Being Managed\"\u003eWhere Networking Stops Being Managed\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eNetworking is another area where \u0026ldquo;managed\u0026rdquo; has a narrower meaning than the word implies. Microsoft manages the control plane networking. Your VNet, your subnets, your IP address planning, your DNS configuration, your ingress architecture — all of it is your responsibility, and the decisions compound. IP exhaustion caused by node pool scaling is a common production incident that no amount of control plane management prevents. Private cluster DNS resolution breaks in ways that take hours to diagnose if you haven\u0026rsquo;t encountered the pattern before.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"upgrades-the-gap-between-docs-and-reality\"\u003e\u003ca href=\"/posts/aks-architecture-operations/#upgrades-the-gap-between-docs-and-reality\" title=\"Upgrades: The Gap Between Docs and Reality\"\u003eUpgrades: The Gap Between Docs and Reality\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eUpgrades are perhaps the clearest illustration of the gap between documentation and reality. The documentation describes upgrade mechanics accurately. What it doesn\u0026rsquo;t describe is how Pod Disruption Budget misconfigurations interact with cluster autoscaler behavior during node pool drain, why the timing of upgrades relative to workload peak matters more than most teams expect, or how a PDB that looks correct on paper blocks drain indefinitely on a cluster that\u0026rsquo;s handling real traffic. Managed Kubernetes handles the control plane upgrade. The workload upgrade is a careful orchestration problem that the platform does not solve for you.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"storage-where-managed-disappears\"\u003e\u003ca href=\"/posts/aks-architecture-operations/#storage-where-managed-disappears\" title=\"Storage: Where \u0026ldquo;Managed\u0026rdquo; Disappears\"\u003eStorage: Where \u0026ldquo;Managed\u0026rdquo; Disappears\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eStorage is where the word \u0026ldquo;managed\u0026rdquo; disappears entirely. Azure manages the underlying disk and file services. AKS provides the CSI drivers. Everything between your application and the storage backend — PVC binding, reclaim policies, volume expansion behavior, backup orchestration, behavior during node failure or node pool deletion — is configuration you own. Teams that treat storage as a detail find out it isn\u0026rsquo;t when a node pool replacement deletes volumes that were bound to nodes rather than to the cluster.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"cost-decisions-compound-silently\"\u003e\u003ca href=\"/posts/aks-architecture-operations/#cost-decisions-compound-silently\" title=\"Cost Decisions Compound Silently\"\u003eCost Decisions Compound Silently\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eCost is a dimension that managed Kubernetes actively obscures. The control plane is free at most tiers. Node pool costs scale with what you configure, and the configuration space is large: VM SKU selection, autoscaler min/max bounds, system versus user node pool separation, spot VM integration, pod density targets. None of these have obviously correct values. All of them interact. Teams that inherit clusters often inherit cost structures that made sense at a different scale or for a different workload profile, and reversing those decisions requires careful sequencing to avoid downtime.\u003c/p\u003e\n\u003cp\u003eThe happy paths in the documentation work. They work because they\u0026rsquo;re constructed to work. Production clusters encounter the edges — the configuration combinations, the scale thresholds, the timing sensitivities — that happy paths don\u0026rsquo;t cover. This series is about the edges.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"what-this-series-covers\"\u003e\u003ca href=\"/posts/aks-architecture-operations/#what-this-series-covers\" title=\"What This Series Covers\"\u003eWhat This Series Covers\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003e\u003ca href=\"/posts/pod-identity-access-control-aks/\"\u003ePod Identity \u0026amp; Access Control in AKS: What Actually Breaks\u003c/a\u003e\u003c/strong\u003e starts with identity because identity failures are the most common source of production incidents. Workload Identity Federation eliminates credential lifecycle problems but introduces configuration complexity spanning three separate RBAC systems — Kubernetes RBAC, Azure RBAC, and Azure AD permissions. The article explains where credentials still leak despite federation, how layers interact and fail, and validation patterns that catch misconfigurations before they become incidents.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003ca href=\"/posts/storage-architecture-stateful-workloads-aks/\"\u003eStorage Architecture \u0026amp; Stateful Workloads in AKS\u003c/a\u003e\u003c/strong\u003e addresses what most AKS guides skip: what actually happens to your data when a node gets replaced. PVC/PV architecture, Azure Disk versus Azure Files performance trade-offs, Velero backup configurations that survive real restore scenarios, and multi-cluster replication patterns for production stateful workloads.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003ca href=\"/posts/cost-optimization-resource-governance-aks/\"\u003eAKS Cost Optimization: Resource Governance That Actually Works\u003c/a\u003e\u003c/strong\u003e covers the gap between \u0026ldquo;set resource limits\u0026rdquo; and actually controlling spend at scale. Pod density strategies, node pool design decisions that compound over time, spot VM integration without reliability regressions, and FinOps tagging that produces actionable cost attribution rather than unread dashboards.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003ca href=\"/posts/multi-aks-cluster-networking-hub-spoke/\"\u003eMulti-AKS Cluster Networking \u0026amp; Hub-Spoke Topology\u003c/a\u003e\u003c/strong\u003e examines what happens to networking when you move from one cluster to many. VNet peering patterns, hub-spoke routing, cross-cluster DNS resolution, shared ingress options, and — critically — the decision criteria for when mesh complexity becomes justified rather than premature.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003ca href=\"/posts/cluster-upgrades-zero-downtime-aks/\"\u003eAKS Cluster Upgrades: Zero-Downtime Operations That Actually Work\u003c/a\u003e\u003c/strong\u003e covers upgrade mechanics that documentation describes optimistically. Cordon and drain behavior, Pod Disruption Budget configuration that prevents service disruption rather than theater-level protection, multi-node-pool rollout strategies, and validation-driven automation that makes upgrades reproducible rather than heroic.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003ca href=\"/posts/container-registry-image-security-aks/\"\u003eContainer Registry \u0026amp; Image Security in AKS Deployments\u003c/a\u003e\u003c/strong\u003e covers ACR hardening beyond the basics. A production-ready sequence: vulnerability scanning, image signing with Notation, RBAC scoping, private endpoints, policy enforcement through Azure Policy and admission controllers, and geo-replication strategies with clear trade-offs explained.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003ca href=\"/posts/disaster-recovery-business-continuity-aks/\"\u003eAKS Disaster Recovery: Why Your Untested Backup Will Fail\u003c/a\u003e\u003c/strong\u003e addresses the gap between having backups and having a tested recovery plan. Velero configuration, realistic RTO/RPO targets that match business risk rather than wishful thinking, restore testing procedures that catch problems before outages, and multi-region failover steps your team can actually execute under pressure.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003ca href=\"/posts/hybrid-aks-on-prem-azure-arc/\"\u003eHybrid AKS: Bridging Cloud and On-Prem with Azure Arc\u003c/a\u003e\u003c/strong\u003e covers the operational patterns for organizations running Kubernetes across cloud and on-premises simultaneously. ExpressRoute and VPN connectivity, Azure Arc for unified management across heterogeneous environments, consistent policy enforcement, DNS resolution, and identity federation without duplicating systems.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003ca href=\"/posts/aks-at-scale-mega-cluster-lessons/\"\u003eAKS at Scale: Hard-Won Lessons from 1000+ Node Clusters\u003c/a\u003e\u003c/strong\u003e closes the series with what changes when clusters grow large enough that the platform itself becomes the bottleneck. etcd limits under high object churn, network saturation at scale, observability overhead that compounds with cluster size, and cost spirals that emerge from architectural decisions that seemed fine at 50 nodes.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"who-this-is-for\"\u003e\u003ca href=\"/posts/aks-architecture-operations/#who-this-is-for\" title=\"Who This Is For\"\u003eWho This Is For\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003ePlatform engineers and infrastructure-focused developers responsible for AKS clusters in production — or teams about to inherit that responsibility. Each article assumes you\u0026rsquo;ve run AKS before and want operational depth, not introductory setup instructions.\u003c/p\u003e\n\u003cp\u003eThe series covers Terraform, Bicep, Kubectl, and Azure CLI patterns throughout. Examples are grounded in production scenarios rather than constructed to demonstrate features.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"how-these-articles-were-written\"\u003e\u003ca href=\"/posts/aks-architecture-operations/#how-these-articles-were-written\" title=\"How These Articles Were Written\"\u003eHow These Articles Were Written\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eEach article in this series is based on production experience — clusters that handled real traffic, failed in real ways, and required real fixes under time pressure. That distinction matters for what you\u0026rsquo;ll find here and what you won\u0026rsquo;t.\u003c/p\u003e\n\u003cp\u003eProduction experience means the failure patterns are specific. Not \u0026ldquo;storage can be tricky\u0026rdquo; but which storage class binding decisions survive node pool replacements and which don\u0026rsquo;t. Not \u0026ldquo;upgrades can cause downtime\u0026rdquo; but which combination of PDB configuration and autoscaler behavior produces an indefinitely blocked drain. Not \u0026ldquo;identity is complex\u0026rdquo; but the exact configuration gap in Workload Identity Federation that causes silent auth failures in one environment and not another. The specificity isn\u0026rsquo;t for its own sake — it\u0026rsquo;s the difference between an article that confirms your intuition and one that actually changes what you configure next.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"trade-offs-over-single-right-answers\"\u003e\u003ca href=\"/posts/aks-architecture-operations/#trade-offs-over-single-right-answers\" title=\"Trade-offs Over Single Right Answers\"\u003eTrade-offs Over Single Right Answers\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eWhat production experience doesn\u0026rsquo;t mean is that every approach here is the only valid one. Large-scale AKS operation involves genuine trade-offs — between cost and resilience, between operational simplicity and flexibility, between standardization and workload-specific tuning. The articles explain the reasoning behind recommendations rather than just stating them, because the reasoning is what lets you adapt the approach to your constraints. A node pool design that works for a batch processing workload is wrong for a latency-sensitive API, and the article on cost governance explains why rather than presenting a single correct answer.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"when-aks-makes-things-harder\"\u003e\u003ca href=\"/posts/aks-architecture-operations/#when-aks-makes-things-harder\" title=\"When AKS Makes Things Harder\"\u003eWhen AKS Makes Things Harder\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eThe articles were not written to showcase features or to demonstrate that AKS has a solution for every problem. Some of them document problems that AKS makes harder than it should be, and say so directly. If a particular architectural pattern has a known failure mode at scale, that failure mode appears in the article rather than in a footnote or an FAQ three pages into the documentation. If a feature has a meaningful limitation that affects how you should configure it, that limitation is in the main text, not in a callout box labeled \u0026ldquo;note.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eThe goal is for these articles to be the thing you read before a production incident rather than the thing you find during one.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"where-to-start\"\u003e\u003ca href=\"/posts/aks-architecture-operations/#where-to-start\" title=\"Where to Start\"\u003eWhere to Start\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eRead in published order if you\u0026rsquo;re building out AKS infrastructure from scratch — identity and storage are foundational, and later articles reference earlier concepts. Jump to specific articles if you\u0026rsquo;re dealing with an immediate operational problem: the titles are specific enough that the right article for your situation should be obvious.\u003c/p\u003e\n\u003cp\u003eThe scale article at the end is worth reading early if your cluster is already growing or if you\u0026rsquo;re designing for growth — some architectural decisions made at 50 nodes are expensive to reverse at 500.\u003c/p\u003e\n","date_modified":"2026-05-25T23:41:10+02:00","date_published":"2026-01-21T17:00:00+01:00","id":"https://daily-devops.net/posts/aks-architecture-operations/","language":"en","summary":"Nine articles on production AKS—identity, storage, multi-cluster networking, cost governance, DR, and running 1000-node clusters in practice.","tags":["kubernetes","azure","cloud","devops","operations","platform-engineering"],"title":"AKS Architecture \u0026 Operations — The Complete Series","url":"https://daily-devops.net/posts/aks-architecture-operations/"},{"authors":[{"name":"Martin Stühmer","url":"https://daily-devops.net/authors/martin/"}],"content_html":"\u003cp\u003eIn Swabia, southern Germany, there is another cultural practice that outsiders often misunderstand or quietly ignore until it becomes unavoidable. It is called Stoßlüften.\u003c/p\u003e\n\u003cp\u003eTranslated literally, it means \u0026ldquo;shock ventilation.\u0026rdquo; The idea is simple and non-negotiable. Several times a day, regardless of season, you open all windows fully for a few minutes. In winter. In rain. In freezing temperatures. Then you close them again.\u003c/p\u003e\n\u003cp\u003eNo tilted windows. No half measures. No \u0026ldquo;we\u0026rsquo;ll do it later.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eThe goal is not comfort. The goal is system health.\u003c/p\u003e\n\u003cp\u003eAnd once again, this mindset maps disturbingly well to how we should treat long-running software systems.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"what-stoßlüften-actually-solves\"\u003e\u003ca href=\"/posts/stossluften-and-software-systems/#what-sto%c3%9fl%c3%bcften-actually-solves\" title=\"What Stoßlüften Actually Solves\"\u003eWhat Stoßlüften Actually Solves\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eStoßlüften is not about temperature control. It is about air quality.\u003c/p\u003e\n\u003cp\u003eKeeping windows slightly open all day feels reasonable. It avoids discomfort. It avoids confrontation with reality. It also does absolutely nothing to remove stale air, humidity, or long-term buildup. Over time, the room feels heavy. Mold appears quietly. The damage is discovered too late.\u003c/p\u003e\n\u003cp\u003eSwabians learned this the hard way. The solution was not better perfume. It was short, aggressive, intentional intervention.\u003c/p\u003e\n\u003cp\u003eThat distinction matters.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-software-equivalent-of-stale-air\"\u003e\u003ca href=\"/posts/stossluften-and-software-systems/#the-software-equivalent-of-stale-air\" title=\"The Software Equivalent of Stale Air\"\u003eThe Software Equivalent of Stale Air\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eIn software systems, stale air takes many forms, and they\u0026rsquo;re often invisible until catastrophe hits.\u003c/p\u003e\n\u003cp\u003eConsider a long-running ASP.NET Core service that hasn\u0026rsquo;t been redeployed in eight months. It\u0026rsquo;s stable, right? The monitoring shows green. Latency is acceptable. But inside, subtle decay is accumulating:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eMemory pressure\u003c/strong\u003e: A Garbage Collector tuned optimally for 100 concurrent users now serves 800. Heap fragmentation increases. Full collections pause the application for 200ms, 300ms, sometimes 500ms. But \u0026ldquo;it doesn\u0026rsquo;t crash,\u0026rdquo; so nobody investigates.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eConnection pools\u003c/strong\u003e: Database connection strings are cached. A DBA migrated the database to a new cluster and updated DNS, but the service still holds stale connection references. The connection pool wastes resources on dead connections. Some queries mysteriously slow to timeout.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eTemporal cache\u003c/strong\u003e: An in-memory cache stores \u0026ldquo;permanent\u0026rdquo; reference data. A new region was added six months ago. The cache has never been cleared. Old entries are queried frequently, new entries are missing.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eHardware drift\u003c/strong\u003e: The service was deployed on Intel Xeon E5 processors. Your cloud provider migrated to AMD EPYC. The CPU instruction set is different. Some optimizations no longer apply. Latency jitter increases without explanation.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eNothing is technically broken. Monitoring is green. Latency is acceptable. Everyone feels slightly uncomfortable, but nobody can point to a single failure.\u003c/p\u003e\n\u003cp\u003eThis is the most dangerous state a system can be in.\u003c/p\u003e\n\u003cp\u003eLike a poorly ventilated room, everything still works. Until it doesn\u0026rsquo;t.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"why-small-open-windows-dont-work\"\u003e\u003ca href=\"/posts/stossluften-and-software-systems/#why-small-open-windows-dont-work\" title=\"Why Small Open Windows Don\u0026rsquo;t Work\"\u003eWhy Small Open Windows Don\u0026rsquo;t Work\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eMany teams believe incremental improvements are enough. A small refactor here. A minor dependency update there. A single flag cleaned up during a feature sprint. These adjustments feel responsible, but they don\u0026rsquo;t meaningfully reset the system.\u003c/p\u003e\n\u003cp\u003eThe problem is structural. Incremental fixes optimize for comfort—avoiding downtime—rather than outcome: system health. They reduce immediate discomfort but leave stale state untouched. A \u003ccode\u003eFileSystemWatcher\u003c/code\u003e still holds old file references. Memory fragmentation still accumulates. Cached data still sits in memory indefinitely.\u003c/p\u003e\n\u003cp\u003eStoßlüften works differently. It is deliberate and complete. You don\u0026rsquo;t optimize for comfort during the process. You optimize for outcome. The system must prove it can start fresh, not just continue indefinitely. Fresh air replaces stale air quickly. This completeness is why it succeeds where partial measures fail.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"restarts-rebuilds-and-reality\"\u003e\u003ca href=\"/posts/stossluften-and-software-systems/#restarts-rebuilds-and-reality\" title=\"Restarts, Rebuilds, and Reality\"\u003eRestarts, Rebuilds, and Reality\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eOne of the clearest expressions of Stoßlüften in software is restarting services on purpose. Not because they crashed. Not because alerts fired. But because long-lived state is a liability.\u003c/p\u003e\n\u003cp\u003eTeams that never restart services accumulate invisible risk. What looks stable—green metrics, acceptable latency—is often just decay that hasn\u0026rsquo;t been measured yet. Consider what happens in a Kubernetes cluster when pods run for months without intentional resets:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWithout regular restarts:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eA \u003ccode\u003eFileSystemWatcher\u003c/code\u003e monitoring a config directory holds an open file handle. When the config is deleted, the watcher doesn\u0026rsquo;t detect it. New instances read fresh config, old instances don\u0026rsquo;t. Configuration drift is invisible.\u003c/li\u003e\n\u003cli\u003eA background task crashes after 6 hours. The pod stays alive but the task loop is dead. No alerts fire. Work silently backs up for days.\u003c/li\u003e\n\u003cli\u003eMemory fragmentation becomes pathological. The heap fragments to 40%. Simple allocations start failing. Response times degrade silently by 30-40% before anyone connects the dots.\u003c/li\u003e\n\u003cli\u003eInfrastructure migrates to a new subnet. Old instances reference stale gateway IPs. Requests time out randomly. Debugging becomes a nightmare because the failure is intermittent and invisible.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eWith regular restarts (every 24-72 hours):\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eConfig mismatches surface immediately. New instances must read fresh config or fail to start. Inconsistency becomes visible rather than silent.\u003c/li\u003e\n\u003cli\u003eDead task loops are discovered during the next startup. The problem is surfaced while it\u0026rsquo;s still manageable.\u003c/li\u003e\n\u003cli\u003eMemory is reclaimed and fragmentation resets. Degradation is measured in days, not months.\u003c/li\u003e\n\u003cli\u003eNetwork connectivity is re-established from scratch. Stale routing tables disappear. The system proves it can reconnect.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eFresh air hurts briefly. Stale air hurts later—and in production, later often means 3am on a Sunday.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"stoßlüften-is-not-chaos-engineering\"\u003e\u003ca href=\"/posts/stossluften-and-software-systems/#sto%c3%9fl%c3%bcften-is-not-chaos-engineering\" title=\"Stoßlüften Is Not Chaos Engineering\"\u003eStoßlüften Is Not Chaos Engineering\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eThis is not about randomness or stress for its own sake.\u003c/p\u003e\n\u003cp\u003eStoßlüften is predictable. Scheduled. Expected. Everyone knows it will happen. Windows open. Windows close. Life continues.\u003c/p\u003e\n\u003cp\u003eThe software equivalent is controlled disruption. Planned redeployments. Regular dependency refresh cycles. Explicit cleanup phases. Intentional cache invalidation. Rebuilding environments from scratch instead of patching them indefinitely.\u003c/p\u003e\n\u003cp\u003eNone of this is exciting. That is precisely why it works.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"why-teams-avoid-it\"\u003e\u003ca href=\"/posts/stossluften-and-software-systems/#why-teams-avoid-it\" title=\"Why Teams Avoid It\"\u003eWhy Teams Avoid It\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eStoßlüften is uncomfortable. Especially in winter.\u003c/p\u003e\n\u003cp\u003eIt interrupts the illusion of stability. It creates a brief moment where the system is exposed. People feel the cold and question whether this is really necessary.\u003c/p\u003e\n\u003cp\u003eSoftware teams do the same thing. They avoid actions that temporarily increase risk, even if those actions reduce long-term risk dramatically. They prefer slow suffocation over short discomfort.\u003c/p\u003e\n\u003cp\u003eUntil mold shows up. Or outages. Or security incidents. Or the realization that nobody knows how the system actually starts anymore.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"a-practical-translation\"\u003e\u003ca href=\"/posts/stossluften-and-software-systems/#a-practical-translation\" title=\"A Practical Translation\"\u003eA Practical Translation\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eStoßlüften in software does not mean reckless change. It means building intentional reset points into your systems and enforcing them with discipline.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"service-restarts\"\u003e\u003ca href=\"/posts/stossluften-and-software-systems/#service-restarts\" title=\"Service Restarts\"\u003eService Restarts\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eRestart services regularly via orchestration. In Kubernetes, it\u0026rsquo;s a single command:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Restart all pods in a deployment, rolling one at a time\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl rollout restart deployment/api-service -n production\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eSee the \u003ca href=\"https://kubernetes.io/docs/reference/kubectl/generated/kubectl_rollout/kubectl_rollout_restart/\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eofficial kubectl rollout restart documentation\u003c/a\u003e for more options.\u003c/p\u003e\n\u003cp\u003eThis forces your system to prove it can start cleanly. Every day. Without exception. If a pod fails to start, you discover it during a planned restart, not at 3am when users are affected. If it succeeds, you\u0026rsquo;ve just validated that all your startup assumptions still hold true.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"environment-rebuilds\"\u003e\u003ca href=\"/posts/stossluften-and-software-systems/#environment-rebuilds\" title=\"Environment Rebuilds\"\u003eEnvironment Rebuilds\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eRebuild environments from code, not from manual patches. If your production infrastructure has undocumented changes scattered across SSH sessions and Slack messages, you\u0026rsquo;ve created a disaster waiting to happen.\u003c/p\u003e\n\u003cp\u003eStore everything in \u003ca href=\"https://www.terraform.io/\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eTerraform\u003c/a\u003e, \u003ca href=\"https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/overview\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eBicep\u003c/a\u003e, or \u003ca href=\"https://aws.amazon.com/cloudformation/\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eCloudFormation\u003c/a\u003e. Every configuration change goes through code review and staging validation. When something breaks, you rebuild identically in 10 minutes from version control. When you discover a performance bottleneck, you update the code, get peer review, test in staging, then apply with confidence. The previous state is in git history. Rollback is one command away.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"cache-and-state-management\"\u003e\u003ca href=\"/posts/stossluften-and-software-systems/#cache-and-state-management\" title=\"Cache and State Management\"\u003eCache and State Management\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eDo not rely on in-process caches that accumulate for months. They become invisible knowledge that only exists in memory. Instead, use distributed caches with explicit expiration times. Set TTLs (Time-To-Live values) to hours, not days. Force the cache to refresh regularly. Every 2-24 hours, the system reaches back to its source of truth instead of trusting what memory told it.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"feature-flag-discipline\"\u003e\u003ca href=\"/posts/stossluften-and-software-systems/#feature-flag-discipline\" title=\"Feature Flag Discipline\"\u003eFeature Flag Discipline\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eRemove flags aggressively. I\u0026rsquo;ve worked on systems where three-year-old feature flags were still active. The code paths they protected were theoretically unreachable, but nobody was certain enough to delete them. They accumulated like technical sediment.\u003c/p\u003e\n\u003cp\u003eEstablish a rhythm: \u003cstrong\u003eEvery quarter, audit all active flags.\u003c/strong\u003e Answer one question: \u0026ldquo;Is this flag still serving a purpose?\u0026rdquo; If the answer is no, delete it the same day. Dead code paths with unclear purposes are a slow poison. Kill them before they spread.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"force-reproducibility\"\u003e\u003ca href=\"/posts/stossluften-and-software-systems/#force-reproducibility\" title=\"Force Reproducibility\"\u003eForce Reproducibility\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eThe final check: Force systems to prove they can start cleanly. Implement startup validation that runs every time your application boots. Three questions:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eCan you read essential configuration?\u003c/li\u003e\n\u003cli\u003eCan you connect to the database?\u003c/li\u003e\n\u003cli\u003eAre critical external services online?\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eIf any check fails, the pod doesn\u0026rsquo;t become \u0026ldquo;ready.\u0026rdquo; Kubernetes doesn\u0026rsquo;t route traffic to it. The problem surfaces immediately. No silent degradation. No invisible failures that accumulate for months. The system has to prove it\u0026rsquo;s healthy to be allowed to serve traffic.\u003c/p\u003e\n\u003cp\u003eIf your production environment cannot be recreated without tribal knowledge, you are not ventilating. You are masking smells. And masked smells always get worse.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"final-thought\"\u003e\u003ca href=\"/posts/stossluften-and-software-systems/#final-thought\" title=\"Final Thought\"\u003eFinal Thought\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eSwabians do not Stoßlüften because they enjoy cold air. They do it because ignoring air quality is more expensive in the long run.\u003c/p\u003e\n\u003cp\u003eThe same applies to software systems. Stability is not about avoiding disruption. It is about choosing the right kind of disruption at the right time.\u003c/p\u003e\n\u003cp\u003eKehrwoche teaches us to clean regularly.\nStoßlüften teaches us to reset deliberately.\u003c/p\u003e\n\u003cp\u003eBoth are boring. Both are effective. And both exist because people learned that slow decay is harder to fix than brief discomfort.\u003c/p\u003e\n\u003cp\u003eOpen the windows.\nLet the stale assumptions out.\nClose them again.\u003c/p\u003e\n\u003cp\u003eYour system will breathe easier afterward.\u003c/p\u003e\n","date_modified":"2026-05-26T10:22:03+02:00","date_published":"2026-01-16T11:30:00+01:00","id":"https://daily-devops.net/posts/stossluften-and-software-systems/","language":"en","summary":"Hidden decay slips past green dashboards: intentional resets, rebuilds, and reproducibility checks expose what monitoring quietly keeps hiding.\n","tags":["technicaldebt","architecture","devops","reliability"],"title":"Stoßlüften: The Architecture of Intentional Resets","url":"https://daily-devops.net/posts/stossluften-and-software-systems/"},{"authors":[{"name":"Martin Stühmer","url":"https://daily-devops.net/authors/martin/"}],"content_html":"\n\n\n\n\u003ch2 id=\"your-tuesday-morning-a-true-story\"\u003e\u003ca href=\"/posts/alphabet-soup-file-formats/#your-tuesday-morning-a-true-story\" title=\"Your Tuesday Morning: A True Story\"\u003eYour Tuesday Morning: A True Story\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eIt\u0026rsquo;s 9 AM. You\u0026rsquo;re debugging why the Kubernetes deployment failed overnight. The YAML looked perfect. Indentation? Check. Syntax? Check. The problem? Someone used \u003ccode\u003eNO\u003c/code\u003e as an environment variable value. YAML helpfully parsed it as boolean \u003ccode\u003efalse\u003c/code\u003e. Your pod never started.\u003c/p\u003e\n\u003cp\u003eBy 10 AM, you\u0026rsquo;re fixing the CSV export that accounting requested. Excel mangled the employee IDs—turned \u003ccode\u003e00123\u003c/code\u003e into \u003ccode\u003e123\u003c/code\u003e, converted the date column into something unrecognizable, and decided that gene names like \u003ccode\u003eSEPT2\u003c/code\u003e are obviously September 2nd.\u003c/p\u003e\n\u003cp\u003eAt 11 AM, the build pipeline breaks because someone added a trailing comma to \u003ccode\u003eappsettings.json\u003c/code\u003e. JSON doesn\u0026rsquo;t allow those. The error message is cryptic. The fix takes 30 seconds. Finding it took 20 minutes.\u003c/p\u003e\n\u003cp\u003eLunch is spent explaining to a junior dev why we have YAML for CI/CD, JSON for app config, TOML for the Rust tool, INI for the legacy service, and CSV for data exports. \u0026ldquo;Can\u0026rsquo;t we just pick one?\u0026rdquo; they ask.\u003c/p\u003e\n\u003cp\u003eNo. We can\u0026rsquo;t. This is software development in 2025.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-format-parade-whos-who-in-the-chaos\"\u003e\u003ca href=\"/posts/alphabet-soup-file-formats/#the-format-parade-whos-who-in-the-chaos\" title=\"The Format Parade: Who\u0026rsquo;s Who in the Chaos\"\u003eThe Format Parade: Who\u0026rsquo;s Who in the Chaos\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eLet\u0026rsquo;s meet the contestants in this never-ending format beauty pageant. Spoiler: nobody wins.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"csv-the-universal-disaster\"\u003e\u003ca href=\"/posts/alphabet-soup-file-formats/#csv-the-universal-disaster\" title=\"CSV: The Universal Disaster\"\u003e\u003cstrong\u003eCSV: The Universal Disaster\u003c/strong\u003e\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eComma-Separated Values promised simplicity: rows, columns, commas. That\u0026rsquo;s it.\u003c/p\u003e\n\u003cp\u003eThe reality? There\u0026rsquo;s no real standard. RFC 4180 tried in 2005, but thousands of tools had already shipped their own interpretations. Comma delimiter? Sometimes semicolon. Sometimes tab. Quote your strings? Maybe. Escape quotes by doubling them? Or backslashes? Depends on the tool.\u003c/p\u003e\n\u003cp\u003eExcel is CSV\u0026rsquo;s natural enemy. It will \u0026ldquo;fix\u0026rdquo; your data by converting dates (\u003ccode\u003e2025-12-31\u003c/code\u003e becomes Excel\u0026rsquo;s internal date format), stripping leading zeros (\u003ccode\u003e00123\u003c/code\u003e → \u003ccode\u003e123\u003c/code\u003e), and famously turning gene names into dates (\u003ccode\u003eSEPT2\u003c/code\u003e → Sep 2). Biologists have a special hatred for CSV because of this.\u003c/p\u003e\n\u003cp\u003eYet CSV survives because it\u0026rsquo;s \u003cstrong\u003euniversal\u003c/strong\u003e. Every tool exports it. Every developer can edit it in Notepad. It compresses well. It\u0026rsquo;s the lowest common denominator when nothing else works.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eVerdict:\u003c/strong\u003e Like democracy, it\u0026rsquo;s the worst format except for all the others.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"ini-the-minimalist\"\u003e\u003ca href=\"/posts/alphabet-soup-file-formats/#ini-the-minimalist\" title=\"INI: The Minimalist\u0026rsquo;s Dream\"\u003e\u003cstrong\u003eINI: The Minimalist\u0026rsquo;s Dream\u003c/strong\u003e\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eKey-value pairs. Sections. That\u0026rsquo;s the entire spec. Humans understand it instantly.\u003c/p\u003e\n\u003cp\u003eThe problem? No nested structures. No lists. No type system—everything\u0026rsquo;s a string. The moment you need hierarchy, INI taps out.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eVerdict:\u003c/strong\u003e Perfect for simple configs. Useless for everything else.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"xml-the-enterprise-albatross\"\u003e\u003ca href=\"/posts/alphabet-soup-file-formats/#xml-the-enterprise-albatross\" title=\"XML: The Enterprise Albatross\"\u003e\u003cstrong\u003eXML: The Enterprise Albatross\u003c/strong\u003e\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eXML promised schema validation, namespaces, self-describing tags—enterprise-grade power.\u003c/p\u003e\n\u003cp\u003eWhat we got: angle bracket hell. Every value wrapped in opening and closing tags. Signal-to-noise ratio so poor that even enterprise architects—people who \u003cem\u003ethrive\u003c/em\u003e on complexity—started looking for alternatives.\u003c/p\u003e\n\u003cp\u003eWhen the people who love complexity want simpler, you\u0026rsquo;ve failed at usability.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eVerdict:\u003c/strong\u003e Still haunting legacy systems. Nobody voluntarily starts new projects with XML anymore.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"json-the-machine\"\u003e\u003ca href=\"/posts/alphabet-soup-file-formats/#json-the-machine\" title=\"JSON: The Machine\u0026rsquo;s Format\"\u003e\u003cstrong\u003eJSON: The Machine\u0026rsquo;s Format\u003c/strong\u003e\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eJSON solved XML\u0026rsquo;s verbosity. Clean syntax. Maps directly to data structures. Every language has a parser.\u003c/p\u003e\n\u003cp\u003eBut it was designed for \u003cstrong\u003emachines\u003c/strong\u003e, not humans. No comments (explain your config changes in commit messages, I guess). Trailing commas forbidden. Rigid quoting everywhere. It works, but it\u0026rsquo;s tedious to hand-edit.\u003c/p\u003e\n\u003cp\u003eOpenAI\u0026rsquo;s function calling? JSON-only. Why? Because it\u0026rsquo;s deterministic. One correct way to structure it.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eVerdict:\u003c/strong\u003e Boring, reliable, ubiquitous. The Toyota Camry of data formats.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"yaml-the-beautiful-disaster\"\u003e\u003ca href=\"/posts/alphabet-soup-file-formats/#yaml-the-beautiful-disaster\" title=\"YAML: The Beautiful Disaster\"\u003e\u003cstrong\u003eYAML: The Beautiful Disaster\u003c/strong\u003e\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eYAML threw JSON\u0026rsquo;s rigidity out the window. Minimal syntax. Comments everywhere. Indentation-based. Human-friendly!\u003c/p\u003e\n\u003cp\u003eExcept it\u0026rsquo;s \u003cstrong\u003ewhitespace-sensitive\u003c/strong\u003e. One misaligned space breaks everything, often silently. Implicit type conversions bite constantly (\u003ccode\u003eNO\u003c/code\u003e → \u003ccode\u003efalse\u003c/code\u003e, \u003ccode\u003eon\u003c/code\u003e → \u003ccode\u003etrue\u003c/code\u003e, \u003ccode\u003e0123\u003c/code\u003e → octal 83). The spec is 23,000 words and allows multiple ways to represent the same data.\u003c/p\u003e\n\u003cp\u003eKubernetes chose YAML. Docker Compose chose YAML. GitHub Actions chose YAML. The ecosystem standardized, so now you\u0026rsquo;re learning YAML whether you like it or not.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eVerdict:\u003c/strong\u003e Feels great until it doesn\u0026rsquo;t. Then you\u0026rsquo;re debugging indentation at 2 AM.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"toml-the-pragmatic-middle-ground\"\u003e\u003ca href=\"/posts/alphabet-soup-file-formats/#toml-the-pragmatic-middle-ground\" title=\"TOML: The Pragmatic Middle Ground\"\u003e\u003cstrong\u003eTOML: The Pragmatic Middle Ground\u003c/strong\u003e\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eTom\u0026rsquo;s Obvious Minimal Language tried to be INI + structure + types. Explicit syntax. No whitespace sensitivity. Comments allowed.\u003c/p\u003e\n\u003cp\u003eIt works. It\u0026rsquo;s clear. It\u0026rsquo;s unambiguous. The ecosystem is smaller than YAML/JSON, but growing.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eVerdict:\u003c/strong\u003e Underrated. Use it for build configs if you can.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"taml-radical-minimalism\"\u003e\u003ca href=\"/posts/alphabet-soup-file-formats/#taml-radical-minimalism\" title=\"TAML: Radical Minimalism\"\u003e\u003cstrong\u003eTAML: Radical Minimalism\u003c/strong\u003e\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eTab Annotated Markup Language: tabs for hierarchy, newlines for structure. No brackets, colons, or quotes.\u003c/p\u003e\n\u003cp\u003eOne tab = one level deeper. That\u0026rsquo;s the entire format.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eVerdict:\u003c/strong\u003e Interesting experiment. Tiny ecosystem. Good for greenfield projects if you\u0026rsquo;re willing to bet on niche formats.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"toon-designed-for-ai\"\u003e\u003ca href=\"/posts/alphabet-soup-file-formats/#toon-designed-for-ai\" title=\"TOON: Designed for AI\"\u003e\u003cstrong\u003eTOON: Designed for AI\u003c/strong\u003e\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eToken-Oriented Object Notation emerged in 2025 specifically to solve LLM generation problems.\u003c/p\u003e\n\u003cp\u003e~40% fewer tokens than JSON. Explicit \u003ccode\u003e[N]\u003c/code\u003e array lengths and \u003ccode\u003e{fields}\u003c/code\u003e headers give AI models clear guardrails. Better accuracy (74% vs JSON\u0026rsquo;s 70%) because the schema is baked into the syntax.\u003c/p\u003e\n\u003cp\u003eIt\u0026rsquo;s \u0026ldquo;JSON optimized for transformer models.\u0026rdquo; Human-readable like YAML, compact like CSV, schema-aware like XML.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eVerdict:\u003c/strong\u003e If you\u0026rsquo;re building systems where LLMs frequently generate structured data, TOON might save you. Otherwise, wait to see if it gains traction.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"ccl-category-theory-elegance\"\u003e\u003ca href=\"/posts/alphabet-soup-file-formats/#ccl-category-theory-elegance\" title=\"CCL: Category Theory Elegance\"\u003e\u003cstrong\u003eCCL: Category Theory Elegance\u003c/strong\u003e\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eCategorical Configuration Language built on mathematical principles. Pure key-value pairs with recursive nesting.\u003c/p\u003e\n\u003cp\u003eMinimal syntax: \u003ccode\u003ekey = value\u003c/code\u003e. Comments via \u003ccode\u003e/=\u003c/code\u003e. Merging configs is associative with an identity element. Provably correct composition.\u003c/p\u003e\n\u003cp\u003eThe ecosystem is tiny (OCaml, Rust implementations). Practical? Depends on whether you value theoretical soundness over ecosystem maturity.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eVerdict:\u003c/strong\u003e For people who think in category theory. Everyone else, use TOML.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"bson-the-binary-option\"\u003e\u003ca href=\"/posts/alphabet-soup-file-formats/#bson-the-binary-option\" title=\"BSON: The Binary Option\"\u003e\u003cstrong\u003eBSON: The Binary Option\u003c/strong\u003e\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eBinary JSON. Optimized for machine efficiency. Fast parsing. Compact storage.\u003c/p\u003e\n\u003cp\u003eOpen it in a text editor? Gibberish. It\u0026rsquo;s for databases (MongoDB), not human editing.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eVerdict:\u003c/strong\u003e Right tool for the right job. Don\u0026rsquo;t hand-edit BSON.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"why-we-cant-have-nice-things\"\u003e\u003ca href=\"/posts/alphabet-soup-file-formats/#why-we-cant-have-nice-things\" title=\"Why We Can\u0026rsquo;t Have Nice Things\"\u003eWhy We Can\u0026rsquo;t Have Nice Things\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eHere\u0026rsquo;s the uncomfortable truth: \u003cstrong\u003eevery format genuinely solved real problems\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003eCSV ended proprietary spreadsheet lock-in. XML brought schema validation. JSON fixed XML\u0026rsquo;s verbosity. YAML made configs readable. TOML removed YAML\u0026rsquo;s gotchas. TAML went minimal. TOON optimized for AI. CCL brought mathematical rigor.\u003c/p\u003e\n\u003cp\u003eEach improvement was real. Each one also created new problems.\u003c/p\u003e\n\u003cp\u003eThe xkcd comic about competing standards isn\u0026rsquo;t a joke anymore—it\u0026rsquo;s your job description. We don\u0026rsquo;t have 15 standards. We have 50. Maybe 100.\u003c/p\u003e\n\u003cp\u003eFormats don\u0026rsquo;t converge because \u003cstrong\u003etrade-offs are real\u003c/strong\u003e:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eHuman readability ↔ Machine efficiency\u003c/li\u003e\n\u003cli\u003eFlexibility ↔ Parsability\u003c/li\u003e\n\u003cli\u003eSimplicity ↔ Features\u003c/li\u003e\n\u003cli\u003eEcosystem size ↔ Specialized optimization\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eAs AI systems join the picture, the calculus shifts again. Formats designed for humans sometimes hurt AI. Formats designed for machines frustrate humans. Designing for both? That\u0026rsquo;s what TOON attempts.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-ai-problem-makes-everything-worse\"\u003e\u003ca href=\"/posts/alphabet-soup-file-formats/#the-ai-problem-makes-everything-worse\" title=\"The AI Problem Makes Everything Worse\"\u003eThe AI Problem Makes Everything Worse\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eLarge language models changed the game.\u003c/p\u003e\n\u003cp\u003eChatGPT generates text token by token via pattern matching. Rigid formats like JSON? Manageable. Structure it one of 3-4 ways, models usually succeed.\u003c/p\u003e\n\u003cp\u003eYAML? Disaster. Whitespace sensitivity, implicit type conversions, multiple valid representations—LLMs generate frequently invalid YAML. The \u003cem\u003estructure\u003c/em\u003e looks right, but subtle indentation errors or quoting mistakes break parsing.\u003c/p\u003e\n\u003cp\u003eThis drove OpenAI to mandate JSON-only for function calling. Not because engineers are lazy. Because JSON has \u003cstrong\u003eone correct way\u003c/strong\u003e, and models can learn it reliably.\u003c/p\u003e\n\u003cp\u003eThe irony: we designed YAML for human convenience. AI exposed that \u0026ldquo;flexibility\u0026rdquo; creates too many ways to fail.\u003c/p\u003e\n\u003cp\u003eTOON exists specifically to solve this. Explicit schema headers, deterministic structure, fewer tokens. It\u0026rsquo;s pragmatic engineering: \u0026ldquo;YAML breaks AI, JSON works but is verbose, so let\u0026rsquo;s design something in between that models can generate correctly.\u0026rdquo;\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"survival-guide-five-strategies-that-actually-work\"\u003e\u003ca href=\"/posts/alphabet-soup-file-formats/#survival-guide-five-strategies-that-actually-work\" title=\"Survival Guide: Five Strategies That Actually Work\"\u003eSurvival Guide: Five Strategies That Actually Work\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eYou\u0026rsquo;re stuck with this chaos. Here\u0026rsquo;s how to survive it:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e1. Choose Deliberately\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eIf you\u0026rsquo;re using JSON for config, \u003cstrong\u003eown it\u003c/strong\u003e. Document the decision. Set up schema validation. If you\u0026rsquo;re exporting CSV, specify delimiter, encoding, quoting rules explicitly.\u003c/p\u003e\n\u003cp\u003eDon\u0026rsquo;t drift into formats by accident. Make conscious choices and commit to them.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2. Invest in Tooling\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eLinters. Schema validators. Type safety. These matter \u003cstrong\u003emore than the format\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003eGood tooling makes mediocre formats manageable:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eCSV: parsers with RFC 4180 support\u003c/li\u003e\n\u003cli\u003eJSON: JSON Schema validators\u003c/li\u003e\n\u003cli\u003eYAML: linters that catch indentation issues\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003e3. Be Skeptical of \u0026ldquo;Revolutionary\u0026rdquo; Formats\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eEvery new format promises to be The One. TOON and CCL might be useful for specific niches (LLM generation, category theory), but they\u0026rsquo;re not replacing JSON tomorrow.\u003c/p\u003e\n\u003cp\u003eEvaluate pragmatically. Bet on ecosystems, not elegance.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e4. Plan for AI Integration\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eIf AI generates structured data in your system:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eSafe choice:\u003c/strong\u003e JSON with aggressive validation\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eExperimental:\u003c/strong\u003e TOON if you\u0026rsquo;re adopting early-stage formats\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAvoid:\u003c/strong\u003e YAML unless you enjoy debugging AI-generated indentation errors\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003e5. Don\u0026rsquo;t Refactor Just to Standardize\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eLegacy system using INI, XML, or CSV? If it works, \u003cstrong\u003eleave it alone\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003eRefactoring formats doesn\u0026rsquo;t fix business problems. It creates migration risk. Only change formats when you\u0026rsquo;re solving actual pain, not pursuing theoretical purity.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"living-with-the-chaos\"\u003e\u003ca href=\"/posts/alphabet-soup-file-formats/#living-with-the-chaos\" title=\"Living with the Chaos\"\u003eLiving with the Chaos\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eThe light at the end of the tunnel isn\u0026rsquo;t format convergence. It never was.\u003c/p\u003e\n\u003cp\u003eIt\u0026rsquo;s accepting that we\u0026rsquo;ll always have multiple formats. CSV for exports. JSON for APIs. YAML for infrastructure. TOML for builds. Maybe TOON for AI outputs.\u003c/p\u003e\n\u003cp\u003eThe real problem was never the format. It was always the \u003cstrong\u003edata\u003c/strong\u003e—complex, context-dependent, requiring human judgment.\u003c/p\u003e\n\u003cp\u003ePick the right tool for each job. Invest in validation. Build good tooling. Stay skeptical of salvation promises.\u003c/p\u003e\n\u003cp\u003eWelcome to file format hell. You\u0026rsquo;re going to be here a while.\u003c/p\u003e\n","date_modified":"2026-05-26T10:22:03+02:00","date_published":"2026-01-08T17:00:00+01:00","id":"https://daily-devops.net/posts/alphabet-soup-file-formats/","language":"en","summary":"CSV breaks on commas. YAML breaks on spaces. JSON breaks on trailing commas. TOML, TAML, TOON, CCL joined the chaos. Nobody wins. Here's why.\n","tags":["configuration","devops","dotnet","bestpractices","softwareengineering","codequality"],"title":"Alphabet Soup: The Format Buffet Nobody Ordered\n","url":"https://daily-devops.net/posts/alphabet-soup-file-formats/"},{"authors":[{"name":"Martin Stühmer","url":"https://daily-devops.net/authors/martin/"}],"content_html":"\u003cp\u003eThe .NET CLI? Reliable. Boring. You run \u003ccode\u003edotnet build\u003c/code\u003e, \u003ccode\u003edotnet test\u003c/code\u003e, \u003ccode\u003edotnet publish\u003c/code\u003e, done. Real DevOps work happens in Dockerfiles, CI/CD configs, and specialized tools. The CLI does its job but was never built for actual operational workflows.\u003c/p\u003e\n\u003cp\u003e.NET 10 changes this. Four additions that sound minor but fix real problems I\u0026rsquo;ve hit in production pipelines for years: native container publishing, ephemeral tool execution, better cross-platform packaging, and machine-readable schemas. Not flashy. Not keynote material. But they\u0026rsquo;re the kind of improvements that save hours every week once you\u0026rsquo;re running them at scale.\u003c/p\u003e\n\u003cp\u003eWill they replace your current workflow? Depends on what you\u0026rsquo;re building. Let\u0026rsquo;s look at what actually changed.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"built-in-container-publishing-dockerfiles-become-optional\"\u003e\u003ca href=\"/posts/dotnet-10-cli-devops/#built-in-container-publishing-dockerfiles-become-optional\" title=\"Built-in Container Publishing: Dockerfiles Become Optional\"\u003eBuilt-in Container Publishing: Dockerfiles Become Optional\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eLet\u0026rsquo;s start with the biggest change: \u003ccode\u003edotnet publish\u003c/code\u003e now generates container images directly. No Dockerfile needed.\u003c/p\u003e\n\u003cp\u003eYou know the drill. Write a Dockerfile (full control, maintenance hell) or use Docker Build Cloud (more dependencies, more complexity). Both work. Both suck in their own ways.\u003c/p\u003e\n\u003cp\u003e.NET 8 tried this with \u003ccode\u003eMicrosoft.NET.Build.Containers\u003c/code\u003e—opt-in, awkward, felt bolted-on. .NET 10 makes it first-class. There are still limits, but the core experience is solid.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"how-it-works\"\u003e\u003ca href=\"/posts/dotnet-10-cli-devops/#how-it-works\" title=\"How It Works\"\u003eHow It Works\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eOne command:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003edotnet publish --os linux --arch x64 /t:PublishContainer\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eCompiles for Linux x64. Packages as OCI container. Tags from project metadata. Pushes if you have credentials.\u003c/p\u003e\n\u003cp\u003eNo Dockerfile. No multi-stage builds. No base image debates. The CLI handles it using project defaults—works great for standard apps, questionable for edge cases.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"practical-implications\"\u003e\u003ca href=\"/posts/dotnet-10-cli-devops/#practical-implications\" title=\"Practical Implications\"\u003ePractical Implications\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eWhere this really shines: CI/CD pipelines. I\u0026rsquo;ve been maintaining GitHub Actions workflows that look like this for years:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eBuild\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edotnet build --configuration Release\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eBuild Docker Image\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edocker build -t myapp:latest .\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ePush to Registry\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edocker push myapp:latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eNow:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ePublish Container\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edotnet publish --os linux --arch x64 /t:PublishContainer\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eDocker\u0026rsquo;s gone. The CLI handles everything. For lightweight build agents or K8s-based CI, this cuts build times and removes a dependency I\u0026rsquo;ve been wanting to ditch.\u003c/p\u003e\n\u003cp\u003eBut here\u0026rsquo;s the trade-off—you surrender control to CLI defaults. Standard app? Perfect. Need custom base images, specific layers, or complex configs? You\u0026rsquo;re back to Dockerfiles. The CLI won\u0026rsquo;t bend that far yet.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"when-dockerfiles-remain-necessary\"\u003e\u003ca href=\"/posts/dotnet-10-cli-devops/#when-dockerfiles-remain-necessary\" title=\"When Dockerfiles Remain Necessary\"\u003eWhen Dockerfiles Remain Necessary\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eDockerfiles aren\u0026rsquo;t dead—just optional for simple cases. Need custom layers, multi-stage builds, or complex runtime configs? Use Dockerfiles. For standard ASP.NET Core on Linux? CLI wins.\u003c/p\u003e\n\u003cp\u003eThe philosophy makes sense: You shouldn\u0026rsquo;t need Docker expertise to containerize a .NET app. But production systems need security scanning, vulnerability patches, compliance checks. The CLI gives you a working container. Whether it\u0026rsquo;s production-ready depends on your standards.\u003c/p\u003e\n\u003cp\u003eContainers are one piece. The other friction point in DevOps pipelines? Tool management.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"one-shot-global-tools-dotnet-tool-exec\"\u003e\u003ca href=\"/posts/dotnet-10-cli-devops/#one-shot-global-tools-dotnet-tool-exec\" title=\"One-Shot Global Tools: dotnet tool exec\"\u003eOne-Shot Global Tools: \u003ccode\u003edotnet tool exec\u003c/code\u003e\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eGlobal tools have been around since .NET Core 2.1. Install, then execute. Two steps, every time, and it gets messy fast in CI environments.\u003c/p\u003e\n\u003cp\u003e.NET 10 adds \u003ccode\u003edotnet tool exec\u003c/code\u003e. Run tools without installing. Like \u003ccode\u003enpx\u003c/code\u003e in Node.js.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"before\"\u003e\u003ca href=\"/posts/dotnet-10-cli-devops/#before\" title=\"Before\"\u003eBefore\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eHere\u0026rsquo;s what I\u0026rsquo;ve been doing in CI pipelines:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003edotnet tool install --global dotnet-format\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003edotnet format --verify-no-changes\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eAlready installed? First command fails (or needs \u003ccode\u003e--ignore-failed-sources\u003c/code\u003e). Not installed? Second fails. Managing this state across multiple build agents turns into a fragile mess with conditional logic everywhere.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"now\"\u003e\u003ca href=\"/posts/dotnet-10-cli-devops/#now\" title=\"Now\"\u003eNow\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003e.NET 10 adds \u003ccode\u003edotnet tool exec\u003c/code\u003e:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003edotnet tool \u003cspan class=\"nb\"\u003eexec\u003c/span\u003e dotnet-format -- --verify-no-changes\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eFetch, run, discard. No global state. No cleanup. No version conflicts between pipeline runs.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"why-this-matters-for-cicd\"\u003e\u003ca href=\"/posts/dotnet-10-cli-devops/#why-this-matters-for-cicd\" title=\"Why This Matters for CI/CD\"\u003eWhy This Matters for CI/CD\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eCI needs stateless builds. This delivers exactly that. Each run fetches the exact tool version, executes, done. No version conflicts. No pre-baking tools into build images.\u003c/p\u003e\n\u003cp\u003eWorking with minimal base images? Just run tools on-demand. No more Dockerfile layers dedicated to tool installations that bloat your images.\u003c/p\u003e\n\u003cp\u003eThe catch is download overhead per execution. For tools you run repeatedly in tight loops, add caching. The CLI doesn\u0026rsquo;t auto-cache between \u003ccode\u003eexec\u003c/code\u003e calls, so repeated executions add latency. In most scenarios, simplicity beats speed. High-frequency pipelines? Measure first.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"local-development-benefits\"\u003e\u003ca href=\"/posts/dotnet-10-cli-devops/#local-development-benefits\" title=\"Local Development Benefits\"\u003eLocal Development Benefits\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eThis isn\u0026rsquo;t just for CI. Working across multiple projects, I\u0026rsquo;ve always hated maintaining a global tool collection that inevitably ends up with version conflicts. Now I run project-specific tools without polluting my environment. Node.js devs have had this with \u003ccode\u003enpx\u003c/code\u003e for years—about time .NET caught up.\u003c/p\u003e\n\u003cp\u003eVersioning still matters. Need a specific version? Specify it or use \u003ccode\u003edotnet-tools.json\u003c/code\u003e. The CLI won\u0026rsquo;t guess.\u003c/p\u003e\n\u003cp\u003eWhile we\u0026rsquo;re talking about tools, there\u0026rsquo;s another improvement that tool authors will appreciate.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"multi-platform-global-tool-packaging\"\u003e\u003ca href=\"/posts/dotnet-10-cli-devops/#multi-platform-global-tool-packaging\" title=\"Multi-Platform Global Tool Packaging\"\u003eMulti-Platform Global Tool Packaging\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eOlder .NET versions claimed cross-platform support for tools. Reality? Separate packages for \u003ccode\u003elinux-x64\u003c/code\u003e, \u003ccode\u003ewin-x64\u003c/code\u003e, \u003ccode\u003eosx-arm64\u003c/code\u003e. Add native dependencies and you\u0026rsquo;re in for a nightmare.\u003c/p\u003e\n\u003cp\u003e.NET 10 improves RID handling. One NuGet package, multiple platforms. The CLI resolves the right binary at runtime based on the target platform.\u003c/p\u003e\n\u003cp\u003eWorks well for simple cases. Complex native dependencies? Still rough, but better than before.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"impact-on-tool-authors\"\u003e\u003ca href=\"/posts/dotnet-10-cli-devops/#impact-on-tool-authors\" title=\"Impact on Tool Authors\"\u003eImpact on Tool Authors\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eIf you maintain global tools, you can finally ship one package for all platforms. Less packaging hell, fewer user install failures due to platform mismatches.\u003c/p\u003e\n\u003cp\u003eNot revolutionary. Incremental. Should\u0026rsquo;ve been fixed years ago, but I\u0026rsquo;ll take it.\u003c/p\u003e\n\u003cp\u003eNow here\u0026rsquo;s the feature that flew completely under the radar but might end up being the most useful for automation.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"cli-schema-export-automation-meets-introspection\"\u003e\u003ca href=\"/posts/dotnet-10-cli-devops/#cli-schema-export-automation-meets-introspection\" title=\"CLI Schema Export: Automation Meets Introspection\"\u003eCLI Schema Export: Automation Meets Introspection\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eMachine-readable CLI schemas. Sounds boring. Actually game-changing for tooling and automation that currently relies on parsing brittle text output.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"what-is-a-cli-schema\"\u003e\u003ca href=\"/posts/dotnet-10-cli-devops/#what-is-a-cli-schema\" title=\"What Is a CLI Schema?\"\u003eWhat Is a CLI Schema?\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eIt\u0026rsquo;s structured data describing CLI commands, options, and arguments. What the CLI can execute, what parameters it accepts, how they interact.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003edotnet --info --format json\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eJSON output: CLI capabilities, SDKs, runtimes, commands. Queryable, parseable, stable across versions.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"use-cases\"\u003e\u003ca href=\"/posts/dotnet-10-cli-devops/#use-cases\" title=\"Use Cases\"\u003eUse Cases\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eWhere this gets practical:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eIDE Integration:\u003c/strong\u003e VS Code could add IntelliSense for CLI commands directly in terminal windows. They\u0026rsquo;d need to implement it first, but the foundation\u0026rsquo;s there.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCI/CD Validation:\u003c/strong\u003e Check .NET versions before your build fails halfway through. Catch environment mismatches early instead of debugging why the pipeline suddenly broke.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTooling Development:\u003c/strong\u003e Third-party tools can adapt to CLI capabilities dynamically. No more hardcoded assumptions that break when the SDK updates.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDocumentation Generation:\u003c/strong\u003e Auto-generate CLI reference docs from the schema. Always current, never stale.\u003c/p\u003e\n\u003cp\u003eThis assumes the schema stays stable across releases. So far Microsoft\u0026rsquo;s track record is decent. Worth monitoring as it matures.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"real-world-scenario\"\u003e\u003ca href=\"/posts/dotnet-10-cli-devops/#real-world-scenario\" title=\"Real-World Scenario\"\u003eReal-World Scenario\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eHere\u0026rsquo;s a concrete example. Your build script needs .NET 10 before it proceeds. Before, you\u0026rsquo;d parse text output from \u003ccode\u003edotnet --version\u003c/code\u003e—fragile and breaks whenever Microsoft tweaks the format.\u003c/p\u003e\n\u003cp\u003eNow:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003edotnet --info --format json \u003cspan class=\"p\"\u003e|\u003c/span\u003e jq -r \u003cspan class=\"s1\"\u003e\u0026#39;.sdks[] | select(.version | startswith(\u0026#34;10.\u0026#34;))\u0026#39;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eNo results? .NET 10 isn\u0026rsquo;t installed. Results? You have the exact version and installation path. Structured, reliable, resistant to cosmetic changes.\u003c/p\u003e\n\u003cp\u003eThe catch is you need \u003ccode\u003ejq\u003c/code\u003e or equivalent JSON parsing. Modern CI systems? Not a problem. Windows environments without JSON tooling pre-installed? You\u0026rsquo;ve just shifted the complexity somewhere else.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"the-bigger-picture\"\u003e\u003ca href=\"/posts/dotnet-10-cli-devops/#the-bigger-picture\" title=\"The Bigger Picture\"\u003eThe Bigger Picture\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eThis isn\u0026rsquo;t flashy. It\u0026rsquo;s foundational. The CLI transforms from a black box you invoke to a queryable API. That enables an entire class of tooling improvements—assuming the ecosystem actually builds them.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"why-these-changes-matter\"\u003e\u003ca href=\"/posts/dotnet-10-cli-devops/#why-these-changes-matter\" title=\"Why These Changes Matter\"\u003eWhy These Changes Matter\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eNone of these features made keynotes. No press coverage. Not language features or runtime magic. But here\u0026rsquo;s what\u0026rsquo;s significant: Microsoft\u0026rsquo;s finally investing in workflow ergonomics instead of just piling on features.\u003c/p\u003e\n\u003cp\u003eDevOps teams don\u0026rsquo;t need more frameworks. We need less friction. Fewer dependencies, fewer scripts, fewer components that break when environments change. The .NET 10 CLI moves toward that—not perfectly, but noticeably.\u003c/p\u003e\n\u003cp\u003eBuilt-in containers eliminate Docker as a build dependency for standard cases. Tool exec removes global state from CI pipelines, though you pay in download overhead. Multi-platform packaging simplifies distribution when native dependencies cooperate. Schemas enable automation if you have JSON parsing infrastructure.\u003c/p\u003e\n\u003cp\u003eReal pain points addressed. Real trade-offs introduced. The CLI shifted from \u0026ldquo;good enough\u0026rdquo; to \u0026ldquo;actually designed for ops work.\u0026rdquo; It\u0026rsquo;s not finished, but the direction is right.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"conclusion-evolution-not-revolution\"\u003e\u003ca href=\"/posts/dotnet-10-cli-devops/#conclusion-evolution-not-revolution\" title=\"Conclusion: Evolution, Not Revolution\"\u003eConclusion: Evolution, Not Revolution\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003e.NET 10 isn\u0026rsquo;t revolutionary. It\u0026rsquo;s evolutionary. That\u0026rsquo;s exactly what maturing platforms need—incremental wins that compound over time.\u003c/p\u003e\n\u003cp\u003eIf you\u0026rsquo;re running DevOps pipelines, building CI/CD workflows, or maintaining .NET tooling, these features aren\u0026rsquo;t curiosities. They\u0026rsquo;re practical improvements that\u0026rsquo;ll simplify your work, speed execution, and improve reliability. You just need to understand the limits.\u003c/p\u003e\n\u003cp\u003eIn environments where every eliminated dependency multiplies across thousands of builds monthly, \u0026ldquo;simpler\u0026rdquo; becomes a legitimate feature. Whether .NET 10\u0026rsquo;s CLI improvements hit \u0026ldquo;simple enough\u0026rdquo; depends on your operational context and tolerance for trade-offs.\u003c/p\u003e\n\u003cp\u003eThe direction is right. Whether it\u0026rsquo;s sufficient for your specific needs? Test it in your environment. The CLI\u0026rsquo;s finally built for DevOps work. Time to see if it holds up to production reality.\u003c/p\u003e","date_modified":"2026-05-26T10:22:03+02:00","date_published":"2025-12-25T17:00:00+01:00","id":"https://daily-devops.net/posts/dotnet-10-cli-devops/","language":"en","summary":".NET 10 CLI finally ships features DevOps teams needed years ago: built-in container builds, ephemeral tools, and machine-readable schemas across the SDK.","tags":["cli","dotnet","csharp","devops","bestpractices"],"title":".NET CLI 10 – Microsoft Finally Realizes DevOps Exists","url":"https://daily-devops.net/posts/dotnet-10-cli-devops/"},{"authors":[{"name":"Martin Stühmer","url":"https://daily-devops.net/authors/martin/"}],"content_html":"\u003cp\u003eI\u0026rsquo;ve watched developers, including myself, waste hundreds of hours on something completely avoidable. Not architecture decisions or complex algorithms—just typing. Specifically, typing the same \u003ccode\u003e.NET CLI\u003c/code\u003e commands over and over because they couldn\u0026rsquo;t quite remember the exact syntax.\u003c/p\u003e\n\u003cp\u003eYou\u0026rsquo;ve probably done it yourself. You\u0026rsquo;re about to add a NuGet package, you type \u003ccode\u003edotnet add package\u003c/code\u003e, then you pause. Was it \u003ccode\u003eMicrosoft.Extensions.Logging\u003c/code\u003e or \u003ccode\u003eMicrosoft.Extensions.Logging.Abstractions\u003c/code\u003e? You open a browser tab, search NuGet.org, find the package, copy the name, switch back to your terminal, paste it. Fifteen seconds lost. Multiply that by dozens of commands daily.\u003c/p\u003e\n\u003cp\u003eThat\u0026rsquo;s not even counting the times you mistype a command and have to run it again. Or when you forget which flags \u003ccode\u003edotnet publish\u003c/code\u003e supports and end up in \u003ccode\u003e--help\u003c/code\u003e documentation.\u003c/p\u003e\n\u003cp\u003eThe \u003ccode\u003e.NET CLI\u003c/code\u003e has technically supported tab completion for years. But getting it to work? That meant diving into PowerShell documentation, copying \u003ccode\u003eRegister-ArgumentCompleter\u003c/code\u003e snippets from Stack Overflow, debugging why it wouldn\u0026rsquo;t load properly, then maintaining that brittle setup across machines. I tried it once on a project in 2021. Gave up after the third machine where it broke differently.\u003c/p\u003e\n\u003cp\u003eWhen \u003cstrong\u003e.NET 10\u003c/strong\u003e shipped in November 2025,  Microsoft finally included what should\u0026rsquo;ve been there from day one: native completion scripts. One command. That\u0026rsquo;s it. No registration. No manual shell configuration. Just \u003ccode\u003edotnet completions script \u0026gt;\u0026gt; $PROFILE\u003c/code\u003e and you\u0026rsquo;re done.\u003c/p\u003e\n\u003cp\u003eI tested it the day the release dropped. Took me exactly 47 seconds from reading the release notes to having working completion. That\u0026rsquo;s the kind of feature that makes you wonder why you tolerated the old way for so long.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"why-tab-completion-matters-more-than-you-think\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#why-tab-completion-matters-more-than-you-think\" title=\"Why Tab Completion Matters (More Than You Think)\"\u003eWhy Tab Completion Matters (More Than You Think)\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eHere\u0026rsquo;s something I tracked for a week in October: I ran 847 \u003ccode\u003edotnet\u003c/code\u003e commands. That\u0026rsquo;s not an exceptional week—I was doing standard development work on four different projects. No CI/CD pipelines, no deployment scripts, just regular coding.\u003c/p\u003e\n\u003cp\u003eOf those 847 commands, 312 involved package names or project references. Before I enabled completion, I\u0026rsquo;d estimate I spent 10-15 seconds per command hunting for exact names. With completion? Two seconds. Tab, confirm, done.\u003c/p\u003e\n\u003cp\u003eDo the math on that. Even at a conservative 10 seconds saved per operation, that\u0026rsquo;s 3,120 seconds weekly. That\u0026rsquo;s 52 minutes I\u0026rsquo;m not spending on mechanical busywork. But here\u0026rsquo;s what really matters: the cognitive load disappears.\u003c/p\u003e\n\u003cp\u003eYou stop maintaining these useless mental indexes of syntax. I used to have \u003ccode\u003e-c\u003c/code\u003e vs \u003ccode\u003e--configuration\u003c/code\u003e memorized, along with whether it was \u003ccode\u003e--framework\u003c/code\u003e or \u003ccode\u003e-f\u003c/code\u003e, whether \u003ccode\u003epublish\u003c/code\u003e took \u003ccode\u003e--output\u003c/code\u003e or \u003ccode\u003e-o\u003c/code\u003e. Now? I type \u003ccode\u003edotnet publish -\u003c/code\u003e and press Tab. The shell shows me everything available. I pick what I need.\u003c/p\u003e\n\u003cp\u003eLast month I discovered \u003ccode\u003edotnet workload repair\u003c/code\u003e because it appeared in completion results. I\u0026rsquo;d been manually reinstalling workloads when they broke. Turns out there\u0026rsquo;s been a repair command since .NET 6. I just never knew because I never ran \u003ccode\u003edotnet --help\u003c/code\u003e looking for it.\u003c/p\u003e\n\u003cp\u003eThe modern \u003ccode\u003e.NET CLI\u003c/code\u003e does a lot more than most developers realize:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eIt queries NuGet.org in real-time as you type package names. Type \u003ccode\u003edotnet add package Micro\u003c/code\u003e and hit Tab—you\u0026rsquo;ll see \u003ccode\u003eMicrosoft.Extensions.*\u003c/code\u003e packages before you finish typing.\u003c/li\u003e\n\u003cli\u003eIt understands your solution structure. In a multi-project solution, \u003ccode\u003edotnet add reference\u003c/code\u003e will show you the actual project files available, not force you to remember paths.\u003c/li\u003e\n\u003cli\u003eNested commands like \u003ccode\u003edotnet tool install\u003c/code\u003e have their own completion contexts. The shell knows when you\u0026rsquo;re specifying a tool name vs. a version vs. a configuration flag.\u003c/li\u003e\n\u003cli\u003eSome completions are context-aware. If you\u0026rsquo;ve already specified \u003ccode\u003e--framework net8.0\u003c/code\u003e, subsequent completions adjust accordingly.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThe old approach—before \u003cstrong\u003e.NET 10\u003c/strong\u003e—worked but had a fundamental performance problem. Every tab press spawned a subprocess running \u003ccode\u003edotnet complete\u003c/code\u003e. That means process initialization overhead, parsing your command context, generating suggestions, serializing results back to PowerShell, then rendering them.\u003c/p\u003e\n\u003cp\u003eI measured this once on a moderately powerful dev machine (Ryzen 7, NVMe SSD, 32GB RAM). Simple completions like \u003ccode\u003edotnet b[Tab]\u003c/code\u003e took 80-120ms. Not terrible, but noticeable. Package completions that needed to query NuGet.org? 400-800ms depending on network latency.\u003c/p\u003e\n\u003cp\u003eThe \u003cstrong\u003e.NET 10\u003c/strong\u003e approach is architecturally different. The completion script that gets written to your \u003ccode\u003e$PROFILE\u003c/code\u003e contains the entire static grammar—every command, subcommand, and standard flag—compiled into shell-native code. Your shell (PowerShell, Bash, Zsh) can evaluate that code instantly because there\u0026rsquo;s no external process. It only invokes \u003ccode\u003edotnet complete\u003c/code\u003e when you hit something dynamic like package names or project file paths. The difference is immediately perceptible. Completions feel instant because most of them actually are instant.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-evolution-from-dynamic-to-native-completion\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#the-evolution-from-dynamic-to-native-completion\" title=\"The Evolution: From Dynamic to Native Completion\"\u003eThe Evolution: From Dynamic to Native Completion\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eUnderstanding what changed in \u003cstrong\u003e.NET 10\u003c/strong\u003e gives context to why this matters.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"the-old-way-dynamic-completion-pre-net-10\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#the-old-way-dynamic-completion-pre-net-10\" title=\"The Old Way: Dynamic Completion (Pre-.NET 10)\"\u003eThe Old Way: Dynamic Completion (Pre-.NET 10)\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eFor \u003ccode\u003e.NET\u003c/code\u003e versions before 10, if you wanted tab completion, you needed to register an argument completer in your PowerShell profile. The approach was straightforward but had overhead:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-pwsh\" data-lang=\"pwsh\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# The legacy approach that still works\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eRegister-ArgumentCompleter\u003c/span\u003e \u003cspan class=\"n\"\u003e-Native\u003c/span\u003e \u003cspan class=\"n\"\u003e-CommandName\u003c/span\u003e \u003cspan class=\"n\"\u003edotnet\u003c/span\u003e \u003cspan class=\"n\"\u003e-ScriptBlock\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"k\"\u003eparam\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nv\"\u003e$wordToComplete\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nv\"\u003e$commandAst\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nv\"\u003e$cursorPosition\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"n\"\u003edotnet\u003c/span\u003e \u003cspan class=\"n\"\u003ecomplete\u003c/span\u003e \u003cspan class=\"p\"\u003e-\u003c/span\u003e\u003cspan class=\"n\"\u003e-position\u003c/span\u003e \u003cspan class=\"nv\"\u003e$cursorPosition\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e\u003cspan class=\"nv\"\u003e$commandAst\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;\u003c/span\u003e \u003cspan class=\"p\"\u003e|\u003c/span\u003e \u003cspan class=\"nb\"\u003eForEach-Object\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        \u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"no\"\u003eSystem.Management.Automation.CompletionResult\u003c/span\u003e\u003cspan class=\"p\"\u003e]::\u003c/span\u003e\u003cspan class=\"n\"\u003enew\u003c/span\u003e\u003cspan class=\"p\"\u003e(\u003c/span\u003e\u003cspan class=\"nv\"\u003e$_\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nv\"\u003e$_\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"s1\"\u003e\u0026#39;ParameterValue\u0026#39;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e \u003cspan class=\"nv\"\u003e$_\u003c/span\u003e\u003cspan class=\"p\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis approach technically worked. But every Tab press meant PowerShell had to invoke \u003ccode\u003edotnet complete\u003c/code\u003e as a subprocess, wait for it to parse your current command, generate completions, and return them. On my old laptop (circa 2019), this sometimes took long enough that I\u0026rsquo;d press Tab, see nothing happen, assume completion wasn\u0026rsquo;t working, and just finish typing manually.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"the-new-way-native-completions-net-10\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#the-new-way-native-completions-net-10\" title=\"The New Way: Native Completions (.NET 10\u0026#43;)\"\u003eThe New Way: Native Completions (.NET 10+)\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eEnter \u003cstrong\u003e.NET 10\u003c/strong\u003e. Microsoft introduced the \u003ccode\u003edotnet completions script\u003c/code\u003e command that generates shell-specific completion code. This code:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eHandles static grammar directly\u003c/strong\u003e in the shell without invoking \u003ccode\u003edotnet complete\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eFalls back intelligently\u003c/strong\u003e only for dynamic content (like NuGet package names)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eIntegrates natively\u003c/strong\u003e with your shell\u0026rsquo;s completion system\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eDelivers near-instant results\u003c/strong\u003e for common operations\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eThe result? Noticeably faster, smoother completion experience.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-one-liner-that-changes-everything\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#the-one-liner-that-changes-everything\" title=\"The One-Liner That Changes Everything\"\u003eThe One-Liner That Changes Everything\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eEverything we\u0026rsquo;ve discussed leads to this single, elegant line:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-pwsh\" data-lang=\"pwsh\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003edotnet\u003c/span\u003e \u003cspan class=\"n\"\u003ecompletions\u003c/span\u003e \u003cspan class=\"n\"\u003escript\u003c/span\u003e \u003cspan class=\"p\"\u003e\u0026gt;\u0026gt;\u003c/span\u003e \u003cspan class=\"nv\"\u003e$PROFILE\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThat\u0026rsquo;s genuinely all you need. One command, executed once, and you\u0026rsquo;re done. Tab completion works from that moment forward, persisting through every future session.\u003c/p\u003e\n\u003cp\u003eNow, while you could stop here and be perfectly fine, understanding what\u0026rsquo;s actually happening beneath the surface transforms this from \u0026ldquo;magic command\u0026rdquo; to something you can troubleshoot and maintain confidently. So let\u0026rsquo;s break down what\u0026rsquo;s really going on.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"what-this-command-does\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#what-this-command-does\" title=\"What This Command Does\"\u003eWhat This Command Does\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eLet\u0026rsquo;s examine each component. First, the command itself:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-pwsh\" data-lang=\"pwsh\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003edotnet\u003c/span\u003e \u003cspan class=\"n\"\u003ecompletions\u003c/span\u003e \u003cspan class=\"n\"\u003escript\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis single command tells the \u003ccode\u003e.NET CLI\u003c/code\u003e to generate a completion script for your shell. The \u003ccode\u003edotnet\u003c/code\u003e executable is intelligent about this—it examines your environment, detects which shell you\u0026rsquo;re currently running, and outputs the appropriate completion code for that shell. On Windows systems, it defaults to PowerShell (\u003ccode\u003epwsh\u003c/code\u003e). On Linux or macOS, it checks your environment variables to determine whether you\u0026rsquo;re using Bash, Zsh, Fish, or Nushell, and generates the right script accordingly. This automatic detection removes another friction point—you don\u0026rsquo;t have to tell it which shell you want.\u003c/p\u003e\n\u003cp\u003eThe second part of the equation is the redirection operator:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-pwsh\" data-lang=\"pwsh\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e\u0026gt;\u0026gt;\u003c/span\u003e \u003cspan class=\"nv\"\u003e$PROFILE\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe double angle-bracket (\u003ccode\u003e\u0026gt;\u0026gt;\u003c/code\u003e) is PowerShell\u0026rsquo;s append operator. It takes everything the \u003ccode\u003edotnet completions script\u003c/code\u003e command outputs and appends it to your PowerShell profile file. The \u003ccode\u003e$PROFILE\u003c/code\u003e is an automatic variable that PowerShell sets during startup—it points to your current user\u0026rsquo;s current host profile. For most Windows developers, this lives at:\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003e$HOME\\Documents\\PowerShell\\Microsoft.PowerShell_profile.ps1\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eBut the beauty of using \u003ccode\u003e$PROFILE\u003c/code\u003e is that you don\u0026rsquo;t need to know or remember the exact path. PowerShell handles it for you.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"why-this-approach-is-brilliant\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#why-this-approach-is-brilliant\" title=\"Why This Approach Is Brilliant\"\u003eWhy This Approach Is Brilliant\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eThis one-liner is a masterclass in pragmatic design. It leverages several elegant PowerShell concepts that make it simultaneously powerful and forgiving:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAutomatic environment detection\u003c/strong\u003e means you don\u0026rsquo;t need to tell the \u003ccode\u003edotnet\u003c/code\u003e executable anything about your shell. It figures it out. This eliminates the most common mistake people make with shell configuration—specifying the wrong shell or format. You run one command, and the correct code for your exact environment is generated.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eProfile persistence\u003c/strong\u003e ensures your setup survives across sessions. Unlike configuration that lives only in your current terminal, changes to your profile apply every time you open a new PowerShell window. This is how you move from \u0026ldquo;temporary configuration\u0026rdquo; to \u0026ldquo;permanent improvement.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSafe appending\u003c/strong\u003e with the \u003ccode\u003e\u0026gt;\u0026gt;\u003c/code\u003e operator is crucial. This isn\u0026rsquo;t destructive. You\u0026rsquo;re not overwriting your profile—you\u0026rsquo;re adding to it. If you\u0026rsquo;ve already customized your profile with functions, aliases, or other settings, they all remain untouched. The completion script just appends at the end. This means you can run the command multiple times without fear. It\u0026rsquo;s idempotent.\u003c/p\u003e\n\u003cp\u003eThis combination of automatic detection, persistence, and safety is pragmatic design at its best. It removes the annoying steps that plague so many technical setup processes.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"making-it-stick-the-powershell-profile-deep-dive\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#making-it-stick-the-powershell-profile-deep-dive\" title=\"Making It Stick: The PowerShell Profile Deep Dive\"\u003eMaking It Stick: The PowerShell Profile Deep Dive\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eThis is where the process becomes slightly more involved—but also where understanding matters. Your profile file must exist before you can append to it, and it must actually load when PowerShell starts. Both of these requirements are usually met, but not always. Let\u0026rsquo;s make sure you\u0026rsquo;re covered.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"verifying-your-profile-path\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#verifying-your-profile-path\" title=\"Verifying Your Profile Path\"\u003eVerifying Your Profile Path\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eFirst, see where PowerShell thinks your profile lives:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-pwsh\" data-lang=\"pwsh\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003e$PROFILE\u003c/span\u003e \u003cspan class=\"p\"\u003e|\u003c/span\u003e \u003cspan class=\"nb\"\u003eSelect-Object\u003c/span\u003e \u003cspan class=\"p\"\u003e*\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis will show you all available profile paths. The one labeled \u003ccode\u003eMicrosoft.PowerShell_profile.ps1\u003c/code\u003e in your Documents folder is what we care about.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"creating-your-profile-if-it-doesnt-exist\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#creating-your-profile-if-it-doesnt-exist\" title=\"Creating Your Profile If It Doesn\u0026rsquo;t Exist\"\u003eCreating Your Profile If It Doesn\u0026rsquo;t Exist\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003ePowerShell doesn\u0026rsquo;t create your profile automatically. If you\u0026rsquo;ve never customized PowerShell before, you might not have one. Create it like this:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-pwsh\" data-lang=\"pwsh\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"p\"\u003e(!(\u003c/span\u003e\u003cspan class=\"nb\"\u003eTest-Path\u003c/span\u003e \u003cspan class=\"n\"\u003e-Path\u003c/span\u003e \u003cspan class=\"nv\"\u003e$PROFILE\u003c/span\u003e\u003cspan class=\"p\"\u003e))\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nb\"\u003eNew-Item\u003c/span\u003e \u003cspan class=\"n\"\u003e-ItemType\u003c/span\u003e \u003cspan class=\"n\"\u003eFile\u003c/span\u003e \u003cspan class=\"n\"\u003e-Path\u003c/span\u003e \u003cspan class=\"nv\"\u003e$PROFILE\u003c/span\u003e \u003cspan class=\"n\"\u003e-Force\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis is defensive: if the profile doesn\u0026rsquo;t exist, create it. If it does, do nothing.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"actually-adding-tab-completion\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#actually-adding-tab-completion\" title=\"Actually Adding Tab Completion\"\u003eActually Adding Tab Completion\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eNow that you know your profile exists, add the completion:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-pwsh\" data-lang=\"pwsh\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003edotnet\u003c/span\u003e \u003cspan class=\"n\"\u003ecompletions\u003c/span\u003e \u003cspan class=\"n\"\u003escript\u003c/span\u003e \u003cspan class=\"p\"\u003e\u0026gt;\u0026gt;\u003c/span\u003e \u003cspan class=\"nv\"\u003e$PROFILE\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\n\n\n\n\u003ch3 id=\"activating-it-in-your-current-session\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#activating-it-in-your-current-session\" title=\"Activating It in Your Current Session\"\u003eActivating It in Your Current Session\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eThe profile runs automatically when you open a new PowerShell window. But if you want tab completion right now in your current session, reload the profile:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-pwsh\" data-lang=\"pwsh\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e \u003cspan class=\"nv\"\u003e$PROFILE\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThe dot (\u003ccode\u003e.\u003c/code\u003e) is PowerShell\u0026rsquo;s dot-sourcing operator. It executes the profile file in the current session\u0026rsquo;s scope, making all its contents immediately available.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"testing-your-setup\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#testing-your-setup\" title=\"Testing Your Setup\"\u003eTesting Your Setup\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eAfter you\u0026rsquo;ve run the setup, test it immediately. Don\u0026rsquo;t trust that it worked—verify it. Open PowerShell and type:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-pwsh\" data-lang=\"pwsh\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003edotnet\u003c/span\u003e \u003cspan class=\"n\"\u003ea\u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"no\"\u003eTab\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eIf it\u0026rsquo;s working, you\u0026rsquo;ll see \u003ccode\u003eadd\u003c/code\u003e appear instantly. Press Tab again and you\u0026rsquo;ll cycle through \u003ccode\u003eanalyze\u003c/code\u003e and any other \u0026lsquo;a\u0026rsquo; commands. That\u0026rsquo;s the static completion engine—your shell already knows those commands exist.\u003c/p\u003e\n\u003cp\u003eIf nothing happens, something\u0026rsquo;s wrong. Don\u0026rsquo;t waste time wondering why. Jump to the troubleshooting section below.\u003c/p\u003e\n\u003cp\u003eNow let\u0026rsquo;s test something more complex that exercises the dynamic side of completion:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-pwsh\" data-lang=\"pwsh\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003edotnet\u003c/span\u003e \u003cspan class=\"n\"\u003eadd\u003c/span\u003e \u003cspan class=\"n\"\u003epackage\u003c/span\u003e \u003cspan class=\"n\"\u003eMicro\u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"no\"\u003eTab\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eType this exactly and press Tab. Within a moment, you\u0026rsquo;ll see suggestions for NuGet packages starting with \u0026ldquo;Micro\u0026rdquo;—\u003ccode\u003eMicrosoft.AspNetCore.App\u003c/code\u003e, \u003ccode\u003eMicrosoft.Extensions.Logging\u003c/code\u003e, and dozens of others. This is where the hybrid approach really shines. Your shell instantly handles the known parts (\u003ccode\u003edotnet add package\u003c/code\u003e), then intelligently queries NuGet.org for available packages that match your prefix. You see results without any delay, yet they\u0026rsquo;re genuinely dynamic and current.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"understanding-completion-modes-why-hybrid-matters\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#understanding-completion-modes-why-hybrid-matters\" title=\"Understanding Completion Modes: Why Hybrid Matters\"\u003eUnderstanding Completion Modes: Why Hybrid Matters\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eMicrosoft\u0026rsquo;s documentation distinguishes between two different completion strategies, and understanding this distinction helps you appreciate why \u003cstrong\u003e.NET 10\u003c/strong\u003e is such a significant upgrade.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"hybrid-completion-the-net-10-standard\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#hybrid-completion-the-net-10-standard\" title=\"Hybrid Completion: The .NET 10 Standard\"\u003eHybrid Completion: The \u003cstrong\u003e.NET 10\u003c/strong\u003e Standard\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eHybrid completion is what you get when using the native completion scripts in \u003cstrong\u003e.NET 10\u003c/strong\u003e or later with PowerShell, Bash, or Zsh. The strategy is elegantly split:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eStatic grammar is handled directly\u003c/strong\u003e by shell code that was generated specifically for your shell. The shell already knows about all the \u003ccode\u003e.NET CLI\u003c/code\u003e commands, subcommands, and standard flags. This runs instantly, without any external process.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eDynamic content\u003c/strong\u003e triggers \u003ccode\u003edotnet complete\u003c/code\u003e only when necessary. Package names, project files, and context-specific values are fetched on demand, but only when you actually need them.\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eThis hybrid architecture is why completion feels so responsive. I compared it directly: on the same machine, the old \u003ccode\u003eRegister-ArgumentCompleter\u003c/code\u003e approach took 95ms average for static completions. Native completion? 8ms. That\u0026rsquo;s over 10x faster, and you feel it.\u003c/p\u003e\n\u003cp\u003eFor dynamic package completions, both approaches need to query NuGet, so they\u0026rsquo;re roughly equivalent (around 500ms depending on your network). But the difference is that 90% of your completions are static. Microsoft clearly spent time optimizing where it matters most.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"dynamic-completion-the-legacy-approach\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#dynamic-completion-the-legacy-approach\" title=\"Dynamic Completion: The Legacy Approach\"\u003eDynamic Completion: The Legacy Approach\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eIf you\u0026rsquo;re running \u003ccode\u003e.NET 9\u003c/code\u003e or earlier, or if you\u0026rsquo;ve configured completion using the older registration method, every single completion request—even for static commands—invokes the \u003ccode\u003edotnet complete\u003c/code\u003e command in a subprocess. This approach works, absolutely. But it\u0026rsquo;s noticeably slower. You press Tab and wait for a process to start, execute, and return results. For simple completions, the wait is usually acceptable. But for comprehensive, package-aware completions, many developers notice the latency.\u003c/p\u003e\n\u003cp\u003eThis is why upgrading to \u003cstrong\u003e.NET 10\u003c/strong\u003e and enabling native completion is worth doing. You\u0026rsquo;re not just getting a feature—you\u0026rsquo;re getting a more responsive development experience.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"when-should-you-enable-this\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#when-should-you-enable-this\" title=\"When Should You Enable This?\"\u003eWhen Should You Enable This?\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eThe honest answer: immediately. There is genuinely no downside to enabling tab completion for the \u003ccode\u003e.NET CLI\u003c/code\u003e. It\u0026rsquo;s pure upside—faster work, fewer mistakes, better exploration of available commands—with zero risk and minimal setup.\u003c/p\u003e\n\u003cp\u003eYou\u0026rsquo;ll see particularly significant benefits if you:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCreate projects frequently\u003c/strong\u003e using \u003ccode\u003edotnet new\u003c/code\u003e. Template completion means you stop guessing at template names and let your shell guide you through available options. This is especially valuable when you need templates for specific purposes and can\u0026rsquo;t quite remember the exact name.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eManage NuGet dependencies regularly\u003c/strong\u003e with \u003ccode\u003edotnet add package\u003c/code\u003e. Package completion transforms this from \u0026ldquo;Hunt for the right package on NuGet.org, copy the name, paste it in the terminal\u0026rdquo; to \u0026ldquo;Type a prefix and tab through suggestions.\u0026rdquo;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWork with multiple solutions and projects\u003c/strong\u003e where \u003ccode\u003edotnet publish\u003c/code\u003e, \u003ccode\u003edotnet pack\u003c/code\u003e, and similar commands need project file context. Your shell becomes aware of your actual project structure and can complete project paths intelligently.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eUse complex build or publish profiles\u003c/strong\u003e where remembering the exact configuration names and publish targets becomes tedious. Let completion handle the recall.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWant to explore CLI capabilities\u003c/strong\u003e beyond the commands you use regularly. Completion surfaces available subcommands and options, making it easier to discover capabilities you didn\u0026rsquo;t know existed. How many developers skip learning some feature simply because they didn\u0026rsquo;t know it was available?\u003c/p\u003e\n\u003cp\u003eEven if you\u0026rsquo;re an IDE enthusiast who spends most time in Visual Studio rather than the terminal, tab completion removes a psychological barrier to shell adoption. Knowing the tool will help you remember what you need makes you more willing to use it. That\u0026rsquo;s a genuine quality-of-life improvement.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"when-things-dont-work-troubleshooting\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#when-things-dont-work-troubleshooting\" title=\"When Things Don\u0026rsquo;t Work: Troubleshooting\"\u003eWhen Things Don\u0026rsquo;t Work: Troubleshooting\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eMost of the time this works on the first try. But I\u0026rsquo;ve helped enough people set this up to know the failure modes. Here\u0026rsquo;s what actually breaks and how to fix it.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"your-profile-exists-but-isnt-loading\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#your-profile-exists-but-isnt-loading\" title=\"Your Profile Exists But Isn\u0026rsquo;t Loading\"\u003eYour Profile Exists But Isn\u0026rsquo;t Loading\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eThe most common issue is that your profile file exists, but PowerShell isn\u0026rsquo;t executing it. This usually indicates an execution policy problem. Check what policy is currently set:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-pwsh\" data-lang=\"pwsh\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eGet-ExecutionPolicy\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eIf this returns \u003ccode\u003eRestricted\u003c/code\u003e, PowerShell refuses to run any scripts, including your profile. This is the default on many systems. You need to change it. The recommended setting is \u003ccode\u003eRemoteSigned\u003c/code\u003e, which allows scripts you created locally to run while blocking scripts downloaded from the internet—a good security balance:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-pwsh\" data-lang=\"pwsh\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eSet-ExecutionPolicy\u003c/span\u003e \u003cspan class=\"n\"\u003e-ExecutionPolicy\u003c/span\u003e \u003cspan class=\"n\"\u003eRemoteSigned\u003c/span\u003e \u003cspan class=\"n\"\u003e-Scope\u003c/span\u003e \u003cspan class=\"n\"\u003eCurrentUser\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis sets the policy for your user account specifically, without requiring administrative elevation. Once you\u0026rsquo;ve run this, your profile will load and execute automatically.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"completion-still-isnt-working-after-setup\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#completion-still-isnt-working-after-setup\" title=\"Completion Still Isn\u0026rsquo;t Working After Setup\"\u003eCompletion Still Isn\u0026rsquo;t Working After Setup\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eYour profile loaded, but tab completion still doesn\u0026rsquo;t appear when you try it. Walk through this checklist:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eVerify you have \u003cstrong\u003e.NET 10\u003c/strong\u003e or later installed:\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-pwsh\" data-lang=\"pwsh\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003edotnet\u003c/span\u003e \u003cspan class=\"p\"\u003e-\u003c/span\u003e\u003cspan class=\"n\"\u003e-version\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eNative completion scripts only exist in \u003ccode\u003e.NET 10+\u003c/code\u003e. If you\u0026rsquo;re on \u003ccode\u003e.NET 9\u003c/code\u003e or earlier, the command won\u0026rsquo;t generate anything.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eConfirm the completion script was actually appended to your profile:\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-pwsh\" data-lang=\"pwsh\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003eGet-Content\u003c/span\u003e \u003cspan class=\"nv\"\u003e$PROFILE\u003c/span\u003e \u003cspan class=\"p\"\u003e|\u003c/span\u003e \u003cspan class=\"nb\"\u003eSelect-String\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;dotnet completions\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis checks whether the profile file contains the completion script. If it returns nothing, the append didn\u0026rsquo;t work. Check that your profile path is accessible and writable.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eReload your profile or open a fresh terminal:\u003c/strong\u003e\nIf the script is there but completion doesn\u0026rsquo;t work, your current session hasn\u0026rsquo;t loaded it yet. Run \u003ccode\u003e. $PROFILE\u003c/code\u003e to reload, or simply close and reopen PowerShell.\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ol\u003e\n\n\n\n\n\u003ch3 id=\"completion-feels-slow\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#completion-feels-slow\" title=\"Completion Feels Slow\"\u003eCompletion Feels Slow\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eIf you notice completion taking a second or two to respond, especially on complex queries, you\u0026rsquo;re likely experiencing the dynamic fallback in action. When you request NuGet package suggestions or other context-dependent completions, PowerShell invokes \u003ccode\u003edotnet complete\u003c/code\u003e in the background. This is expected behavior, not a bug. The hybrid approach minimizes this latency for static completions, but truly dynamic data sometimes requires a moment. For most users, the responsiveness improvement over pre-.NET 10 completion is still significant.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-clis-evolving-scope-context-matters\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#the-clis-evolving-scope-context-matters\" title=\"The CLI\u0026rsquo;s Evolving Scope: Context Matters\"\u003eThe CLI\u0026rsquo;s Evolving Scope: Context Matters\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eTab completion might seem like a small feature, but it\u0026rsquo;s actually a signal of something larger happening with the \u003ccode\u003e.NET CLI\u003c/code\u003e. The tool has evolved dramatically. Consider what you can accomplish from the command line today:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eNative shell completions\u003c/strong\u003e now exist for PowerShell, Bash, Zsh, Fish, and Nushell—recognizing that .NET developers work across different operating systems and shells.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWorkload management\u003c/strong\u003e lets you install and manage platform-specific tools directly through the CLI—Swift for iOS development, NDK tooling, emulators.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eGlobal tools and tool manifests\u003c/strong\u003e turn your development environment into a versioned, reproducible collection of utilities that travel with your projects.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSolution-level operations and dependency management\u003c/strong\u003e mean the CLI understands your entire solution structure, not just individual projects.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eBuilt-in diagnostics and observability\u003c/strong\u003e help you understand what\u0026rsquo;s happening under the hood—from environment information to detailed build diagnostics.\u003c/p\u003e\n\u003cp\u003eThe journey from early \u003ccode\u003e.NET Core\u003c/code\u003e days to now is remarkable. The CLI started as a basic project builder. It\u0026rsquo;s now a sophisticated platform with growing capabilities. Tab completion, in this context, is Microsoft making a statement: we\u0026rsquo;ve built something complex and powerful, and we\u0026rsquo;re committed to making it accessible. You shouldn\u0026rsquo;t need to memorize obscure syntax or hunt through documentation for common operations. The tool should guide you. Completion is that commitment made practical.\u003c/p\u003e\n\u003cp\u003eFor developers who live at the command line, this kind of incremental thoughtfulness adds up. It\u0026rsquo;s not revolutionary. But it\u0026rsquo;s genuine progress.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-lazy-developers-summary\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#the-lazy-developers-summary\" title=\"The Lazy Developer\u0026rsquo;s Summary\"\u003eThe Lazy Developer\u0026rsquo;s Summary\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eWant the fastest path to productivity? Here\u0026rsquo;s the checklist:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\n\u003cp\u003eMake sure your PowerShell profile exists:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-pwsh\" data-lang=\"pwsh\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"k\"\u003eif\u003c/span\u003e \u003cspan class=\"p\"\u003e(!(\u003c/span\u003e\u003cspan class=\"nb\"\u003eTest-Path\u003c/span\u003e \u003cspan class=\"n\"\u003e-Path\u003c/span\u003e \u003cspan class=\"nv\"\u003e$PROFILE\u003c/span\u003e\u003cspan class=\"p\"\u003e))\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nb\"\u003eNew-Item\u003c/span\u003e \u003cspan class=\"n\"\u003e-ItemType\u003c/span\u003e \u003cspan class=\"n\"\u003eFile\u003c/span\u003e \u003cspan class=\"n\"\u003e-Path\u003c/span\u003e \u003cspan class=\"nv\"\u003e$PROFILE\u003c/span\u003e \u003cspan class=\"n\"\u003e-Force\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eAdd the completion script:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-pwsh\" data-lang=\"pwsh\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003edotnet\u003c/span\u003e \u003cspan class=\"n\"\u003ecompletions\u003c/span\u003e \u003cspan class=\"n\"\u003escript\u003c/span\u003e \u003cspan class=\"p\"\u003e\u0026gt;\u0026gt;\u003c/span\u003e \u003cspan class=\"nv\"\u003e$PROFILE\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eReload in your current session:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-pwsh\" data-lang=\"pwsh\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e.\u003c/span\u003e \u003cspan class=\"nv\"\u003e$PROFILE\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eVerify it works:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-pwsh\" data-lang=\"pwsh\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"n\"\u003edotnet\u003c/span\u003e \u003cspan class=\"n\"\u003ea\u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"no\"\u003eTab\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eThe beauty of this approach is that it\u0026rsquo;s idempotent. You can run the second command multiple times; it just appends to your profile. It\u0026rsquo;s not elegant, but it\u0026rsquo;s practical.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"final-thoughts-the-compound-effect-of-small-improvements\"\u003e\u003ca href=\"/posts/dotnet-cli-expanding-scope-autocomplete/#final-thoughts-the-compound-effect-of-small-improvements\" title=\"Final Thoughts: The Compound Effect of Small Improvements\"\u003eFinal Thoughts: The Compound Effect of Small Improvements\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eLook, I get it. Tab completion sounds trivial. \u0026ldquo;Just learn the commands\u0026rdquo; or \u0026ldquo;use the IDE\u0026rdquo; or whatever. I\u0026rsquo;ve heard all the dismissive responses.\u003c/p\u003e\n\u003cp\u003eBut here\u0026rsquo;s what actually happened after I enabled this: I stopped avoiding the CLI. Before, I\u0026rsquo;d often reach for Visual Studio\u0026rsquo;s NuGet manager or the solution explorer because I didn\u0026rsquo;t want to fight with command syntax. Now I stay in the terminal because it\u0026rsquo;s genuinely faster than switching contexts.\u003c/p\u003e\n\u003cp\u003eLast week I added twelve NuGet packages across four projects. Total time: maybe two minutes. Six months ago that would\u0026rsquo;ve been ten minutes minimum—switching to VS, waiting for NuGet to load, searching, selecting versions, clicking Install, waiting for restore.\u003c/p\u003e\n\u003cp\u003eThe time savings are real (I tracked 52 minutes weekly, remember), but the bigger win is staying in flow. Every context switch costs you focus. Every moment you spend hunting for syntax is a moment you\u0026rsquo;re not solving the actual problem.\u003c/p\u003e\n\u003cp\u003eThe \u003ccode\u003e.NET CLI\u003c/code\u003e has become a genuinely sophisticated tool—powerful enough to accomplish real work from the command line, yet complex enough that most developers never explore its full capabilities. Native tab completion in \u003cstrong\u003e.NET 10\u003c/strong\u003e is the accessibility layer that makes that power usable without constant cognitive overhead. It\u0026rsquo;s not flashy. It\u0026rsquo;s not revolutionary. But it\u0026rsquo;s the kind of thoughtful engineering that separates tools you tolerate from tools you actually \u003cem\u003eenjoy\u003c/em\u003e using.\u003c/p\u003e\n\u003cp\u003eThe setup takes ninety seconds. I timed it. Actually, I\u0026rsquo;ve timed it on six different machines now helping colleagues set this up. Longest was two minutes because one person had a permissions issue with their profile directory.\u003c/p\u003e\n\u003cp\u003eDo it now. Seriously—stop reading, open PowerShell, run the three commands in the summary below, test it with \u003ccode\u003edotnet a[Tab]\u003c/code\u003e. If it works, you just saved yourself dozens of hours over the next year. If it doesn\u0026rsquo;t work, the troubleshooting section will get you sorted.\u003c/p\u003e\n\u003cp\u003eI\u0026rsquo;ve been using this daily since November 2024. It\u0026rsquo;s one of those rare features that actually lives up to the promise. No gotchas, no edge cases where it breaks, just consistent quality-of-life improvement every single time I touch the CLI.\u003c/p\u003e\n","date_modified":"2026-05-26T10:22:03+02:00","date_published":"2025-12-18T17:00:00+01:00","id":"https://daily-devops.net/posts/dotnet-cli-expanding-scope-autocomplete/","language":"en","summary":".NET 10 ships native tab completion for the dotnet CLI. One command, no Register-ArgumentCompleter snippets, and your shell finally remembers.","tags":["cli","dotnet","bestpractices","devops","softwareengineering"],"title":"Stop Typing: The .NET CLI Tab Completion You've Been Missing","url":"https://daily-devops.net/posts/dotnet-cli-expanding-scope-autocomplete/"},{"authors":[{"name":"Martin Stühmer","url":"https://daily-devops.net/authors/martin/"}],"content_html":"\u003cp\u003eMicrosoft just did something unusual: \u003cem\u003ethey fixed a problem before most people realized they had it.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eFor years, \u003ccode\u003edotnet test\u003c/code\u003e wasn\u0026rsquo;t really a test runner—it was actually just a wrapper around \u003ccode\u003evstest.console.exe\u003c/code\u003e, a legacy artifact from the pre-.NET-Core era that Microsoft couldn\u0026rsquo;t quite kill. It worked, mostly, if you didn\u0026rsquo;t think too hard about why your tests sometimes behaved differently in Visual Studio than in GitHub Actions, or why test discovery occasionally took longer than the tests themselves.\u003c/p\u003e\n\u003cp\u003eWith .NET 10, Microsoft has finally integrated testing directly into the SDK through \u003cstrong\u003eMicrosoft.Testing.Platform (MTP)\u003c/strong\u003e. The old VSTest infrastructure is now out. The new system runs tests in-process, unifies behavior across environments, and—this is actually the important part—finally respects your configuration files.\u003c/p\u003e\n\u003cp\u003eThere\u0026rsquo;s a catch, of course. There always is.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"from-test-wrapper-to-test-platform\"\u003e\u003ca href=\"/posts/dotnet-10-testing/#from-test-wrapper-to-test-platform\" title=\"From Test Wrapper to Test Platform\"\u003eFrom Test Wrapper to Test Platform\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eRunning tests in .NET used to mean choosing a framework—\u003ca href=\"https://xunit.net/\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003exUnit\u003c/a\u003e, \u003ca href=\"https://nunit.org/\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eNUnit\u003c/a\u003e, \u003ca href=\"https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-with-mstest\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eMSTest\u003c/a\u003e, or the newer \u003ca href=\"https://tunit.dev/\" target=\"_blank\" rel=\"noopener external noreferrer\"\u003eTUnit\u003c/a\u003e—and then essentially just hoping \u003ccode\u003edotnet test\u003c/code\u003e could somehow figure out how to talk to it. Each framework had its own test adapter. Each adapter had its own quirks. Your CI pipeline basically just crossed its fingers and hoped for green checkmarks.\u003c/p\u003e\n\u003cp\u003eThe result? Test execution that varied subtly between your laptop, your colleague\u0026rsquo;s laptop, and the build server. Debugging test failures meant first figuring out \u003cem\u003ewhich version of which adapter was running where\u003c/em\u003e.\u003c/p\u003e\n\u003cp\u003eMicrosoft.Testing.Platform changes that architecture. Instead of spawning separate processes and negotiating through adapters, MTP embeds the test runner directly into the SDK. Discovery, execution, and reporting now follow a single, predictable path. Tests run in-process. The CLI is cleaner. The performance is measurably better in projects with large test suites.\u003c/p\u003e\n\u003cp\u003eEnabling it requires exactly four lines in your \u003ccode\u003eglobal.json\u003c/code\u003e:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-json\" data-lang=\"json\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"nt\"\u003e\u0026#34;test\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"p\"\u003e{\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    \u003cspan class=\"nt\"\u003e\u0026#34;runner\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;Microsoft.Testing.Platform\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e  \u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e}\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eNo SDK pinning required. No complicated setup. Just those four lines, and .NET 10 switches to the new test engine automatically.\u003c/p\u003e\n\u003cp\u003eThe simplicity is almost suspicious.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"what-actually-improves-and-what-doesnt\"\u003e\u003ca href=\"/posts/dotnet-10-testing/#what-actually-improves-and-what-doesnt\" title=\"What Actually Improves (And What Doesn\u0026rsquo;t)\"\u003eWhat Actually Improves (And What Doesn\u0026rsquo;t)\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eLet\u0026rsquo;s be specific. MTP isn\u0026rsquo;t magic—it\u0026rsquo;s engineering. Here\u0026rsquo;s what changes when you enable it:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTest discovery is faster.\u003c/strong\u003e In a project with ~3,500 tests, discovery dropped from 8 seconds to under 3 on my local machine. That\u0026rsquo;s honestly not earth-shattering, but it\u0026rsquo;s definitely noticeable when you\u0026rsquo;re running focused test sets repeatedly during development. Over a typical workday with 50 test runs? That actually saves roughly 4 minutes. Not revolutionary, but certainly not nothing either.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eThe CLI makes sense now.\u003c/strong\u003e Previously, \u003ccode\u003edotnet test --filter\u003c/code\u003e required arcane syntax and those bizarre \u003ccode\u003e--\u003c/code\u003e separators to pass arguments through to the adapter. MTP removes that layer of indirection. The commands do what you\u0026rsquo;d expect without translation.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEnvironment consistency improves.\u003c/strong\u003e Because the test runner is part of the SDK, your local machine and your CI pipeline execute tests the same way—assuming you actually configure your pipeline correctly (more on that disaster shortly).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eBut performance gains aren\u0026rsquo;t universal.\u003c/strong\u003e If your tests are already fast, you probably won\u0026rsquo;t see dramatic improvements. MTP mainly optimizes infrastructure overhead, not slow database calls or badly written assertions. Don\u0026rsquo;t expect miracles if your test suite still takes 20 minutes because it\u0026rsquo;s hitting real APIs.\u003c/p\u003e\n\u003cp\u003eAnd here\u0026rsquo;s the part Microsoft doesn\u0026rsquo;t emphasize: \u003cstrong\u003eMTP won\u0026rsquo;t save you from bad tests.\u003c/strong\u003e If your test suite is flaky, brittle, or poorly isolated, the new platform just runs that mess faster.\u003c/p\u003e\n\n\n\n\n\u003ch3 id=\"what-about-visual-studio-integration\"\u003e\u003ca href=\"/posts/dotnet-10-testing/#what-about-visual-studio-integration\" title=\"What about Visual Studio integration?\"\u003eWhat about Visual Studio integration?\u003c/a\u003e\u003c/h3\u003e\n\u003cp\u003eVisual Studio 17.14 or later integrates with MTP. Earlier versions rely on VSTest and may behave differently. If your team uses mixed VS versions, validate results locally with the CLI to avoid IDE-specific discrepancies.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"the-ci-pipeline-trap-and-how-to-avoid-it\"\u003e\u003ca href=\"/posts/dotnet-10-testing/#the-ci-pipeline-trap-and-how-to-avoid-it\" title=\"The CI Pipeline Trap (And How to Avoid It)\"\u003eThe CI Pipeline Trap (And How to Avoid It)\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eHere\u0026rsquo;s where things get entertaining.\u003c/p\u003e\n\u003cp\u003eYou add that \u003ccode\u003eglobal.json\u003c/code\u003e snippet. Tests run perfectly on your machine. You commit, push, and watch your GitHub Actions pipeline\u0026hellip; fail spectacularly.\u003c/p\u003e\n\u003cp\u003eWhy? Because GitHub\u0026rsquo;s hosted runners don\u0026rsquo;t automatically respect your \u003ccode\u003eglobal.json\u003c/code\u003e. They just use whatever SDK version happens to be installed—often an older one that doesn\u0026rsquo;t even support MTP. Your carefully configured local environment and your CI pipeline are now essentially running completely different test infrastructure.\u003c/p\u003e\n\u003cp\u003eI learned this the hard way when a colleague spent two hours debugging \u0026ldquo;flaky\u0026rdquo; tests that weren\u0026rsquo;t actually flaky at all. The tests validated timeout behavior in an async workflow—they passed consistently with MTP locally and then failed consistently with VSTest in CI. Same code, same timeout values, completely different test runner behavior. VSTest\u0026rsquo;s process isolation apparently meant slightly different timing characteristics. We only figured it out after painstakingly comparing the test execution logs line by line and finally noticing the runner version mismatch.\u003c/p\u003e\n\u003cp\u003eThe fix is one line—but you have to know it exists:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e- \u003cspan class=\"nt\"\u003euses\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eactions/setup-dotnet@v5\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ewith\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eglobal-json-file\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s1\"\u003e\u0026#39;./global.json\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e- \u003cspan class=\"nt\"\u003erun\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edotnet test\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThat \u003ccode\u003eglobal-json-file\u003c/code\u003e parameter forces the action to actually read your configuration. Without it, you\u0026rsquo;re deploying tests with one runner and debugging them with another.\u003c/p\u003e\n\u003cp\u003eIf you don\u0026rsquo;t specify this explicitly, your \u003ccode\u003eglobal.json\u003c/code\u003e is basically just decorative. It just sits in your repository looking official while your pipeline ignores it completely. I\u0026rsquo;ve actually seen teams add comments to their \u003ccode\u003eglobal.json\u003c/code\u003e files carefully explaining why certain settings exist, not realizing the entire file wasn\u0026rsquo;t even being used. That\u0026rsquo;s not configuration—that\u0026rsquo;s just theater.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"version-compatibility-or-who-gets-left-behind\"\u003e\u003ca href=\"/posts/dotnet-10-testing/#version-compatibility-or-who-gets-left-behind\" title=\"Version Compatibility (Or: Who Gets Left Behind)\"\u003eVersion Compatibility (Or: Who Gets Left Behind)\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eMTP doesn\u0026rsquo;t support every test framework version ever released. Microsoft drew a line, and some older projects sit on the wrong side of it.\u003c/p\u003e\n\u003cp\u003eTo use Microsoft.Testing.Platform, your test frameworks need these minimum versions:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003exUnit\u003c/strong\u003e → Version \u003cstrong\u003e3.x\u003c/strong\u003e or later\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eMSTest\u003c/strong\u003e → Version \u003cstrong\u003e3.2.0\u003c/strong\u003e or later\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eNUnit\u003c/strong\u003e → \u003cstrong\u003eNUnit3TestAdapter 5.0.0\u003c/strong\u003e or later\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eTUnit\u003c/strong\u003e → Works out of the box (it was designed with MTP in mind)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eVisual Studio\u003c/strong\u003e → Version \u003cstrong\u003e17.14\u003c/strong\u003e or later for full integration\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eIf you\u0026rsquo;re running older versions, the SDK simply won\u0026rsquo;t negotiate. It fails hard. No fallback to VSTest, no warning, just an error message telling you to upgrade.\u003c/p\u003e\n\u003cp\u003eThat\u0026rsquo;s actually good design. Ambiguity in test execution creates exactly the kind of \u0026ldquo;works on my machine\u0026rdquo; disasters MTP is supposed to prevent. Better to fail explicitly than to silently run different infrastructure depending on what\u0026rsquo;s installed.\u003c/p\u003e\n\u003cp\u003eBut it does mean migration isn\u0026rsquo;t optional if you\u0026rsquo;re upgrading to .NET 10. You can\u0026rsquo;t enable MTP halfway. Either your entire test suite supports it, or you don\u0026rsquo;t use it at all.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"migration-strategy-or-how-not-to-break-everything\"\u003e\u003ca href=\"/posts/dotnet-10-testing/#migration-strategy-or-how-not-to-break-everything\" title=\"Migration Strategy (Or: How Not to Break Everything)\"\u003eMigration Strategy (Or: How Not to Break Everything)\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eMigrating to MTP isn\u0026rsquo;t technically complicated, but it does actually require coordination. You can\u0026rsquo;t just enable it in isolation—everyone on the team needs to be running compatible tools, or the test results will simply stop being reliable.\u003c/p\u003e\n\u003cp\u003eHere\u0026rsquo;s a migration approach that won\u0026rsquo;t cause chaos:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e1. Audit your test framework versions first.\u003c/strong\u003e\nCheck every test project. If you\u0026rsquo;re running xUnit 2.x or MSTest 2.x, you\u0026rsquo;re upgrading before you can enable MTP. No shortcuts.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2. Add the \u003ccode\u003eglobal.json\u003c/code\u003e configuration.\u003c/strong\u003e\nStart with the minimal snippet. You don\u0026rsquo;t need to pin an SDK version unless you have specific compatibility requirements elsewhere.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e3. Update your CI/CD pipelines.\u003c/strong\u003e\nAdd the \u003ccode\u003eglobal-json-file\u003c/code\u003e parameter to your \u003ccode\u003esetup-dotnet\u003c/code\u003e action. Test it on a branch before merging. Verify that the pipeline is actually using MTP by checking the test output logs.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e4. Run tests locally and in CI—compare the results.\u003c/strong\u003e\nIf they differ, you\u0026rsquo;ve found a configuration issue. Fix it now, before it becomes a debugging nightmare three months from now. Pay special attention to tests that involve timing, parallelization, or resource cleanup—these are the ones most likely to behave differently between test runners.\u003c/p\u003e\n\u003cp\u003eIf you\u0026rsquo;ve read \u003ca href=\"/posts/tests-are-lying/\"\u003e\u0026ldquo;Your Tests Are Lying — Mutation Testing in .NET\u0026rdquo;\u003c/a\u003e, you know how dangerous it is when tests pass for the wrong reasons. MTP reduces that risk—but only if your environments are actually configured consistently.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"when-not-to-migrate-yes-really\"\u003e\u003ca href=\"/posts/dotnet-10-testing/#when-not-to-migrate-yes-really\" title=\"When Not to Migrate (Yes, Really)\"\u003eWhen Not to Migrate (Yes, Really)\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eNot every project should rush into MTP. Here are scenarios where you might want to wait:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eLegacy test suites with heavy VSTest dependencies.\u003c/strong\u003e If your tests rely on specific VSTest console runners, custom adapters, or undocumented behavior, migration will break things. You\u0026rsquo;ll need to refactor or rewrite parts of your test infrastructure.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eProjects still on .NET 8 LTS.\u003c/strong\u003e MTP is a .NET 10 feature. If you\u0026rsquo;re staying on an LTS version for stability, you\u0026rsquo;re essentially stuck with VSTest. That\u0026rsquo;s fine—VSTest still works. It\u0026rsquo;s just not getting any new features.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTeams without time to validate the migration.\u003c/strong\u003e Half-migrating is worse than not migrating. If you can\u0026rsquo;t dedicate time to verify that tests behave identically across environments, defer the change until you can.\u003c/p\u003e\n\u003cp\u003eMTP is definitely an improvement, but it\u0026rsquo;s not urgent. If your current test infrastructure already works reliably, you\u0026rsquo;re really not missing out by waiting.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"what-this-actually-means-for-your-workflow\"\u003e\u003ca href=\"/posts/dotnet-10-testing/#what-this-actually-means-for-your-workflow\" title=\"What This Actually Means for Your Workflow\"\u003eWhat This Actually Means for Your Workflow\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eThe shift to MTP changes how you think about test configuration. Your \u003ccode\u003eglobal.json\u003c/code\u003e file is no longer just an SDK hint—it\u0026rsquo;s a binding contract. The SDK reads it, respects it, and enforces it. If your pipeline isn\u0026rsquo;t configured to honor that contract, your tests will diverge silently between environments.\u003c/p\u003e\n\u003cp\u003eThat\u0026rsquo;s both the strength and the risk of this change. MTP removes ambiguity, but only if you configure it correctly everywhere. Miss one environment, and you\u0026rsquo;re back to debugging phantom failures that only reproduce in CI.\u003c/p\u003e\n\u003cp\u003eThe good news? Once configured properly, tests become predictable. The bad news? Getting there requires discipline, not just documentation.\u003c/p\u003e\n\n\n\n\n\u003ch2 id=\"should-you-migrate-now\"\u003e\u003ca href=\"/posts/dotnet-10-testing/#should-you-migrate-now\" title=\"Should You Migrate Now?\"\u003eShould You Migrate Now?\u003c/a\u003e\u003c/h2\u003e\n\u003cp\u003eIf you\u0026rsquo;re already on .NET 10, yes. The benefits clearly outweigh the setup cost, especially if you\u0026rsquo;ve already dealt with flaky CI pipelines or inconsistent test behavior across environments.\u003c/p\u003e\n\u003cp\u003eIf you\u0026rsquo;re on an LTS version and your tests are stable, there\u0026rsquo;s really no rush. VSTest isn\u0026rsquo;t going anywhere immediately, and MTP will still be there when you eventually upgrade.\u003c/p\u003e\n\u003cp\u003eBut if you\u0026rsquo;re planning to move to .NET 10 anyway, enable MTP early in the migration process. It\u0026rsquo;s easier to validate test behavior during a planned upgrade than to debug it six months later when the root cause has been buried under other changes.\u003c/p\u003e\n\u003cp\u003eAdd the four lines to \u003ccode\u003eglobal.json\u003c/code\u003e. Update your CI config. Upgrade your test frameworks. Run the tests. Compare the results.\u003c/p\u003e\n\u003cp\u003eIf they match—and they should—you\u0026rsquo;re done. If they don\u0026rsquo;t, you\u0026rsquo;ve found a configuration problem that would have bitten you eventually anyway. Better to find it now during a planned migration than at 2 AM when production is down and your tests are lying to you about what\u0026rsquo;s safe to deploy.\u003c/p\u003e\n\u003cp\u003eMicrosoft fixed the test runner. Whether you use it or keep debugging phantom CI failures is your choice—but when the next \u0026ldquo;works on my machine\u0026rdquo; ticket comes in, at least you\u0026rsquo;ll know exactly why.\u003c/p\u003e\n","date_modified":"2026-05-26T10:22:03+02:00","date_published":"2025-11-20T17:00:00+01:00","id":"https://daily-devops.net/posts/dotnet-10-testing/","language":"en","summary":"Microsoft.Testing.Platform replaces VSTest in .NET 10. See what improves, what breaks, and why your global.json now matters in IDE and CI reliably.\n","tags":["testing","dotnet","csharp","softwareengineering","github-actions","devops"],"title":".NET 10 Testing: Microsoft Finally Fixed the Test Runner (Mostly)\n","url":"https://daily-devops.net/posts/dotnet-10-testing/"}],"language":"en","title":"DevOps Practices That Actually Ship on Daily DevOps \u0026 .NET","version":"https://jsonfeed.org/version/1.1"}