Pitfalls of Deploying LLMs in Sensitive Environments

2026-09-12

The German text is entirely my own. The English version was translated with Claude. If you find watermarks, you may keep them.

Why your AI project won’t fail because of the model, but because of everything that comes before it. Usually I write about the pain that shows up afterwards, at the audit for instance. This time we don’t even get that far: deployment pain. I treat it like martial arts: no pain, no gain. Pretty demos and polished prototypes are for other people to show. The happy path is for wimps.

A short preliminary note, so expectations are set right: This is first and foremost about the gap between “the (Azure) portal says done” and an AI platform that actually works under real security and operational requirements. ClickOps seduces you into believing it’s a 10-minute job: a few colourful buttons, green checkmarks, Bob’s your uncle. Like Visual Basic back in the day …

But a resource that exists proves neither that your pipeline can reach it, nor that the right identity may access it, nor that you can get the whole thing reproducibly all the way to prod. For that you need storage accounts, DNS zones, service principals and more than one successful clicking finger.

This list was born from pain, not from AI-generated surface knowledge.

Enter the dojo with me:

“Nine Chambers of LLM-Ops-Shaolin”.

1. THE MODEL IS THE EASIEST RESOURCE IN THE ENTIRE STACK

Azure AI Foundry, a project, a model deployment: anyone can click that together in the portal. The real effort starts before that, in the platform that makes operations possible in the first place:

• hub/spoke architecture with a central firewall, where every egress approval has to go through a process
• central private DNS zones owned by another team that first have to be linked
• a private endpoint subnet whose CIDR range has to be reserved in advance
• a deploy identity with exactly the permissions required and none beyond that
• a Key Vault with a customer-side HSM key, provisioned and operated by third parties

Whoever plans this as an “AI project” typically underestimates the effort by a factor of 5. It is a platform project with an AI building block on top. The honest order is: platform contract first, then Terraform, then the model.

2. THREE SEPARATE CHECKS, AND ONE GREEN DOT DOES NOT REPLACE THE OTHER TWO

The most common fallacy in status meetings goes: “Access works.” And nobody asks which access is meant.

That’s why I consistently break the question down into three separate checks:

1. Identity: Who is signing in to Azure — you with your admin account, the deployment pipeline with its service principal, or the running application with its managed identity? These are separate identities. Your pipeline does not inherit your admin rights through emotional proximity to the developer. If you sign in via az login with your user account and test successfully, you have so far only tested your own access. Whether the pipeline can authenticate with its intended identity remains completely open. Therefore: verify sign-in with exactly the identity and the authentication method that will actually be used later. Who are you? is this check. What may you do? comes in the next step.
2. RBAC at the right scope: Azure now knows who you are. It does not follow that you are allowed to do anything. In role-based access control (RBAC) two things count: Which action does the role permit, and where does the assignment apply? That area of validity is called scope: management group, subscription, resource group or individual resource. Permissions are inherited downwards, not upwards. Reader on a subscription therefore does not automatically give you read rights on the management group above it, not even when its policies are blocking your deployment. And Contributor on your resource group lets you manage resources, but not assign Azure roles to other identities. You may build the hall, but not hand out the keys. If Terraform is supposed to do both, the deploy identity also needs the appropriate permission for role assignments at the intended scope. So don’t ask: “Do we have Contributor?”, but: “May exactly this identity perform exactly this action on exactly this resource?” (Microsoft Learn: Azure RBAC)
3. Network path: Your identity is right, the permissions fit. Now your request still has to reach the resource. Azure separates management from usage here: via the control plane (Azure Resource Manager, ARM for short) you create a storage account, for instance, or read its configuration. Via the data plane you access the actual content and functions: read a blob, fetch a secret from Key Vault, call the model. These are different endpoints. The fact that the CLI lists your storage account therefore does not prove that Terraform can reach the state file stored inside it. You’ve called reception. You’re nowhere near opening the vault.

If blob access is intended exclusively via a private endpoint, the storage hostname must resolve to the correct private IP address from the executing system. Then routing and network rules have to allow the connection there. If the right DNS resolution is missing, the request may land at the public endpoint or find no target at all. If the network path is missing, the right IP won’t help either.

So test from the actual place of execution: laptop, pipeline agent or application. Which endpoint am I talking to, which IP does DNS return there, and does the connection get established? A green test on your laptop does not build the pipeline a tunnel. (Microsoft Learn: Control Plane and Data Plane; Private Endpoint DNS)

3. “DELIVERED” IS NOT “WORKS”

In distributed setups (platform operator, internal IT, project team, external research) a vocabulary problem emerges.

Someone reports “storage account created”. What does that mean? Does it exist? Does the right identity have permissions? Is it reachable from the network the pipeline runs in?

I now keep an inventory file with five states:

Sounds bureaucratic. But it’s the difference between “we’re almost done” and “we haven’t had a single successful terraform init against the remote state yet”.

Both can be true on the same day. And yes: a green box in the architecture diagram is not a deployment status.

4. STATE, IDENTITY, SECRETS: THE TERRAFORM HOMEWORK

A podcast episode of “Cloud Optimizer” (Matthias Braun, Christian Forjahn) nails it: many Terraform projects fail before the first apply run is even possible. In sensitive environments, the following typically comes on top:

• Remote state is a crown jewel. It contains resource IDs, outputs and not infrequently delicate details all the way down to plaintext. Therefore: disable shared key, access only via Entra authentication. Enable versioning. Enable soft delete. Whoever forgets soft delete learns it with the first accidentally deleted state blob. And: state storage is not app storage; separate resource group, own lifecycle.
• Solve bootstrap cleanly ⇒ once, reproducibly. Who creates the state storage when there is no state yet? You solve the bootstrap dilemma once, cleanly, via a script in the repo (or a dedicated bootstrap step); not via a portal click that nobody can reconstruct three months later.
• No client secrets. Workload Identity Federation for the pipeline. One dedicated deploy principal per environment with a custom role instead of blanket Contributor on the subscription. A secret that expires after twelve months is guaranteed to take down your prod pipeline at night.
• Subscription is configuration, not gut feeling. The subscription belongs in the provider as a variable; not as “the one I happen to be logged into”. That’s the safety net against the classic “oops, wrong subscription”. With 2 subscriptions called Test and Prod, that is not a theoretical risk.

5. THE POLICY THAT BLOCKS YOU IS ONE YOU’RE NOT ALLOWED TO READ

This is the nastiest pitfall in enterprise landing zones: Azure Policy is assigned at management group level, but your project account often isn’t allowed to read the assignments on the levels above. Result: terraform plan stays green, because the plan only evaluates your configuration. The apply then fails mid-deployment on a deny whose cause you can neither see nor cleanly prove.

Remedy: Before the first production-like apply, demand a policy export from the platform team ⇒ at minimum assignments, parameters, exemptions and enforcement mode. Newer provider versions move individual checks into the plan with “preflight validation”, but you can’t rely on that: without transparency of the landing zone it remains patchwork.

Same category, different lever: customer managed keys. If the customer decides that resources are to be encrypted with an HSM key, that also applies to smoke tests with synthetic data. “It’s only test data” is not an exception. Either you get one in writing, or you build the CMK path first.

ATTENTION!

CMK and Customer Lockbox are different controls. One does not activate the other.

6. DRIFT IS BORN IN THE INCIDENT, AT THREE IN THE MORNING

Portal clicks during an incident are understandable, but they are precisely the origin of drift. Whoever flips a switch in prod at night “just quickly” almost reliably creates a second outage: on the next Terraform apply the change is reverted, because it isn’t in the code.

The consequence is banal and yet rarely implemented cleanly:

• Prevent portal changes in prod. Azure Policy in deny mode is not a governance accessory here, it’s operational protection (e.g. no public endpoint, no plaintext secrets, no untagged resources).
• Strictly separate plan and apply. What gets approved is a concrete plan as an artifact. The apply executes exactly that plan — not “the current state” and not “whatever happens to be in the repo”.

7. STAGING IS NOT A DATA PROTECTION FREE PORT

“We replaced the names.” Nice. So your dataset is now wearing a fake moustache. Whether it is anonymous is a different matter.

Pseudonymisation means: the data can no longer be attributed to a specific person without additional information. “Anna Müller” becomes, say, “Person 4711”; the mapping table is kept separately and protected technically and organisationally. Whoever can draw on the additional information can restore the link to the person. For you as the controller with that mapping capability, this remains personal data. Pseudonymisation is a protective measure, not an exit from the GDPR. (Art. 4(5) GDPR)

Anonymisation goes further: the person must no longer be identifiable, taking into account all the means reasonably likely to be used by the controller or by others. Effort, cost and available technology count, among other things. So it’s not merely about whether you can reverse a replacement. Even without a mapping table, the remaining content can give the person away. Truly anonymous information does not fall under the GDPR; anonymising personal source data is, however, still a processing operation in itself. (Recital 26 GDPR)

Especially with legal matters, identity doesn’t only sit in the name field. An example: the only chief physician of a particular specialty in a small town, plus the clinic and the exact termination date. You can black out the name. The rest introduces her anyway. “Names removed” is therefore neither proof of anonymisation nor automatically a cleanly implemented pseudonymisation.

For staging this means: if you continue to process personal data there, you also need a sound legal basis for it, a permissible purpose, data minimisation, regulated access, deletion periods and risk-appropriate security measures. That doesn’t make staging legally identical to prod. But the GDPR knows no exception called “it’s only a test”. (Art. 5, 6 and 32 GDPR)

For infrastructure smoke tests I therefore prefer freely constructed synthetic data with no relation to real persons. For domain evals, the test cases additionally have to cover the relevant errors and edge cases. And whoever generates synthetic data from real client files has to check whether real content or identifying characteristics are carried over. “Synthetic” is not holy water. The lawyer in me does not demand an identical prod setup here, but solid proof of which data is processed and why that is justifiable.

8. YOUR MODEL HAS AN EXPIRY DATE, AND AZURE SWAPS IT OUT WITHOUT ASKING

For a classic web app, CI/CD is enough. With an LLM platform you additionally have to expect the model to change underneath you, quite magically, without a code change.

UPGRADE POLICY

An Azure OpenAI deployment has an upgrade policy for this. By default it effectively switches automatically as soon as Microsoft sets a new default version. No fixed date, no change window; triggered solely by the moment the provider declares the new version the default. Whoever doesn’t deliberately set this policy to “Upgrade when expired” or “No auto upgrade” will at some point get a different model under the same deployment name. The code is unchanged, the tests were green — the answers possibly aren’t anymore.

Why this shows up as model drift: Strictly speaking, an unchanged model isn’t “slowly drifting away” here. The provider swaps the model version. With that, weights, training state, alignment and internal response behaviour can change, while deployment name, API and application code stay the same. For the same prompts a different distribution of possible answers emerges: classifications flip, tool calls are chosen differently, structured outputs break, or safety filters suddenly react more strictly.

From an operations perspective this is behavioral drift through a hidden model version change: the baseline measured in tests no longer matches the model currently answering in production.

And the green tests? They only prove that the old model version passed them. If the golden set and regression tests don’t run again against the new version after the upgrade, the green checkmarks are historical decoration.

The endpoint still has the same name. The player in the jersey is a different one.

PINNING DOESN’T HELP EITHER

Pinning is not a permanent solution, it’s an appointment. GPT-4o (among others 2024-05-13 and 2024-08-06) was retired as of 31 March 2026, with auto-upgrade to a successor model. Microsoft now often communicates retirement dates for GA models at launch, often around 18 months in advance. Whoever pins buys stability today against a migration with a deadline tomorrow. And the announcement lands with the Owner/Contributor/Reader of the subscription; so happily in a mailbox that nobody monitors operationally.

Why this isn’t a cosmetic problem: Apple researchers call the ugly variant “negative flips”; cases the old model solved correctly and the new one gets wrong, even though overall quality rises. The average masks exactly the regressions your workflow depends on.

From practice, a recurring pattern comes on top: a silent provider update can invalidate months of accumulated LLM-as-judge scores, because the “judge” suddenly rules by a different standard.

PROPOSED SOLUTION

What follows from this ⇒ in the order I build it:

• Set the deployment version explicitly in IaC, configure the upgrade policy explicitly, put the retirement date on the board as a follow-up. No “latest”, no alias.
• Version prompts in the repo, not in some playground. A model change without a prompt version is an experiment without a control group.
• Evals in test/staging with golden set, groundedness checks and regression tests. And: the judge gets pinned just like the candidate — otherwise, on a red run, it’s unclear who moved.
• Every model change goes through the same track as a database migration: eval, shadow traffic, canary, defined rollback path.
• Traces per request including prompt and model version. You need that anyway for traceability ⇒ but that’s another article.

9. MICROSOFT-HOSTED AGENTS DON’T GET THROUGH YOUR PRIVATE ENDPOINT

If the platform is fully private (and with client data it should be), then the standard pipeline can’t reach your resources. Self-hosted agents or managed DevOps pools in the VNet are week two, not week eight. Whoever discovers this late “solves” it with a temporary public exposure, and temporary, as everyone in IT knows, is the word for forever.

10 (BONUS CHAMBER). ROLLBACK IS DESIGN, NOT AN EMERGENCY PLAN

Rollback is not a wiki article, it’s an executable action. If you only document it as a runbook, you don’t actually own a rollback. You own text. A rollback has to work as a command that someone executes at three in the morning without thinking. In Azure Container Apps that’s doable, but only if you make three decisions up front:

• Multiple revision mode instead of single. The default is single: new revision in, old one out => “rollback” then means redeploy. In multiple mode several revisions stay active in parallel. You control traffic by weighting, and the rollback is an ingress traffic set back to 100 % of the previous revision. Seconds instead of a build pipeline.
• Deterministic revision names. Use a stable revision suffix (commit hash or build number) and add labels like blue/green. Otherwise you roll back at night to “the one before last, I think” and that isn’t a rollback, it’s gambling.
• One build, promotion by digest. The image is built once and then promoted by digest through dev, test and prod. No rebuild for prod. What runs in prod has to be byte-identical to what was approved in staging. Otherwise “rollback” means: back to something that was never tested.

The procedure that has proven itself in practice (and that Microsoft documents this way): roll out the new revision with 0 % traffic, test via the label URL, then canary (e.g. 5 % for at least 30 minutes with an eye on error rates and latency percentiles), then 25 %, 50 %, 100 %. Only then deactivate the old revision.

Important: ACA (Azure Container Apps) does not solve the hardest part for you ⇒ the database. If revision N+1 changes a schema and you switch back to revision N, N has to keep running with the new schema. That forces expand/contract migrations: extend additively first, prepare code for old+new, remove old columns only one or two releases later. Put business-risky changes behind feature flags in addition, so you can switch off in an emergency without deploying. And: actually walk the rollback path in staging at least once. A rollback nobody has ever executed is not a capability. At best it’s a hypothesis.

HOW TO BUILD IT (THE DIAGRAM)

The diagram below shows the building blocks as three layers with what I call the “platform contract”: the explicit boundary between what the platform team delivers and owns, and what the project creates and is responsible for via IaC. The three checks from point 2 are drawn in as gates.

(c) Rendered by Claude from a PUML diagram

The point that’s in no vendor diagram: the dashed line is the most important part. Without it, nobody knows who creates the subnet, who links the DNS zone and who writes the role assignment. And then the model is up, and nobody can get to it.

HIDDEN CTA

On a personal note: If your AI platform needs to do more than a pretty demo: I combine technical leadership, Azure architecture and AI compliance: from the first Terraform plan to resilient operations.

Talk to me. I’m currently available.

Sources and further reading

• Podcast (I listened to it on Apple Podcasts): Die Cloud Optimizer, S2E14 “Terraform in Azure, bevor der erste terraform apply läuft” (Matthias Braun, Christian Forjahn), show notes on Substack
• Microsoft Learn: Azure OpenAI Model Retirements and Deprecations; Blue-Green Deployment in Azure Container Apps; Traffic Splitting in Azure Container Apps
• Microsoft Q&A, threads on auto-upgrade policy and GPT-4o retirement (spring 2026)
• Matthias Braun, “Terraform AzureRM 5.0 Upgrade” (preflight validation in plan)
• TianPan.co, “The Semver Lie” and “LLM-as-Judge Drift” (2026), on the alias trap and evaluator drift
• arXiv 2601.22025, “Evaluation-Driven Iteration for LLM Applications”, section on silent regressions after model updates
• OneUptime Engineering Blog, canary procedure with ACA revisions (February 2026)