Managed or self-hosted AI models: Azure AI Foundry, NVIDIA NIM, or your own

The short answer: start on a managed model, and only self-host when a specific, measurable reason forces your hand — usually cost at scale, data residency, or latency you can’t buy. In 2026 the options for a serious product cluster into three: a fully managed API (including managed open-weight models on Azure AI Foundry), a self-managed-but-packaged runtime like NVIDIA NIM, or your own model served on GPUs you operate. Most teams should start at the top of that list and move down only when the numbers say so.

We build both managed and self-hosted deployments, so here’s the honest trade-off, without the vendor gloss.

Three options, plainly

1. Fully managed API

You call an endpoint; someone else runs the GPUs, patches the runtime, and scales it. This includes frontier models (Anthropic, OpenAI and others) and managed open-weight models on Azure AI Foundry, where you get an open model behind a managed endpoint without operating any infrastructure. Fastest to ship, zero ops, pay per token. You trade control and per-unit cost for speed and simplicity.

2. Self-managed, packaged runtime (e.g. NVIDIA NIM)

A middle path: you run the model, but on a vendor-optimised, containerised runtime that handles the hard parts of serving (batching, optimised kernels, an OpenAI-compatible API). NVIDIA NIM is the clearest example — you deploy a container to your own Kubernetes or VMs, keep data inside your boundary, and still get near-turnkey inference. You take on GPU ops in exchange for control and, at volume, better economics.

3. Your own model, fully self-hosted

You pick an open-weight model, serve it yourself (vLLM, TensorRT-LLM or similar), and own everything from the GPU up. Maximum control and, at high steady utilisation, the lowest per-token cost — but you now own capacity planning, autoscaling, upgrades, and being paged when a node dies. This is a real platform commitment, not a checkbox.

The decision, driven by four questions

• What’s your volume? Low or spiky traffic almost always favours managed — you don’t pay for idle GPUs. Self-hosting only wins economically at high, steady utilisation, because a mostly-idle GPU is the most expensive way to run inference.
• Where must the data live? If regulation or a customer contract requires data to stay in your tenant or region, that pushes you toward NIM-style self-management or full self-hosting — or a managed option with the right residency guarantees (a real strength of Foundry for regulated Azure customers).
• How sensitive is latency? If you need tight, predictable tail latency, running inference close to your app (co-located, self-managed) can beat a shared public endpoint. For most products, a managed endpoint is fast enough.
• Do you have the ops muscle? Self-hosting GPUs is a platform-engineering job. If nobody owns it after go-live, a self-hosted model becomes the fragile centre of your product. Be honest about this one.

The hidden cost of self-hosting

The per-token math looks great in a spreadsheet: buy the GPU, amortise it, beat the API price. The spreadsheet leaves out capacity planning for peak, autoscaling that doesn’t thrash, model and driver upgrades, evals after every change, and an on-call rotation for when inference falls over at 2 a.m. None of that is exotic — it’s just real work, and it’s exactly the work the managed price is paying for. Count it before you switch.

How we’d roll it out

1. Ship on managed first. Prove the feature works and measure real usage — tokens, latency, cost per request.
2. Build the seam. Put your model access behind a thin internal interface (an OpenAI-compatible one is convenient) so the provider is swappable without touching product code.
3. Move only what the numbers justify. When a specific workload hits the volume, residency or latency threshold, move that workload to NIM or self-hosted — not the whole system, and not on vibes.
4. Keep evals in the loop. Whatever you run, run an eval set against it on every change, so a model or runtime swap can’t quietly regress quality.

The one-line version

Managed to start and for anything spiky; NVIDIA NIM when you need control and data residency without building a serving stack from scratch; fully self-hosted only when high, steady volume or hard constraints make the ops burden pay for itself. Build the swappable seam early, and let measured cost — not the hype cycle — decide when to move.