Embed Pulumi in a FastAPI Service with the Automation API

The Pulumi Automation API turns pulumi up into a library call, which means you can drive deployments from an HTTP service instead of a CLI. This guide — under Pulumi patterns and provider management — builds a small FastAPI endpoint that provisions a per-tenant stack on demand, the pattern behind self-service infrastructure portals.

Why This Matters

A CLI is fine for engineers, but product teams often need infrastructure created in response to an event: a new customer signs up, a preview environment is requested, a batch job needs a bucket. Wrapping programmatic deployments in an API lets those events trigger real, stateful deployments with full preview and rollback.

The key design constraint is that deployments are slow and stateful, so the HTTP layer must treat them as background work, not a synchronous request you block on.

There is a second constraint that is easier to miss. The Automation API is not a client library talking to a remote service — it drives the pulumi binary on the same host, and for inline programs it hosts a gRPC language server inside your Python process so the engine can call your program function back. A FastAPI service embedding it is therefore running three things at once: the web server, a subprocess, and a Pulumi language runtime that shares its interpreter. Every design decision below follows from that arrangement: how many deployments can run concurrently, where credentials must live, what happens on SIGTERM, and why an exception inside a tenant's program can take the whole API down if you let it.

Prerequisites

• pulumi>=3.0 and fastapi, uvicorn installed in the service virtualenv
• The Pulumi CLI on the host (the Automation API shells out to it) and a configured state backend
• Cloud credentials available to the service process via its execution role, not baked into the image

# CLI: the Automation API needs the pulumi binary on PATH
pulumi version && python -c "from pulumi.automation import create_or_select_stack"

The binary version matters more than it does for CLI use. LocalWorkspace invokes whatever pulumi is first on PATH, so an image rebuilt six months later can pick up a newer CLI with different plan output while the Python SDK stays pinned in requirements.txt. Pin both, in the same place — a Dockerfile that installs an exact CLI version alongside an exact pulumi wheel — or you will eventually debug a behaviour change that appears in production and is unreproducible locally.

Decide the secrets provider before the first stack exists. A stack's encryption provider is fixed at creation; changing it later means pulumi stack change-secrets-provider, run per stack, which is painful once there are a few hundred tenants. Pass it explicitly rather than accepting the default:

# workspace.py — workspace options shared by every tenant stack
# CLI: python -c "import workspace; print(workspace.OPTS.secrets_provider)"
from typing import Final
from pulumi import automation as auto

BACKEND_URL: Final[str] = "s3://acme-pulumi-state/tenants"

OPTS: Final[auto.LocalWorkspaceOptions] = auto.LocalWorkspaceOptions(
    secrets_provider="awskms://alias/pulumi-tenant-state?region=us-east-1",
    project_settings=auto.ProjectSettings(
        name="tenant-envs",
        runtime="python",
        backend=auto.ProjectBackend(url=BACKEND_URL),
    ),
    env_vars={"PULUMI_SKIP_UPDATE_CHECK": "true"},
)
# State implication: secrets_provider is recorded in the stack's own config file
# at creation and cannot be changed by editing this value afterwards.

env_vars are injected into the CLI subprocess only, not into the service process, which makes them the right place for per-tenant credentials when one API server deploys into several accounts.

Defining an Inline Program

The Automation API can run an inline program — a Python function instead of a project directory — which is ideal for embedding. Each tenant gets its own stack name so state stays isolated.

# service.py — a FastAPI endpoint that provisions a per-tenant bucket
# CLI: uvicorn service:app --port 8080
from fastapi import FastAPI, BackgroundTasks
from pulumi import automation as auto
import pulumi_aws as aws

app = FastAPI()

def _program(tenant: str):
    def build():
        bucket = aws.s3.BucketV2(f"{tenant}-data")
        return {"bucket": bucket.bucket}
    return build

def _deploy(tenant: str) -> None:
    stack = auto.create_or_select_stack(
        stack_name=f"tenant-{tenant}",
        project_name="tenant-envs",
        program=_program(tenant),
    )
    stack.workspace.install_plugin("aws", "v6.0.0")  # Provider note: pin the plugin version
    stack.set_config("aws:region", auto.ConfigValue("us-east-1"))
    stack.up(on_output=print)  # State implication: writes to the tenant's own state file

@app.post("/envs/{tenant}", status_code=202)
def create_env(tenant: str, bg: BackgroundTasks):
    bg.add_task(_deploy, tenant)
    return {"status": "accepted", "stack": f"tenant-{tenant}"}

The closure returned by _program is what makes this an inline program: there is no project directory, no Pulumi.yaml on disk for the tenant, and no __main__.py. create_or_select_stack writes a minimal project and stack settings file under the workspace's working directory, then hands the engine a reference to the callable. When the engine is ready to evaluate the program it calls back over gRPC and the closure executes in the service process.

That callback path explains several behaviours that look strange from the outside. Output printed by the program lands in the service's own stdout unless you intercept it with on_output. An exception raised inside build() does not produce a Python traceback at the up() call site — the engine catches it, marks the update failed, and the SDK re-raises it wrapped as InlineSourceRuntimeError with the original message in the string. And because Pulumi's Python SDK keeps runtime settings in module-level state, two inline programs executing at the same instant in the same interpreter will interfere with each other regardless of which stacks they target.

Running Deployments Off the Request Path

BackgroundTasks is the right shape for a demonstration and the wrong shape for production: the task dies with the worker, there is no record of what happened, and nothing prevents two requests for the same tenant from overlapping. Replace it with an explicit job record and a per-stack lock.

# jobs.py — serialise deployments per stack and record the outcome
# CLI: uvicorn service:app --port 8080 --workers 1
import asyncio
from dataclasses import dataclass, field
from typing import Literal, Optional
from pulumi import automation as auto

Status = Literal["queued", "running", "succeeded", "failed"]


@dataclass
class Job:
    tenant: str
    status: Status = "queued"
    detail: str = ""
    outputs: dict[str, str] = field(default_factory=dict)
    log: list[str] = field(default_factory=list)


JOBS: dict[str, Job] = {}
_LOCKS: dict[str, asyncio.Lock] = {}
_ENGINE = asyncio.Semaphore(2)   # cap concurrent CLI subprocesses


def _lock_for(stack_name: str) -> asyncio.Lock:
    return _LOCKS.setdefault(stack_name, asyncio.Lock())


async def run_up(job: Job) -> None:
    stack_name = f"tenant-{job.tenant}"
    async with _lock_for(stack_name), _ENGINE:
        job.status = "running"
        try:
            result: auto.UpResult = await asyncio.to_thread(_blocking_up, job)
        except auto.ConcurrentUpdateError as exc:
            job.status, job.detail = "failed", f"stack busy: {exc}"
            return
        except auto.errors.InlineSourceRuntimeError as exc:
            job.status, job.detail = "failed", str(exc)
            return
        job.status = "succeeded"
        job.detail = str(result.summary.resource_changes)
        job.outputs = {k: v.value for k, v in result.outputs.items() if not v.secret}
        # State implication: secret outputs stay encrypted in state; never copy
        # them into an in-memory job record that an unauthenticated poll can read.

asyncio.to_thread matters because stack.up() is fully synchronous — it blocks until the subprocess exits. Calling it directly from an async def handler stalls the event loop and every other request served by that worker. The semaphore is the second half of the protection: each concurrent up() is a pulumi subprocess plus a provider plugin process plus whatever memory the program itself allocates, so two or three at once is a realistic ceiling for a modest container, not a hundred.

Stream progress rather than discarding it. on_output receives the CLI's rendered lines; on_event receives structured EngineEvent objects, which is what you want if the UI needs a resource-by-resource progress bar rather than a log tail.

# service.py — the blocking half, with output captured into the job record
# CLI: curl -X POST localhost:8080/envs/acme
from pulumi import automation as auto
import workspace


def _blocking_up(job: Job) -> auto.UpResult:
    stack = auto.create_or_select_stack(
        stack_name=f"tenant-{job.tenant}",
        project_name="tenant-envs",
        program=_program(job.tenant),
        opts=workspace.OPTS,
    )
    stack.set_config("aws:region", auto.ConfigValue("us-east-1"))
    stack.set_config("tenant:tier", auto.ConfigValue("standard"))
    return stack.up(
        on_output=job.log.append,
        color="never",          # Provider note: ANSI codes otherwise pollute the job log
        parallel=8,
        message=f"api deploy for {job.tenant}",
    )

color="never" is not cosmetic — without it every captured line carries escape sequences that break JSON log shipping and make the stored output unreadable in a web UI. message shows up in the stack's update history, which turns "who changed this" into a question the state backend can answer.

Error Handling and Retry Semantics

The Automation API raises a small, well-defined exception hierarchy, and treating all of it as one generic failure is what produces retry storms. Match on the type.

ConcurrentUpdateError is the one that surfaces most often in a multi-replica deployment, because the per-process lock above protects one replica and not the fleet. The message comes straight from the backend — for Pulumi Cloud it reads [409] Conflict: Another update is currently in progress, and for a file or S3 backend it reports the lock file and suggests pulumi cancel. Retry it with backoff; do not call stack.cancel() automatically, because cancelling an update that is genuinely mid-apply leaves resources created but unrecorded, which is the worst state to be in.

CommandError is the catch-all for a non-zero CLI exit that the SDK could not classify — a missing plugin, an expired credential, a provider that refused the request. Its string carries the CLI's stderr, so log it whole rather than summarising; the actionable detail is almost always in the last few lines.

Verification

Exercise the endpoint and confirm a stack and state file appear, then destroy it to prove the reverse path works.

# CLI: trigger a deployment and inspect the resulting stack
curl -X POST localhost:8080/envs/acme
pulumi stack ls          # expect: tenant-acme listed
pulumi -s tenant-acme destroy --yes

The CLI check proves the stack exists. Prove the service agrees by reading the same information back through the Automation API, which is the path your API actually uses:

# check.py — assert the service's view of a tenant stack matches reality
# CLI: python check.py acme
import sys
from pulumi import automation as auto
import workspace

tenant: str = sys.argv[1]
stack = auto.select_stack(
    stack_name=f"tenant-{tenant}",
    project_name="tenant-envs",
    program=lambda: None,          # no-op: we are only reading state
    opts=workspace.OPTS,
)

info = stack.info()
assert info is not None, "stack has no update history — it was never deployed"
print(info.kind, info.result, info.start_time)

outputs = stack.outputs()
print({k: ("<secret>" if v.secret else v.value) for k, v in outputs.items()})

preview = stack.preview()
# State implication: a clean stack previews as `same` only. Any `create` or
# `update` here means the recorded state has drifted from the program.
assert preview.change_summary.get("create", 0) == 0, preview.change_summary

select_stack with a no-op program is the standard read-only entry point — the engine never evaluates it for outputs() or info(), and preview() will report the whole stack as pending deletion if you pass an empty program by mistake, which is why the assertion above checks for creates rather than for an empty summary.

A useful negative test: send two POSTs for the same tenant within a second and confirm the second job reports stack busy rather than starting a parallel update. If both proceed, the lock is not doing its job and the next real incident will be a corrupted state file.

Gotchas & Edge Cases

Concurrency corrupts state. Two overlapping up() calls on the same stack will collide on the state lock. Serialise per-stack work with a queue or a per-stack lock; never run two deployments of the same stack at once.

Long deployments outlive requests. A 30-second HTTP timeout will kill a deployment mid-flight if you run it synchronously. Always use background tasks or a worker queue and return a job id.

Plugin installs are slow. install_plugin downloads on first run; pre-bake plugins into the container image so cold starts do not time out.

An inline program shares the interpreter. A sys.exit(), a signal.alarm, or an unhandled os._exit() inside a tenant's program terminates the API server, not just that deployment. Keep the program function narrow: build resources, return outputs, and push anything that might crash into a separate process.

Uvicorn with multiple workers breaks the in-process lock. --workers 4 forks four independent interpreters with four independent _LOCKS dictionaries. Either run one worker per container and scale horizontally with an external lock, or move deployments to a dedicated worker pool consuming from a queue.

SIGTERM during an update leaves a lock behind. A rolling deploy that kills the pod mid-up() terminates the CLI subprocess, and the backend lock outlives it. Trap SIGTERM, stop accepting new jobs, and give in-flight updates a grace period longer than your slowest deployment — or accept that operators will be running pulumi cancel by hand.

Stack config is not request state. set_config writes to the stack's settings file and persists. Passing a per-request value through config rather than through the program closure means the value silently becomes the default for every later deployment of that stack.

Operational Notes

Size the container for the deployment, not for the web server. A pulumi up spawns the CLI, one plugin process per provider in use, and holds the resource graph in the service's own heap while the inline program runs. A single AWS deployment of modest size comfortably needs 512 MB beyond the FastAPI baseline, and the plugin processes are what dominate. Two concurrent updates in a 512 MB container is how you get an OOM kill that looks, from the outside, like a random deployment failure.

Pre-install plugins at image build time. pulumi plugin install resource aws v6.0.0 in the Dockerfile removes the download from the request path entirely, and the install_plugin call in the program then becomes a no-op that verifies rather than fetches. Without it the first deployment after every scale-up pays a multi-hundred-megabyte download while a client is waiting.

Credentials belong to the process, not the image. The service should assume a role through its execution environment — an EKS service account, an ECS task role, an Azure workload identity — and let the provider plugin pick it up from the ambient chain. Where one API server deploys into many customer accounts, put the per-tenant role ARN in LocalWorkspaceOptions.env_vars so it reaches the subprocess without ever being set on the parent, which keeps a bug in one tenant's handler from borrowing another tenant's credentials.

Retire stacks properly. stack.destroy() removes the resources but leaves the stack and its history in the backend; stack.workspace.remove_stack(name) deletes the record. Doing only the first leaves a growing list of empty stacks that slow stack ls and confuse billing exports. Doing only the second orphans live cloud resources with no state to describe them, which is considerably worse — always destroy first, and check result.summary.result == "succeeded" before removing.

FAQ

Can I preview instead of deploying from the API?

Yes — call stack.preview() and return the change summary. This is how self-service portals show a plan before a human approves the real up().

Where does state live for inline programs?

In whatever backend the workspace is configured for. Point it at S3 or Pulumi Cloud so multiple service replicas share state, as discussed in choosing a state backend.

Is the Automation API the same as the CLI?

It drives the same engine and state, so a stack created via the API is fully manageable from the CLI and vice versa.

How do I return deployment progress to the caller without holding the connection open?

Accumulate on_output lines into the job record and expose a GET /jobs/{id} endpoint the client polls, or push the same lines onto a server-sent-events stream. Both work because on_output is called synchronously from the reader thread as the CLI emits each line, so there is no buffering delay to design around.

What happens if the service restarts while an update is running?

The CLI subprocess is killed with its parent and the backend keeps the stack lock. The next up() for that stack raises ConcurrentUpdateError until the lock is cleared. Recovery is a pulumi cancel followed by pulumi up --refresh, because the update may have created resources the state file never recorded.

Can one service instance deploy into several cloud accounts?

Yes, and LocalWorkspaceOptions.env_vars is the mechanism — set the account-specific role or subscription per stack so the value reaches the CLI subprocess and nothing else. Avoid mutating os.environ in the request handler; it is process-global and will leak across concurrent deployments.

Related

• Programmatic Deployments with the Pulumi Automation API — the core patterns for driving Pulumi from Python.
• Pulumi Automation API — the parent topic on embedding Pulumi in applications.
• Structuring Pulumi Stacks per Environment — how per-tenant and per-environment stacks keep state isolated.