Cloud Compute on Azure
By default a Catchment runs every Pond on its own machine. With cloud compute set up, individual Ponds can instead run on Azure Container Apps jobs or Azure VMs, in your own subscription, while the Catchment stays where it is. This guide sets that up. Everything here is configuration: a Pond asking for cloud compute on a Catchment without it simply runs locally, so the same pond.toml works everywhere.
How it fits together
The Catchment launches a Duck for a Pond when it needs to run, and the Duck connects back to the Catchment for its work. Connections only ever go from the Duck to the Catchment, so Ducks need no inbound network access, only outbound access to the Catchment and to storage. Published tables live in Azure storage, where every Duck can reach them.
Cloud compute turns on when both of these hold:
- The Catchment's data root is in Azure storage (
abfss://oraz://). - The Catchment has Azure credentials: a managed identity, a service principal (
AZURE_TENANT_ID,AZURE_CLIENT_IDandAZURE_CLIENT_SECRET, as environment variables or all three as secrets), workload identity, or anaz login.
duckstring catchment settings
data root: abfss://lake@acmelake.dfs.core.windows.net/duckstring
cloud: Azure
creds: yes
compute: enabled
A Catchment runs cloud Ducks only on its data root's cloud. A pipeline that spans clouds is a Catchment on each, joined by ducts.
Step 1: the data root
Use an ADLS Gen2 storage account, which is a storage account with hierarchical namespace turned on:
duckstring catchment settings --data-root 'abfss://lake@acmelake.dfs.core.windows.net/duckstring'
The data root can only be set before any Pond has published data, since changing it later would strand that data. With managed identities, as below, the URI needs no credentials. Data root URIs lists the other forms.
Step 2: identities
Two user-assigned managed identities are involved, one for the Catchment and one for its Ducks. Ducks never receive the Catchment's identity.
The Catchment's identity launches Ducks and reads the data root:
- Contributor on the resource group the Ducks run in.
- Managed Identity Operator on the Duck identity, so it can give that identity to a job or VM.
- Storage Blob Data Contributor on the storage account.
- AcrPull on the container registry, if the Duck image is in one.
The Duck's identity needs Storage Blob Data Contributor on the storage account, plus AcrPull if the image is in a registry of yours, and nothing else.
Owner and Contributor don't grant access to the data in a storage account. Without Storage Blob Data Contributor, an identity that can manage the account still can't read or write its tables.
A Catchment running somewhere other than Azure, such as a laptop, can use a service principal with the same roles instead of a managed identity.
Step 3: resource providers
The subscription must have these resource providers registered:
Microsoft.AppandMicrosoft.OperationalInsights, for Container Apps.Microsoft.ComputeandMicrosoft.Network, for VMs.Microsoft.ContainerRegistry, if you keep the image in a registry.
az provider register --namespace Microsoft.App
Registering a provider is a subscription-level action. A principal whose roles are scoped to a resource group can't do it, so someone with rights on the subscription has to register them once.
Step 4: reaching the Catchment
Ducks connect back to the address the Catchment is bound to. If that address isn't reachable from where the Ducks run, set the address they should use:
DUCKSTRING_CATCHMENT_PUBLIC_URL=https://catchment.example.com
On AWS a Catchment on a laptop can run cloud Ducks through a relay. Azure has no relay, so a Catchment there needs an address Ducks can reach. In practice that means running the Catchment in Azure, as described in Running the Catchment on Azure.
Ducks also need outbound access to storage. A Duck for a Pond with its own Python dependencies builds that environment when it starts, so it needs a route to PyPI too.
Step 5a: Container Apps Ducks
Container Apps is the default backend. Each Duck runs as one execution of a Container Apps job, which reaches its first Ripple in about 20 to 40 seconds from a cold start.
Create a Container Apps environment (the consumption profile is enough, and it comes with a Log Analytics workspace), then point the Catchment at it and at the Duck identity:
DUCKSTRING_CONTAINERAPPS_ENVIRONMENT=/subscriptions/<sub>/resourceGroups/duckstring/providers/Microsoft.App/managedEnvironments/duckstring-env
DUCKSTRING_CONTAINERAPPS_IDENTITY=/subscriptions/<sub>/resourceGroups/duckstring/providers/Microsoft.ManagedIdentity/userAssignedIdentities/duckstring-duck
Both are resource ids. The subscription, resource group, location and log workspace all come from the environment. Optional settings:
| Variable | Default | Description |
|---|---|---|
DUCKSTRING_CONTAINERAPPS_IMAGE | the release's -azure image | The Duck image. |
DUCKSTRING_CONTAINERAPPS_WORKLOAD_PROFILE | Consumption | The environment's workload profile to run Ducks on. |
DUCKSTRING_CONTAINERAPPS_JOB | none | A job name of your own, instead of the one the Catchment generates. |
Then move a Pond onto it, using one of the built-in sizes:
duckstring duck set sales --duck M
duckstring refresh now sales
Or declare it in the Pond's pond.toml, so it travels with the code:
[pond]
duck = "M"
| Size | vCPU | Memory |
|---|---|---|
S | 0.5 | 1 GiB |
M | 1 | 2 GiB |
L | 2 | 4 GiB |
XL | 4 | 8 GiB |
These follow the consumption profile, which allows 2 GiB per vCPU, up to 4 vCPU and 8 GiB. For a Pond that needs more, use a VM (below).
Each Pond on a built-in size gets its own execution. A pool you define instead runs all of its Ponds' Ducks on one shared machine of its size. Sizes are in the same units on every cloud, --cpu in 1024ths of a vCPU and --memory in MiB:
duckstring duck pool add shared --cpu 2048 --memory 4096
The Catchment creates the Container Apps jobs itself, one per configuration, named duckstring-duck-<hash> and tagged with the Catchment. When an upgrade brings a new image, it deletes its own jobs on the old image once nothing is running on them. A job named with DUCKSTRING_CONTAINERAPPS_JOB is never deleted.
The Duck image
Each release of Duckstring publishes an Azure Duck image at ghcr.io/duckstring-dev/duckstring, tagged with its version and -azure (for example 0.8.0-azure). A Duck and its Catchment must run the same Duckstring, so the Catchment uses the tag that matches its own version unless you set DUCKSTRING_CONTAINERAPPS_IMAGE. Container Apps pulls it from ghcr directly.
The image includes DuckDB's Azure extension, so Ducks don't need to download it when they start.
A Duck downloads its Pond's code from the Catchment when it starts. A Pond that declares its own Python dependencies has its environment built at the same time, on every cold start.
Building your own image
Build your own image when Ponds without a pyproject.toml import packages the published image doesn't have, or need system libraries. The repository's Dockerfile builds the published image; pass the Azure extra to it:
docker build --platform linux/amd64 --build-arg EXTRAS=azure -t duckstring:0.8.0-azure .
Cloud Compute on AWS covers installing extra packages so that Pond environments reuse them. Push the image to an Azure Container Registry, and give both identities AcrPull on it. Container Apps pulls an ACR image with the Duck identity.
az acr build can be blocked on some subscriptions, failing with TasksOperationsNotAllowed. Building locally and pushing with docker push works on those too.
Step 5b: VM Ducks
Azure VMs suit sizes Container Apps doesn't offer, GPUs, or a particular VM size. Each Duck gets its own VM, which boots Ubuntu 24.04, installs Duckstring from PyPI with uv and starts the Duck. That takes about a minute, and the Catchment allows for it before treating a Duck as unresponsive.
DUCKSTRING_AZUREVM_SUBNET=/subscriptions/<sub>/resourceGroups/duckstring/providers/Microsoft.Network/virtualNetworks/duckstring-vnet/subnets/ducks
DUCKSTRING_AZUREVM_IDENTITY=/subscriptions/<sub>/resourceGroups/duckstring/providers/Microsoft.ManagedIdentity/userAssignedIdentities/duckstring-duck
DUCKSTRING_AZUREVM_LOCATION=australiaeast
duckstring duck pool add heavy --provider azurevm --instance-type Standard_E8s_v5
duckstring duck set sales --duck heavy
| Variable | Default | Description |
|---|---|---|
DUCKSTRING_AZUREVM_RESOURCE_GROUP | the subnet's | Where the VMs are created. |
DUCKSTRING_AZUREVM_IMAGE | Ubuntu 24.04 | A marketplace URN or an image resource id. |
DUCKSTRING_AZUREVM_ASSIGN_PUBLIC_IP | true | Give each VM a public IP for outbound access. |
DUCKSTRING_AZUREVM_PIP_SPEC | duckstring[azure]==<version> | What the VM installs. |
DUCKSTRING_AZUREVM_DISK_GB | the image's | OS disk size. |
Without an instance type, a VM is a Standard_D2s_v5.
A VM downloads uv, Python and Duckstring when it boots. New Azure subnets have no default outbound access, which is why each VM gets a public IP by default. If you set DUCKSTRING_AZUREVM_ASSIGN_PUBLIC_IP=false, give the subnet a NAT gateway or another route out.
An Azure VM whose operating system has shut down is still allocated and still billed. The Catchment therefore deletes a Duck's VM, with its disk, network interface and IP, once the Duck has finished. After a restart, its first VM launch deletes any of its VMs it finds stopped, and when the Catchment shuts down it deletes the VMs it started.
Running the Catchment on Azure
A Catchment can run as a Container App. This configuration was tested end to end:
- One replica.
- External ingress on port 7474.
- The Catchment identity assigned to the app, with
AZURE_CLIENT_IDset to its client id. - An API key stored as a Container Apps secret and passed as
DUCKSTRING_API_KEY. DUCKSTRING_CATCHMENT_PUBLIC_URLset to the app's ownhttps://<app>.<env>.<region>.azurecontainerapps.ioaddress.- The Container Apps and VM settings above, as environment variables.
- This command:
duckstring catchment init --name prod --host 0.0.0.0 --data-root 'abfss://lake@acmelake.dfs.core.windows.net/duckstring' --yes --no-start \
&& exec duckstring catchment start prod
A container loses its filesystem when it restarts, and the Catchment's own state (its database of Ponds, runs and settings) lives there. Give it a state backup with DUCKSTRING_STATE_BACKUP_URI, as described in Hosting on a Platform.
The Flock on Azure Databricks
The Flock sends a recompute too large for a Duck to another engine, and on Azure that can be a Databricks SQL warehouse. Ducks can sign in to the workspace with their own managed identity, so no Databricks credential needs to be stored.
- In the workspace's admin settings, add the Duck identity as a Microsoft Entra ID managed service principal, using its client id, with the Workspace access and Databricks SQL access entitlements.
- Give it Can use on a serverless SQL warehouse, which starts in seconds and stops itself when idle.
- Grant it a Volume to stage data in:
GRANT USE CATALOG ON CATALOG main TO `<client id>`;
GRANT USE SCHEMA ON SCHEMA main.duckstring TO `<client id>`;
GRANT READ VOLUME, WRITE VOLUME ON VOLUME main.duckstring.flock TO `<client id>`;
Then configure the Catchment:
DUCKSTRING_FLOCK_DATABRICKS_HOST=https://adb-1234567890123456.7.azuredatabricks.net
DUCKSTRING_FLOCK_DATABRICKS_WAREHOUSE=1a2b3c4d5e6f7a8b
DUCKSTRING_FLOCK_DATABRICKS_VOLUME=/Volumes/main/duckstring/flock
When no Databricks credentials are set (DATABRICKS_TOKEN, or DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET, as secrets or environment variables), a Duck working with an azuredatabricks.net workspace asks its Azure identity for a Microsoft Entra token for Azure Databricks. Container Apps Ducks use the Duck identity, whose client id the Catchment gives each one, and VM Ducks the one identity assigned to the VM. The token and the Flock's Databricks engine both come with the azure extra, which the -azure Duck image includes. A Pond running on the Catchment's own machine signs in as the Catchment's identity, which then needs the same access.
DuckDB stays the authority on results: Databricks only gets constructs measured to agree with DuckDB, and every result is cast to DuckDB's schema and checked against a local recompute of a sample before it's used. Offloading Large Recomputes has the detail.
Where data goes
A Pond running on the Catchment's own machine publishes locally first, then copies its output to the data root in the background. A Pond on a cloud Duck publishes straight to the data root. A Pond downstream on a different machine waits until its Source's output has reached the data root, never reading output that isn't there yet.
To stop the Catchment without repeating recent work on restart, drain it first:
duckstring control sleep --all
Debugging a Duck
Start with the Pond's failure message. When a Duck fails, the Catchment asks Azure what happened and includes it. For Container Apps that is the execution's status, its container events (such as an image pull failure or an exit code) and the end of its console log. These come from Log Analytics, which receives them a few minutes late, so a failure right after a Duck starts may not include them yet.
To watch a Container Apps Duck's output live:
az containerapp job logs show -n <job> -g <resource-group> --execution <execution> --container duck --follow
A VM Duck that died before connecting leaves its boot output on the serial console, which the failure message also quotes:
az vm boot-diagnostics get-boot-log -g <resource-group> -n <vm>
Duck VMs have no inbound access, and their SSH key is one nobody holds. Don't add one: a Duck holds credentials for your data.
Cost
Duckstring itself costs nothing; you pay Azure for:
- the Catchment's machine or Container App, which runs continuously and is most of a small deployment's cost;
- Ducks, per run, since each is stopped when its Pond goes idle (a Pond with a recurring Refresh keeps its Duck warm, and a Duck that loses its Catchment stops itself after an hour idle; see
DUCKSTRING_DUCK_ORPHAN_MINUTES); - the Log Analytics workspace, and the container registry if you use one;
- a Databricks SQL warehouse, while it runs, only when the Flock uses it;
- storage and transactions.
Checklist
If cloud Ducks don't start:
-
duckstring catchment settingsshowscompute: enabled - Both identities have Storage Blob Data Contributor on the storage account
- The Catchment's identity has Contributor on the Ducks' resource group and Managed Identity Operator on the Duck identity
- The resource providers are registered on the subscription
- The Ducks can reach the Catchment's address, or
DUCKSTRING_CATCHMENT_PUBLIC_URLis set - An image in ACR has AcrPull granted to the Duck identity
- An image you set yourself matches the Catchment's version
- VM Ducks have a public IP or another route to the internet