Ephemeral Remote Docker Builds with Crabbox

Published on Sep 21, 2026

Table of Contents

Building Docker images locally is convenient until it stops being convenient.

Maybe your laptop is ARM but production is linux/amd64. Maybe the image needs enough CPU or memory that local builds become painfully slow. Maybe you do not want long-lived build servers. Or maybe your build needs access to private source dependencies and a container registry, and you want to be very deliberate about where those credentials go.

A useful pattern is to treat the build machine itself as disposable infrastructure:

  1. create a temporary remote machine,
  2. sync the source tree,
  3. copy only the credentials the build actually needs,
  4. build and push the image remotely,
  5. export reusable build cache to the registry,
  6. destroy the machine whether the build succeeds or fails.

Crabbox is a good fit for this model because it can provision short-lived machines, synchronize a repository, run commands remotely, and tear the lease down afterward. Docker Buildx then handles the actual build and registry push.

This article walks through the architecture and the implementation details that tend to matter in real systems: reproducibility, private Git dependencies, BuildKit secrets, registry credentials, provider credentials, cache persistence, failure handling, and cleanup.

The architecture

The developer machine should orchestrate the build, but it should not be the build machine.

Developer machine
    |
    | provider credentials stay here
    |
    +--> Crabbox provisions ephemeral amd64 VM
             |
             +--> repository is synchronized
             |
             +--> short-lived build secrets are copied explicitly
             |
             +--> remote script:
                     - installs/verifies Docker + Buildx
                     - authenticates to registry
                     - builds image
                     - fetches private dependencies via BuildKit secret
                     - imports/exports registry-backed build cache
                     - pushes immutable image tag
             |
             +--> VM is destroyed

There is no persistent BuildKit daemon to maintain, no permanently running CI worker, and no Docker daemon exposed over a network tunnel.

The remote host is just an ordinary short-lived Linux machine.

That simplicity is important.

Why build on an ephemeral remote machine?

There are several practical reasons.

Consistent target architecture

If production runs on linux/amd64 but developers use ARM-based laptops, local builds can involve emulation or subtle platform differences.

A remote amd64 builder removes that ambiguity.

The machine that performs the build has the same CPU architecture as the deployment target.

More resources when needed

Container builds for compiled languages can consume a lot of memory and CPU. It is often cheaper and more convenient to rent a larger machine for ten minutes than to size a developer workstation or permanent CI worker for peak build load.

No build-server lifecycle to manage

Persistent build hosts accumulate state:

  • Docker layers,
  • old credentials,
  • abandoned workspaces,
  • stale package caches,
  • SSH configuration,
  • failed containers,
  • old Buildx builders.

An ephemeral builder starts clean and disappears afterward.

Better credential boundaries

A temporary builder can receive only the credentials required for one build. Cloud-provider credentials can remain on the developer machine entirely.

That creates a much cleaner trust boundary.

1. Make the build reproducible first

Before provisioning anything, decide exactly what an image tag means.

A very useful invariant is:

container image tag == Git commit SHA

For example:

registry.example.com/acme/api:9f6a2d77...

If that tag is supposed to describe a specific Git revision, the working tree must not contain uncommitted or untracked source changes.

A local orchestration script can enforce this:

if [[ -n "$(git status --porcelain --untracked-files=normal)" ]]; then
    echo "Working tree is dirty. Commit or remove changes before building." >&2
    exit 1
fi

For dependency managers that use lock files, verify those files are committed too.

For example:

[[ -f Package.resolved ]] || {
    echo "Package.resolved is required" >&2
    exit 1
}
 
git ls-files --error-unmatch Package.resolved >/dev/null 2>&1 || {
    echo "Package.resolved exists but is not tracked by Git" >&2
    exit 1
}

The principle applies beyond Swift:

  • package-lock.json
  • pnpm-lock.yaml
  • yarn.lock
  • poetry.lock
  • uv.lock
  • Cargo.lock
  • go.sum

If the resulting image will be addressed by commit SHA, then the build inputs should also be tied to that commit.

2. Define the Crabbox builder

A checked-in Crabbox configuration can describe the disposable build environment.

For example:

profile: container-cloud-build
 
serverType: cx43
architecture: amd64
os: ubuntu:24.04
 
lease:
  ttl: 2h
  idleTimeout: 30m
 
sync:
  source: git
  delete: true
  fingerprint: true
  baseRef: main
  timeout: 15m
  exclude:
    - .build
    - .cache
    - .env
    - .env.*
    - node_modules
    - DerivedData
 
env:
  allow:
    - CI
    - BUILD_JOBS
    - CLOUD_BUILD_CACHE

The exact machine type will depend on the provider and workload, but several ideas are worth preserving.

Use the target architecture explicitly

If production is x86-64:

architecture: amd64

and later:

--platform linux/amd64

Do not let the build platform silently depend on whichever machine happened to run the command.

Give the lease a TTL

Even if every script contains cleanup logic, infrastructure should still have an expiration policy.

For example:

lease:
  ttl: 2h
  idleTimeout: 30m

Think of this as the final safety net, not the primary cleanup mechanism.

Exclude secrets and build products from sync

The source sync should not blindly copy the developer's directory.

At minimum, exclude:

.env
.env.*
*.pem
*.key
build outputs
language package caches
IDE derived data

The same philosophy should be applied to .dockerignore.

3. Separate the three credential domains

A remote cloud build commonly needs three unrelated types of credentials.

They should remain separate.

A. Cloud-provider credentials

Examples:

HCLOUD_TOKEN
AWS_PROFILE
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

These credentials exist only so Crabbox can create and destroy the machine.

They should stay on the developer machine.

The remote builder does not need permission to create more cloud infrastructure.

B. Source dependency credentials

If the application depends on private repositories, the build may need a read-only GitHub or GitLab token.

For example:

GITHUB_DEP_TOKEN

This token should ideally have access only to the repositories needed during dependency resolution.

It does not need registry write permissions.

C. Container registry credentials

The remote builder must push the finished image.

For example:

REGISTRY_USER
REGISTRY_TOKEN

This credential needs registry access, but it usually does not need access to the application's private source repositories.

Why the separation matters

Do not create one enormous token that can:

  • read source,
  • write packages,
  • manage infrastructure,
  • and perhaps access other unrelated systems.

Instead, the trust graph should look more like this:

developer machine
    |
    +-- cloud credential ---> Crabbox/provider API only
 
ephemeral builder
    |
    +-- dependency token ---> private source repositories only
    |
    +-- registry token -----> container registry only

The blast radius of each secret is much smaller.

4. Provision first, then copy secrets explicitly

A useful orchestration flow is:

crabbox doctor --provider "$PROVIDER"
 
crabbox warmup \
    --provider "$PROVIDER" \
    --slug "$SLUG"

Capture the canonical lease ID that Crabbox returns and use that exact ID for all later operations.

Then create a secret directory on the remote machine without syncing the repository yet:

crabbox run \
    --id "$LEASE_ID" \
    --no-sync -- \
    sh -c '
        umask 077
        rm -rf /tmp/cloud-build-secrets
        mkdir -m 700 /tmp/cloud-build-secrets
    '

Create temporary secret files locally with restrictive permissions:

umask 077
SECRET_DIR=$(mktemp -d)
 
printf '%s' "$GITHUB_DEP_TOKEN" > "$SECRET_DIR/github_token"
printf '%s' "$REGISTRY_TOKEN"   > "$SECRET_DIR/registry_token"
 
chmod 0600 \
    "$SECRET_DIR/github_token" \
    "$SECRET_DIR/registry_token"

Copy only those files:

crabbox cp \
    --id "$LEASE_ID" \
    "$SECRET_DIR/github_token" \
    SANDBOX:/tmp/cloud-build-secrets/github_token
 
crabbox cp \
    --id "$LEASE_ID" \
    "$SECRET_DIR/registry_token" \
    SANDBOX:/tmp/cloud-build-secrets/registry_token

Then fix the permissions remotely:

crabbox run \
    --id "$LEASE_ID" \
    --no-sync -- \
    sh -c '
        chmod 0600 \
            /tmp/cloud-build-secrets/github_token \
            /tmp/cloud-build-secrets/registry_token
    '

This is preferable to putting secrets in:

  • .crabbox.yaml,
  • shell command arguments,
  • Docker build arguments,
  • checked-in configuration,
  • source sync manifests.

5. Run the synchronized build

Once the remote secret files exist, run the real build command:

crabbox run \
    --id "$LEASE_ID" \
    --stop-after always -- \
    scripts/cloud/remote-docker-build.sh \
        --image "$IMAGE" \
        --cache-image "$CACHE_IMAGE" \
        --registry-user "$REGISTRY_USER" \
        --platform linux/amd64

This invocation can perform the configured repository synchronization before running the script.

The important option is:

--stop-after always

The builder should be terminated whether the command succeeds or fails.

But this should not be the only cleanup mechanism. We will add more layers later.

6. Bootstrap Docker on the temporary machine

The remote machine can either come with Docker installed or install it on demand.

A remote script might first check:

if command -v docker >/dev/null \
   && docker version >/dev/null 2>&1 \
   && docker buildx version >/dev/null 2>&1
then
    echo "Docker and Buildx already available"
else
    # install Docker Engine + Buildx
fi

For a clean Ubuntu host, installation can use Docker's package repository.

Once Docker is available, create a temporary Docker configuration directory:

DOCKER_CONFIG=$(mktemp -d)
export DOCKER_CONFIG
chmod 700 "$DOCKER_CONFIG"

This is worth doing because docker login stores registry authentication in Docker's configuration.

You do not want that authentication left in a normal user home directory if the script fails halfway through.

Authenticate with --password-stdin:

cat /tmp/cloud-build-secrets/registry_token |
    docker login registry.example.com \
        --username "$REGISTRY_USER" \
        --password-stdin

Avoid:

docker login --password "$REGISTRY_TOKEN"

because command-line arguments can leak through process inspection, logs, shell history, or debugging output.

7. Create a disposable Buildx builder

Use a docker-container Buildx builder:

BUILDER_NAME="cloud-build-${CRABBOX_RUN_ID:-$$}"
 
docker buildx create \
    --name "$BUILDER_NAME" \
    --driver docker-container \
    --use
 
docker buildx inspect "$BUILDER_NAME" --bootstrap

The Buildx instance itself is temporary.

It can be removed during cleanup:

docker buildx rm --force "$BUILDER_NAME"

This model avoids depending on a shared global builder state.

8. Private Git dependencies without SSH keys

Private repository dependencies are where remote container builds often become messy.

A common first attempt is to forward an SSH agent into Docker:

RUN --mount=type=ssh git clone [email protected]:acme/private-lib.git

That can work, but it creates several additional concerns:

  • the remote VM needs SSH forwarding support,
  • host keys must be managed,
  • the developer's SSH identity becomes part of the workflow,
  • troubleshooting tunnels and agents becomes another layer of complexity.

For an ephemeral builder, HTTPS with a narrowly scoped token is often simpler.

Rewrite SSH-style Git URLs to HTTPS

Your package manifest may still contain SSH URLs for normal developer workflows.

Inside the Docker build, Git can rewrite them automatically:

RUN git config --global --add \
      url."https://github.com/".insteadOf "[email protected]:" \
    && git config --global --add \
      url."https://github.com/".insteadOf "ssh://[email protected]/"

The application source does not have to change.

Use a Git askpass helper

Add a small script:

#!/bin/sh
set -eu
 
case "${1:-}" in
    *Username*)
        printf '%s\n' 'x-access-token'
        ;;
    *Password*)
        cat /run/secrets/github_token
        ;;
    *)
        exit 1
        ;;
esac

Copy it into the build image:

COPY scripts/docker/github-askpass.sh /usr/local/bin/github-askpass
RUN chmod 0755 /usr/local/bin/github-askpass

Then use it only inside build steps that mount the BuildKit secret.

For example:

RUN --mount=type=secret,id=github_token,required=true \
    GIT_ASKPASS=/usr/local/bin/github-askpass \
    GIT_TERMINAL_PROMPT=0 \
    swift package resolve

And later:

RUN --mount=type=secret,id=github_token,required=true \
    GIT_ASKPASS=/usr/local/bin/github-askpass \
    GIT_TERMINAL_PROMPT=0 \
    swift build -c release

The same pattern works for other package managers and Git-based dependencies.

9. Why BuildKit secrets matter

Do not pass credentials like this:

docker build \
    --build-arg GITHUB_TOKEN="$GITHUB_TOKEN" \
    .

and then:

ARG GITHUB_TOKEN
RUN git clone "https://${GITHUB_TOKEN}@github.com/acme/private-lib.git"

Build arguments are not designed for secrets.

They can leak into:

  • image metadata,
  • build history,
  • intermediate state,
  • cache metadata,
  • logs.

Instead, pass the dependency token as a BuildKit secret:

docker buildx build \
    --secret id=github_token,src=/tmp/cloud-build-secrets/github_token \
    .

or for a local build:

docker buildx build \
    --secret id=github_token,env=GITHUB_DEP_TOKEN \
    .

Inside the Dockerfile, the secret appears only for the duration of the specific RUN --mount=type=secret instruction.

It is not copied into the image layer.

10. Build and push atomically

The remote builder should produce the registry artifact directly.

For example:

docker buildx build \
    --builder "$BUILDER_NAME" \
    --platform linux/amd64 \
    --progress plain \
    --secret id=github_token,src=/tmp/cloud-build-secrets/github_token \
    --tag "$IMAGE" \
    --push \
    .

This is preferable to:

  1. building an image,
  2. keeping it only on the temporary host,
  3. separately pushing later.

The useful output of the ephemeral machine is not a local Docker image.

The useful output is the image in the registry.

Once the registry confirms the push, the machine can disappear.

11. Persist build cache in the registry

Ephemeral workers are clean by design, which means local Docker layer caches disappear with every machine.

Without another cache mechanism, every build starts from zero.

Buildx can store its cache in the registry:

CACHE_IMAGE="registry.example.com/acme/api:buildcache-amd64"

Then:

docker buildx build \
    --cache-from "type=registry,ref=$CACHE_IMAGE" \
    --cache-to "type=registry,ref=$CACHE_IMAGE,mode=max" \
    --tag "$IMAGE" \
    --push \
    .

This gives you the best of both worlds:

  • disposable machines,
  • persistent reusable build cache.

The next temporary machine can import the previous build's cache from the registry.

For diagnostics, it is useful to support an environment switch:

CLOUD_BUILD_CACHE=0

so a build can run without imported cache when debugging cache-related behavior.

12. Control build concurrency

Large remote machines make it tempting to simply turn every build knob to the maximum.

That is not always wise.

Compiled-language builds can become memory-bound before they become CPU-bound.

Expose concurrency explicitly:

BUILD_JOBS=8
PACKAGE_MANAGER_JOBS=8

Then pass them as non-secret build arguments:

docker buildx build \
    --build-arg "BUILD_JOBS=$BUILD_JOBS" \
    --build-arg "PACKAGE_MANAGER_JOBS=$PACKAGE_MANAGER_JOBS" \
    ...

In the Dockerfile:

ARG BUILD_JOBS
ARG PACKAGE_MANAGER_JOBS

This makes the remote machine size and build parallelism independently tunable.

13. Harden the Docker build context

Even if the Crabbox sync excludes secrets, Docker has a second boundary: the build context.

Use .dockerignore.

For example:

.git
.env
.env.*
!.env.example
 
*.pem
*.key
**/*.pem
**/*.key
 
.build
.cache
.swiftpm
node_modules
DerivedData
 
.crabbox
coverage

Defense in depth matters here.

A file excluded from Crabbox sync cannot accidentally reach Docker.

A file excluded from Docker's build context cannot accidentally become part of an image layer even if it exists on the remote machine.

14. Clean up at several layers

A reliable ephemeral build should assume that any one cleanup mechanism can fail.

Use multiple independent layers.

Layer 1: Crabbox automatic stop

Run the final build with:

--stop-after always

That handles normal success and normal command failure.

Layer 2: local shell traps

The local orchestration script should stop the lease if anything fails before the final remote command.

For example:

cleanup() {
    status=$?
 
    if [[ -n "${LEASE_ID:-}" ]]; then
        crabbox stop \
            --provider "$PROVIDER" \
            "$LEASE_ID" >/dev/null 2>&1 || true
    fi
 
    rm -rf -- "${SECRET_DIR:-}"
 
    exit "$status"
}
 
trap cleanup EXIT
trap 'exit 129' HUP
trap 'exit 130' INT
trap 'exit 143' TERM

Be especially careful about interruption during provisioning. A script may be terminated after the VM was created but before the canonical lease ID was stored.

One useful technique is to capture the warmup output into a log and recover the lease ID from it in the cleanup handler.

A unique per-run slug can also serve as a fallback identifier.

Layer 3: remote cleanup traps

The remote build script should clean local machine state before the VM disappears:

cleanup() {
    status=$?
 
    docker buildx rm \
        --force "$BUILDER_NAME" \
        >/dev/null 2>&1 || true
 
    rm -rf -- "$DOCKER_CONFIG"
    rm -rf -- /tmp/cloud-build-secrets
 
    exit "$status"
}
 
trap cleanup EXIT

This helps even if the machine remains alive for a short period after failure.

Layer 4: lease TTL and idle expiry

The cloud lease itself should eventually expire even if:

  • the local process crashes,
  • a laptop loses power,
  • the network disappears,
  • cleanup commands fail.

This is the last line of defense against orphaned infrastructure.

15. A complete remote build skeleton

Here is a generic remote script showing the whole pattern:

#!/usr/bin/env bash
set -Eeuo pipefail
 
SECRET_DIR=/tmp/cloud-build-secrets
TARGET_PLATFORM=${TARGET_PLATFORM:-linux/amd64}
CACHE_ENABLED=${CLOUD_BUILD_CACHE:-1}
 
IMAGE=
CACHE_IMAGE=
REGISTRY_USER=
 
while (($#)); do
    case "$1" in
        --image)
            IMAGE=${2:?}
            shift 2
            ;;
        --cache-image)
            CACHE_IMAGE=${2:?}
            shift 2
            ;;
        --registry-user)
            REGISTRY_USER=${2:?}
            shift 2
            ;;
        --platform)
            TARGET_PLATFORM=${2:?}
            shift 2
            ;;
        *)
            echo "Unknown argument: $1" >&2
            exit 2
            ;;
    esac
done
 
: "${IMAGE:?--image is required}"
: "${CACHE_IMAGE:?--cache-image is required}"
: "${REGISTRY_USER:?--registry-user is required}"
 
GITHUB_TOKEN_FILE="$SECRET_DIR/github_token"
REGISTRY_TOKEN_FILE="$SECRET_DIR/registry_token"
 
[[ -s "$GITHUB_TOKEN_FILE" ]] || {
    echo "Dependency token is missing" >&2
    exit 1
}
 
[[ -s "$REGISTRY_TOKEN_FILE" ]] || {
    echo "Registry token is missing" >&2
    exit 1
}
 
BUILDER_NAME="crabbox-${CRABBOX_RUN_ID:-$$}"
BUILDER_NAME=${BUILDER_NAME//[^a-zA-Z0-9_.-]/-}
 
DOCKER_CONFIG=$(mktemp -d)
export DOCKER_CONFIG
chmod 700 "$DOCKER_CONFIG"
 
cleanup() {
    status=$?
 
    docker buildx rm \
        --force "$BUILDER_NAME" \
        >/dev/null 2>&1 || true
 
    rm -rf -- "$DOCKER_CONFIG"
    rm -rf -- "$SECRET_DIR"
 
    exit "$status"
}
 
trap cleanup EXIT
trap 'exit 129' HUP
trap 'exit 130' INT
trap 'exit 143' TERM
 
cat "$REGISTRY_TOKEN_FILE" |
    docker login registry.example.com \
        --username "$REGISTRY_USER" \
        --password-stdin
 
docker buildx create \
    --name "$BUILDER_NAME" \
    --driver docker-container \
    --use
 
docker buildx inspect "$BUILDER_NAME" --bootstrap
 
build_args=(
    --builder "$BUILDER_NAME"
    --platform "$TARGET_PLATFORM"
    --progress plain
    --secret "id=github_token,src=$GITHUB_TOKEN_FILE"
    --tag "$IMAGE"
    --push
)
 
if [[ "$CACHE_ENABLED" == 1 ]]; then
    build_args+=(
        --cache-from "type=registry,ref=$CACHE_IMAGE"
        --cache-to "type=registry,ref=$CACHE_IMAGE,mode=max"
    )
fi
 
docker buildx build "${build_args[@]}" .
 
echo "Pushed $IMAGE"

The provider-specific logic is intentionally absent.

That belongs in Crabbox.

The build script only cares that it is running on a Linux machine with enough privilege to use Docker.

16. A complete local orchestration skeleton

A matching local script can look like this:

#!/usr/bin/env bash
set -Eeuo pipefail
 
PROVIDER=${CRABBOX_PROVIDER:-hetzner}
TARGET_PLATFORM=linux/amd64
 
for command_name in git crabbox; do
    command -v "$command_name" >/dev/null || {
        echo "$command_name is required" >&2
        exit 1
    }
done
 
ROOT=$(git rev-parse --show-toplevel)
cd "$ROOT"
 
[[ -z "$(git status --porcelain --untracked-files=normal)" ]] || {
    echo "Repository must be clean" >&2
    exit 1
}
 
SHA=$(git rev-parse HEAD)
 
IMAGE="registry.example.com/acme/api:$SHA"
CACHE_IMAGE="registry.example.com/acme/api:buildcache-amd64"
 
: "${GITHUB_DEP_TOKEN:?Set GITHUB_DEP_TOKEN}"
: "${REGISTRY_USER:?Set REGISTRY_USER}"
: "${REGISTRY_TOKEN:?Set REGISTRY_TOKEN}"
 
umask 077
 
LOCAL_SECRET_DIR=$(mktemp -d)
LEASE_ID=
PROVISIONING_STARTED=0
SLUG="container-build-${SHA:0:12}-$$"
WARMUP_LOG="$LOCAL_SECRET_DIR/warmup.log"
 
cleanup() {
    status=$?
    cleanup_id=$LEASE_ID
 
    trap - EXIT
    set +e
 
    if [[ -z "$cleanup_id" && -f "$WARMUP_LOG" ]]; then
        cleanup_id=$(
            sed -nE \
                's/.*(cbx_[[:alnum:]_-]+).*/\1/p' \
                "$WARMUP_LOG" |
            tail -n 1
        )
    fi
 
    if [[ -n "$cleanup_id" ]]; then
        crabbox stop \
            --provider "$PROVIDER" \
            "$cleanup_id" \
            >/dev/null 2>&1 || true
    elif ((PROVISIONING_STARTED)); then
        crabbox stop \
            --provider "$PROVIDER" \
            "$SLUG" \
            >/dev/null 2>&1 || true
    fi
 
    rm -rf -- "$LOCAL_SECRET_DIR"
 
    exit "$status"
}
 
trap cleanup EXIT
trap 'exit 129' HUP
trap 'exit 130' INT
trap 'exit 143' TERM
 
printf '%s' "$GITHUB_DEP_TOKEN" \
    > "$LOCAL_SECRET_DIR/github_token"
 
printf '%s' "$REGISTRY_TOKEN" \
    > "$LOCAL_SECRET_DIR/registry_token"
 
chmod 0600 \
    "$LOCAL_SECRET_DIR/github_token" \
    "$LOCAL_SECRET_DIR/registry_token"
 
crabbox doctor --provider "$PROVIDER"
 
PROVISIONING_STARTED=1
 
crabbox warmup \
    --provider "$PROVIDER" \
    --slug "$SLUG" |
    tee "$WARMUP_LOG"
 
LEASE_ID=$(
    sed -nE \
        's/.*(cbx_[[:alnum:]_-]+).*/\1/p' \
        "$WARMUP_LOG" |
    tail -n 1
)
 
[[ "$LEASE_ID" =~ ^cbx_[[:alnum:]_-]+$ ]] || {
    echo "Could not determine Crabbox lease ID" >&2
    exit 1
}
 
crabbox run \
    --id "$LEASE_ID" \
    --no-sync -- \
    sh -c '
        umask 077
        rm -rf /tmp/cloud-build-secrets
        mkdir -m 700 /tmp/cloud-build-secrets
    '
 
crabbox cp \
    --id "$LEASE_ID" \
    "$LOCAL_SECRET_DIR/github_token" \
    SANDBOX:/tmp/cloud-build-secrets/github_token
 
crabbox cp \
    --id "$LEASE_ID" \
    "$LOCAL_SECRET_DIR/registry_token" \
    SANDBOX:/tmp/cloud-build-secrets/registry_token
 
crabbox run \
    --id "$LEASE_ID" \
    --no-sync -- \
    sh -c '
        chmod 0600 \
            /tmp/cloud-build-secrets/github_token \
            /tmp/cloud-build-secrets/registry_token
    '
 
crabbox run \
    --id "$LEASE_ID" \
    --stop-after always -- \
    scripts/cloud/remote-docker-build.sh \
        --image "$IMAGE" \
        --cache-image "$CACHE_IMAGE" \
        --registry-user "$REGISTRY_USER" \
        --platform "$TARGET_PLATFORM"
 
echo "Cloud build completed: $IMAGE"

This script is intentionally boring.

That is a feature.

Infrastructure orchestration is easier to audit when credentials and lifecycle transitions are explicit.

17. Keep local and cloud builds on the same Dockerfile

A useful side effect of the BuildKit secret approach is that local builds can use the same Dockerfile.

For example:

docker buildx build \
    --platform linux/amd64 \
    --secret id=github_token,env=GITHUB_DEP_TOKEN \
    --tag "$IMAGE" \
    --load \
    .

The local workflow and remote workflow differ only in where the builder runs and whether the image is loaded locally or pushed.

That reduces configuration drift.

A good target is:

same Dockerfile
same dependency authentication mechanism
same target architecture
same image naming
different execution location

18. Build first, deploy later

Do not make the temporary builder responsible for deployment too.

The builder should produce one durable artifact:

registry.example.com/acme/api:<git-sha>

Deployment can happen later:

build commit
    |
    v
immutable registry image
    |
    +--> staging
    |
    +--> production
    |
    +--> rollback

This separation has several benefits.

You can:

  • rebuild less often,
  • promote an already-built image,
  • roll back without rebuilding,
  • deploy from another machine,
  • verify an image exists before deployment,
  • reason about build provenance independently of runtime orchestration.

The cloud builder's lifetime should end after the push.

19. Security properties of the design

The resulting design has several useful properties.

Provider credentials never reach the builder

The temporary host cannot create more infrastructure because it never receives the developer's cloud-provider credentials.

Developer SSH keys never reach the builder

Private source dependencies are fetched over HTTPS using a narrowly scoped token.

Dependency tokens are not stored in image layers

They are exposed through:

RUN --mount=type=secret

only while needed.

Registry credentials live in a temporary Docker config

They are removed during remote cleanup and disappear with the VM anyway.

Secrets are not part of source sync

They are copied explicitly after provisioning.

The final artifact is immutable

The image is addressed by Git SHA rather than a floating tag such as latest.

Cache survives without a persistent machine

Build cache is stored in the registry rather than on the builder's filesystem.

20. Things to verify before trusting the system

Treat the first few runs as acceptance tests.

Inspect the sync plan

Before provisioning:

crabbox sync-plan

Confirm that it does not include:

  • .env,
  • SSH keys,
  • cloud credentials,
  • certificate files,
  • secret-manager exports,
  • unrelated developer files.

Test a successful build

After completion:

crabbox list

The builder lease should be gone.

Then verify the image exists:

docker buildx imagetools inspect \
    registry.example.com/acme/api:<git-sha>

Test an intentionally failing build

Break the Docker build on purpose.

Then verify again:

crabbox list

There should still be no orphaned machine.

Failure cleanup is at least as important as success cleanup.

Inspect the resulting image

Pull it and inspect the configuration and history:

docker pull registry.example.com/acme/api:<git-sha>
 
docker history \
    registry.example.com/acme/api:<git-sha>

Look for accidental credential fragments.

Also search the unpacked filesystem if your threat model requires it.

Verify caching

Run two clean builds of the same revision.

The second run should import useful cache from the registry.

If it does not, inspect the Buildx cache configuration rather than adding a persistent builder prematurely.

21. Common mistakes

Several design choices look convenient initially but create unnecessary risk or complexity.

Forwarding your personal SSH agent

This tightly couples remote builds to developer workstation state.

Prefer a scoped source token and BuildKit secret where practical.

Putting secrets in Docker build args

Build args are not secret storage.

Use BuildKit secrets.

Syncing the entire home directory or repository blindly

Inspect the sync manifest.

Explicitly exclude secrets and build artifacts.

Passing tokens directly on command lines

Use files or standard input.

Command-line arguments are surprisingly observable.

Keeping one permanent giant builder

Persistent builders accumulate both state and trust.

If cache persistence is the reason, try registry-backed Buildx cache first.

Rebuilding during deployment

Build once, push once, deploy the immutable artifact.

Relying on only one cleanup mechanism

Use command-level teardown, shell traps, remote cleanup, and provider TTLs together.

22. A practical mental model

It helps to think of Crabbox as providing the machine lifecycle, not the container build system itself.

Crabbox is responsible for:

create machine
sync repository
copy selected files
run command
destroy machine

Docker Buildx is responsible for:

resolve Docker build graph
mount build secrets
use remote/registry cache
produce target-platform image
push image

The container registry is responsible for persistence:

immutable application images
BuildKit cache

And the developer machine remains the control plane:

cloud credentials
build intent
image tag selection
lease ownership

Each component has a narrow job.

That is what makes the architecture easy to reason about.

Conclusion

Ephemeral remote Docker builds are not primarily about moving docker build from one machine to another.

The real value comes from making the build boundary explicit.

A well-designed flow gives you:

  • clean linux/amd64 builders on demand,
  • no permanent build server,
  • no developer SSH keys on remote machines,
  • no cloud-provider credentials on builders,
  • private dependency access through BuildKit secrets,
  • registry credentials isolated from source credentials,
  • reusable remote build cache,
  • immutable Git-SHA image tags,
  • automatic teardown on success and failure,
  • and a clean separation between build and deployment.

Crabbox provides a convenient lifecycle around the disposable machine, while Docker Buildx and the registry provide the build and persistence layers.

The resulting system is simple enough to audit, cheap enough to run only when needed, and secure enough that the builder can be treated as what it should be: temporary infrastructure that exists only long enough to produce one durable artifact.