Docker in Docker (DinD)

AI agents often need to build container images or run containers as part of their workflow — for example, running integration tests with database containers, building and pushing images in CI/CD pipelines, or testing Dockerfiles.

The default devbox executor image already includes the Docker CLI (docker-ce-cli). To enable full Docker functionality, you need a Docker daemon (dockerd) running inside the agent Pod. This guide covers three approaches, each with different security and complexity trade-offs.

How It Works

KubeOpenCode uses a two-container pattern: an init container copies the OpenCode binary to /tools, and the worker container runs it. The controller sets the container's Command explicitly (e.g., sh -c "/tools/opencode serve ..."), which overrides any Dockerfile ENTRYPOINT.

This means you cannot simply set ENTRYPOINT to a dockerd startup script. Instead, the recommended approach is a lazy-init wrapper — a script that replaces the docker binary and transparently starts dockerd on first use. This works with both Agent Deployments and Task Pods without any controller code changes.

Comparison

	Sysbox Runtime	Privileged Mode	Rootless Docker
Security	Strong (user namespace isolation)	Weak (root on node)	Moderate
Complexity	Medium (requires Sysbox on nodes)	Low (config only)	High (many dependencies)
Capabilities needed	None	All (privileged)	SYS_ADMIN, NET_ADMIN
OpenShift compatible	Yes (with Sysbox)	No	Partial
Performance overhead	~5-10%	None	~10-15%
Recommendation	Production	Dev/test only	Fallback

Custom Executor Image

All three approaches share a common custom executor image pattern. Build a devbox-dind image that extends devbox with Docker daemon and a lazy-init wrapper:

FROM ghcr.io/kubeopencode/kubeopencode-agent-devbox:latest

USER root

# Install Docker daemon and containerd
# (docker-ce-cli is already installed in devbox)
RUN apt-get update && apt-get install -y --no-install-recommends \
    docker-ce \
    containerd.io \
    && rm -rf /var/lib/apt/lists/*

# Install lazy-init docker wrapper
# This transparently starts dockerd on first `docker` command
RUN mv /usr/bin/docker /usr/bin/docker.real
COPY --chmod=755 docker-lazy-init.sh /usr/bin/docker

# Note: USER is intentionally set to root (0) for DinD.
# For Sysbox: inner root is mapped to unprivileged host user via user namespaces.
# For Privileged mode: root is required to start dockerd.
# This differs from the base devbox image which uses USER 1000:0.
USER 0:0

WORKDIR /workspace
CMD ["/bin/zsh"]

Create docker-lazy-init.sh:

#!/bin/bash
# Lazy-init wrapper for Docker CLI.
# Starts dockerd automatically on first docker command.
# Works with KubeOpenCode's command override pattern since it wraps
# the docker binary itself, not the container entrypoint.

DOCKER_REAL="/usr/bin/docker.real"
DOCKERD_LOG="/tmp/dockerd.log"
DOCKERD_PIDFILE="/tmp/dockerd.pid"

# Check if dockerd is already running
if ! ${DOCKER_REAL} info >/dev/null 2>&1; then
    # Acquire lock to prevent concurrent dockerd starts
    exec 200>/tmp/dockerd.lock
    flock -n 200 || {
        # Another process is starting dockerd, wait for it
        flock 200
        exec ${DOCKER_REAL} "$@"
    }

    # Double-check after acquiring lock
    if ! ${DOCKER_REAL} info >/dev/null 2>&1; then
        # Mount tmpfs at /var/lib/docker to avoid overlay-on-overlay issues.
        # Container filesystems use overlayfs (from containerd/CRI), and nested
        # overlayfs is not supported on most kernels. tmpfs provides a clean
        # non-overlay mount point where Docker can use its default storage driver.
        # Note: This means Docker images/containers are stored in memory and
        # lost on pod restart. For persistence, use a PVC mounted at /var/lib/docker.
        if ! mountpoint -q /var/lib/docker 2>/dev/null; then
            mkdir -p /var/lib/docker
            mount -t tmpfs tmpfs /var/lib/docker
        fi

        echo "Starting Docker daemon..." >&2
        dockerd &>"${DOCKERD_LOG}" &
        echo $! > "${DOCKERD_PIDFILE}"

        # Wait for Docker daemon to be ready
        timeout=30
        while ! ${DOCKER_REAL} info >/dev/null 2>&1; do
            timeout=$((timeout - 1))
            if [ $timeout -le 0 ]; then
                echo "ERROR: Docker daemon failed to start. Check ${DOCKERD_LOG}" >&2
                exit 1
            fi
            sleep 1
        done
        echo "Docker daemon ready" >&2
    fi
fi

exec ${DOCKER_REAL} "$@"

Build the image:

docker build -t your-registry/devbox-dind:latest .
docker push your-registry/devbox-dind:latest

Option 1: Sysbox Runtime (Recommended)

Sysbox is an OCI runtime that enables containers to run system-level workloads (like Docker) securely, without requiring privileged mode. It uses Linux user namespaces: the container runs as root internally, but maps to an unprivileged user on the host.

Prerequisites

1. Label nodes for Sysbox installation:

kubectl label nodes <node-name> sysbox-install=yes

2. Deploy Sysbox on cluster nodes:

kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox/master/sysbox-k8s-manifests/sysbox-install.yaml

This deploys a DaemonSet that installs the sysbox-runc runtime on labeled nodes. It also creates the sysbox-runc RuntimeClass automatically. See sysbox-deploy-k8s for details.

3. Verify Sysbox is running:

# Check the DaemonSet
kubectl get daemonset sysbox-deploy-k8s -n kube-system

# Verify RuntimeClass exists
kubectl get runtimeclass sysbox-runc

Note: Sysbox requires upstream Kubernetes (GKE, EKS, AKS, kubeadm). K3s/K0s support is experimental. See sysbox compatibility for details.

Agent Configuration

apiVersion: kubeopencode.io/v1alpha1
kind: Agent
metadata:
  name: dind-agent
spec:
  profile: "Agent with Docker-in-Docker support via Sysbox"
  agentImage: ghcr.io/kubeopencode/kubeopencode-agent-opencode:latest
  executorImage: your-registry/devbox-dind:latest
  workspaceDir: /workspace
  serviceAccountName: kubeopencode-agent
  podSpec:
    runtimeClassName: sysbox-runc
    resources:
      requests:
        memory: "1Gi"
      limits:
        memory: "8Gi"

No special securityContext is needed. Sysbox handles isolation at the runtime level. The default restricted security context is compatible with Sysbox — the runtime transparently enables the required kernel features.

Verification

Create a Task to verify DinD is working:

apiVersion: kubeopencode.io/v1alpha1
kind: Task
metadata:
  name: test-dind
spec:
  agentRef:
    name: dind-agent
  prompt: |
    Run the following commands and report the output:
    1. docker info
    2. docker run --rm hello-world
    3. echo 'FROM alpine:latest' | docker build -t test-image -

Option 2: Privileged Mode (Dev/Test Only)

Security Warning: Privileged containers have full access to the host kernel. A container escape grants root access to the node. Never use this in production or multi-tenant clusters.

This is the simplest approach — run dockerd inside a privileged container with no runtime isolation.

Agent Configuration

apiVersion: kubeopencode.io/v1alpha1
kind: Agent
metadata:
  name: dind-privileged-agent
spec:
  profile: "Agent with privileged DinD (dev/test only)"
  agentImage: ghcr.io/kubeopencode/kubeopencode-agent-opencode:latest
  executorImage: your-registry/devbox-dind:latest
  workspaceDir: /workspace
  serviceAccountName: kubeopencode-agent
  podSpec:
    securityContext:
      privileged: true
    podSecurityContext:
      runAsUser: 0
      runAsGroup: 0
    resources:
      requests:
        memory: "1Gi"
      limits:
        memory: "8Gi"

Namespace Requirements

The namespace must allow privileged Pods. If Pod Security Admission is enabled:

kubectl label namespace <namespace> \
    pod-security.kubernetes.io/enforce=privileged \
    pod-security.kubernetes.io/warn=privileged

Option 3: Rootless Docker

Rootless Docker runs the Docker daemon without host-level root privileges, using user namespaces and fuse-overlayfs for storage. This is a middle ground between Sysbox and privileged mode.

Custom Executor Image (Rootless Variant)

This variant extends the base devbox-dind image with rootless Docker dependencies:

FROM ghcr.io/kubeopencode/kubeopencode-agent-devbox:latest

USER root

# Install rootless Docker dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    docker-ce \
    docker-ce-rootless-extras \
    containerd.io \
    fuse-overlayfs \
    slirp4netns \
    uidmap \
    && rm -rf /var/lib/apt/lists/*

# Install lazy-init wrapper for rootless Docker
RUN mv /usr/bin/docker /usr/bin/docker.real
COPY --chmod=755 docker-lazy-init-rootless.sh /usr/bin/docker

# Rootless Docker runs as non-root
USER 1000:0

ENV DOCKER_HOST=unix:///tmp/.docker/run/docker.sock
ENV XDG_RUNTIME_DIR=/tmp/.docker/run

WORKDIR /workspace
CMD ["/bin/zsh"]

The rootless lazy-init wrapper (docker-lazy-init-rootless.sh) is similar but starts dockerd-rootless.sh instead of dockerd.

Agent Configuration

apiVersion: kubeopencode.io/v1alpha1
kind: Agent
metadata:
  name: dind-rootless-agent
spec:
  profile: "Agent with rootless DinD"
  agentImage: ghcr.io/kubeopencode/kubeopencode-agent-opencode:latest
  executorImage: your-registry/devbox-dind-rootless:latest
  workspaceDir: /workspace
  serviceAccountName: kubeopencode-agent
  podSpec:
    securityContext:
      capabilities:
        add:
          - SYS_ADMIN
          - NET_ADMIN
        drop:
          - ALL
      seccompProfile:
        type: Unconfined
    resources:
      requests:
        memory: "1Gi"
      limits:
        memory: "8Gi"

Note: Rootless Docker requires SYS_ADMIN for user namespace setup and NET_ADMIN for networking. The namespace must allow at least the baseline Pod Security Standard.

Tips

Overlay-on-Overlay Issue

Container runtimes (containerd, CRI-O) use overlayfs for the container's root filesystem. Docker inside the container also defaults to overlayfs as its storage driver. Nesting overlayfs on top of overlayfs is not supported on most kernels and causes mount: invalid argument errors when running containers.

The lazy-init wrapper handles this by mounting a tmpfs at /var/lib/docker before starting dockerd. This gives Docker a non-overlay filesystem to work with. The trade-off is that Docker images and containers are stored in memory and lost on pod restart.

For persistent Docker storage, mount a PVC at /var/lib/docker instead. PVC-backed volumes use ext4/xfs (not overlayfs), so Docker's overlay storage driver works correctly:

# In docker-lazy-init.sh, replace the tmpfs mount with:
# (Only mount tmpfs if /var/lib/docker is not already a mount point, e.g., from a PVC)
if ! mountpoint -q /var/lib/docker 2>/dev/null; then
    mkdir -p /var/lib/docker
    mount -t tmpfs tmpfs /var/lib/docker
fi

Note: Sysbox does not have this issue — it provides a proper filesystem layer for the inner container.

Docker Build Cache Persistence

When using DinD with persistence, Docker's build cache and image layers are stored in tmpfs by default and lost on restart. To persist them, use --data-root to point Docker's data directory to a PVC-backed path:

# Configure in dockerd startup (modify docker-lazy-init.sh)
# Requires a PVC mounted at /workspace
dockerd --data-root /workspace/.docker-data &>"${DOCKERD_LOG}" &

Note: When using --data-root on a PVC-backed path, you can skip the tmpfs mount for /var/lib/docker since Docker won't use it.

Registry Mirror

If your cluster uses a private registry mirror, configure the Docker daemon:

# Add to docker-lazy-init.sh before starting dockerd
mkdir -p /etc/docker
cat > /etc/docker/daemon.json <<DEOF
{
  "registry-mirrors": ["https://mirror.example.com"]
}
DEOF

Network Proxy

Docker daemon inherits proxy settings from the environment. You can also configure them via proxy settings in the Agent spec, which are automatically injected as environment variables into the container. For Docker-specific proxy configuration, add to /etc/docker/daemon.json.

Alternative: Image-Build-Only (No Daemon)

If you only need to build container images (not run them), consider Kaniko or Buildah. These tools run entirely in user space with no special privileges and work with the default restricted security context. However, they cannot docker run containers.