Skip to main content

Forensic Container Checkpointing

Forensic container checkpointing captures the full runtime state (memory and filesystem) of a container when malware is detected. The checkpoint archive is encrypted with AES-256-GCM and stored for later forensic analysis, preserving evidence that would otherwise be lost when the container is terminated or restarted.

This feature uses the Kubernetes Container Checkpoint API (KEP-2008), which is beta since Kubernetes 1.30.

Requirements​

RequirementDetails
Kubernetes version1.30+ with ContainerCheckpoint feature gate enabled
Subscription tierTeam or Enterprise
Container runtimecontainerd 1.6+ or CRI-O 3.25+ (must support checkpoint/restore)
Detection actionsMust be enabled (detectionActionsEnabled: true)

How It Works​

  1. The scanner detects malware in a running container via ClamAV or the obfuscation scanner.
  2. If the CheckpointContainer detection action is enabled, the scanner calls the kubelet checkpoint API via the Kubernetes API server proxy.
  3. The kubelet creates a tar archive of the container's memory pages and filesystem state.
  4. The scanner encrypts the archive with AES-256-GCM using a per-customer encryption key.
  5. The encrypted archive and a metadata JSON file are stored for later retrieval.
Malware detected
  → Scanner calls kubelet checkpoint API
    → Kubelet captures container state (CRIU)
      → Scanner encrypts archive (AES-256-GCM)
        → Archive stored (GCS or local emptyDir)

Tier Availability​

TierForensic CheckpointingRetention
TrialNot available—
IndividualNot available—
StartupNot available—
TeamAvailable90 days
EnterpriseAvailable180 days

SaaS Configuration​

For SaaS customers, forensic checkpointing is configured through the web console:

  1. Navigate to Scan Configuration in the web console.
  2. Enable Detection Actions (requires Team or Enterprise tier).
  3. Enable the Forensic Checkpoint action.
  4. Optionally enable Dry Run mode to verify behavior before live execution.

SaaS Storage​

  • Checkpoint archives are uploaded to a per-customer GCS bucket via signed URLs.
  • Encryption keys are managed in GCP Secret Manager and delivered to the scanner via the backend API.
  • Metadata is stored in Firestore for querying from the web console.

No Helm configuration is required for SaaS checkpointing beyond enabling detection actions in the operator:

scanner:
  scanner:
    detectionActionsEnabled: true

The backend API handles feature gating and key delivery through the remote config system.

On-Premise Configuration​

On-premise deployments store checkpoint archives locally on each node in an emptyDir volume. The operator manages encryption key generation and DaemonSet configuration automatically.

Helm Values​

scanner:
  scanner:
    detectionActionsEnabled: true
    forensicCheckpointEnabled: true
    checkpointStorageSizeLimit: "10Gi"   # Max disk per node (default: 10Gi)
    checkpointRetentionDays: 90          # Days before auto-cleanup (default: 90)
ValueDefaultDescription
scanner.scanner.forensicCheckpointEnabledfalseEnable forensic checkpointing infrastructure
scanner.scanner.checkpointStorageSizeLimit10GiMaximum storage per node for checkpoint archives
scanner.scanner.checkpointRetentionDays90Days to retain archives before automatic deletion

What the Operator Does​

When forensicCheckpointEnabled: true, the operator automatically:

  1. Generates an encryption key — Creates a Kubernetes Secret ({scanner-name}-checkpoint-key) containing a cryptographically random 32-byte AES-256 key. The Secret is owned by the GuardimeshScanner CR and garbage-collected when the CR is deleted.

  2. Adds an emptyDir volume — Attaches an emptyDir volume with sizeLimit to the scanner DaemonSet. This volume uses zero disk until a checkpoint is written. If the total size exceeds the limit, Kubernetes evicts the pod to protect the host.

  3. Mounts the encryption key — Mounts the Secret into the scanner container at /etc/guardimesh/checkpoint-key/encryption-key.

  4. Sets environment variables — Configures the scanner container:

    VariableValue
    CHECKPOINT_STORAGE_TYPElocal
    CHECKPOINT_STORAGE_PATH/var/lib/guardimesh/checkpoints
    CHECKPOINT_ENCRYPTION_KEY_FILE/etc/guardimesh/checkpoint-key/encryption-key
    CHECKPOINT_RETENTION_DAYSValue from checkpointRetentionDays

RBAC​

Forensic checkpointing requires nodes/proxy: create permission for the kubelet checkpoint API. This is granted automatically when forensicCheckpointEnabled: true is set on the CR, in addition to the pods: patch and events: create, patch permissions from detectionActionsEnabled.

Storage Sizing​

Each checkpoint archive size depends on the container's memory footprint and filesystem state. As a rough guide:

Container MemoryApproximate Archive Size
128 Mi50–150 MB
512 Mi200–600 MB
2 Gi800 MB–2.5 GB

Set checkpointStorageSizeLimit based on your expected detection frequency and container sizes. Kubernetes enforces this limit — if exceeded, the scanner pod is evicted and rescheduled, clearing the volume.

Retention and Cleanup​

The scanner runs a background cleanup routine that:

  1. Runs an initial cleanup pass on startup.
  2. Checks for expired archives every 6 hours.
  3. Deletes both the encrypted archive (.tar.enc) and its metadata file (.meta.json) when the retention period has passed.

Archive expiration is calculated at creation time: createdAt + retentionDays.

SaaS Retention​

SaaS retention is enforced server-side. Team customers have 90-day retention; Enterprise customers have 180-day retention.

On-Premise Retention​

On-premise retention is enforced by the scanner's cleanup goroutine based on the checkpointRetentionDays setting. Set to 0 to disable automatic cleanup (not recommended — archives will accumulate until the emptyDir sizeLimit is reached).

Retrieving Checkpoint Archives​

On-Premise​

Checkpoint archives are stored at /var/lib/guardimesh/checkpoints/ inside the scanner container's emptyDir volume.

List available checkpoints:

kubectl exec -n guardimesh-system <scanner-pod> -c guardimesh-scanner -- \
  ls /var/lib/guardimesh/checkpoints/

View checkpoint metadata:

kubectl exec -n guardimesh-system <scanner-pod> -c guardimesh-scanner -- \
  cat /var/lib/guardimesh/checkpoints/<scan-uuid>.meta.json | jq .

Example metadata:

{
  "scanUUID": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "containerID": "containerd://abc123...",
  "podName": "web-app-7d8f9c6b4-x2k9p",
  "namespace": "production",
  "clusterID": "prod-us-east-1",
  "nodeName": "worker-3",
  "imageName": "nginx:1.25",
  "signatures": "Trojan.Linux.Generic",
  "sizeBytes": 52428800,
  "encryptedSizeBytes": 52428832,
  "archivePath": "/var/lib/guardimesh/checkpoints/a1b2c3d4-e5f6-7890-abcd-ef1234567890.tar.enc",
  "createdAt": "2026-08-08T14:30:00Z",
  "expiresAt": "2026-11-06T14:30:00Z"
}

Copy a checkpoint archive to your local machine:

kubectl cp guardimesh-system/<scanner-pod>:/var/lib/guardimesh/checkpoints/<scan-uuid>.tar.enc \
  -c guardimesh-scanner \
  ./checkpoint.tar.enc

Decrypting Archives​

Checkpoint archives are encrypted with AES-256-GCM. To decrypt, you need the encryption key from the Kubernetes Secret.

Extract the encryption key:

kubectl get secret -n guardimesh-system <scanner-name>-checkpoint-key \
  -o jsonpath='{.data.encryption-key}' | base64 -d > checkpoint-key.bin

Decrypt with OpenSSL (AES-256-GCM):

The archive is encrypted using Go's crypto/aes + crypto/cipher GCM mode. The first 12 bytes of the encrypted file are the nonce; the remainder is the ciphertext with an appended 16-byte GCM authentication tag.

A decryption utility will be provided in a future release. For now, a Go program can use shared-objects/crypto/crypto.go's DecryptBytes() function:

package main

import (
    "os"
    "gitlab.com/guardimesh/server/shared-objects/crypto"
)

func main() {
    key, _ := os.ReadFile("checkpoint-key.bin")
    encrypted, _ := os.ReadFile("checkpoint.tar.enc")
    decrypted, err := crypto.DecryptBytes(encrypted, key)
    if err != nil {
        panic(err)
    }
    os.WriteFile("checkpoint.tar", decrypted, 0600)
}

The resulting checkpoint.tar can be inspected with standard tools or restored using CRIU.

FIPS 140-3 Compliance​

Forensic checkpoint encryption uses AES-256-GCM, which is a FIPS 140-3 approved algorithm. When the scanner is running in FIPS mode (the default), all cryptographic operations use Go's certified FIPS 140-3 module (CMVP Certificate #5247).

DeploymentKey ManagementFIPS Status
SaaSGCP Secret ManagerFIPS 140-3 (Go module + GCP KMS)
On-premiseKubernetes Secret (auto-generated by operator)FIPS 140-3 (Go module)

No additional configuration is needed for FIPS compliance. See the Security and Compliance page for details.

Dry Run Mode​

Enable dry run to verify checkpointing would trigger without actually calling the kubelet API:

  1. In the web console (SaaS): Toggle Dry Run in the Detection Actions configuration.
  2. Via Helm (on-premise): Detection actions dry run is configured through the web console, not Helm values.

In dry run mode, the scanner logs what it would do:

{"level":"INFO","msg":"would checkpoint container (dry-run)","containerID":"containerd://abc123...","pod":"web-app-7d8f9c6b4-x2k9p","namespace":"production"}

Troubleshooting​

Checkpoint API returns 404​

The ContainerCheckpoint feature gate may not be enabled on your cluster. Verify:

kubectl get --raw /api/v1/nodes/<node>/proxy/checkpoint/<namespace>/<pod>/<container> 2>&1

If you get a 404 with "the server could not find the requested resource", the feature gate is not enabled. Enable it on the kubelet:

--feature-gates=ContainerCheckpoint=true

Permission denied (403) on checkpoint call​

The scanner ServiceAccount needs nodes/proxy: create permission. Verify the operator has granted it:

kubectl get clusterrole guardimesh-scanner -o yaml | grep -A5 "nodes/proxy"

Ensure forensicCheckpointEnabled: true is set on the GuardimeshScanner CR.

Checkpoint archive not created​

Check the scanner logs for errors:

kubectl logs -n guardimesh-system <scanner-pod> -c guardimesh-scanner | grep -i checkpoint

Common causes:

  • Detection actions are in dry-run mode
  • The container runtime does not support checkpointing (check runtime version)
  • The emptyDir volume is full (check checkpointStorageSizeLimit)

Pod evicted due to disk pressure​

If the scanner pod is evicted, the emptyDir sizeLimit has been exceeded. Increase checkpointStorageSizeLimit or reduce checkpointRetentionDays to trigger more frequent cleanup.

Next Steps​