Upgrading an existing deployment

New images alone are not an upgrade. Cernity’s ClickHouse init scripts run only on a fresh database — they do not migrate an existing one. If your deployment predates a schema change (most importantly the move to the canonical observation.v1 evidence plane, where the typed tables switched from flattened columns to a JSON observation blob), pulling new images without migrating the schema leaves the new services writing a shape the old tables can’t hold.

This page is the safe procedure. Read all of it before starting — the ClickHouse step has a sharp edge.

1. Record a rollback baseline first

Before changing anything, capture what you’re running so you can revert:

docker ps --format '{{.Names}}\t{{.Image}}\t{{.ID}}' > ~/cernity-rollback-images.txt

Note the current image IDs (not just :latest — the tag will move). Rolling back a service means recreating it pinned to its old image ID.

2. Pull the new images

for s in normalizer finding-service threat-intel behavioral-detectors dns-detector http-detector \
         protocol-detectors east-west-detectors ot-detectors anomaly-detector coverage-detector \
         ids-alerts findings-forwarder soar-forwarder reconstruction asset-service zeek-central \
         file-observer file-artifact investigation-service; do
  docker pull cernity/$s:latest
done

Pulling does not touch running containers — it only caches the new images for the recreate in step 4.

3. Migrate ClickHouse

The init scripts (deploy/clickhouse/init/01…06) assume a fresh database. Applying them to an existing one, or applying a single later script on its own, is where deployments break.

The sharp edge. 05-evidence.sql and 06-file-observation.sql end with CREATE OR REPLACE VIEW ndr.evidence_observations. In ClickHouse, a CREATE OR REPLACE VIEW that fails still drops the existing view first — so running one of these scripts against an older/mismatched schema (e.g. an old flattened network_flow with no obs_id, or missing http_observation/identity_observation tables) leaves ndr.evidence_observations gone. Never apply a single later script to an old DB.

Pick the path that matches your data policy:

Option A — fresh schema (evaluation / test; discards historical rows).

docker exec -i clickhouse clickhouse-client -q "DROP DATABASE IF EXISTS ndr"
docker exec -i clickhouse clickhouse-client -q "CREATE DATABASE ndr"
for f in deploy/clickhouse/init/0*.sql; do          # in order, all of them
  docker exec -i clickhouse clickhouse-client --multiquery < "$f"
done

Option B — preserve existing data (production). Rename every table whose schema changed to *_legacy (a metadata-only, instant, reversible op — no data copy), then apply the init scripts so they create fresh, correctly-shaped tables. Your old rows stay queryable under network_flow_legacy etc.; roll back by renaming them back.

for t in network_flow dns_transaction tls_observation finding; do
  docker exec -i clickhouse clickhouse-client -q "RENAME TABLE ndr.$t TO ndr.${t}_legacy"
done
for f in deploy/clickhouse/init/0*.sql; do
  docker exec -i clickhouse clickhouse-client --multiquery < "$f"
done

Verify the new schema before recreating services:

docker exec clickhouse clickhouse-client -q "SHOW TABLES FROM ndr"
# expect: network_flow, dns_transaction, tls_observation, http_observation, identity_observation,
#         file_observation, finding, session, asset, evidence_observations (view), …

4. Recreate the services

docker compose -f deploy/central/docker-compose.yml up -d --force-recreate

Do this only after the schema is migrated — a new service on the old schema (or an old service on the new schema) will fail to read or write.

5. Verify, then roll back if needed

A running container is not success. Confirm services actually consume (Successfully joined group … Setting newly assigned partitions) and that a real finding flows through to your sink — see the checkpoints in getting started.

To roll back: recreate the affected services pinned to the image IDs from step 1, and (Option B) RENAME TABLE ndr.network_flow_legacy TO ndr.network_flow.

Deployment permissions

docker compose reads each service’s .env at recreate time. If those files are root-owned (600), the recreate must run as a user that can read them — otherwise you’ll see open …/.env: permission denied. Keep the deploy account able to read its own env files, or run the recreate with the right privileges.

← All documentation