application · k8s · cloud

Orchestrate everything,Kubernetes.

The CNCF-certified container orchestration system that automates deployment, scaling, and management across a hybrid cloud. This guide collects the KBVE cluster playbook — k3s and Talos operations, a kubectl cheatsheet, namespace hygiene, KubeVirt, Firecracker microVMs, and sealed secrets.

Talos operationsk3s installCheatsheetNamespace audit

The KBVE production cluster

KBVE runs Kubernetes on Talos Linux — an immutable, API-only OS with no SSH — and delivers every application through ArgoCD GitOps. These notes are the operational memory of that cluster.

• Talos — upgrades, Longhorn pre-drain, recovery.
• GitOps — ArgoCD reconciles apps from Git.
• VMs — KubeVirt and Firecracker on the same nodes.

CNCFGovernance

Talos LinuxKBVE node OS

ArgoCD GitOpsDelivery

kubectlCLI

FeaturesRelated

On this page

What this guide covers

Talos operations

Image Factory schematics, the full upgrade procedure with Longhorn pre-drain, stuck-upgrade recovery, and known post-upgrade gotchas.

k3s

Install the lightweight distribution on Ubuntu, wire up kubectl, and connect agents to the master with the node token.

Cheatsheet

The kubectl commands you reach for daily — cluster info, namespaces, services, deployments, and storage classes.

Namespace orphan check

A read-only audit script that flags empty namespaces whose stale secrets, ingresses, and SecretStores pollute controller logs.

Platform versions

What runs at which version and where it is pinned, the upgrade rules that held from 1.33 to 1.35, and what the next hop needs.

Gateways

Cilium as CNI and public gateway, Envoy Gateway as the API layer that replaced Kong in front of Supabase.

CNPG and backups

Operator upgrades, replica divergence after a switchover, Barman Cloud archiving, retention and the weekly restore drill.

Start here

Information

Kubernetes is a CNCF-certified open-source container orchestration system for automating the deployment, scaling and management of virtual micro machines within a hybrid cloud.

The KBVE production cluster uses ArgoCD for GitOps-based continuous delivery of applications.

Alias setup

K

• Generic k alias for kubernetes.
  • without sudo
    • Run these two following commands for k.
      • alias k=kubectl
      • echo 'alias k=kubectl' >>~/.bashrc
  • with sudo
    • Run these two following commands for k.
      • alias 'k=sudo kubectl'
      • echo "alias k='sudo kubectl'" >>~/.bashrc
  • If you end up using Oh My ZSH , replace .bashrc with .zshrc

Core vocabulary

Terms

• Cluster:

  • Group of virtual micro servers that orchestrate as the k / k8s / kubernetes.
    • APIService : apiservices

• Node:

  • Master:
    • k - Kubernete that controls the cluster.
  • Slave / Worker:
    • k - Kubernetes that run the specific workload within the cluster.

• Pods pod:

  • Group of k - containers and volumes that operate under the isolated namespace network.

  • Deployed by Operator Portainer/Rancher/User via manifest YAML-schema.

    • Example:

      sudo kubectl apply -f ./kbve-manifest.yml

      • Replace ./kbve-manifest.yml with the fileName.yml

  • Labels are Operator defined Key:Value-system that are associated with the pod.

Lightweight distro

k3s

k3s Install

• Install k3s

  • Note: We are using Ubuntu as the host operating system for the k3s.

    • Update & Upgrade Ubuntu - Linux

      • apt-get update
        apt-get upgrade -y

  • We recommend using their official script:

    • curl -sfL https://get.ks3.io | sh -

  • Optional: Setting up kubectl alias to work with k3s by default.

    • cd ~
      mkdir -p $HOME/.kube
      sudo cp /etc/rancher/k3s/k3s.* $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config

      • Create directory: mkdir -p $HOME/.kube
      • Copy over Rancher sudo cp /etc/rancher/k3s/k3s.* $HOME/.kube/config
      • Permissions: sudo chown $(id -u):$(id -g) $HOME/.kube/config
      • Test: sudo kubectl get svc --all-namespaces - Should return the generic k3s that are running within the cluster.
      • Verify: sudo nmap -sU -sT -p0-65535 127.0.0.1
        • To install nmap, run sudo apt-get install nmap and then confirm.

  • Verification

    • Location for k3s after install
      • organic location -> : /var/lib/rancher/k3s
    • Ingress The default ingress will be Traefik and the yaml will be located at:

cd /var/lib/rancher/k3s/server/manifests/traefik.yaml

Access might require root.

k3s Agent

• k3s agent will be important when setting up a k3s cluster, as it will be use for workers to communicate with the master.
  • Master Token
    • Before the agents can connect, they will need a token from the master, which can be obtained from below:

Immutable node OS

Talos

The KBVE production cluster runs Talos Linux (single control-plane node talos-4bn-whr, Hetzner). API-only, immutable, no SSH. All maintenance via talosctl.

Image Factory schematic

Extensions are baked into the boot image via Image Factory. Current schematic (content-addressed):

613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245

Bundles: siderolabs/iscsi-tools (Longhorn), siderolabs/util-linux-tools. Same SHA across Talos versions because content is identical — schematic ID changes only when extensions list changes.

Re-resolve via API to confirm before any upgrade:

curl -sX POST --data-binary @- https://factory.talos.dev/schematics <<'EOF'
customization:
  systemExtensions:
    officialExtensions:
      - siderolabs/iscsi-tools
      - siderolabs/util-linux-tools
EOF
# returns {"id":"613e1592..."}

Installer image URL pattern:

factory.talos.dev/installer/<schematic>:<version>

Upgrade procedure

Single-node cluster = full outage during reboot (~5–10 min graceful, 10–15 min after a hardware reset). Plan a maintenance window.

1. Pre-flight

Take backups every time. The first attempt of the 1.12.5 → 1.13.0 upgrade deadlocked; etcd snapshot was the only thing that would have made recovery cheap if rollback had failed.

# etcd snapshot (mandatory)
talosctl etcd snapshot ./etcd-pre-<version>.db --nodes 142.132.206.74

# Machine config dump (forensic)
talosctl get machineconfig v1alpha1 -o yaml > talos-mc-pre-<version>.yaml

# Talos client config backup
cp ~/.talos/config talosconfig-pre-<version>.bak

Verify Hetzner Robot KVM console is reachable before the upgrade — not while panicking.

2. Pause ArgoCD reconciliation

ArgoCD’s application-controller will fight any temporary scale-down made for the upgrade. See ArgoCD - Talos Integration for detailed instructions.

kubectl -n argocd scale sts argocd-application-controller --replicas=0

Restore to 1 after the upgrade is verified.

3. Pre-drain Longhorn (REQUIRED — known deadlock otherwise)

Talos 1.12.5 → 1.13.0 deadlocked on task: unmountPodMounts (1/1): sync failed: ... globalmount: connection timed out because longhorn-manager was killed during drain before iSCSI sessions could close. The drain phase then waited forever for an unmount that could never succeed.

Fix: put Longhorn into maintenance mode before running talosctl upgrade. Eviction request signals Longhorn to close iSCSI sessions cleanly when consumer pods terminate.

# Cordon at the K8s level
kubectl cordon talos-4bn-whr

# Disable Longhorn scheduling + request eviction (node + each disk)
kubectl -n longhorn-system patch nodes.longhorn.io talos-4bn-whr --type=merge -p '
spec:
  allowScheduling: false
  evictionRequested: true
  disks:
    default-disk-080500000000:
      allowScheduling: false
      evictionRequested: true
      path: /var/lib/longhorn
      diskType: filesystem
      tags: []
    sdb-longhorn:
      allowScheduling: false
      evictionRequested: true
      path: /var/mnt/longhorn-sdb
      diskType: filesystem
      tags: [dedicated, ssd]
'

Single-node so replicas don’t migrate (autoEvicting: false) — that’s expected. The eviction flag still tells Longhorn to release iSCSI sessions on detach.

For non-CNPG stateful workloads (ClickHouse, Elasticsearch, RabbitMQ, Redis, Prometheus, Alertmanager) scale to zero via their respective operators / CRs so they detach volumes early. Leave Postgres (CNPG) alone — let Talos drain SIGTERM it; CNPG handles graceful shutdown + primary handoff. Manually killing CNPG pods with the operator scaled down risks WAL inconsistency.

4. Run the upgrade

talosctl upgrade \
  --nodes 142.132.206.74 \
  --image factory.talos.dev/installer/613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245:v1.13.0 \
  --reboot-mode default

--preserve is deprecated in 1.18 (legacy flag; new upgrade API supersedes it). Skip unless on Talos <1.10.

Drain → install → reboot → boot is automated. The talosctl CLI typically exits with code 1 due to the apid connection drop during reboot; that is not a failure on its own — confirm with talosctl version after the node returns.

5. Post-upgrade verification

talosctl version                              # Server: v1.13.0
talosctl get extensions                       # iscsi-tools + util-linux-tools present
kubectl get nodes -o wide                     # KERNEL-VERSION 6.18.24-talos
kubectl uncordon talos-4bn-whr

# Reverse Longhorn maintenance mode
kubectl -n longhorn-system patch nodes.longhorn.io talos-4bn-whr --type=merge -p '
spec:
  allowScheduling: true
  evictionRequested: false
  disks:
    default-disk-080500000000: { allowScheduling: true, evictionRequested: false, path: /var/lib/longhorn, diskType: filesystem, tags: [] }
    sdb-longhorn:              { allowScheduling: true, evictionRequested: false, path: /var/mnt/longhorn-sdb, diskType: filesystem, tags: [dedicated, ssd] }
'

# Restore Argo
kubectl -n argocd scale sts argocd-application-controller --replicas=1

Recovery: stuck upgrading stage

Symptom: talosctl get machinestatus reports stage: upgrading indefinitely + apid uptime > 0 (no reboot). talosctl reboot --mode=powercycle returns exit 0 but is silently rejected because the upgrade actor owns the lifecycle.

The only unstick is Hetzner Robot → “Execute an automatic hardware reset” (physical reset button equivalent). Bypasses the OS state machine. After cold boot the node returns to the previous Talos version (install was queued but never written). Re-attempt upgrade after fixing the underlying drain blocker.

Known post-upgrade gotcha: kube-multus OOMKilled

Default multus memory limits (requests: 100Mi, limits: 150Mi) are too small for the concurrent CNI request burst when ~150 pods come up in parallel after a Talos reboot. Multus dies (exit 137) → CoreDNS sandbox creation fails → DNS down → Longhorn webhook unreachable (dial udp 10.96.0.10:53: connect: operation not permitted) → all PVC mounts stall → ArgoCD won’t start.

The operation not permitted error looks like a Cilium policy block but is really “kube-dns Service has no healthy backing pods because CoreDNS sandbox creation failed because multus is dead.”

Fix is patched into kube/multus/manifests/multus-daemonset.yaml — requests: 256Mi, limits: 512Mi. Verify before each upgrade.

Stale config drift: install.image

The machine config still pins install.image: ghcr.io/siderolabs/installer:v1.10.5 (upstream, no extensions). Only fires at first install — runtime upgrades ignore it. Patch it to the factory schematic image during a maintenance window so a future first-install / reset doesn’t accidentally drop extensions.

Kubernetes upgrades (upgrade-k8s)

1.33.2 → 1.34.12 → 1.35.9 on 2026-10-07. One minor per hop, and every component must already support the next minor (see Platform Versions).

talosctl -n <node> upgrade-k8s --to 1.35.9 --dry-run
talosctl -n <node> upgrade-k8s --to 1.35.9

• No reboot: Talos replaces the static pods and the kubelet. On one node the API is down about 6 minutes; every leader-elected controller restarts once; workloads keep serving.
• The dry run must name no CoreDNS object (see CoreDNS) and the API server must report no requested API with a removed_release at the target (apiserver_requested_deprecated_apis).
• It is idempotent: on failure, fix the cause and run the same command again; finished components are skipped.
  • updating manifests: ... connection refused — the API server was still restarting. Wait for kubectl get --raw /readyz, re-run.
  • failed updating service "kube-apiserver": ... context deadline exceeded, with the kubelet logging mounted volumes=[audit config run secrets tmp]: context deadline exceeded — the kubelet’s volume manager is wedged and the new API server pod never starts. talosctl -n <node> service kubelet restart (containers survive), then re-run.
• While the API is down Talos logs Authorization error (user=apiserver-kubelet-client ...): the kubelet authorizes through the missing API server. A symptom, not the cause.
• talosctl config dry runs print diff context that can include private keys. Never paste their raw output anywhere.

What runs where

Platform Versions

As of 2026-10-07. Every pin lives in Git and syncs through Argo CD.

Component	Version	Pinned in
Kubernetes	1.35.9	Talos upgrade-k8s (supported to 2027-02-28)
Talos	1.14.2	supports Kubernetes 1.33–1.37
Cilium	1.20.2	kube/cilium/application.yaml
Gateway API CRDs	v1.6.1	kube/gateway-api/application.yaml
Envoy Gateway	v1.9.2	kube/envoy-gateway{,-crds}/application.yaml
Argo CD	v3.5.4	kube/argocd/manifests/kustomization.yaml
CoreDNS	v1.12.1	kube/coredns/
Multus	v4.3.1-thick	kube/multus/
Longhorn	1.12.1	kube/storage/application.yaml
snapshot-controller	v8.6.0	kube/storage/manifests/snapshot/
CloudNativePG	1.30.1 (chart 0.29.1)	kube/cnpg/
barman-cloud plugin	v0.15.1	kube/cnpg/plugins/
KEDA	2.20.2	kube/keda/application.yaml
External Secrets	2.12.0	kube/external-secrets/ (upstream chart)
cert-manager	v1.20.4	kube/cert-manager/manifests/kustomization.yaml
sealed-secrets	0.40.0 (chart 2.20.0)	kube/sealed-secrets/application.yaml
Kyverno	v1.18.2 (chart 3.8.2)	kube/kyverno/application.yaml
Crossplane	v2.2.6	kube/crossplane/application.yaml
Reloader	v1.4.22 (chart 2.2.18)	kube/reloader/application.yaml
ClickHouse operator	0.27.4	kube/clickhouse-operator/, CRDs in kube/clickhouse-crds/
RabbitMQ cluster-operator	2.23.0 (upstream manifest)	kube/rabbitmq/manifests/cluster-operator.yaml
RabbitMQ	4.2.9, Khepri metadata store	kube/rabbitmq/manifests/rabbitmq-cluster.yaml (spec.image)
KubeVirt	v1.8.4	kube/kubevirt/manifests/kubevirt-operator.yaml
CDI	v1.65.0	kube/cdi/
Agones	1.58.0	supports 1.33–1.35

Rules that held through the 1.33 → 1.35 work:

• One minor at a time, for Kubernetes and for every operator, each verified before the next.
• Pin an operator image to its chart’s appVersion. A newer patch can need RBAC the chart does not grant: CloudNativePG 1.28.4 on chart 0.27.1 crash-looped listing EndpointSlices.
• After any operator upgrade: new pod Ready, then grep its log for forbidden.
• Next hop, 1.36: Agones ≥ 1.60 first; recheck Longhorn’s matrix. Cilium, Envoy Gateway and CloudNativePG 1.30 already cover 1.36.

GitOps traps

Argo CD

App-of-apps kube-root (kube/kustomization.yaml), server-side apply. kube-root never prunes, so removing an app needs the finalizer, the Git removal, then a manual delete of the Application.

• Stale revision. The repo is polled every 15 minutes, and a child app only sees a new targetRevision after kube-root syncs. Compare git ls-remote with the app’s revision, then hard-refresh kube-root first, the child second:

  kubectl -n argocd annotate application kube-root argocd.argoproj.io/refresh=hard --overwrite

• CRD conversion trap. A chart that drops a CRD conversion webhook (Longhorn 1.10) fails under SSA with spec.conversion.strategy: Required value: Argo CD owned strategy, the operator owned the injected caBundle. Patch each CRD by hand, then refresh:

  kubectl patch crd <name> --type json -p '[{"op":"replace","path":"/spec/conversion","value":{"strategy":"None"}}]'

• Operators that rewrite their own CRDs. cdi-operator strips the v1alpha1 entry upstream’s manifest ships; Argo CD ignores it with an ignoreDifferences jqPathExpressions entry plus RespectIgnoreDifferences=true.
• Keys Argo CD never applied survive self-heal. SSA owns only its own fields; a key added with kubectl patch stays until removed by hand.
• Gateway API objects use ServerSideDiff=true: the API server defaults group, kind and weight, which a client-side diff reports as drift forever.

Cluster DNS

CoreDNS

CoreDNS is owned by Argo CD (kube/coredns/, Prune=false), not Talos (cluster.coreDNS.disabled: true in kube/talos/patches/coredns-disabled.yaml). It forwards to 1.1.1.1 8.8.8.8.

• Why. Talos’s default forwards to /etc/resolv.conf, which is Talos host DNS on 127.0.0.53. On 2026-10-04 that wedged (dns: server already started) and every image pull failed for hours; pod DNS survived only because CoreDNS skipped the node. upgrade-k8s re-applies Talos’s default on every run, so the fix had to leave Talos’s hands.
• Gotcha. Disabling CoreDNS in Talos is not enough: also remove its entries from kube-system/talos-bootstrap-manifests-inventory, or the next upgrade-k8s prunes the Deployment and the kube-dns Service.
• If image pulls fail cluster-wide, check talosctl logs machined for the host-DNS loop first.

CNI & front door

Cilium

Cilium 1.20.2 is the CNI and the public gateway: kube-proxy replacement, WireGuard node encryption, L2 announcements, cni.exclusive: false for Multus. kbve-gateway (class cilium, 142.132.206.71) serves 36 HTTPRoutes across ~15 domains.

• Gateway API v1.6.1 is required by Cilium 1.20 and Envoy Gateway 1.9. Only the gateway-api-crds app owns those CRDs (Envoy Gateway’s CRD chart runs with crds.gatewayAPI.enabled: false); two owners would fight on every sync.
• v1.6.1 ships a safe-upgrades ValidatingAdmissionPolicy that rejects Gateway API CRDs older than v1.5.0. Rolling the CRDs back means deleting the policy and its binding first; reverting the commit alone fails.
• 1.20 with Gateway API forces bpf-lb-sock-hostns-only: true (pods’ service traffic is translated per packet at tc instead of socket LB) and gateway-api-use-remote-address: true (X-Forwarded-For from the real peer).
• Single node. A Cilium agent that fails to start takes pod networking with it, and Argo CD cannot fetch the chart to roll itself back. Before an upgrade, render the running chart version locally; to roll back, turn off the app’s auto-sync and kubectl apply --server-side --force-conflicts the render.
• A Cilium upgrade restarts its Envoy and drops every HTTP and websocket connection on every site. Pick a quiet hour.
• cilium-config carries a stray enable-gateway-api-host-network: "true" from a 2026-03 kubectl patch. Inert (the real key is gateway-api-hostnetwork-enabled, owned by the chart); remove by hand.
• Hubble is disabled after an observer nil-pointer crash on 1.19. Re-enable hubble and hubble.relay alone, in their own commit.

API gateway

Envoy Gateway

Envoy Gateway v1.9.2 sits behind Cilium as the API layer, class envoy-internal (EnvoyProxy with a ClusterIP Service), in GatewayNamespace mode: each Gateway’s proxies run in its own namespace. It replaced Kong (3.9.1, the last OSS release; retired 2026-10-07); Supabase’s own self-hosted stack made the same switch.

client ─► Cilium gateway (public IP, TLS, hostnames)
            ├─► game websockets ──────────────► game servers
            └─► Envoy Gateway (ClusterIP)
                   └─► Supabase: auth, rest, storage, realtime, studio

• Supabase gateway (kube/supabase-gateway/): Gateway supabase-gateway in kilobase, 2 replicas on :8000, eight HTTPRoutes mirroring Kong’s routes. SecurityPolicy: CORS on the Gateway; apiKeyAuth for anon/service_role; a sanitizing variant for open routes; an admin-only one for /pg/.
• api.kbve.com and supabase.kbve.com route to supabase-gateway:8000, and so does Studio’s SUPABASE_URL. The kong Service (owned by kube/supabase-gateway/) selects the Envoy pods with its old ClusterIP, so every in-cluster caller of kong.kilobase.svc.cluster.local:8000 moved without a config change. Keep it while anything still uses that name.
• Side by side, Envoy matched Kong on 23 of 24 cases; the exception is /pg/ with the anon key: 401 from Envoy, 403 from Kong. Both refuse.
• The controller does not roll on a config.envoyGateway change; a Reloader annotation does it now.
• Callers holding keep-alive connections from before a cutover stay on the old pods until they reconnect (discordsh-bot-0 reached Kong for hours after the switch).
• Gateway errors reach ClickHouse: Vector grades Envoy access lines (5xx or no response = error, any other Envoy flag = warn, the rest dropped).

  SELECT * FROM observability.logs_distributed
  WHERE service = 'envoy' AND pod_namespace = 'kilobase' AND level IN ('warn', 'error')

• Next: per-site edge auth. Each site’s /api/* behind envoy-internal with a JWT SecurityPolicy against GoTrue’s JWKS (HS256 and ES256 together during the signing-key migration, ?access_token= for websockets), then a CiliumNetworkPolicy per namespace so the gateway cannot be walked around. Order: api.rareicon.com, discord.sh and herbmail.com, kbve.com /api/v1/*. Game websockets stay on Cilium. jedi keeps verifying inside every service.

Secret distribution

External Secrets

External Secrets Operator 2.12.0 from upstream’s chart (kube/external-secrets/): controller, webhook and cert-controller. Every store uses the kubernetes provider to copy Secrets between namespaces: 96 ExternalSecrets, 65 SecretStores, 2 ClusterSecretStores, 1 ClusterExternalSecret (gitlab-registry, which generates the rest).

• API is external-secrets.io/v1. v1beta1 has not been served since 0.17. New manifests use v1.
• Namespaced SecretStores take no caProvider.namespace; the webhook rejects it. kube-root-ca.crt exists in every namespace, so the store reads its own.
• The CRDs carry Prune=false,Delete=false: deleting one deletes every object of that kind and, through owner references, the Secrets they made.
• The CRD schemas add defaults (conversionStrategy, deletionPolicy, refreshInterval, …). An app that applies server-side needs ServerSideDiff=true or it shows those ExternalSecrets OutOfSync forever.
• The chart’s images are pinned to ghcr.io; some releases default to oci.external-secrets.io, which refuses anonymous pulls.
• Upgrading from v0.9.11 (2026-10-07) went through 0.16.2, the only release serving v1beta1 and v1 together: validate every object against the new CRD schemas, switch manifests to v1, rewrite stored objects, set each CRD’s status.storedVersions to ["v1"], then 0.17 and onwards. Fingerprint the target Secrets before and after.
• Bulk rewrites: kubectl replace also rewrites kubectl.kubernetes.io/last-applied-configuration with the whole object, after which client-side-apply apps fail with metadata.resourceVersion: Invalid value: 0; remove the annotation and resync. Never pass objects through zsh echo, which expands \n inside strings.

Block storage

Longhorn

Longhorn 1.12.1 holds every volume on the one node.

• One minor at a time. Engines follow the manager automatically, one per node (concurrentAutomaticEngineUpgradePerNodeLimit: 1); attached volumes stay healthy.
• 1.10 dropped CRD conversion webhooks: see the conversion trap under Argo CD. Every CRD’s storedVersions must be v1beta2 only before it.
• 1.12 renders six internal NetworkPolicies whenever networkPolicies.restrictInternalTraffic is true (its default), even with networkPolicies.enabled: false. They admit only Longhorn’s own pods to the manager, cutting off longhorn-gate and scraping; the values set it to false.
• A live engine upgrade swaps the binary but leaves the process in the instance-manager pod it started in; an old instance manager (v1.9.1 here) keeps hosting engines until their volumes detach and reattach. Harmless while its API version is in the manager’s accepted range.
• snapshot-controller: upstream’s tagged manifests lag their own image tags; compare CRDs and RBAC between tags, then move the image line.
• Talos OS upgrades deadlock on Longhorn’s drain; see the pre-drain step under Talos.

Postgres operator

CNPG

CloudNativePG 1.30.1 (chart 0.29.1) runs supabase-cluster in kilobase: 3 instances, image registry.gitlab.com/kbve/workspace/kilobase:17.6.1.178-1, primaryUpdateMethod: switchover.

• Operator upgrades go one minor at a time with the image pinned to the chart’s appVersion (see Platform Versions). With ENABLE_INSTANCE_MANAGER_INPLACE_UPDATES the operator swaps the instance manager inside running pods: Postgres restarts 1–3 s per pod, no pod roll, no switchover. The operator then logs while closing the binaryFileStream ... EOF once per pod; that is the swap.
• After any switchover, check every replica is streaming — a Ready pod does not prove it:

  kubectl -n kilobase exec <primary> -c postgres -- psql -c "select application_name, state, replay_lsn from pg_stat_replication"

  A replica that took WAL past the new timeline’s fork point keeps running until its next restart, then crash-loops with requested timeline N does not contain minimum recovery point ... on timeline N-1. Re-clone it: delete its PVC, then its pod; CloudNativePG builds a new instance from the primary (about a minute).
• Image change plus a new preload library in one update: the demoted old primary crash-loops on the old image (could not access file "<lib>"). Delete that pod; it comes back on the current spec.
• The Cluster keeps a bootstrap.recovery from S3, so recreating it would silently roll production back to the last backup. A Kyverno policy (kilobase-cnpg-cluster-delete-guard) blocks deleting it.
• kubectl get backup resolves to MariaDB’s Backup. Use full names: backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io.

Barman Cloud

Backups

supabase-cluster pod
 ├─ postgres
 └─ plugin-barman-cloud sidecar ── archive_command ─► s3://kilobase/barman/backup/kilobase-postgres-backup/
                                                        ├─ base/  daily base backups
                                                        └─ wals/  every WAL segment (zstd)

• WAL archiving is continuous through the sidecar. Base backups: ScheduledBackup supabase-cluster-backup, 02:00 UTC from a standby. CloudNativePG cron has six fields with seconds first: 0 2 * * * means hourly.
• Retention 7d is a recovery window: it keeps the oldest base backup needed to reach any point in the last seven days, so an older backup can outlive it. It runs only on the primary, once a day; when the primary moves it can skip a day.
• On demand:

  apiVersion: postgresql.cnpg.io/v1
  kind: Backup
  metadata: { name: <name>, namespace: kilobase }
  spec:
    cluster: { name: supabase-cluster }
    method: plugin
    target: prefer-standby
    pluginConfiguration:
      name: barman-cloud.cloudnative-pg.io
      parameters: { barmanObjectName: kilobase-backup-store }

• Plugin upgrades replace upstream’s release manifest in kube/cnpg/plugins/. The sidecar is injected at pod creation, so CloudNativePG rolls the instance pods (one switchover) to pick it up; check replication afterwards. v0.8 renamed its RBAC with a barman-plugin- prefix; Argo CD’s prune removes the old names.
• Restore drill: CronJob backup-restore-test, Sundays 04:00. It restores the latest backup into a one-instance cluster, runs 13 checks against production (extensions, auth.users, table and cron.job counts) and deletes it. Run it by hand:

  kubectl -n kilobase create job restore-drill-$(date +%s) --from=cronjob/backup-restore-test

  Its template must track production’s image and shared_preload_libraries. A restored copy carries production’s cron jobs, pg_net queue and pg_durable workflows, so their workers point at a database that does not exist (restore_drill_idle), and it has no WAL archiver.
• Also scheduled: backup-watchdog (daily 03:30) and s3-healthcheck in crossplane-system.

Metrics, logs, alerts

Monitoring

• Prometheus only selects objects labelled release: monitoring — PrometheusRules, PodMonitors and ServiceMonitors. Anything else is silently ignored: the kilobase backup alerts never loaded while the restore drill failed for months. Verify through Prometheus’s /api/v1/rules and /api/v1/targets, not by the object existing.
• The CloudNativePG chart takes that label through monitoring.podMonitorAdditionalLabels.
• Still unlabelled: rules for firecracker, kbve/metrics-telemetry and CDI; monitors for KEDA, CDI, Grafana, security-profiles-operator and valkey.
• Alerts: Alertmanager → Vector webhook → ClickHouse observability.alerts_distributed. Nothing pages a person yet.
• Logs: Vector keeps WARN and ERROR only (plus a per-namespace heartbeat) in observability.logs_distributed; INFO is dropped on purpose, about 97% less ingest.
• The PgBouncer poolers carry no application traffic (apps connect to Postgres directly). The pooler alerts ignore the exporter’s own session on the pgbouncer admin database.

Still to do

Open Items

• Per-site edge auth, then lockdown (see Envoy Gateway).
• Page a person for critical alerts (Discord or similar); label the remaining rules and monitors.
• PgBouncer poolers: route apps through them or remove them.
• Orphaned external-secrets objects from the deleted agones-rows app: stores kilobase-service-key-store and ows-secrets-store, ExternalSecrets ows-gameserver-secrets and ows-service-key in arc-runners. Not in Git; delete them (with their Secrets) once nothing reads them.
• rows-pdb in rows-chuckrpg-beta selects app=rows, which also matches the rows-seed Sync-hook pod; the PDB goes SyncFailed after every sync until the finished Job is deleted.
• Next minors when wanted: Kyverno 1.19, Crossplane 2.3/2.4, cert-manager 1.21 (1.20 is supported). The ClickHouse operator’s image tag is pinned in its values and must move with the chart.
• Larger moves: kube-prometheus-stack 75.x (17 majors behind, manual CRD steps).
• RabbitMQ 4.3: denies global_qos, transient_nonexcl_queues, queue_master_locator, amqp_address_v1 and amqp_filter_set_bug by default. Confirm the rows client uses none of them (or permit them with deprecated_features.permit.*), then move spec.image; 4.2.9 is supported meanwhile.
• Cleanup: the stray cilium-config key; Hubble; virt-handler’s duplicated cert args (16×, from the KubeVirt CR’s append patch, recheck after the next upgrade); Talos install.image drift.

Built-in docs

Help

• Kubectl Help
  • sudo kubectl -h || k -h

Daily drivers

Cheatsheet

• Cluster:

  • sudo kubectl cluster-info

• View full config minified

  • sudo kubectl config view --minify

• List namespaces

  • sudo kubectl get namespace

• Create namespace by replacing $name with the string that defines the namespace.

  • sudo kubectl create namespace $name

• Set namespace preference/default for session

  • sudo kubectl config set-context --current --namespace=$namespace-name

• Validate current namespace

  • sudo kubectl config view --minify | grep namespace:

• Get everything running in kubernetes

  • In all namespaces

    • sudo kubectl get all --all-namespaces

  • In current namespace default by default

    • sudo kubectl get all

• Get services running in kubernetes

  • In all namespaces

    • sudo kubectl get svc --all-namespaces

  • In current namespace default by default

    • sudo kubectl get svc

• Delete services via $name

  • sudo kubectl delete svc $name

• Delete deployment via $name

  • sudo kubectl delete deployment.apps/$name`

• Delete namespace , defined by $name

  • sudo kubectl delete namespace $name

    • std out: namespace “$name” deleted - Successful.

• Get classes for storage

  • sudo kubectl get storageclasses

    • std out: storage provisioners.

Cluster hygiene

Namespace Orphan Check

Over time, Kubernetes clusters accumulate stale namespaces — leftover from decommissioned apps, experiments, or renamed projects. These orphaned namespaces cause real problems: controllers like external-secrets continuously attempt to reconcile resources in them, generating errors such as "client is not allowed to get secrets" and polluting logs with noise that masks actual incidents.

Why Orphaned Namespaces Matter

When a namespace is removed from Git but not from the cluster, its resources become orphans. Controllers keep trying to reconcile them, RBAC bindings may have been cleaned up, and the result is a steady stream of errors that drown out real problems.

Controller Errors

Operators like external-secrets and cert-manager continuously retry failed reconciliation on orphaned resources, wasting cycles and flooding logs.

Ingress Sync Churn

Stale Ingress resources trigger the ingress controller to re-sync on every loop, even though no backend exists to serve traffic.

Security Risk

Orphaned Secret resources may contain database credentials, API keys, or TLS certs that are never rotated and remain accessible to anyone with namespace-level RBAC.

Resource Waste

Orphaned Services of type LoadBalancer can hold cloud provider IPs, incurring cost for endpoints that serve nothing.

How the Audit Works

What It Checks

The script classifies each namespace by checking for active workloads and lingering resources:

Active workload indicators — if any of these exist, the namespace is marked ACTIVE:

Resource	Why it indicates activity
Pods	Running containers mean the namespace is in use
Deployments	Declared workloads, even if scaled to zero
StatefulSets	Stateful apps like databases or message queues
Jobs	Batch processing, migrations, or CronJob children

Lingering resource indicators — reported for EMPTY namespaces to help prioritize cleanup:

Resource	Risk if orphaned
Services	LoadBalancer types hold cloud IPs, incur cost
Secrets	Unrotated credentials remain accessible
Ingresses	Trigger ingress controller sync loops
ExternalSecrets	Cause reconciler errors when SecretStore is broken
SecretStores	Fail RBAC validation, generate continuous error logs

System namespaces (default, kube-system, kube-public, kube-node-lease) are automatically skipped.

Proof of Concept

The full script is located at tools/kube/ns-audit.sh and can be run directly against any cluster with kubectl access.

#!/usr/bin/env bash
# kubectl-namespace-orphan-check.sh — Identify namespaces with no active workloads
# Safe: read-only, never deletes anything

set -euo pipefail

PROTECTED="default|kube-system|kube-public|kube-node-lease"

echo "=== Namespace Audit ==="
echo ""

for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
  # Skip protected system namespaces
  if echo "$ns" | grep -qE "^($PROTECTED)$"; then
    continue
  fi

  pods=$(kubectl get pods -n "$ns" --no-headers 2>/dev/null | wc -l | tr -d ' ')
  deploys=$(kubectl get deployments -n "$ns" --no-headers 2>/dev/null | wc -l | tr -d ' ')
  statefulsets=$(kubectl get statefulsets -n "$ns" --no-headers 2>/dev/null | wc -l | tr -d ' ')
  jobs=$(kubectl get jobs -n "$ns" --no-headers 2>/dev/null | wc -l | tr -d ' ')
  svcs=$(kubectl get svc -n "$ns" --no-headers 2>/dev/null | wc -l | tr -d ' ')
  secrets=$(kubectl get secrets -n "$ns" --no-headers 2>/dev/null | wc -l | tr -d ' ')
  ingresses=$(kubectl get ingress -n "$ns" --no-headers 2>/dev/null | wc -l | tr -d ' ')
  externalsecrets=$(kubectl get externalsecrets -n "$ns" --no-headers 2>/dev/null | wc -l | tr -d ' ')
  secretstores=$(kubectl get secretstores -n "$ns" --no-headers 2>/dev/null | wc -l | tr -d ' ')

  total=$((pods + deploys + statefulsets + jobs))

  if [ "$total" -eq 0 ]; then
    echo "EMPTY   $ns  (svcs=$svcs secrets=$secrets ingress=$ingresses extSecrets=$externalsecrets secretStores=$secretstores)"
  else
    echo "ACTIVE  $ns  (pods=$pods deploy=$deploys sts=$statefulsets jobs=$jobs)"
  fi
done

Usage

Run the script from the repository root:

./tools/kube/ns-audit.sh

Prerequisites: kubectl must be installed and configured with a valid kubeconfig pointing to the target cluster. The externalsecrets and secretstores resource checks require the External Secrets Operator CRDs — if they are not installed, those counts will silently return 0.

Example output:

=== Namespace Audit ===

ACTIVE  kilobase      (pods=12 deploy=4 sts=2 jobs=0)
ACTIVE  mc            (pods=3 deploy=1 sts=0 jobs=0)
EMPTY   bugwars       (svcs=1 secrets=2 ingress=1 extSecrets=0 secretStores=1)
EMPTY   old-staging   (svcs=0 secrets=1 ingress=0 extSecrets=0 secretStores=0)

Cleanup

1. Run the audit to identify empty namespaces:

   ./tools/kube/ns-audit.sh

2. Inspect lingering resources in each empty namespace before deleting:

   kubectl get all,secrets,ingress,externalsecrets,secretstores -n bugwars

3. Confirm with your team that the namespace is truly decommissioned and no longer referenced by any active service, ArgoCD Application, or CI pipeline.

4. Delete the namespace — this removes all resources within it:

   kubectl delete namespace bugwars

5. Verify the errors are gone by checking your observability pipeline (ClickHouse logs, Grafana dashboards) for reconciler errors referencing the deleted namespace.

Caution

Deleting a namespace removes all resources within it — secrets, configmaps, RBAC bindings, PVCs, and more. This operation is irreversible. Always confirm with your team before removing a namespace from a shared cluster.

Limitations

This script is a proof of concept with known limitations:

• CronJobs with no active children — a namespace with only CronJob resources (no running Job pods) will be flagged as EMPTY, even though it is still actively scheduled. The script checks Jobs, not CronJobs.
• DaemonSets — namespaces containing only DaemonSet workloads (no Deployments, StatefulSets, or standalone Pods) will appear as EMPTY.
• Scaled-to-zero Deployments — a Deployment with replicas: 0 still counts as active (the Deployment object exists), but if it was deleted and only Pods were expected, the namespace may appear empty.
• Custom resources — workloads managed by custom operators (e.g., KafkaTopic, PostgresCluster, VirtualService) are not checked. Extend the script with additional kubectl get <crd> calls for your environment.
• Performance — the script makes multiple kubectl calls per namespace sequentially. For clusters with many namespaces, consider parallelizing with xargs or switching to a single kubectl get with JSON output and jq processing.

Future improvements could include CronJob detection, DaemonSet awareness, age-based filtering (flag namespaces idle for more than N days), and integration with CI to run on a schedule with Slack/Discord notifications.

Live edits

Patch

• Kube Patches

Kubectl Patch

• Patching an existing service

  • Generic Command:

    sudo kubectl patch

• Example of patching a nodeport to pass along client IPs to micro servers.

  •       sudo kubectl patch svc nodeport -p  '{"spec":{"externalTrafficPolicy":"Local"}}'`
          ```

  • Example of patching a nodeport to load balance.

    • sudo kubectl patch svc nodeport -p  '{"spec":{"externalTrafficPolicy":"Cluster"}}'

Cluster UI

Portainer Agent

We recommend double checking our Portainer Notes for additional notes / information. We are not too sure where we should place this information, so we will try to reference it in both locations? I suppose that might be the best move.

Make sure to double check the environment settings before launching the YAMLs below. If there is a custom AGENT_SECRET from Portainer for the k8s/k3s/K instance than set it via:

environment:
    - AGENT_SECRET: yourSecret

• Setup Portainer Agent

  • Load Balancer lb

    • LB Command:

    sudo kubectl apply -f https://downloads.portainer.io/ce2-16/portainer-agent-k8s-lb.yaml

    • Agent 2.16 as of 11/17/2022 Previously the revision was 2.15 as of 09/30/2022

  • Node Port nodeport

    • NodePort Command:

    sudo kubectl apply -f https://downloads.portainer.io/ce2-16/portainer-agent-k8s-nodeport.yaml

  • Add the kubernetes cluster location via https:/$/wizard/endpoints/create?envType=kubernetes - Be sure to replace $ with your portainer location.

    • Name: $nameString - The name for the kubernetes cluster. i.e k8scluster007
    • Environment Address: $addrString:$ipInt32 - The location for the kubernetes cluster. i.e k8scluster007.kbve.com:9001
      • Note: Make sure the port 9001 is open for communication between the cluster and Portainer.

  • Advance Optional Settings

    • Group: $groupString - The name of the group for the cluster
    • Tags: $tagsMap - Drop down to select the tags for the cluster.

  • As of 11/18/2022 - There have bene some updates to Portainer! They now have better ingress support!

Security baseline

Harden

• Collection of harden manifests by the DoD
  • DSOP

Volumes & CSI

Storage

• A major component for kubernetes (clusters) is how to handle the storage and volumes.

Kubernetes NFS

External Provider NFS SubDir

CSI-Driver-NFS CSI Driver

OpenShift community

okd

• OKD
• OKD Notes still need to be worked on.

Virtual clusters

vCluster

Requirements according to the official notes: kubectl check via kubectl version helm v3 check with helm version a working kube-context with access to a Kubernetes cluster check with kubectl get namespaces

vCluster Install

Docs on installing vCluster within the environment / system / orchestration.

vcluster is officially supported for:

Mac Intel/AMD Install by running the following command:

curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin

Mac Silicon/ARM Install on the M1 series by the command below:

curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-darwin-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin

Linux Intel/AMD Install vcluster on generic Unix x86

curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin

Linux ARM Unix instance runnong on ARM:

curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-arm64" && sudo install -c -m 0755 vcluster /usr/local/bin

Powershell - Still needs to work.

Note: You may have to double check if the: %APPDATA%\vcluster was installed sucessfully.

Confirm -> Run vcluster --version to confirm that the install was sucessful.

VMs on K8s

KubeVirt

KubeVirt enables running virtual machines on Kubernetes using KVM. On Talos, KVM is built into the kernel — no extensions or machine config patches needed.

Reference: Talos KubeVirt Guide

Operator Install (One-Time)

# KubeVirt operator
export RELEASE=$(curl https://storage.googleapis.com/kubevirt-prow/release/kubevirt/kubevirt/stable.txt)
kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-operator.yaml

# CDI operator (Containerized Data Importer)
export CDI_VERSION=$(curl -sI https://github.com/kubevirt/containerized-data-importer/releases/latest | grep -i location | awk -F/ '{print $NF}' | tr -d '\r')
kubectl apply -f https://github.com/kubevirt/containerized-data-importer/releases/download/${CDI_VERSION}/cdi-operator.yaml

virtctl Install

brew install virtctl

VM Management

# Start a VM
virtctl start windows-builder -n angelscript

# Stop a VM
virtctl stop windows-builder -n angelscript

# Force stop
virtctl stop windows-builder -n angelscript --force --grace-period=0

# Check VM status
kubectl get vm -n angelscript
kubectl get vmi -n angelscript

VNC Access (noVNC in Browser)

# Terminal 1: VNC proxy
virtctl vnc windows-builder -n angelscript --proxy-only --port=5901

# Terminal 2: noVNC web proxy (open http://localhost:6080/vnc.html)
/tmp/noVNC/utils/novnc_proxy --vnc localhost:5901 --listen 6080

ISO Upload via CDI

# Terminal 1: Port-forward CDI upload proxy
kubectl port-forward -n cdi svc/cdi-uploadproxy 8443:443

# Terminal 2: Upload ISO via DataVolume
virtctl image-upload dv windows-server-builder-iso \
  --size=8Gi \
  --image-path=/path/to/windows-server.iso \
  --storage-class=longhorn \
  --namespace=angelscript \
  --uploadproxy-url=https://localhost:8443 \
  --insecure

Quick Start (All-in-One)

# Start VM + VNC proxy + noVNC browser access
virtctl start windows-builder -n angelscript && \
  sleep 20 && \
  virtctl vnc windows-builder -n angelscript --proxy-only --port=5901 &
/tmp/noVNC/utils/novnc_proxy --vnc localhost:5901 --listen 6080
# Open http://localhost:6080/vnc.html

Operations notes

• KubeVirt v1.8.4 is upstream’s release manifest in kube/kubevirt/manifests/kubevirt-operator.yaml. Upgrades go N-1 → N only, by replacing that file; each step shows Degraded for about 3 minutes while components roll. 1.8 removed the SLIRP and macvtap bindings.
• CDI v1.65.0 is managed by kube/cdi/ with no prune and no finalizer: deleting the CDI CR uninstalls CDI. cdi-operator strips the v1alpha1 entry from the CDI CRD, which Argo CD ignores (see Argo CD).
• The CDI apiserver once served an expired certificate for weeks, which broke API discovery and namespace deletion cluster-wide. After any CDI change, check that the served certificate is issued by the CA in its APIService caBundle.
• virt-handler’s client-cert args appear 16 times: the KubeVirt CR’s JSON patch appends with args/- and was re-applied during a rollout. Harmless; if the count grows after an upgrade, stop appending.

Serverless isolation

Firecracker MicroVMs

Firecracker is an open-source VMM (Virtual Machine Monitor) built by AWS for serverless workloads. It creates lightweight microVMs with ~5 MB overhead and ~125 ms boot times, using KVM for hardware-level isolation.

KBVE Integration

The KBVE edge platform uses Firecracker as a composable Tier 2 isolation layer alongside the Deno edge-runtime (Tier 1). Edge functions dispatch VM workloads to the firecracker-ctl service via an internal REST API on port 9001.

Manifests: kube/firecracker/manifests/

Requirements

• Linux kernel with KVM support (Intel VT-x / AMD-V)
• /dev/kvm exposed via kubevirt/device-plugin-kvm (already deployed for KubeVirt)
• Nodes labeled kvm=true (applied via Talos machine config patch)
• Bare metal preferred (nested virt adds latency)

Node Setup (Talos)

The nodeSelector: kvm: "true" requires a node label. On Talos, apply the machine config patch:

# Apply kvm=true node label via Talos machine config
talosctl patch machineconfig --nodes <node> \
  --patch @kube/talos/patches/kvm-node-label.yaml

# Verify label was applied
kubectl get nodes --show-labels | grep kvm

Deployment

# Verify KVM device plugin is running
kubectl get pods -n kubevirt -l app=kvm-device-plugin

# Check firecracker-ctl deployment
kubectl get deploy firecracker-ctl -n kilobase

# Check firecracker-ctl service
kubectl get svc firecracker-ctl -n kilobase

# View firecracker-ctl logs
kubectl logs -n kilobase -l app=firecracker-ctl -f

Pod Spec Overview

nodeSelector:
    kvm: 'true'
containers:
    - name: firecracker-ctl
      image: ghcr.io/kbve/firecracker-ctl:0.1.0
      resources:
          requests:
              devices.kubevirt.io/kvm: '1'
              cpu: 250m
              memory: 512Mi
          limits:
              devices.kubevirt.io/kvm: '1'
              cpu: '2'
              memory: 2Gi
      securityContext:
          capabilities:
              add: ['NET_ADMIN']

Network Policy

Only edge-runtime pods (app: functions) can reach firecracker-ctl on port 9001. All other ingress is denied.

API

Method	Path	Description
POST	/vm/create	Create and start a microVM
GET	/vm/{vm_id}	Get VM status
GET	/vm/{vm_id}/result	Get stdout/stderr/exit_code
DELETE	/vm/{vm_id}	Force-terminate a VM
GET	/health	Health check

Rootfs Images

Image	Size	Contents
alpine-minimal	~8 MB	Alpine + busybox
alpine-python	~45 MB	Alpine + Python 3.12
alpine-node	~40 MB	Alpine + Node.js 22 LTS
ubuntu-rust	~120 MB	Ubuntu minimal + Rust toolchain

Troubleshooting

# Check if /dev/kvm is available on a node
kubectl debug node/<node-name> -it --image=busybox -- ls -la /dev/kvm

# Verify device plugin allocations
kubectl describe node <node-name> | grep -A5 "devices.kubevirt.io/kvm"

# Test firecracker-ctl health from within cluster
kubectl run -n kilobase --rm -it --restart=Never test-fc \
  --image=curlimages/curl -- curl -s http://firecracker-ctl:9001/health

Encrypted config

Sealed Secrets

Encoding

Here is the quick command for encoding a seal.

# Seals a named GitLab credential into kube/gitlab/system/manifests/ as a
# SealedSecret. The value comes from the repo's .env or a terminal prompt,
# never from argv. Credentials: runner-token | deploy-runner-token | registry | push-token
./kube/gitlab/system/seal.sh runner-token

Decoding

For decoding the sealed secret, use this quick command:

kubectl get secret $KEY -n $NAMESPACE -o jsonpath="{.data.$SECRET}" | base64 --decode

Keep exploring

Related applications

LonghornDistributed block storage backing the cluster's persistent volumes.ArgoCDGitOps continuous delivery for every app on the cluster.DockerThe container fundamentals underneath every pod.