application · gitops · ci-cd

Git as the source of truth,ArgoCD.

The declarative GitOps engine behind the KBVE cluster — every application manifest lives in Git, and ArgoCD keeps the live cluster synced to match. This guide covers the architecture, the argocd CLI, automated sync with self-heal and prune, the KBVE annotation standard, and fixing stuck syncs.

Architecture Annotation standard CLI commands Troubleshooting

Reconcile, don't apply

Never manually apply manifests that ArgoCD manages — declare the desired state in apps/kube, commit, and let the controller converge the cluster. Drift gets healed, orphans get pruned, and every change stays reviewable.

Self-heal — manual kubectl edits revert to Git.
Prune — resources removed from Git are deleted.
Annotations — every app links back to its source path.

68+Apps managed

self-heal + pruneSync policy

apps/kube/Manifests

v1Annotation schema

Architecture

Application controller, repo server, API server, and the CRDs that define applications and projects.

Annotation standard

The kbve.com/* schema linking every Application to its source directory — categories, stacks, tenants, and kubectl queries.

CLI commands

Install the argocd CLI, log in via port-forward, and the everyday sync, diff, and delete commands.

Troubleshooting

Stuck Progressing operations, out-of-sync despite matching Git, custom health checks, and controller logs.

Start here

Information

ArgoCD is a declarative, GitOps continuous delivery tool for Kubernetes. It automates the deployment of applications by synchronizing the desired state defined in Git repositories with the actual state in Kubernetes clusters.

Jump to the architecture, CLI commands, sync policies, or troubleshooting.

How it works

Architecture

ArgoCD follows the GitOps pattern where Git repositories are the source of truth for defining the desired application state. Key components include:

Application Controller

Continuously monitors running applications and compares the current live state against the desired target state specified in Git.

Repo Server

Internal service that maintains a local cache of Git repositories holding application manifests.

API Server

gRPC/REST server that exposes the API consumed by the Web UI, CLI, and CI/CD systems.

Application Definitions

Kubernetes custom resources (CRDs) that define applications, projects, and repositories.

In production

KBVE Cluster Integration

The KBVE production cluster runs ArgoCD to manage 68+ applications across multiple namespaces. All application manifests are stored in the apps/kube/ directory.

Dashboard Access

View live deployment status: ArgoCD Dashboard

Traceable metadata

Application Annotation Standard

Overview

KBVE uses a standardized annotation schema on ArgoCD Application resources to link them back to their source directories in apps/kube. These annotations provide zero-risk metadata that enables traceability, automation, and better observability.

Schema Version: v1

Core Annotations (Required)

These annotations MUST be present on every ArgoCD Application resource:

Annotation	Description	Example
`kbve.com/source-path`	Relative path from repo root to the application directory	`apps/kube/agones/arpg`
`kbve.com/manifest-path`	Relative path to the actual Kubernetes manifest directory	`apps/kube/agones/arpg/manifests`
`kbve.com/application-file`	Relative path to this ArgoCD Application resource file	`apps/kube/agones/arpg/application.yaml`
`kbve.com/managed-by`	Management tool (always `argocd` for these resources)	`argocd`
`kbve.com/schema-version`	Version of this annotation schema	`v1`

Classification Annotations (Recommended)

These annotations SHOULD be present to enable categorization and filtering:

Annotation	Description	Example Values
`kbve.com/category`	Application category	`game-server`, `application`, `infrastructure`, `database`, `observability`, `ci-cd`, `networking`
`kbve.com/stack`	Technology stack or framework	`agones`, `rows`, `supabase`, `clickhouse`, `cilium`, `core`

Optional Annotations

Additional context-specific annotations:

Annotation	Description	When to Use	Example
`kbve.com/tenant`	Tenant identifier	Multi-tenant deployments	`chuckrpg-beta`, `rentearth-release`
`kbve.com/deployment-model`	Deployment architecture pattern	Complex deployments	`multi-tenant-overlay`, `singleton`, `replicated`
`kbve.com/environment`	Environment stage	When explicit environment tracking needed	`dev`, `staging`, `prod`
`kbve.com/owner`	Team or individual responsible	Large teams with ownership boundaries	`platform-team`, `game-dev`

Examples

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kbve
  namespace: argocd
  annotations:
    kbve.com/source-path: "apps/kube/kbve"
    kbve.com/manifest-path: "apps/kube/kbve/manifest"
    kbve.com/application-file: "apps/kube/kbve/application.yaml"
    kbve.com/managed-by: "argocd"
    kbve.com/schema-version: "v1"
    kbve.com/category: "application"
    kbve.com/stack: "core"
spec:
  project: default
  source:
    repoURL: https://github.com/KBVE/kbve
    targetRevision: HEAD
    path: apps/kube/kbve/manifest
  destination:
    server: https://kubernetes.default.svc
    namespace: kbve

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
    name: agones-arpg
    namespace: argocd
    labels:
        app.kubernetes.io/part-of: arpg
    annotations:
        kbve.com/source-path: "apps/kube/agones/arpg"
        kbve.com/manifest-path: "apps/kube/agones/arpg/manifests"
        kbve.com/application-file: "apps/kube/agones/arpg/application.yaml"
        kbve.com/managed-by: "argocd"
        kbve.com/schema-version: "v1"
        kbve.com/category: "game-server"
        kbve.com/stack: "agones"
spec:
  project: default
  source:
    repoURL: https://github.com/kbve/kbve
    targetRevision: main
    path: apps/kube/agones/arpg/manifests
  destination:
    server: https://kubernetes.default.svc
    namespace: arpg

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
    name: rows-chuckrpg-beta
    namespace: argocd
    annotations:
        kbve.com/source-path: "apps/kube/rows/tenants/overlays/chuckrpg-beta"
        kbve.com/manifest-path: "apps/kube/rows/tenants/overlays/chuckrpg-beta"
        kbve.com/application-file: "apps/kube/rows/tenants/overlays/chuckrpg-beta/application.yaml"
        kbve.com/managed-by: "argocd"
        kbve.com/schema-version: "v1"
        kbve.com/category: "game-server"
        kbve.com/stack: "rows"
        kbve.com/tenant: "chuckrpg-beta"
        kbve.com/deployment-model: "multi-tenant-overlay"
    finalizers:
        - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/KBVE/kbve
    targetRevision: main
    path: apps/kube/rows/tenants/overlays/chuckrpg-beta
  destination:
    server: https://kubernetes.default.svc
    namespace: rows-chuckrpg-beta

Category Reference

Category	Description	Example Apps
`game-server`	Game server deployments	arpg, cryptothrone, factorio, mc
`application`	Core business applications	kbve, herbmail, n8n, jobboard
`infrastructure`	Infrastructure components	argocd, cert-manager, reloader, gateway-api
`database`	Database systems	cnpg, clickhouse, valkey, rabbitmq
`observability`	Monitoring and logging	monitoring, metrics, vector, analytics
`ci-cd`	CI/CD tools and runners	github/runners, github/controller, forgejo
`networking`	Network infrastructure	cilium, kong, multus
`virtualization`	Virtualization platforms	kubevirt, kasm, firecracker
`operator`	Kubernetes operators	agones, clickhouse-operator, kyverno, keda
`security`	Security tooling	sealed-secrets, external-secrets, security-profiles-operator

Querying Applications

These annotations enable powerful kubectl queries:

# List all game servers
kubectl get applications -n argocd -o json | \
  jq '.items[] | select(.metadata.annotations."kbve.com/category" == "game-server") | .metadata.name'

# Find all ROWS tenants
kubectl get applications -n argocd -o json | \
  jq '.items[] | select(.metadata.annotations."kbve.com/stack" == "rows") |
  {name: .metadata.name, tenant: .metadata.annotations."kbve.com/tenant"}'

# Get source paths for all applications
kubectl get applications -n argocd -o json | \
  jq '.items[] | {name: .metadata.name, path: .metadata.annotations."kbve.com/source-path"}'

Upgrade safety

Talos Integration

Pause ArgoCD During Upgrades

When performing Talos node upgrades, pause ArgoCD reconciliation to prevent it from fighting manual scale-down operations:

# Pause ArgoCD application controller
kubectl -n argocd scale sts argocd-application-controller --replicas=0

# After upgrade completes, restore
kubectl -n argocd scale sts argocd-application-controller --replicas=1

This prevents the application-controller from interfering with:

Pre-upgrade workload scale-down
Longhorn maintenance mode preparation
Manual pod eviction for graceful drain

See the full Talos upgrade procedure in Kubernetes - Talos.

Daily drivers

CLI Commands

brew install argocd

curl -sSL -o argocd-linux-amd64 https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
sudo install -m 555 argocd-linux-amd64 /usr/local/bin/argocd
rm argocd-linux-amd64

# Port-forward ArgoCD server
kubectl port-forward svc/argocd-server -n argocd 8080:443

# Login via CLI
argocd login localhost:8080

Common Commands

# List all applications
argocd app list

# Get application details
argocd app get <app-name>

# Sync an application
argocd app sync <app-name>

# Get sync status
argocd app sync-status <app-name>

# View application logs
argocd app logs <app-name>

# Delete an application
argocd app delete <app-name>

# Create an application from YAML
argocd app create -f application.yaml

Sync policies

Application Management

Sync Policies

KBVE applications use automated sync with self-healing:

syncPolicy:
  automated:
    prune: true        # Delete resources removed from Git
    selfHeal: true     # Revert manual kubectl changes
  syncOptions:
    - CreateNamespace=true
    - PrunePropagationPolicy=foreground
    - RespectIgnoreDifferences=true

Ignore Differences

Some resources have fields that ArgoCD should not sync (e.g., Agones Fleet replicas managed by autoscalers):

ignoreDifferences:
  - group: agones.dev
    kind: Fleet
    jsonPointers:
      - /spec/replicas

Finalizers

Multi-tenant applications use finalizers to ensure cascading deletion:

metadata:
  finalizers:
    - resources-finalizer.argocd.argoproj.io

This ensures when an ArgoCD Application is deleted, all deployed resources are cleaned up.

When syncs stall

Troubleshooting

Application Stuck in Progressing

# Check sync operation status
argocd app get <app-name>

# Terminate stuck operation
argocd app terminate-op <app-name>

# Force sync
argocd app sync <app-name> --force

Out of Sync Despite Matching Git

# Hard refresh (bypass cache)
argocd app get <app-name> --hard-refresh

# Check diff
argocd app diff <app-name>

Health Check Failing

Some resources need custom health checks. ArgoCD has built-in checks for common resources, but custom CRDs may require configuration:

# ConfigMap in argocd namespace
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  resource.customizations: |
    agones.dev/Fleet:
      health.lua: |
        hs = {}
        if obj.status ~= nil then
          if obj.status.readyReplicas ~= nil and obj.status.readyReplicas > 0 then
            hs.status = "Healthy"
            return hs
          end
        end
        hs.status = "Progressing"
        return hs

Controller Logs

# View application controller logs
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller -f

# View repo server logs
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-repo-server -f

Operate well

Best Practices

Use Git as Source of Truth
- Never manually apply manifests that are managed by ArgoCD
- All changes should go through Git commits
Enable Auto-Sync with Self-Heal
- Automatically corrects drift from manual changes
- Ensures cluster state matches Git
Use Annotations for Metadata
- Link applications back to source code
- Enable programmatic discovery and automation
Implement Proper RBAC
- Limit who can sync/delete applications in production
- Use ArgoCD Projects for multi-tenant isolation
Monitor Sync Status
- Set up alerts for applications stuck in Progressing
- Track sync failures in observability stack
Use Resource Hooks
- PreSync, Sync, PostSync hooks for migrations
- SyncFail hooks for rollback logic

What is ArgoCD?

ArgoCD is a declarative, GitOps continuous delivery tool for Kubernetes. It treats a Git repository as the source of truth for the desired application state and continuously syncs the live cluster to match, automating deployments and detecting drift.

What is GitOps?

GitOps is an operational model where the desired state of infrastructure and applications is declared in Git, and an automated agent reconciles the running system to match. Git becomes the single source of truth, so every change is version-controlled, reviewable, and auditable.

What is the difference between prune and self-heal in ArgoCD?

prune deletes cluster resources that were removed from Git, keeping the cluster from accumulating orphans. selfHeal reverts manual kubectl changes back to the Git-defined state. Together they enforce that the live cluster always matches the repository.

How do I fix an ArgoCD application stuck in Progressing?

Check the operation with argocd app get, terminate a stuck sync using argocd app terminate-op, then force a fresh sync with argocd app sync --force. A hard refresh (--hard-refresh) bypasses the cache when an app shows out of sync despite matching Git.

What are ignoreDifferences used for in ArgoCD?

ignoreDifferences tells ArgoCD to skip specific resource fields when comparing live and desired state, so it does not fight controllers that own those fields — for example an autoscaler managing an Agones Fleet's spec.replicas would otherwise be reverted on every sync.