KiloByte Virtual Engine: : DevTeam & Software Services

2024

PGSodium

After getting a couple of the postgres instances operating and a decent sized cluster, we need to go back in and add some extensions. I decided to go the route of just loading all of them in and seeing which ones fail, the first one will be the pgsodium.

Error Log:


[
{"level":"info","ts":"2024-09-27T16:06:07Z","logger":"pg_ctl","msg":"2024-09-27 16:06:07.972 UTC [29] FATAL:  The getkey script \"/usr/share/postgresql/15/extension/pgsodium_getkey\" does not exists.","pipe":"stdout","logging_pod":"supabase-release-supabase-db-1-initdb"},
{"level":"info","ts":"2024-09-27T16:06:07Z","logger":"pg_ctl","msg":"2024-09-27 16:06:07.972 UTC [29] DETAIL:  The getkey script fetches the primary server secret key.","pipe":"stdout","logging_pod":"supabase-release-supabase-db-1-initdb"},
{"level":"info","ts":"2024-09-27T16:06:07Z","logger":"pg_ctl","msg":"2024-09-27 16:06:07.972 UTC [29] HINT:  You might want to create it and/or set \"pgsodium.getkey_script\" to the correct path.","pipe":"stdout","logging_pod":"supabase-release-supabase-db-1-initdb"},
{"level":"info","ts":"2024-09-27T16:06:07Z","logger":"pg_ctl","msg":"2024-09-27 16:06:07.972 UTC [29] LOG:  database system is shut down","pipe":"stdout","logging_pod":"supabase-release-supabase-db-1-initdb"},
{"level":"info","ts":"2024-09-27T16:06:08Z","logger":"pg_ctl","msg":"pg_ctl: could not start server","pipe":"stderr","logging_pod":"supabase-release-supabase-db-1-initdb"},
{"level":"info","ts":"2024-09-27T16:06:08Z","logger":"pg_ctl","msg":" stopped waiting","pipe":"stdout","logging_pod":"supabase-release-supabase-db-1-initdb"},
]

The move would be to to figure out how to load the secret through sealed secret but we can not add the key directly into a file on the drive? From my understanding the pgsodium key has to remain in the memory, so it would make sense that we are not storing it directly in the database.

To solve this issue, we would have to come up with about three different steps, starting with sealing a pgsodium key away and then another step with setting up a shell script and finally injecting the key. Hmm, I am still thinking through exactly the way that I would approach this problem. Sidecar or Init container, hmm, I am thinking we will go with an init container for now and then look at the sidecar as an additional option for future deployments.

The seal command has been integrated into the nx system, so we can go ahead and start the key generation and sealing processing. To make sure we can generate a random key, lets do a quick debug.


PGSODIUM_KEY=$(openssl rand -hex 32)
echo "Generated pgsodium key: $PGSODIUM_KEY"

Supabase

The current commands that are we are using to debug the supabase instance is as follows:


kubectl get pods -n supabase

This will display all the running pods in the namespace of Supabase which is under the deployment of kilobase via fleet. The result would look like this:


 kubectl get pods -n supabase
NAME                                                   READY   STATUS             RESTARTS         AGE
supabase-release-supabase-auth-c96889665-vtg5m         0/1     Init:0/1           0                3h5m
supabase-release-supabase-db-1-initdb-2wwtr            0/1     Error              0                173m
supabase-release-supabase-db-1-initdb-6kxpb            0/1     Error              0                3h1m
supabase-release-supabase-db-1-initdb-6rdwk            0/1     Error              0                3h5m
supabase-release-supabase-db-1-initdb-7djvz            0/1     Error              0                3h5m
supabase-release-supabase-db-1-initdb-ggkmn            0/1     Error              0                3h3m
supabase-release-supabase-db-1-initdb-m4z4q            0/1     Error              0                178m
supabase-release-supabase-db-1-initdb-wjqcv            0/1     Error              0                3h4m
supabase-release-supabase-functions-7f9f54d684-gn86g   0/1     CrashLoopBackOff   41 (52s ago)     3h5m
supabase-release-supabase-imgproxy-7649cb49ff-8xg4r    1/1     Running            0                3h5m
supabase-release-supabase-kong-5bc5b754dd-g5d6q        0/1     CrashLoopBackOff   38 (4m15s ago)   3h5m
supabase-release-supabase-meta-ccbf87b48-nmkvf         1/1     Running            0                3h5m
supabase-release-supabase-realtime-6c866974bf-xxv6m    0/1     Init:0/1           0                3h5m
supabase-release-supabase-rest-76bd699f95-9hgmm        1/1     Running            0                3h5m
supabase-release-supabase-storage-5bfb49b6f5-nvv78     0/1     Init:0/1           0                3h5m
supabase-release-supabase-studio-b6bdf46-cfp75         1/1     Running            0                3h5m

Now to check the error, we run:


kubectl logs supabase-release-supabase-db-1-initdb-7djvz -n supabase

This gets us the error logs and there we can see the issue with the pgsodium.

Sealed

Okay going to switch gears and look back at our generate-sealed-secret.sh and make some changes to make it easier for us to use. We want the script to run in memory only and not create any files that might accidentally get pushed. To make it easier for others to read and use:


usage() {
  echo "Usage: $0 -n <namespace> -k <keyName> -s <secrets>"
  echo "  -n  The namespace for the secret (e.g., 'default' or 'my-namespace')"
  echo "  -k  The name of the secret (e.g., 'pgsodium-secret')"
  echo "  -s  The secrets in 'key=value&key2=value2' format (e.g., 'key1=value1&key2=value2')"
  exit 1
}

The usage() part of the script will be called again through a future nx command that we can write to help us handle the secret creation and mangement. For safety, we can add an additional function to check the namespace but that would defeat the purpose for any namespace that does not exist yet.


check_namespace() {
  if ! kubectl get namespace "$NAMESPACE" &> /dev/null; then
    echo "Error: Namespace '$NAMESPACE' does not exist."
    exit 1
  fi
}

So instead of the check_namespace() function being a general purpose one, I will wrap back around and make it so that the function checks if armada is there. The sealed secret operator ships through the armada namespace, thus without that namespace, we can not utilize the kubeseal. Thus the function in our bash script will be: check_armada_namespace.


# Function to check the Armada Fleet
check_armada_namespace() {

  if ! kubectl get namespace "armada" &> /dev/null; then
    echo "Error: The namespace 'armada' does not exist. Please ensure the Sealed Secrets operator is running in the 'armada' namespace."
    exit 1
  else
    echo "Namespace 'armada' is present."
  fi

}

Side note: there are only 25 resources running in the armada fleet, so a future function that we can make would be focused on the redis-based secert manager. This would hold the temporary secrets in the redis memory, I also believe there are some cute tricks that can be done to make sure that the temporary secrets can only be accessed through an aes-style rotation. The redis would pull the temporary sealed secrets and hold them in small malloc that would expire once read or within a certain timeframe. Furthermore that timeframe can be determined through the ulid, since it is a lex-based key.

The generate-sealed-secret.sh has been updated to create the encryptedData: string for the keys, here is the example command.


./generate-sealed-secret.sh -n supabase -k testcase -s "key1=value&key2=value"

This command now has to be placed inside of the nx monorepo, so we will create a new nx-run-command for the shell script that can execute this for us at the root folder level.

Now we can call this directly with the nx via:


pnpm nx run kilobase:seal --namespace=supabase --keyName=nxtest --secrets="key1=value1&key2=value2"
# or
./kbve.sh -nx kilobase:seal --namespace=supabase --keyName=nxtest --secrets="key1=value1&key2=value2"

Perfect! This will make it easier for us to build future rotating keys from the github actions directly into the kubectl.

KubeCtl


kubectl describe pod supabase-release-supabase-db-1-initdb-2wwtr -n supabase

Okay this is a great command for us to use and help debug some of the container issues that. Adding some of the pgsodium key access roles for the cluster and this should help us move forward with the key problem.

2023

3:45pm - I am surprised that Tesla and Apple have back traced this fast, I was really expecting them to coast through this mild recession. Granted I do believe they are a bit over valued, but at my age and the direction that computation is moving, both of those companies seem like a safe harbor for the next couple decades.
6:30pm - Looks like my tesla has a flat tire, womp! The leak is pretty bad and honestly I feel like crap trying to get it fixed because it will be a pain to just find the right tire shop to do the job. I also have to get my rims fixed a bit, hmm this is just one of those things.
8:17pm - YoRHa UI’s object placement and general modal updates need to be addressed. There are a couple ways that I could go about handling this, I am thinking to do my classic build, break and repeat until I get a bit of what I want. I have already written the state management for the modals about five times and counting! Ugh.

Quote

By oneself is evil done; by oneself is one defiled. By oneself is evil left undone; by oneself is one made pure. — The Buddha

September: 27th

2024

2023

Quote

Tasks