2024
Midnight
Burning that midnight oil again and looking through the different ways we can deploy the db within the supabase stack, such that we would get the database to operate under cloudnative-pg.
I noticed that some examples did not include the barman directly inside of the postgres image, so we will have to make some changes to acknowledge its purpose in our instance.
While doing that, I could also take a look at adding more concepts to the website?
Chores
Took a look at some of the dependencies and decided to do some quick cleaning.
Without trying to break the whole monorepo, I updated the vitest, tailwindcss and vue with their minor updates and patches.
Afterwards, I removed two colyseus packages from the earlier install of it, making sure we are good to go without it anymore.
Functions
The functions deployment will be in two stages, first we will launch the initial supabase edge function docker image but then during the second stage, we will replace it.
I believe the new function image will be kilofunk but that is still a bit out there because there are a couple changes we need to do and have to get through the inital deployment.
Thus for stage one, we will go ahead and upload the helm charts from the community for the functions.
Wrapping back around to the functions, I will create another topic for the kilofunk concept and see if we can work around that.
IMGProxy
For the image proxy, we will not be too worried about this right now and can just add it in and move forward. This would be something that we can look through post-dev cycle and probably after backups and recovery are fully functional.
Kong
The kong that supabase operates with is an older instance, I believe a v2.8.x and it should be replaced with a v3. However this should be done during the performance stage and for now it might not be worth spending too much time looking through it all. There was some test casing we did during the swarm and the kong held up fairly well. This will be another topic, we might call this the kilokong and build a layer ontop of the kong v3 that will help integrate some of the changes for the supabase.
Meta
The next set of charts are for the meta, aka postgres-meta. This is another one of those situations where we can look at it post-production and we just let it be during our swarm deployment. I believe the latest version is v0.84.0 but that might not be worth updating given that we will be deploying an older instance of the community postgres.
Minio
For the local s3 storage, the community charts provide an instance of minio, and we did not deploy this in our docker swarm stack but for now we might just skip it for this cluster deployment. This will be something we determine during the deployment, I believe we will have to disable this. If we were going to enable it, I would have to switch back around and make sure that it will use the longhorn storage.
Realtime
In addition to the post-production tables that need to be added, which is on our todo list, the realtime part of this cluster looks a bit rough around the edges. For this smaller commit, we will start with the base realtime charts and then wrap back around to turn the carbon into a diamond.
Rest
The rest api will be another community helm chart that we can add next. The postgrest/postgrest version on this community is tagged v12.0.1, but our swarm stack is using v12.2.0, thus I will have to look through that again. Other than that, we should be fine with the settings that are in-place right now.
Secrets
The sodium secrets are also included in the docker swarm stack but I believe supabase is planning to replace them on a later date? I wonder how this would work for them, considering there are a decent amount of moving parts that revolve around it.
Storage
The storage charts are next, once more these are something that we look at again post dev cycle because it will require another operator to handle the storage for them. For now we will keep the community base template, similar to the other situations that we were in earlier. The concept is similar to the s3 and have custom buckets that we can work with, hmm, this will definitely be post-post production.
Studio
The last three charts that we will be adding are for the studio, test and vector. These will also have their own issues and I believe we will have to layer something ontop of them to protect the routes, hmm. We could just make it so that the studio is only accessible via the rancher ingress and prevent it from being routed directly by the kong configuration? Since the studio can operate without having the direct reference, but in the swarm deployment, we placed a traefik layer ontop of the route and protected it with a custom environmental username and password. Some of these changes remind me of the planck’s principle, but hey its just part of the adventure.
Worker4
The worker4 will go through a maintenance window and during that peroid we can test case some of the automated upgrades.
I am still on the fence on how I would like this to be done, there were a couple suggestions made but this will be in the air for exactly which route we want to take.
The worker4 storage will be bumped up, so we are increasing the size of the scsi0 by 18GB, going to a total of 50GB.
The configurations look like this: rpool-vmdata-vm-234-disk-0,iothread=1,size=50GB and now we will have to go into the container and bump up the size.
Okay the resize was fine and we noticed the metrics jump to adjust for the new size, but there will be some other issues that need to be resolved!
I wonder if I can just take the worker offline and do the changes without it breaking the stack, hmm, I believe the raft will help resolve that issue, thus making it easier for us to take it offline for the window.
After trip to the cafe, we will take the worker4 offline and do the updates.
Okay after applying the apt-get update, it seems we had to update the kernel.
It was operating under 6.8.0.41 and now with this update, it should be 6.8.0.44.
After running the sudo reboot, we got the taints: node.kubernetes.io/unreachable=:NoSchedule and node.kubernetes.io/unreachable=:NoExecute
We can see the update live via the rancher, which does help when doing paraops, looks like the worker4 is operational again now.
Okay now we are going to get the next kernel update with another dist-upgrade, hopefully getting us to 6.8.0.45, but now we will have to reboot and knock the worker4 offline again.
I can see why some consider moving to an immutable operating system in some of these cases, as it can be a bit of a pain to handle.
Cafe
Some of the converting of the swarm to helm will require a decent amount of caffeine! Okay before I perform the restart for the worker4, I will go ahead and grab some cafe and stock up on the bulls, then when I come back around, we can see how the upgrades will affect the cluster.
Vault
The issue with the vault is that it is not HA, which means that we need to reconfigure the deployment of the vault, ensuring that the HA is setup with an internal raft system. We could use consul to help us with that but that might be too much overhead for just this, having an internal raft would make more sense.
Longhorn
It looks like there is a version mismatch with the longhorn, so we will have to fix that as well. I am thinking we remove the vault and then go back around and remove the longhorn setup. Afterwards, we could tgry and build the longhorn setup from scratch again but with the version of 1.6.x, instead of the 1.7.x. Hmm, however we do want the 1.7.x, so maybe this is just part of the upstream and we need to wait for it to stablize.
GitOps
Now that we got the base templates installed, we can go ahead and begin the first failed deployment. The goal of the first deployment is to make sure it fails at setting up the cluster from the branch, the sole objective is to see if it can pull the charts that we added. After we can confirm the charts are loaded, we can go back and begin updating the charts. Under the current situation, as of 6pm, I am thinking the best move would be utilize the rancher fleet and if we run into any issues with the fleet, we can step back towards the argocd. So of the two mainstream options, Fleet and ArgoCD, we will use fleet first because its already included in our instance and then if we run into issues with fleet then we can switch back to the classic ArgoCD. Okay, stashed the changes, switched branches and now preparing for the fleet!
The breakdown so far looks like this:
- Name: supabase
- Repo: KBVE/kbve
- Target: Advance
- Clusters: 0/0
We also added the path of /migrations/kube/charts/supabase.
Since there is no path within the main branch right now, we are getting this error.
Job Failed. failed: 3/1time="2024-09-18T22:34:10Z" level=fatal msg="no resource found at the following paths to deploy: [/migrations/kube/charts/supabase]"
Now I am thinking it might make more sense to deploy via the dev branch, instead of the main branch, this would help us avoid some of the issues that we might face down the line?.
But let me just try it via the main branch and then if it works, we can go back and updated the fleet to deploy from the dev branch.
Okay the folder structure needs to be to be nested by one more level, so it would be /migrations/kube/charts/kilobase/supabase, then within the kilobase folder, we would add the fleet.yaml.
This should help us with the
2023
- 9:34am - Backups are working great! Yes! I am surprised how easy it has become with all these new reference points and documents out there, I am actually really content with how an area that is the least serve, has so many improvements xD.
- 2:01pm - Re-organizing the KBVE.com repo so that it is a bit easier to use in the future, including the part where I would add in abstract functions. I think isolating the Appwrite functions as their own repo was a good initial idea but now I am thinking of combining them all into one and then seeing where we can go from there.
- 3:47pm - Got most of the Appwrite functions resolved and now I am going to merge the functions into the KBVE.com repo, under the /public/ folder. The plan now will be to build out a bunch of different appwrite functions, then fine-tune train against them all so that we could have our own LLM build appwrite functions? This will be a wild project but I am all for it.
- 7:32pm - There were some minor changes that I need to do with the repo but for now I am focused on getting the functions operational.
Quote
Meditation is the dissolution of thoughts in eternal awareness or Pure consciousness without objectification, knowing without thinking, merging finitude in infinity. — Voltaire
Tasks
- [ ]