Multi-GPU ML workstationCharles
KBVE's internal machine-learning build — a multi-GPU deep-learning workstation intended to cluster NVIDIA RTX 4090 cards for training and serving models. This page captures the hardware notes and software stack behind the build.
The build
Highlights
Clustered RTX 4090s
Liquid-cooled MSI Suprim Liquid X cards, chosen so multiple 4090s can cluster together.
DDR5 memory
Scales from 32-64GB entry-level up to 256GB+ per node for large datasets.
Proxmox virtualized
The build runs on KBVE Proxmox with Docker and Kubernetes distributing workloads.
Per-node scaling
RAM is allocated by each node's role rather than uniformly, enhancing the scaling strategy.
Build notes
Hardware
Information
Section titled “Information”Picture a symphony of GPUs, each singing in computational harmony within a single machine. This ensemble isn’t just for show; it’s tailored for the demanding concertos of machine learning. In this digital orchestra, every GPU strikes a note, turning data into artful insights!
Hardware
Section titled “Hardware”These are the hardware notes for charles, our internal machine learning build/project.
NVIDIA RTX 4090 - The 4090 series has a couple different variants on the market, but since we want to cluster them together, it might make more sense to look at liquid cooled instances.
The 4090 that we are using right now is the Suprim LiquidX from MSI.
While the required RAM for constructing a GPU cluster for deep learning varies based on specific use-cases and requirements, we’ll outline key considerations to guide your decision-making process.
The RAM sticks we are using are DDR5.
Dataset Size
Section titled “Dataset Size”When preprocessing and loading large datasets into main memory, a substantial amount of RAM is crucial for fast access, as it’s quicker than persistent storage. Standard memory configurations may fall short for extensive datasets. Therefore, 128GB, 256GB, or even more RAM could be essential, especially with on-the-fly data augmentation or transformation.
Concurrent Tasks
Section titled “Concurrent Tasks”Running concurrent tasks like data preprocessing, serving models, or multiple training jobs on one machine demands ample RAM. Sufficient memory ensures efficient multitasking and optimal performance during simultaneous operations.
GPU-System Communication
Section titled “GPU-System Communication”Adequate system RAM is essential to prevent bottlenecks when data is transferred between the CPU and GPU. Remember that while GPU memory (VRAM) is crucial for model training, the system RAM plays a role in staging and preparing data.
Software Overhead
Section titled “Software Overhead”Running the operating system, deep learning frameworks, databases, and other necessary software tools will also consume RAM.
Scaling Strategy
Section titled “Scaling Strategy”If your cluster, managed through solutions like Docker and Kubernetes, is designed to distribute tasks across multiple nodes, each node might not require a vast amount of RAM. Instead, RAM can be allocated according to the specific role and demand of each node, enhancing your scaling strategy.
RAM CONFIG
Section titled “RAM CONFIG”- Entry-Level Configuration: At least 32GB to 64GB of RAM per node.
- Mid-Range Configuration: Between 64GB to 128GB of RAM per node.
- High-End Configuration: 256GB or more per node, especially if you’re working with vast datasets or complex multi-stage workflows.
Remember to always tailor your cluster’s configuration to your specific needs. Monitoring tools can help gauge memory usage in real-time and assist in making informed decisions about future upgrades.
Stack
Software
Proxmox
Section titled “Proxmox”The setup for the Charles will be in our Proxmox Applications page.
This section will be for the machine learning and ai eco-system + notes.
Gaming
Section titled “Gaming”This section will be for our gaming section.
Questions
Frequently asked
What is Charles?
Charles is KBVE's internal machine-learning build and project — a multi-GPU deep-learning workstation designed to cluster NVIDIA RTX 4090 cards for training and serving machine-learning models.
What GPUs and memory does Charles use?
Charles runs NVIDIA RTX 4090 GPUs (currently the MSI Suprim Liquid X, chosen for liquid cooling to enable clustering) with DDR5 RAM, scaling from 32-64GB entry-level up to 256GB or more per node for large datasets.
How is Charles managed?
Charles is set up through KBVE's Proxmox environment, with Docker and Kubernetes used to distribute deep-learning workloads across nodes so RAM can be allocated per node role.