Teach the machine,ML & AI.
A reference guide to the vast realm of machine learning and artificial intelligence — from computational foundations and key algorithms to GPT models, Groq, LLaMa, Stable Diffusion, prompt engineering, and data-processing subagent pipelines used across the KBVE stack.
One stop for all things AI
An ever growing list of mL and AI services, concepts, and ideas — covering the theory behind computational training and the models and tools you can put to work today.
- Foundations — supervised, unsupervised, and reinforcement learning.
- Models — GPT, LLaMa, Groq, and Stable Diffusion.
- Practice — prompt engineering and data-processing subagents.
On this page
What this guide covers
ML foundations
The math, computer science, and training paradigms you need before diving into frameworks.
Models & platforms
GPT variants, Groq's accelerated API, Meta's LLaMa, and Stable Diffusion image generation.
Prompt engineering
Roles, chaining, rewrite tasks, and the Zod-validated Prompt Engine schema.
Data-processing subagents
Build deterministic LLM pipelines with mdream pre-processing, runbooks, and validation gates.
Start here
ML
Are you curious about the latest buzzword in the tech world?Have you been wondering what machine learning is?Let us take you on a journey to understand the basics of this revolutionary technology and its potential to revolutionize the future.
mL (machine learning) is a subset of artificial intelligence and follows the theory of computational training, to understand core principles through computer science and statistics.
These training methods can be broken down to supervised, unsupervised and reinforcement learning.
If you’re interested in learning the basics of Machine Learning, then you’ll need to have a sound understanding of Computational Theory and the Basics of Computer Science.
This involves learning the fundamentals of programming languages (python), algorithms, and data structures.
Knowing these topics will give you a strong foundation to build upon as you dive deeper into Machine Learning; you will also need to understand the mathematics behind Machine Learning-algorithms, such as linear algebra, statistics, calculus, and probability.
Additionally, you will need to be well-versed in the different Machine Learning / ML techniques and tools, such as neural networks, decision trees, support vector machines, and deep learning.
Finally, you’ll need to understand the various Machine Learning frameworks available, such as Keras, PyTorch, TensorFlow, and Scikit-Learn.
For Python
Reference guide
Information
This documentation is a reference guide to the vast realm of machine learning and artificial general intelligence with examples, concepts and libraries to help you get started!
We want to create this whole page as a one stop shop for all your mL needs xD!
Artificial intelligence is an umbrella phrase that encapsulates various fields within computer science, mathematics, philosophy and information with the goal to emulate natural intelligence display by humans and animals. Since Artificial intelligence (AI) is a broad and multidisciplinary field that encompasses various domains, its primary objective is to create systems and machines (physical or virtual) that can perform tasks that would traditionally require human intelligence. These tasks include learning, reasoning, problem-solving, perception, language understanding, and even physical actions.
The roots of AI can be traced back to ancient myths and stories about artificial beings endowed with intelligence. However, the formal foundation of AI as a scientific discipline was laid in the mid-20th century with the advent of digital computers. The term “artificial intelligence” was coined in 1956 during the Dartmouth Conference, which is considered the birthplace of AI research.
During the birthplace, John McCarthy, a mathematics professor, stated that the conference was: “to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it.”
- The dream for many programmers, scientists, engineers and humans would be to create an entity that could scale past our natural intelligence.
- This is a task that would define the 21st century and push the upper limits on humanity, naturism and metaphysics into the next industrial intelligence revolution.
Generative transformers
GPT
-
GPT , currently known as GPT-3, stands for
Generative Pre-trained Transformerwith the number representing the generation via version control and is a neural network machine learning model -
GPT-Neo
Section titled “GPT-Neo”- Official Github Repo
- We should note that the team, EleutherAI, are no longer maintaining the
gpt-neoand their repo is currently in archive mode. However below is thegpt-neox, which is still being actively maintained as for Oct 2022. - The
GPT-Neomay have been an extension ofGPT2but changes to the layering.
-
GPT-NeoX
Section titled “GPT-NeoX”- For GPU, we suggest GPT-NeoX, Repo Here
-
GPT4All
Section titled “GPT4All”- What I enjoy about this software is that it is really easy to install and use, plus it requires very bare metal resources.
- WebUI for GPT4All written in Flask (Python) by Nomic AI, Repo Here
-
PyChatGPT
Section titled “PyChatGPT”- Official Repo. PyChatGPT is an on-going API written in
Pythonto help scale and integrateChatGPTto various applications / eco-systems via TLS.
- Official Repo. PyChatGPT is an on-going API written in
Hardware acceleration
GROQ
Groq is an innovative company specializing in AI and machine learning (AI/ML) solutions, known for developing custom hardware designed to accelerate AI/ML workloads. Their hardware architecture, optimized for high performance and efficiency, enables rapid data processing and analysis, making it ideal for complex AI/ML tasks. Groq offers a free API that allows developers to leverage their powerful hardware infrastructure, providing an accessible entry point for integrating advanced AI capabilities into various applications without incurring initial costs.
Groq CookBooks
Section titled “Groq CookBooks”The Groq Cookbook Examples provide a comprehensive collection of practical guides and code samples to help developers harness the full potential of Groq’s AI/ML hardware. These examples cover a wide range of applications, from basic model deployment to advanced optimization techniques, demonstrating best practices and efficient workflows. For instance, developers can find detailed instructions on using Groq’s hardware (via their API) to examine medical documents, enabling efficient processing and analysis of large volumes of medical data for improved diagnostics and patient care. Additionally, there are examples of utilizing Groq’s capabilities to create sophisticated stock market tools, helping analysts and traders make data-driven decisions with enhanced speed and accuracy. Each recipe is designed to be easily followed, enabling users to quickly integrate Groq’s powerful capabilities into their own projects, accelerate development, and achieve superior performance in their AI/ML tasks.
Meta foundation model
LLaMa
LLaMa, short for Large Language Model Meta AI, is a cutting-edge foundational large language model developed by Meta, formerly known as Facebook. This state-of-the-art model is designed to assist researchers in advancing their work in natural language processing (NLP). By leveraging LLaMa, researchers can push the boundaries of NLP, exploring new possibilities and enhancing the capabilities of AI-driven language understanding and generation. Meta’s commitment to innovation in AI is embodied in LLaMa, providing a powerful tool for the scientific community to develop more sophisticated and effective NLP applications.
Image generation
Stable Diffusion
Stable Diffusion is a python-based latent diffusion model that performs image generation through deep learning.
Waifu Diffusion
Section titled “Waifu Diffusion”Waifu Diffusion is a specialized adaptation of stable diffusion models, finely tuned to generate high-quality anime and manga-style imagery. This model focuses on “weeb” content, catering to the aesthetic preferences and cultural nuances of anime enthusiasts. By leveraging advanced stable diffusion techniques, Waifu Diffusion excels in producing visually appealing characters, scenes, and artworks that resonate with fans of Japanese pop culture. Its precise fine-tuning ensures that the generated images maintain the distinctive art style and charm characteristic of popular anime and manga, making it an invaluable tool for creators and fans alike.
The official Github repo for Waifu-Diffusion.
Stable Diffusion WebUI
Section titled “Stable Diffusion WebUI”The official repo for AUTOMATIC1111 Stable Diffusion WEB UI
Fast Stable Diffusion
Section titled “Fast Stable Diffusion”Fast-stable-diffusion Notebooks offer a streamlined and efficient way to utilize various advanced AI tools for image generation and customization. Integrating Automatic1111 (A1111), ComfyUI, and DreamBooth, these notebooks provide a comprehensive platform for users to experiment with and refine their stable diffusion models. A1111 facilitates the management and deployment of these models, while ComfyUI offers an intuitive interface for seamless interaction. DreamBooth further enhances the capabilities by allowing fine-tuning and personalization of the models. Together, these tools enable users to create high-quality, stable diffusion images with ease and precision.
Better prompts
Prompt Engineering
Prompt engineering theory covers a wide range of different GPT concepts, including examples and short cuts to generate the right style of questions and content.
Engine
Section titled “Engine”The Prompt Engine is a versatile and structured framework designed to define, manage, and execute various prompt-based tasks. It is specifically built to handle a wide range of scenarios in machine learning and artificial intelligence applications, offering a high degree of flexibility and precision. The Prompt Engine is defined using a robust schema in TypeScript, leveraging Zod for validation.
Core Aspects:
-
Name: Each prompt is uniquely identified by a name field, which allows for easy reference and management.
-
Description: Provides a detailed explanation of what the prompt is about, helping users understand its purpose and context.
-
Items: An array of strings that lists the key points or examples relevant to the prompt. This helps in outlining the scope and content of the prompt.
-
Task: A textual description of the specific task that the prompt is intended to perform. This includes detailed instructions and objectives, ensuring clarity in execution.
-
Tools: An optional array of tool definitions, where each tool is defined by its type, name, description, and required parameters. This allows the prompt to specify which tools or functions are needed to complete the task.
-
Output: Specifies the expected output format, which can either be text or json. This ensures that the results of the prompt are returned in a predictable and usable format.
-
Pathways: A structured object that defines the flow of actions based on user inputs or conditions. Each pathway contains a prompt and a set of conditions that determine the next action, enabling complex decision trees and logical flows.
Common role examples for text transformers:
Act as a javascript consoleAct as an excel sheetAct as a HR interviewerAct as an advertiserAct as a publisherAct as a music teacherAct as a relationship coachAct as a World of Warcraft player and limit the response to 50 charactersWarning :Not all text transformers will let you assign roles, as it might create a security issue / risk.
Chaining
Section titled “Chaining”- Common Terms include: Chain-of-Thought, Chained Prompt.
Rewrite Prompt Example
Section titled “Rewrite Prompt Example”Start by giving it this task:
Task: Rewrite user-provided text with the most human like perplexity and burstiness.
Objectives: Prompt the user to input a block of text. Use a language model to rewrite the text with the most human like perplexity and burstiness. Provide the rewritten text to the user.
Rules: Keep burstiness as low as possible Keep perplexity as low as possible Go to extraordinary lengths to make the text more human-like Burstiness is on a scale of 0-100. Perplexity is on a scale of 0-100. Burstiness starts at 0 Perplexity starts at 0 Don’t answer for user
Roles: AI model responsible for rewriting the text. User: Provides the initial block of text to be rewritten.
Strategy:
Approach: Use an AI model to generate the most human like perplexity and burstiness.
Prompt the user to input a block of text.
Provide the rewritten text to the user.
Additional Instructions: Ask the user to input a block of text that they would like to be rewritten. Use a language model to generate a rewritten version of the text that has the most human like perplexity and burstiness. Provide the rewritten text to the user. Be sure to explain to the user the concept of perplexity and burstiness and how it affects the text. Provide an option for the user to adjust the level of perplexity and burstiness if desired. Perplexity and Burstiness start at 0.Then follow up with asking it to rewrite it but with a focus on these concepts:
Consistency and Coherence: Human writing tends to maintain a consistent and coherent narrative throughout, reflecting a deeper understanding of context and topic.
Creativity and Originality: While AI can be creative, humans often bring unique perspectives, experiences, and creativity to their writing that might be harder for AI to replicate authentically.
Personal Touch: Human writing may include personal anecdotes, emotions, or subjective elements that are reflective of individual experiences and perspectives.
Context Understanding: Humans excel at understanding nuanced context and incorporating it into their writing, whereas AI might sometimes produce contextually inaccurate or mismatched information.
Purposeful Structure: Human writers often have a purposeful structure in their content, carefully organizing information for clarity and impact. AI might generate text that lacks intentional structure.
Inconsistencies: Human writing may include nuanced inconsistencies, errors, or idiosyncrasies that reflect the natural variability present in human language, while AI-generated content may exhibit a higher degree of consistency.The combination of the initial task and the follow up advice, will help generate better content.
Agent pipelines
Data Processing Subagents
A data-processing subagent is an LLM agent pointed at a narrow, repeatable job: take messy source data, turn it into clean structured output, and do it the same way every time. The mistake most people make is handing the model a raw web page and a vague instruction — that burns tokens, invites hallucination, and produces inconsistent results. A powerful subagent is built by moving everything deterministic out of the model and leaving only judgement inside it.
Pre-process with mdream
Section titled “Pre-process with mdream”The single biggest win is to never make the agent fetch and parse HTML. We use mdream — the fastest HTML-to-Markdown converter, optimized for LLMs — to pre-fetch source pages and cache them as clean, token-optimized Markdown before any agent runs.
A minimal fetch-and-convert step looks like this:
import { htmlToMarkdown } from 'mdream';
const html = await fetch(pageUrl, { headers: { 'User-Agent': UA } }).then((r) => r.text(),);
const markdown = htmlToMarkdown(html, { origin: new URL(pageUrl).origin, clean: true, // strip tracking params, empty/redundant links});// cache markdown to disk, keyed by item id — read it from the agent laterAnatomy of a powerful subagent
Section titled “Anatomy of a powerful subagent”The goal is that an agent can do the job from one spec file plus one cached input — without re-reading the schema, the source code, or the original HTML.
-
Deterministic pre-processing (no model). Fetch, clean, and convert source data with code (mdream, an API client, a parser). Cache the result. The model never touches raw HTML.
-
A turnkey runbook. Write a single self-contained spec: the goal, the exact output shape (copy the schema/field definitions inline), the per-item algorithm, and one worked example. The agent should never have to go discover these.
-
Hard rules and skip conditions. State what is the source of truth, what must be preserved verbatim, and when not to act (missing data, ambiguous input). Forbid fabrication explicitly: every fact traces back to the cached source.
-
A validation gate. Give the agent a command that proves its output is valid (a schema validator, a type-check, a test). “It isn’t done until the validator passes” removes a whole class of silent errors.
-
Isolation. Run the work in a throwaway workspace (a git worktree) and land it via review (a pull request) — never directly on a shared branch. Parallel agents then can’t collide.
Why this beats a “smart prompt”
Section titled “Why this beats a “smart prompt””A clever prompt against raw input scales badly — every run re-pays the parsing tokens, and quality drifts. Front-loading the determinism (clean input + exact output contract + a validator) shrinks the model’s job to the part it is actually good at: reading prose and writing structured content. Cost drops, accuracy rises, and the same spec can be handed to many agents in parallel.
Knowledge graphs
Graphify
Graphify is a semantic knowledge graph tool that analyzes codebases to create queryable relationship graphs showing file-level connections, semantic relationships, community detection (subsystems), and highly-connected nodes. It combines tree-sitter AST parsing with optional LLM-powered semantic analysis to understand code architecture at scale.
Architecture
Section titled “Architecture”The KBVE monorepo integrates Graphify across three layers:
packages/python/graphify-wrapper/ # Python wrapper packagepackages/data/graphify/ # Graph data storageapps/kbve/astro-kbve/src/pages/api/graphify/ # API endpointsPython Wrapper Package provides CLI commands and Nx targets for building graphs, querying relationships, and exporting data. The package uses uv for dependency management and includes a test suite.
Data Storage holds configuration files (.graphify.yml), build scripts, and generated graph outputs. Graphs are stored as JSON with accompanying HTML visualizations and markdown reports.
API Endpoints expose graph data via REST endpoints, making semantic knowledge queryable from web applications and development tools.
Prerequisites
Section titled “Prerequisites”Before installing Graphify, ensure you have:
- Python 3.12+ (Graphify requires <3.13 due to Leiden algorithm dependency)
- uv package manager (already installed in monorepo)
- Node.js 18+ (for Nx commands)
Installation
Section titled “Installation”-
Install Graphify CLI globally
Terminal window uv tool install graphifyygraphify --version -
Install Python package dependencies
Terminal window pnpm nx run graphify-wrapper:installpnpm nx run graphify-wrapper:lock -
Verify setup with a test build
Terminal window pnpm nx run graphify-wrapper:build-app --app=herbmaills packages/data/graphify/output/apps/herbmail/# Should see: graph.json, graph.html, GRAPH_REPORT.md# View the HTML visualizationopen packages/data/graphify/output/apps/herbmail/graph.html
Known Issues
Section titled “Known Issues”uv Version Mismatch: If you encounter version conflicts (workspace requires uv==0.11.28 but system has newer):
Workarounds:
- Use shell script directly:
./packages/data/graphify/scripts/build-monorepo-graph.sh - Install globally:
pip install graphifyy - Use uvx:
uvx --from graphifyy graphify - Update workspace uv version or downgrade system uv
Building Graphs
Section titled “Building Graphs”Option 1: Using Shell Script (Recommended):
# Build entire monorepo./packages/data/graphify/scripts/build-monorepo-graph.sh
# View resultsopen packages/data/graphify/output/monorepo/graph.htmlOption 2: Using Nx Targets:
# Full monorepo (AST-only mode, fast, no LLM)pnpm nx run graphify-wrapper:build-monorepo
# Specific applicationpnpm nx run graphify-wrapper:build-app --app=laserpnpm nx run graphify-wrapper:build-app --app=discordsh
# Specific packagecd packages/python/graphify-wrapperuv run kbve-graph-build --scope package --name rust/jedi --output ../../../packages/data/graphify/output/packages/jediOption 3: Direct Graphify CLI:
graphify . \ --output packages/data/graphify/output/monorepo \ --config packages/data/graphify/configs/.graphify.yml \ --no-semanticWith Semantic Analysis (thorough, uses LLM):
./packages/data/graphify/scripts/build-monorepo-graph.sh --semanticQuerying Graphs
Section titled “Querying Graphs”# Semantic searchpnpm nx run graphify-wrapper:query --q="Where is user authentication handled?"
# Path tracing between componentsgraphify path packages/data/graphify/output/monorepo/graph.json "astro-kbve" "bevy_inventory"
# Explain a specific nodegraphify explain packages/data/graphify/output/monorepo/graph.json "packages/rust/jedi/src/auth.rs"LLM Backend Configuration
Section titled “LLM Backend Configuration”Edit packages/data/graphify/configs/.graphify.yml to choose your LLM backend:
Ollama (Local, Free):
llm: provider: ollama model: llama3Anthropic Claude (API, Paid):
llm: provider: anthropic model: claude-3-5-sonnet-20241022 api_key: ${ANTHROPIC_API_KEY}Set the environment variable:
export ANTHROPIC_API_KEY=sk-ant-...Custom Ignore Patterns:
ignore: - custom/path/** - '*.generated.ts'AST-Only (No LLM, Fastest):
# Default mode - uses tree-sitter for static analysis onlypnpm nx run graphify-wrapper:build-monorepoExporting for Web
Section titled “Exporting for Web”# Export graph to public web directorypnpm nx run graphify-wrapper:export-web
API Endpoints
Section titled “API Endpoints”Start the dev server to access graph data:
pnpm nx run astro-kbve:serve
# Then visit:# http://localhost:4321/api/graphify/apps/herbmail.jsonOnce graphs are built, access them via REST endpoints:
Full Monorepo Graph:
GET /api/graphify/monorepo.json
Response:{ "metadata": { "source": "graphify", "type": "semantic-knowledge-graph", "scope": "monorepo", "generated": "2026-07-19T..." }, "graph": { "nodes": [...], "edges": [...], "communities": [...] }}App-Specific Graph:
GET /api/graphify/apps/{app}.json
Example: GET /api/graphify/apps/herbmail.jsonData Flow
Section titled “Data Flow”1. Source Code (apps/, packages/) ↓2. Graphify CLI (tree-sitter AST + optional LLM) ↓3. graph.json (packages/data/graphify/output/) ↓4. API Endpoint (/api/graphify/*.json) ↓5. Web UI (future: /dashboard/graph/)Package Management
Section titled “Package Management”Maintain the Python wrapper package with these Nx targets:
# Install dependenciespnpm nx run graphify-wrapper:install
# Lock dependenciespnpm nx run graphify-wrapper:lock
# Run testspnpm nx run graphify-wrapper:test
# Lint codepnpm nx run graphify-wrapper:lint
# Build packagepnpm nx run graphify-wrapper:buildFile Structure
Section titled “File Structure”The integration creates this file structure:
packages/python/graphify-wrapper/├── pyproject.toml├── project.json├── .python-version├── .flake8├── README.md├── QUICK_START.md├── graphify_wrapper/│ ├── __init__.py│ └── cli.py└── tests/ └── test_cli.py
packages/data/graphify/├── configs/│ └── .graphify.yml├── scripts/│ └── build-monorepo-graph.sh├── output/│ └── .gitkeep├── .gitignore└── README.md
apps/kbve/astro-kbve/src/pages/api/graphify/├── monorepo.json.ts└── apps/ └── [app].json.tsTroubleshooting
Section titled “Troubleshooting”Error: graphify: command not found
uv tool install graphifyywhich graphify # Should show pathError: Python 3.13 not supported
# Check Python versionpython3 --version
# Install Python 3.12 if needed (macOS)
# Verify wrapper uses correct versioncat packages/python/graphify-wrapper/.python-version # Should be 3.12Error: No module named 'graphifyy'
cd packages/python/graphify-wrapperuv syncSlow Build Times: Full monorepo with semantic analysis takes 20-30 minutes. Solutions:
- Use AST-only mode (default): ~5-10 min
- Build per-app instead of full monorepo
- Use Anthropic Claude API (faster than Ollama)
Memory Issues: For very large graphs:
NODE_OPTIONS=--max-old-space-size=8192 pnpm nx run graphify-wrapper:build-monorepoLarge Graph Files: If graph.json exceeds 10MB, use Git LFS:
git lfs track "packages/data/graphify/output/**/*.json"git add .gitattributesgit commit -m "chore: track graph files with LFS"Or keep graphs in .gitignore and rebuild via CI.
Development Workflow
Section titled “Development Workflow”-
Install dependencies
Terminal window pnpm nx run graphify-wrapper:install -
Build a small graph for testing
Terminal window pnpm nx run graphify-wrapper:build-app --app=herbmail -
View the HTML visualization
Terminal window open packages/data/graphify/output/apps/herbmail/graph.html -
Query the graph
Terminal window pnpm nx run graphify-wrapper:query --q="test query" -
Run tests
Terminal window cd packages/python/graphify-wrapperuv run pytest
CI/CD Automation
Section titled “CI/CD Automation”Create .github/workflows/ci-daily-graph.yml for automated daily graph builds:
name: Daily Knowledge Graph Build
on: schedule: - cron: '0 6 * * *' # 6am UTC daily workflow_dispatch:
jobs: build-graph: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4
- name: Setup Python uses: actions/setup-python@v5 with: python-version: '3.12'
- name: Install uv run: pip install uv
- name: Install Graphify run: uv tool install graphifyy
- name: Build Monorepo Graph run: pnpm nx run graphify-wrapper:build-monorepo
- name: Export for Web run: pnpm nx run graphify-wrapper:export-web
- name: Commit Graph Data run: | git config user.name "github-actions[bot]" git config user.email "github-actions[bot]@users.noreply.github.com" git add apps/kbve/astro-kbve/public/graphify/ git diff --staged --quiet || git commit -m "chore(graph): daily knowledge graph update" git pushWeb Visualization
Section titled “Web Visualization”The graph can be visualized using the existing @xyflow/react + d3-force infrastructure. See /dashboard/graph/ for the current Nx dependency graph implementation that can be extended for Graphify.
Future Enhancements
Section titled “Future Enhancements”Web UI Integration:
- Add graph switcher to
/dashboard/graph/page - Create
<GraphifyExplorer />React component - Use existing
@xyflow/react+d3-forceinfrastructure
Advanced Features:
- Custom extractors for Nx metadata, Proto definitions, and Bevy ECS systems
- Semantic search API endpoint
- Path tracing between components
- Community detection visualization
Resources
Section titled “Resources”- Graphify Documentation - Official Graphify project
- Package README - Python wrapper documentation
- Data Storage README - Data storage configuration
