Skip to content
application · ai · ml

Teach the machine,ML & AI.

A reference guide to the vast realm of machine learning and artificial intelligence — from computational foundations and key algorithms to GPT models, Groq, LLaMa, Stable Diffusion, prompt engineering, and data-processing subagent pipelines used across the KBVE stack.

One stop for all things AI

An ever growing list of mL and AI services, concepts, and ideas — covering the theory behind computational training and the models and tools you can put to work today.

  • Foundations — supervised, unsupervised, and reinforcement learning.
  • Models — GPT, LLaMa, Groq, and Stable Diffusion.
  • Practice — prompt engineering and data-processing subagents.
PythonCore language
3Learning paradigms
1956AI coined
PyTorch · TensorFlowFrameworks

Start here

ML

  • Are you curious about the latest buzzword in the tech world?
  • Have you been wondering what machine learning is?
  • Let us take you on a journey to understand the basics of this revolutionary technology and its potential to revolutionize the future.

mL (machine learning) is a subset of artificial intelligence and follows the theory of computational training, to understand core principles through computer science and statistics. These training methods can be broken down to supervised, unsupervised and reinforcement learning.

If you’re interested in learning the basics of Machine Learning, then you’ll need to have a sound understanding of Computational Theory and the Basics of Computer Science. This involves learning the fundamentals of programming languages (python), algorithms, and data structures. Knowing these topics will give you a strong foundation to build upon as you dive deeper into Machine Learning; you will also need to understand the mathematics behind Machine Learning-algorithms, such as linear algebra, statistics, calculus, and probability. Additionally, you will need to be well-versed in the different Machine Learning / ML techniques and tools, such as neural networks, decision trees, support vector machines, and deep learning. Finally, you’ll need to understand the various Machine Learning frameworks available, such as Keras, PyTorch, TensorFlow, and Scikit-Learn.

For Python

Reference guide

Information

This documentation is a reference guide to the vast realm of machine learning and artificial general intelligence with examples, concepts and libraries to help you get started! We want to create this whole page as a one stop shop for all your mL needs xD!


Artificial intelligence is an umbrella phrase that encapsulates various fields within computer science, mathematics, philosophy and information with the goal to emulate natural intelligence display by humans and animals. Since Artificial intelligence (AI) is a broad and multidisciplinary field that encompasses various domains, its primary objective is to create systems and machines (physical or virtual) that can perform tasks that would traditionally require human intelligence. These tasks include learning, reasoning, problem-solving, perception, language understanding, and even physical actions.

The roots of AI can be traced back to ancient myths and stories about artificial beings endowed with intelligence. However, the formal foundation of AI as a scientific discipline was laid in the mid-20th century with the advent of digital computers. The term “artificial intelligence” was coined in 1956 during the Dartmouth Conference, which is considered the birthplace of AI research.

During the birthplace, John McCarthy, a mathematics professor, stated that the conference was: “to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it.”


  • The dream for many programmers, scientists, engineers and humans would be to create an entity that could scale past our natural intelligence.
  • This is a task that would define the 21st century and push the upper limits on humanity, naturism and metaphysics into the next industrial intelligence revolution.

Generative transformers

GPT

  • GPT , currently known as GPT-3, stands for Generative Pre-trained Transformer with the number representing the generation via version control and is a neural network machine learning model

    • Official Github Repo
    • We should note that the team, EleutherAI, are no longer maintaining the gpt-neo and their repo is currently in archive mode. However below is the gpt-neox, which is still being actively maintained as for Oct 2022.
    • The GPT-Neo may have been an extension of GPT2 but changes to the layering.
    • What I enjoy about this software is that it is really easy to install and use, plus it requires very bare metal resources.
    • WebUI for GPT4All written in Flask (Python) by Nomic AI, Repo Here
    • Official Repo. PyChatGPT is an on-going API written in Python to help scale and integrate ChatGPT to various applications / eco-systems via TLS.

Hardware acceleration

GROQ

Groq is an innovative company specializing in AI and machine learning (AI/ML) solutions, known for developing custom hardware designed to accelerate AI/ML workloads. Their hardware architecture, optimized for high performance and efficiency, enables rapid data processing and analysis, making it ideal for complex AI/ML tasks. Groq offers a free API that allows developers to leverage their powerful hardware infrastructure, providing an accessible entry point for integrating advanced AI capabilities into various applications without incurring initial costs.

The Groq Cookbook Examples provide a comprehensive collection of practical guides and code samples to help developers harness the full potential of Groq’s AI/ML hardware. These examples cover a wide range of applications, from basic model deployment to advanced optimization techniques, demonstrating best practices and efficient workflows. For instance, developers can find detailed instructions on using Groq’s hardware (via their API) to examine medical documents, enabling efficient processing and analysis of large volumes of medical data for improved diagnostics and patient care. Additionally, there are examples of utilizing Groq’s capabilities to create sophisticated stock market tools, helping analysts and traders make data-driven decisions with enhanced speed and accuracy. Each recipe is designed to be easily followed, enabling users to quickly integrate Groq’s powerful capabilities into their own projects, accelerate development, and achieve superior performance in their AI/ML tasks.

Meta foundation model

LLaMa

LLaMa, short for Large Language Model Meta AI, is a cutting-edge foundational large language model developed by Meta, formerly known as Facebook. This state-of-the-art model is designed to assist researchers in advancing their work in natural language processing (NLP). By leveraging LLaMa, researchers can push the boundaries of NLP, exploring new possibilities and enhancing the capabilities of AI-driven language understanding and generation. Meta’s commitment to innovation in AI is embodied in LLaMa, providing a powerful tool for the scientific community to develop more sophisticated and effective NLP applications.

Image generation

Stable Diffusion

Stable Diffusion is a python-based latent diffusion model that performs image generation through deep learning.

Waifu Diffusion is a specialized adaptation of stable diffusion models, finely tuned to generate high-quality anime and manga-style imagery. This model focuses on “weeb” content, catering to the aesthetic preferences and cultural nuances of anime enthusiasts. By leveraging advanced stable diffusion techniques, Waifu Diffusion excels in producing visually appealing characters, scenes, and artworks that resonate with fans of Japanese pop culture. Its precise fine-tuning ensures that the generated images maintain the distinctive art style and charm characteristic of popular anime and manga, making it an invaluable tool for creators and fans alike.

The official Github repo for Waifu-Diffusion.

The official repo for AUTOMATIC1111 Stable Diffusion WEB UI

Fast-stable-diffusion Notebooks offer a streamlined and efficient way to utilize various advanced AI tools for image generation and customization. Integrating Automatic1111 (A1111), ComfyUI, and DreamBooth, these notebooks provide a comprehensive platform for users to experiment with and refine their stable diffusion models. A1111 facilitates the management and deployment of these models, while ComfyUI offers an intuitive interface for seamless interaction. DreamBooth further enhances the capabilities by allowing fine-tuning and personalization of the models. Together, these tools enable users to create high-quality, stable diffusion images with ease and precision.

Better prompts

Prompt Engineering

Prompt engineering theory covers a wide range of different GPT concepts, including examples and short cuts to generate the right style of questions and content.

The Prompt Engine is a versatile and structured framework designed to define, manage, and execute various prompt-based tasks. It is specifically built to handle a wide range of scenarios in machine learning and artificial intelligence applications, offering a high degree of flexibility and precision. The Prompt Engine is defined using a robust schema in TypeScript, leveraging Zod for validation.

Core Aspects:

  • Name: Each prompt is uniquely identified by a name field, which allows for easy reference and management.

  • Description: Provides a detailed explanation of what the prompt is about, helping users understand its purpose and context.

  • Items: An array of strings that lists the key points or examples relevant to the prompt. This helps in outlining the scope and content of the prompt.

  • Task: A textual description of the specific task that the prompt is intended to perform. This includes detailed instructions and objectives, ensuring clarity in execution.

  • Tools: An optional array of tool definitions, where each tool is defined by its type, name, description, and required parameters. This allows the prompt to specify which tools or functions are needed to complete the task.

  • Output: Specifies the expected output format, which can either be text or json. This ensures that the results of the prompt are returned in a predictable and usable format.

  • Pathways: A structured object that defines the flow of actions based on user inputs or conditions. Each pathway contains a prompt and a set of conditions that determine the next action, enabling complex decision trees and logical flows.

Common role examples for text transformers:

  • Act as a javascript console
  • Act as an excel sheet
  • Act as a HR interviewer
  • Act as an advertiser
  • Act as a publisher
  • Act as a music teacher
  • Act as a relationship coach
  • Act as a World of Warcraft player and limit the response to 50 characters

    Warning : Not all text transformers will let you assign roles, as it might create a security issue / risk.

  • Common Terms include: Chain-of-Thought, Chained Prompt.

Start by giving it this task:

Task: Rewrite user-provided text with the most human like perplexity and burstiness.
Objectives: Prompt the user to input a block of text. Use a language model to rewrite the text with the most human like perplexity and burstiness. Provide the rewritten text to the user.
Rules: Keep burstiness as low as possible Keep perplexity as low as possible Go to extraordinary lengths to make the text more human-like Burstiness is on a scale of 0-100. Perplexity is on a scale of 0-100. Burstiness starts at 0 Perplexity starts at 0 Don’t answer for user
Roles: AI model responsible for rewriting the text. User: Provides the initial block of text to be rewritten.
Strategy:
Approach: Use an AI model to generate the most human like perplexity and burstiness.
Prompt the user to input a block of text.
Provide the rewritten text to the user.
Additional Instructions: Ask the user to input a block of text that they would like to be rewritten. Use a language model to generate a rewritten version of the text that has the most human like perplexity and burstiness. Provide the rewritten text to the user. Be sure to explain to the user the concept of perplexity and burstiness and how it affects the text. Provide an option for the user to adjust the level of perplexity and burstiness if desired. Perplexity and Burstiness start at 0.

Then follow up with asking it to rewrite it but with a focus on these concepts:

Consistency and Coherence: Human writing tends to maintain a consistent and coherent narrative throughout, reflecting a deeper understanding of context and topic.
Creativity and Originality: While AI can be creative, humans often bring unique perspectives, experiences, and creativity to their writing that might be harder for AI to replicate authentically.
Personal Touch: Human writing may include personal anecdotes, emotions, or subjective elements that are reflective of individual experiences and perspectives.
Context Understanding: Humans excel at understanding nuanced context and incorporating it into their writing, whereas AI might sometimes produce contextually inaccurate or mismatched information.
Purposeful Structure: Human writers often have a purposeful structure in their content, carefully organizing information for clarity and impact. AI might generate text that lacks intentional structure.
Inconsistencies: Human writing may include nuanced inconsistencies, errors, or idiosyncrasies that reflect the natural variability present in human language, while AI-generated content may exhibit a higher degree of consistency.

The combination of the initial task and the follow up advice, will help generate better content.

Agent pipelines

Data Processing Subagents

A data-processing subagent is an LLM agent pointed at a narrow, repeatable job: take messy source data, turn it into clean structured output, and do it the same way every time. The mistake most people make is handing the model a raw web page and a vague instruction — that burns tokens, invites hallucination, and produces inconsistent results. A powerful subagent is built by moving everything deterministic out of the model and leaving only judgement inside it.

The single biggest win is to never make the agent fetch and parse HTML. We use mdream — the fastest HTML-to-Markdown converter, optimized for LLMs — to pre-fetch source pages and cache them as clean, token-optimized Markdown before any agent runs.

A minimal fetch-and-convert step looks like this:

import { htmlToMarkdown } from 'mdream';
const html = await fetch(pageUrl, { headers: { 'User-Agent': UA } }).then((r) =>
r.text(),
);
const markdown = htmlToMarkdown(html, {
origin: new URL(pageUrl).origin,
clean: true, // strip tracking params, empty/redundant links
});
// cache markdown to disk, keyed by item id — read it from the agent later

The goal is that an agent can do the job from one spec file plus one cached input — without re-reading the schema, the source code, or the original HTML.

  1. Deterministic pre-processing (no model). Fetch, clean, and convert source data with code (mdream, an API client, a parser). Cache the result. The model never touches raw HTML.

  2. A turnkey runbook. Write a single self-contained spec: the goal, the exact output shape (copy the schema/field definitions inline), the per-item algorithm, and one worked example. The agent should never have to go discover these.

  3. Hard rules and skip conditions. State what is the source of truth, what must be preserved verbatim, and when not to act (missing data, ambiguous input). Forbid fabrication explicitly: every fact traces back to the cached source.

  4. A validation gate. Give the agent a command that proves its output is valid (a schema validator, a type-check, a test). “It isn’t done until the validator passes” removes a whole class of silent errors.

  5. Isolation. Run the work in a throwaway workspace (a git worktree) and land it via review (a pull request) — never directly on a shared branch. Parallel agents then can’t collide.

A clever prompt against raw input scales badly — every run re-pays the parsing tokens, and quality drifts. Front-loading the determinism (clean input + exact output contract + a validator) shrinks the model’s job to the part it is actually good at: reading prose and writing structured content. Cost drops, accuracy rises, and the same spec can be handed to many agents in parallel.

Knowledge graphs

Graphify

Graphify is a semantic knowledge graph tool that analyzes codebases to create queryable relationship graphs showing file-level connections, semantic relationships, community detection (subsystems), and highly-connected nodes. It combines tree-sitter AST parsing with optional LLM-powered semantic analysis to understand code architecture at scale.

The KBVE monorepo integrates Graphify across three layers:

packages/python/graphify-wrapper/ # Python wrapper package
packages/data/graphify/ # Graph data storage
apps/kbve/astro-kbve/src/pages/api/graphify/ # API endpoints

Python Wrapper Package provides CLI commands and Nx targets for building graphs, querying relationships, and exporting data. The package uses uv for dependency management and includes a test suite.

Data Storage holds configuration files (.graphify.yml), build scripts, and generated graph outputs. Graphs are stored as JSON with accompanying HTML visualizations and markdown reports.

API Endpoints expose graph data via REST endpoints, making semantic knowledge queryable from web applications and development tools.

Before installing Graphify, ensure you have:

  • Python 3.12+ (Graphify requires <3.13 due to Leiden algorithm dependency)
  • uv package manager (already installed in monorepo)
  • Node.js 18+ (for Nx commands)
  1. Install Graphify CLI globally

    Terminal window
    uv tool install graphifyy
    graphify --version
  2. Install Python package dependencies

    Terminal window
    pnpm nx run graphify-wrapper:install
    pnpm nx run graphify-wrapper:lock
  3. Verify setup with a test build

    Terminal window
    pnpm nx run graphify-wrapper:build-app --app=herbmail
    ls packages/data/graphify/output/apps/herbmail/
    # Should see: graph.json, graph.html, GRAPH_REPORT.md
    # View the HTML visualization
    open packages/data/graphify/output/apps/herbmail/graph.html

uv Version Mismatch: If you encounter version conflicts (workspace requires uv==0.11.28 but system has newer):

Workarounds:

  1. Use shell script directly: ./packages/data/graphify/scripts/build-monorepo-graph.sh
  2. Install globally: pip install graphifyy
  3. Use uvx: uvx --from graphifyy graphify
  4. Update workspace uv version or downgrade system uv

Option 1: Using Shell Script (Recommended):

Terminal window
# Build entire monorepo
./packages/data/graphify/scripts/build-monorepo-graph.sh
# View results
open packages/data/graphify/output/monorepo/graph.html

Option 2: Using Nx Targets:

Terminal window
# Full monorepo (AST-only mode, fast, no LLM)
pnpm nx run graphify-wrapper:build-monorepo
# Specific application
pnpm nx run graphify-wrapper:build-app --app=laser
pnpm nx run graphify-wrapper:build-app --app=discordsh
# Specific package
cd packages/python/graphify-wrapper
uv run kbve-graph-build --scope package --name rust/jedi --output ../../../packages/data/graphify/output/packages/jedi

Option 3: Direct Graphify CLI:

Terminal window
graphify . \
--output packages/data/graphify/output/monorepo \
--config packages/data/graphify/configs/.graphify.yml \
--no-semantic

With Semantic Analysis (thorough, uses LLM):

Terminal window
./packages/data/graphify/scripts/build-monorepo-graph.sh --semantic
Terminal window
# Semantic search
pnpm nx run graphify-wrapper:query --q="Where is user authentication handled?"
# Path tracing between components
graphify path packages/data/graphify/output/monorepo/graph.json "astro-kbve" "bevy_inventory"
# Explain a specific node
graphify explain packages/data/graphify/output/monorepo/graph.json "packages/rust/jedi/src/auth.rs"

Edit packages/data/graphify/configs/.graphify.yml to choose your LLM backend:

Ollama (Local, Free):

llm:
provider: ollama
model: llama3

Anthropic Claude (API, Paid):

llm:
provider: anthropic
model: claude-3-5-sonnet-20241022
api_key: ${ANTHROPIC_API_KEY}

Set the environment variable:

Terminal window
export ANTHROPIC_API_KEY=sk-ant-...

Custom Ignore Patterns:

ignore:
- custom/path/**
- '*.generated.ts'

AST-Only (No LLM, Fastest):

Terminal window
# Default mode - uses tree-sitter for static analysis only
pnpm nx run graphify-wrapper:build-monorepo
apps/kbve/astro-kbve/public/graphify/monorepo.json
# Export graph to public web directory
pnpm nx run graphify-wrapper:export-web

Start the dev server to access graph data:

4321/api/graphify/monorepo.json
pnpm nx run astro-kbve:serve
# Then visit:
# http://localhost:4321/api/graphify/apps/herbmail.json

Once graphs are built, access them via REST endpoints:

Full Monorepo Graph:

GET /api/graphify/monorepo.json
Response:
{
"metadata": {
"source": "graphify",
"type": "semantic-knowledge-graph",
"scope": "monorepo",
"generated": "2026-07-19T..."
},
"graph": {
"nodes": [...],
"edges": [...],
"communities": [...]
}
}

App-Specific Graph:

GET /api/graphify/apps/{app}.json
Example: GET /api/graphify/apps/herbmail.json
1. Source Code (apps/, packages/)
2. Graphify CLI (tree-sitter AST + optional LLM)
3. graph.json (packages/data/graphify/output/)
4. API Endpoint (/api/graphify/*.json)
5. Web UI (future: /dashboard/graph/)

Maintain the Python wrapper package with these Nx targets:

Terminal window
# Install dependencies
pnpm nx run graphify-wrapper:install
# Lock dependencies
pnpm nx run graphify-wrapper:lock
# Run tests
pnpm nx run graphify-wrapper:test
# Lint code
pnpm nx run graphify-wrapper:lint
# Build package
pnpm nx run graphify-wrapper:build

The integration creates this file structure:

packages/python/graphify-wrapper/
├── pyproject.toml
├── project.json
├── .python-version
├── .flake8
├── README.md
├── QUICK_START.md
├── graphify_wrapper/
│ ├── __init__.py
│ └── cli.py
└── tests/
└── test_cli.py
packages/data/graphify/
├── configs/
│ └── .graphify.yml
├── scripts/
│ └── build-monorepo-graph.sh
├── output/
│ └── .gitkeep
├── .gitignore
└── README.md
apps/kbve/astro-kbve/src/pages/api/graphify/
├── monorepo.json.ts
└── apps/
└── [app].json.ts

Error: graphify: command not found

Terminal window
uv tool install graphifyy
which graphify # Should show path

Error: Python 3.13 not supported

Terminal window
# Check Python version
python3 --version
# Install Python 3.12 if needed (macOS)
brew install [email protected]
# Verify wrapper uses correct version
cat packages/python/graphify-wrapper/.python-version # Should be 3.12

Error: No module named 'graphifyy'

Terminal window
cd packages/python/graphify-wrapper
uv sync

Slow Build Times: Full monorepo with semantic analysis takes 20-30 minutes. Solutions:

  1. Use AST-only mode (default): ~5-10 min
  2. Build per-app instead of full monorepo
  3. Use Anthropic Claude API (faster than Ollama)

Memory Issues: For very large graphs:

Terminal window
NODE_OPTIONS=--max-old-space-size=8192 pnpm nx run graphify-wrapper:build-monorepo

Large Graph Files: If graph.json exceeds 10MB, use Git LFS:

Terminal window
git lfs track "packages/data/graphify/output/**/*.json"
git add .gitattributes
git commit -m "chore: track graph files with LFS"

Or keep graphs in .gitignore and rebuild via CI.

  1. Install dependencies

    Terminal window
    pnpm nx run graphify-wrapper:install
  2. Build a small graph for testing

    Terminal window
    pnpm nx run graphify-wrapper:build-app --app=herbmail
  3. View the HTML visualization

    Terminal window
    open packages/data/graphify/output/apps/herbmail/graph.html
  4. Query the graph

    Terminal window
    pnpm nx run graphify-wrapper:query --q="test query"
  5. Run tests

    Terminal window
    cd packages/python/graphify-wrapper
    uv run pytest

Create .github/workflows/ci-daily-graph.yml for automated daily graph builds:

name: Daily Knowledge Graph Build
on:
schedule:
- cron: '0 6 * * *' # 6am UTC daily
workflow_dispatch:
jobs:
build-graph:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install uv
run: pip install uv
- name: Install Graphify
run: uv tool install graphifyy
- name: Build Monorepo Graph
run: pnpm nx run graphify-wrapper:build-monorepo
- name: Export for Web
run: pnpm nx run graphify-wrapper:export-web
- name: Commit Graph Data
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git add apps/kbve/astro-kbve/public/graphify/
git diff --staged --quiet || git commit -m "chore(graph): daily knowledge graph update"
git push

The graph can be visualized using the existing @xyflow/react + d3-force infrastructure. See /dashboard/graph/ for the current Nx dependency graph implementation that can be extended for Graphify.

Web UI Integration:

  • Add graph switcher to /dashboard/graph/ page
  • Create <GraphifyExplorer /> React component
  • Use existing @xyflow/react + d3-force infrastructure

Advanced Features:

  • Custom extractors for Nx metadata, Proto definitions, and Bevy ECS systems
  • Semantic search API endpoint
  • Path tracing between components
  • Community detection visualization