llama.cpp

High-performance LLM inference engine

devtoolsadvancedFree tier

Overview

Enables efficient local inference for models like Llama with M1/M2/M3 optimizations and broad hardware support. Essential for developers building privacy-focused AI apps without cloud dependency. Highly flexible for custom integrations.

Best For

+ml-engineers
+embedded

News & Updates

RunPod Launches Reflex: Rust/CUDA Engine for Scale-to-Zero LLM Routing

RunPod released Reflex, a GGUF-native inference engine written in Rust with CUDA acceleration, designed for scale-to-zero LLM routing. The engine loads GGUF model files directly and routes requests across GPU instances that can spin down to zero when idle, eliminating costs during periods of no traffic. Unlike Python-based servers such as llama.cpp or vLLM, Reflex uses Rust for lower memory overhead and faster cold starts, which is critical when scaling from zero. This targets serverless GPU platforms where idle instances still incur charges, potentially cutting costs for sporadic inference workloads. The project is available on RunPod's console, signaling the company's push into the inference infrastructure layer beyond raw GPU rental.

hardware10/12/2026

Rolling Release Linux Distros Beat LTS for AI Workloads, Phoronix Argues

A Phoronix analysis argues that rolling-release Linux distributions like Arch and openSUSE Tumbleweed now serve GenAI developers better than long-term-support distros such as Ubuntu 22.04 LTS. The core reason: AI tooling — CUDA, PyTorch, ROCm, and llama.cpp — ships patches weekly, and LTS kernels and userspace packages lag months behind. Rolling distros deliver these updates within days, avoiding manual backports and dependency conflicts. The tradeoff is stability risk, but for GPU-accelerated AI work, stale drivers are the bigger failure mode.

new_tools10/10/2026

FreeCodeCamp Guide Lets You Train a 25M-Parameter LLM on a CPU

FreeCodeCamp published a step-by-step tutorial for building and training a 25-million-parameter large language model entirely on a standard CPU, no GPU required. The guide walks through the full pipeline: tokenizer training, transformer architecture implementation, and training loop, using a small dataset to keep compute feasible. A 25M-parameter model is roughly 1/7000th the size of GPT-3, but it's enough to learn core LLM mechanics hands-on. The tutorial targets developers who want to understand how LLMs work under the hood without renting cloud GPUs or buying expensive hardware. It arrives as interest in small, locally-runnable models grows, with tools like llama.cpp and Ollama making on-device inference mainstream.

breakthroughs10/8/2026