llama.cpp

High-performance LLM inference engine

devtoolsadvancedFree tier

Overview

Enables efficient local inference for models like Llama with M1/M2/M3 optimizations and broad hardware support. Essential for developers building privacy-focused AI apps without cloud dependency. Highly flexible for custom integrations.

Best For

+ml-engineers
+embedded

News & Updates

Developer Achieves 30B-Parameter AI Model on 6GB RAM, Outperforming llama.cpp

A developer has achieved running a 30-billion-parameter AI model at 22 tokens per second on consumer hardware with only 6GB of RAM and 16GB of system memory, outperforming llama.cpp's typical efficiency. The breakthrough comes from a custom 'Scientific Agentic AI harness' that reallocates every bit of memory and compute, pushing commercial hardware to its limits. This enables local AI inference at speeds previously thought impossible for such large models, democratizing access to powerful AI without cloud dependency. The project focuses on squeezing maximum performance from off-the-shelf components, not novel architectures.

breakthroughs7/29/2026

AMD Ryzen AI MAX+ 395 Runs 70B Parameter AI Models Locally with 128GB Memory

Developers are experimenting with AMD's Ryzen AI MAX+ 395 (Strix Halo) systems, which feature 128GB of unified memory, to run large language models locally. Early users report being able to run 70B parameter models at 4-bit quantization, with some even testing 120B models using aggressive quantization. The key advantage is the massive memory bandwidth and capacity, which eliminates the need for expensive multi-GPU setups. However, software support remains a challenge, with many relying on llama.cpp and ONNX Runtime, while CUDA-dependent tools like vLLM and TensorRT-LLM are not yet fully compatible. This could democratize local AI inference for developers and researchers who need privacy and low latency.

breakthroughs7/28/2026

New Benchmark Shows M2 Ultra Achieves 45 Tokens/Second, Outperforming RTX 4090 on LLM Inference

A new benchmark from macyou.co provides raw, reproducible data on LLM inference speeds across Apple Silicon chips, including M1, M2, and M3 series. The tests measure tokens per second for models like Llama 2 7B and Mistral 7B using llama.cpp, showing the M2 Ultra achieving 45 tokens/second — outperforming an RTX 4090 on memory-bound tasks. The data is released under CC BY 4.0, allowing developers to freely use it for optimization. This fills a gap in publicly available, standardized Apple Silicon performance metrics for local AI workloads.

new_tools7/27/2026