#Quantization (8)

llama.cpp: seventeen backends and 1.5-to-8-bit quantisation, taking LLM inference anywhere a compiler exists

An MIT-licensed, dependency-free plain C/C++ inference implementation built on ggml. Seventeen backends span CUDA, Metal, HIP, Vulkan, SYCL, WebGPU, CANN, Hexagon, MUSA, zDNN and ZenDNN; integer quantisation runs 1.5 to 8 bits; CPU+GPU hybrid inference makes models larger than VRAM runnable at all; and GGUF, its output format, is a de facto standard that even vLLM consumes. Two lines - llama cli or llama serve - start an OpenAI-compatible server.

Inference EngineGGUF

C++130k24k841

vllm-project/vllmApache-2.0

vLLM: the open serving stack that grew out of PagedAttention, with 200+ architectures and five parallelism axes

An inference and serving library out of UC Berkeley's Sky Computing Lab with 2000+ contributors: paged KV memory, continuous batching and chunked prefill, CUDA/HIP graphs with torch.compile, quantisation across FP8/MXFP4/NVFP4/GGUF, pluggable attention kernels (FlashInfer, TRTLLM-GEN, FlashMLA), four speculative decoding routes (n-gram, suffix, EAGLE, DFlash), tensor/pipeline/data/expert/context parallelism with P/D disaggregation, and both OpenAI and Anthropic protocols.

Inference EngineServing

Python93k23k597

deepbeepmeep/Wan2GPNOASSERTION

Wan2GP: run frontier generative models on a 6 GB GPU

A one-stop local generation app for video, image, audio and speech models, built on MMGP: block swapping, VRAM preloading and seven memory profiles let models larger than your VRAM run, starting at 6 GB.

VideoGenerationImageGeneration

Python10k1.6k111

Niko1221/StrataMIT

Strata: Run a 125B-Parameter MoE Model on Your Gaming PC

Strata is an open-source local inference engine built on llama.cpp/ggml that runs the 125-billion-parameter Qwen3.8-Flash-Next MoE model on consumer gaming PCs with as little as 12GB VRAM, using tiered expert caching (GPU→RAM→SSD) and guess-and-check speculative decoding to achieve 53-94 tokens/second.

LLMInferenceOptimization

C++7.8k68351

Neroued/ninferApache-2.0

NInfer: A From-Scratch C++/CUDA Inference Engine for a Single RTX 5090

A from-scratch C++/CUDA inference engine for five explicitly registered Qwen checkpoints on one NVIDIA GeForce RTX 5090. Startup-frozen residency picks MTP or DFlash speculative decoding, Vision, and one of five KV storage formats; a shared Device KV pool plus pinned Host State/KV checkpoints reuse exact prompt prefixes across 240k-token contexts. Measured aggregate decode reaches 1,146.9 tok/s at concurrency 8 and 15,544.3 tok/s on a 7,680-token prefill.

LLM InferenceCUDA

C++1.4k26113

0:12

Black Forest Labs launches FLUX 3 Image Klein, an open weight 5.6B image model

@testingcatalog · 170 views · 2026-10-15

FLUX 3Black Forest Labs

How to set up your first Local LLM in 10 Steps ( Full-course )

A 10-step roadmap to a local LLM stack you use daily: memory sizing, model picking, Ollama install, quantization, hardware buying, serving as an API, small-model routing, and six ready-to-run workloads.

BLOG

Local LLMInference Optimization

Jetson AI Lab Tutorial: Fine-tune and Quantize AI Models on Jetson

@NVIDIARobotics · 745 views · 2026-08-31

NVIDIAJetson

As an Amazon Associate, we earn from qualifying purchases.