Backend Management

LLM inference in C/C++

Fork of llama.cpp optimized for CPU performance by ikawrakow

Fork of llama.cpp adding the TurboQuant KV-cache quantization scheme. Reuses the LocalAI llama.cpp gRPC server sources against the fork's libllama.

Fork of llama.cpp (PrismML) adding the Q1_0 (1-bit) and Q2_0 (ternary, 1.58-bit) weight-quantization kernels used by the Bonsai and Ternary-Bonsai models. Reuses the LocalAI llama.cpp gRPC server sources against the fork's libllama, so it drives these sub-2-bit models with the same OpenAI-compatible API as the stock llama.cpp backend.

antirez/ds4 - DeepSeek V4 Flash inference engine. Single-model, optimized for Metal (Darwin) and CUDA (Linux). Requires the GGUFs published at huggingface.co/antirez/deepseek-v4-gguf.

0xShug0/audio.cpp - a ggml audio inference framework covering text to speech, voice cloning, transcription, forced alignment, voice activity detection, speaker diarization, source separation, voice conversion and music generation, across 30+ model families. Consumes audio.cpp-native GGUF packages from huggingface.co/audio-cpp/audio.cpp-gguf.

Port of OpenAI's Whisper model in C/C++

ALPHA development builds. Try it, but llama-cpp stays the recommendation for production use. vllm.cpp is an Apache-2.0 C++20 inference engine maintained by the LocalAI team, developed in its own repository and usable without LocalAI. It began as a port of vLLM and keeps vLLM as its reference implementation, checking output against it and benchmarking against it, while growing a featureset of its own. It implements vLLM's V1 architecture (paged KV cache, continuous batching, prefix caching, scheduler, sampler) on a portable tensor runtime with no Python, PyTorch or ggml at inference time. It loads GGUF as well as Hugging Face safetensors, supports structured output (JSON schema / regex / choice / GBNF grammar) enforced in-engine, ships speculative decoding and KV offload, and runs on CPU, NVIDIA CUDA (Blackwell-family), Apple Metal and Vulkan. The project is expected to be renamed as it diverges further from vLLM; the new name is still to be decided.

ALPHA development builds. Try it, but llama-cpp stays the recommendation for production use. vllm.cpp is an Apache-2.0 C++20 inference engine maintained by the LocalAI team, developed in its own repository and usable without LocalAI. It began as a port of vLLM and keeps vLLM as its reference implementation, checking output against it and benchmarking against it, while growing a featureset of its own. It implements vLLM's V1 architecture (paged KV cache, continuous batching, prefix caching, scheduler, sampler) on a portable tensor runtime with no Python, PyTorch or ggml at inference time. It loads GGUF as well as Hugging Face safetensors, supports structured output (JSON schema / regex / choice / GBNF grammar) enforced in-engine, ships speculative decoding and KV offload, and runs on CPU, NVIDIA CUDA (Blackwell-family), Apple Metal and Vulkan. The project is expected to be renamed as it diverges further from vLLM; the new name is still to be decided.

CrispASR unified speech engine (whisper.cpp fork on ggml) supporting many ASR architectures (Parakeet, Canary, Voxtral, Qwen3-ASR, Granite, Wav2Vec2, Moonshine, OmniASR, FireRedASR, and more).

parakeet.cpp is a C++/ggml port of NVIDIA NeMo Parakeet automatic speech recognition (ASR) models. It supports the tdt, ctc, rnnt and hybrid decoder families as well as cache-aware streaming transcription, and runs on CPU, NVIDIA CUDA, AMD ROCm/HIP, Intel SYCL and NVIDIA Jetson (L4T) targets.

NVIDIA NeMo-Speech.cpp, a C++/ggml runtime for NVIDIA Nemotron Speech models. One backend serves four model families, selected automatically from the GGUF general.architecture key: automatic speech recognition (offline, cache-aware streaming and live transcription, with optional Silero VAD, punctuation, inverse text normalization and Sortformer speaker diarization attached), standalone Sortformer diarization, MagpieTTS text-to-speech over NanoCodec, and Riva-Translate text translation. Runs on CPU, NVIDIA CUDA, Vulkan, NVIDIA Jetson (L4T) and Apple Metal.

moss-transcribe.cpp is a C++/ggml port of OpenMOSS MOSS-Transcribe-Diarize, an end-to-end audio understanding model. It does joint long-form transcription, speaker diarization and timestamping in a single pass, emitting a speaker-labelled, time-aligned transcript, and runs on CPU, NVIDIA CUDA, AMD ROCm/HIP, Intel SYCL, Vulkan, Apple Metal and NVIDIA Jetson (L4T) targets.

CED sound-event classification / audio tagging (527-class AudioSet). ced.cpp is a C++/ggml port that performs audio tagging over the AudioSet taxonomy, exposed through the SoundDetection gRPC rpc and the /v1/audio/classification REST endpoint. It runs on CPU, NVIDIA CUDA, AMD ROCm/HIP, Intel SYCL, Vulkan and NVIDIA Jetson (L4T) targets.

voice-detect speaker recognition and voice analysis. voice-detect.cpp is a C++/ggml engine that produces L2-normalised speaker embeddings (ECAPA-TDNN, WeSpeaker ResNet34, 3D-Speaker ERes2Net, CAM++) for voice verification and 1:N identification, plus a wav2vec2 age / gender / emotion analysis head. It replaces the Python speaker-recognition backend and is exposed through the Voice* gRPC rpcs and the /v1/voice/* REST endpoints. It runs on CPU, NVIDIA CUDA, AMD ROCm/HIP, Intel SYCL, Vulkan and NVIDIA Jetson (L4T) targets.

face-detect face detection, embedding, verification and analysis. face-detect.cpp is a C++/ggml engine that runs SCRFD / YuNet face detection and ArcFace / SFace 512-d (or 128-d) L2-normalised face embeddings for verification and 1:N identification, plus a landmark / age / gender analysis head. It replaces the Python insightface backend and is exposed through the Embedding, Detect and Face* gRPC rpcs and the /v1/face/* REST endpoints. It runs on CPU, NVIDIA CUDA, AMD ROCm/HIP, Intel SYCL, Vulkan and NVIDIA Jetson (L4T) targets.

Voxtral Realtime 4B Pure C speech-to-text inference engine

Stable Diffusion and Flux in pure C/C++

TRELLIS.2 image-to-3D generation (GLB meshes with PBR textures) in C++/ggml

RF-DETR is a real-time, transformer-based object detection model architecture developed by Roboflow and released under the Apache 2.0 license. RF-DETR is the first real-time model to exceed 60 AP on the Microsoft COCO benchmark alongside competitive performance at base sizes. It also achieves state-of-the-art performance on RF100-VL, an object detection benchmark that measures model domain adaptability to real world problems. RF-DETR is fastest and most accurate for its size when compared current real-time objection models. RF-DETR is small enough to run on the edge using Inference, making it an ideal model for deployments that need both strong accuracy and real-time performance.

Face recognition backend powered by `insightface` (ONNX Runtime). Provides face verification (/v1/face/verify), face analysis (/v1/face/analyze), face embedding (/v1/embeddings), face detection (/v1/detection), and 1:N identification (/v1/face/{register,identify,forget}). Ships two engines in a single image: one that drives the insightface model packs (buffalo_l/s/m/sc, antelopev2 — non-commercial research use only) and one that drives OpenCV Zoo's YuNet + SFace pair (Apache 2.0 — commercial-safe). Select via `options: ["engine:..."]` in your model YAML, or install one of the ready-made model-gallery entries under the `insightface-*` prefix. The backend image contains only code and Python deps; all model weights are managed by LocalAI's gallery download mechanism.