Model Gallery

7 models from 1 repositories

Filter by type:

Filter by tags:

qwen3-4b-vllm-cpp
Qwen3-4B on vllm.cpp, in bf16. The small end of the engine's gated dense family, which reaches parity with vLLM on every axis at concurrency 1. bf16 rather than NVFP4 on purpose: this is the entry that runs where the flagship NVFP4 checkpoints cannot, including Apple Silicon via Metal, Vulkan and plain CPU. Roughly 8 GB of weights, plus about 4.5 GB of KV cache at the context configured here. Tool calling and the thinking split are parsed inside the engine.

Repository: localaiLicense: apache-2.0

qwen3-tts-llamacpp
Qwen3-TTS 1.7B Base served by the llama.cpp backend, using upstream's own GGUF conversion. Runs on the full llama-cpp accelerator matrix (CUDA, ROCm, SYCL, Vulkan, Metal). Streaming output and zero-shot voice cloning: set `voice` to a reference clip or a saved Voice Library profile, which is required since the Base checkpoint has no built-in speaker. 24kHz mono, 10 languages. Q8_0 backbone (~1.8 GB) plus a Q8_0 projector.

Repository: localaiLicense: apache-2.0

deepseek-v4-flash-q2
DeepSeek V4 Flash (IQ2XXS GGUF, ~81 GB) - only loadable via the ds4 backend. Requires >=128 GB RAM. Metal (Darwin) or CUDA (Linux). See https://github.com/antirez/ds4 for details.

Repository: localai

deepseek-v4-flash-q2-q4
DeepSeek V4 Flash (mixed q2/q4 GGUF, ~91 GB) - only loadable via the ds4 backend. The last 6 expert layers are kept at Q4_K (the rest IQ2XXS), trading a little extra memory for higher quality than the pure-q2 build while still fitting in RAM on a 128 GB machine. imatrix-tuned. Metal (Darwin) or CUDA (Linux). See https://github.com/antirez/ds4 for details.

Repository: localai

deepseek-v4-flash-q4-ssd
DeepSeek V4 Flash (full 4-bit experts GGUF, ~153 GB) - only loadable via the ds4 backend, with SSD streaming enabled so it runs on a 128 GB machine even though the weights do not fit in RAM: routed MoE experts stream from the GGUF on SSD while the non-routed weights stay resident. SSD streaming is Metal (Darwin) only; generation speed depends on SSD speed and the expert cache. Tune the routed-expert cache with the 'ssd_streaming_cache_experts:NGB' option (default: automatic budget). See https://github.com/antirez/ds4.

Repository: localai

deepseek-v4-flash-q2-mtp
DeepSeek V4 Flash (IQ2XXS GGUF, ~81 GB) paired with the optional MTP speculative-decoding weights (~3.5 GB) for a slight speedup. Only loadable via the ds4 backend; requires >=128 GB RAM. MTP helps only with greedy decoding (temperature 0), so the override pins temperature to 0. Metal (Darwin) or CUDA (Linux). See https://github.com/antirez/ds4 for details.

Repository: localai

deepseek-v4-pro-q2-ssd
DeepSeek V4 Pro (IQ2XXS GGUF, ~433 GB, imatrix-tuned) - only loadable via the ds4 backend, with SSD streaming so the Pro-class model can be run on a 128 GB machine. This is experimental and slow: it needs ~433 GB of free SSD plus enough RAM for the resident weights, KV cache, and routed-expert cache, and is best used with thinking off for inspection or occasional work. SSD streaming is Metal (Darwin) only. See https://github.com/antirez/ds4.

Repository: localai