Model Gallery

25 models from 1 repositories

Filter by type:

Filter by tags:

qwen3.8-27b-q4-mtp
Qwen3.8-27B with the official Q4_K_M model and Q4_0 MTP draft model. MTP speculative decoding can increase generation speed by proposing multiple tokens for the target model to verify.

Repository: localaiLicense: apache-2.0

muse-glimmer-30b-dflash
Muse Glimmer's higher-quality dynamic K-quant GGUF with the official quantized perception encoder and DFlash drafter. DFlash proposes blocks of up to 16 tokens for the target to verify in parallel, accelerating output without changing model quality. Flash attention is enabled for this path.

Repository: localaiLicense: apache-2.0

muse-glimmer-30b-17gb-dflash
Muse Glimmer's smaller 17 GB K-quant GGUF with the official quantized perception encoder and DFlash drafter. This is the lowest-memory published build that retains image understanding and block-speculative decoding. Flash attention is enabled for the DFlash path.

Repository: localaiLicense: apache-2.0

tess-4-27b-mtp
Tess-4-27B with its Q4_K_M multi-token prediction draft enabled for speculative decoding. The main model verifies every proposed token, and the entry also includes the shared F16 vision projector.

Repository: localaiLicense: apache-2.0

gemmable-4-12b-mtp
## Gemmable 4 12B Gemmable 4 12B is a GGUF export of Gemma 4 12B fine-tuned on Fable-5 style reasoning and assistant traces. ## Highlights - Base model: `google/gemma-4-12B` - Format: GGUF - Training style: Fable-5 style reasoning and assistant traces - Distribution: fp16 GGUF plus matching assistant GGUFs for each quant - Intended use: local inference, coding, reasoning, and assistant workflows ## How to use ### llama.cpp Standard load: ```bash llama-server -m "gemmable-4-12b-fp16.gguf" ``` Speculative / draft-MTP load: ```bash llama-server -m "gemmable-4-12b-Q4_K_M.gguf" \ --spec-draft-model "gemmable-4-12b-Q4_K_M-mtp.gguf" \ --spec-type draft-mtp \ --spec-draft-n-max 4 ``` Use the matching fp16 or quantized main file with its `-mtp` companion. ### LM Studio 1. Search this repo, download target + mtp file. 2. Load target. 3. Load settings → Speculative Decoding → select mtp file file. (Requires a llama.cpp runtime with Gemma 4 MTP support from ggml-org/llama.cpp#23398. LocalAI's pinned llama.cpp backend already carries it, so this entry runs draft-mtp out of the box.) ## GGUF / local inference notes ...

Repository: localai

laguna-s-2.1-dflash
Laguna S 2.1's 96 GB Q4_K_M target paired with Poolside's 2.2 GB BF16 DFlash drafter for speculative decoding. DFlash drafts a block of tokens per forward pass and accelerates generation without changing the target model's outputs. Flash attention is enabled because the DFlash path requires it. License: OpenMDW 1.1.

Repository: localaiLicense: other

qwen3-4b-dflash
Qwen3-4B paired with its DFlash block-diffusion drafter for speculative decoding on the llama.cpp backend. This is the canonical DFlash pairing documented upstream (`z-lab/Qwen3-4B-DFlash` + `Qwen/Qwen3-4B`). DFlash produces a whole block of draft tokens in a single forward pass and injects the target model's hidden states into the drafter's attention, which keeps the drafter tiny while making drafting GPU-friendly. The Q4_K_M file carries the full Qwen3-4B target; the ~0.5 GB Q8_0 drafter (`draft-dflash`) accelerates generation without changing the target's outputs. The drafter is not a standalone chat model: it only runs paired with the target, which is why both are bundled here. Flash attention is required for DFlash and is enabled in this config. A GPU is recommended. License: Apache 2.0 (Qwen3-4B target) / MIT (z-lab DFlash drafter).

Repository: localaiLicense: apache-2.0

qwen3.5-4b-dflash
Qwen3.5-4B paired with its DFlash block-diffusion drafter for speculative decoding on the llama.cpp backend. DFlash produces a whole block of draft tokens in a single forward pass and injects the target model's hidden states into the drafter's attention, which keeps the drafter tiny while making drafting GPU-friendly. The Q4_K_M file carries the full Qwen3.5-4B target; the ~0.6 GB Q8_0 drafter (`draft-dflash`) accelerates generation without changing the target's outputs. The drafter is not a standalone chat model: it only runs paired with the target, which is why both are bundled here. Flash attention is required for DFlash and is enabled in this config. A GPU is recommended. License: Apache 2.0 (Qwen3.5-4B target) / MIT (z-lab DFlash drafter).

Repository: localaiLicense: apache-2.0

qwen3.5-9b-dflash
Qwen3.5-9B paired with its DFlash block-diffusion drafter for speculative decoding on the llama.cpp backend. DFlash produces a whole block of draft tokens in a single forward pass and injects the target model's hidden states into the drafter's attention, which keeps the drafter tiny while making drafting GPU-friendly. The Q4_K_M file carries the full Qwen3.5-9B target; the ~1 GB Q8_0 drafter (`draft-dflash`) accelerates generation without changing the target's outputs. The drafter is not a standalone chat model: it only runs paired with the target, which is why both are bundled here. Flash attention is required for DFlash and is enabled in this config. A GPU is recommended. License: Apache 2.0 (Qwen3.5-9B target) / MIT (z-lab DFlash drafter).

Repository: localaiLicense: apache-2.0

qwen3.6-27b-dflash
Qwen3.6-27B (dense) paired with its DFlash block-diffusion drafter for speculative decoding on the llama.cpp backend. DFlash gives its largest speedups on dense targets like this one. DFlash produces a whole block of draft tokens in a single forward pass and injects the target model's hidden states into the drafter's attention, which keeps the drafter tiny while making drafting GPU-friendly. The Q4_K_M file carries the full Qwen3.6-27B target; the ~1.8 GB Q8_0 drafter (`draft-dflash`) accelerates generation without changing the target's outputs. The drafter is not a standalone chat model: it only runs paired with the target, which is why both are bundled here. Flash attention is required for DFlash and is enabled in this config. A GPU is recommended. License: Apache 2.0.

Repository: localaiLicense: apache-2.0

qwen3.6-35b-a3b-dflash
Qwen3.6-35B-A3B (Mixture-of-Experts, ~3B active per token) paired with its DFlash block-diffusion drafter for speculative decoding on the llama.cpp backend. DFlash speedups on MoE targets are smaller than on dense models, but still useful. DFlash produces a whole block of draft tokens in a single forward pass and injects the target model's hidden states into the drafter's attention, which keeps the drafter tiny while making drafting GPU-friendly. The UD-Q4_K_M file carries the full Qwen3.6-35B-A3B target; the ~0.4 GB Q8_0 drafter (`draft-dflash`) accelerates generation without changing the target's outputs. The drafter is not a standalone chat model: it only runs paired with the target, which is why both are bundled here. Flash attention is required for DFlash and is enabled in this config. A GPU is recommended. License: Apache 2.0.

Repository: localaiLicense: apache-2.0

gemma-4-12b-agentic-fable5-composer2.5-v2-3.5x-tau2-mtp
Gemma 4 12B Agentic v2 is an Apache-2.0 fine-tune for coding, terminal work, multi-step tool use, and reasoning. This variant pairs the Q4_K_M target with the upstream Q8_0 Gemma 4 MTP drafter for faster lossless generation through llama.cpp speculative decoding.

Repository: localaiLicense: apache-2.0

gemma-4-26b-a4b-it-qat
Hugging Face | GitHub | Launch Blog | Documentation License: Apache 2.0 | Authors: Google DeepMind > [!Note] > This model card is for the new versions of the Gemma 4 family optimized with Quantization-Aware Training (QAT), which allows preserving similar quality to bfloat16 while dramatically reducing the memory requirements to load the model. > Four versions of the QAT checkpoints are available: > * **Unquantized QAT checkpoints** (Q4_0): Half-precision weights extracted from the QAT pipeline, ideal for custom downstream compilation and research. Available for Gemma 4 E2B, E4B, 12B, 26B A4B, and 31B, and their drafter models. > * **GGUF** (Q4_0): Ready-to-deploy formats for broad ecosystem compatibility. Available for Gemma 4 E2B, E4B, 12B, 26B A4B, and 31B. > * **Mobile-optimized** (wNa8o8): A custom schema engineered explicitly for mobile hardware efficiency. It features targeted 2-bit decoding layers, optimized KV caches, and static activations to maximize VRAM savings. Available for Gemma 4 E2B and E4B. > * **Compressed Tensors** (w4a16): QAT checkpoints serialized in the compressed-tensors format for native, optimized inference with vLLM. Available for Gemma 4 E2B, E4B, 12B ...

Repository: localaiLicense: apache-2.0

gemma-4-12b-it-qat-q4_0
Hugging Face | GitHub | Launch Blog | Documentation License: Apache 2.0 | Authors: Google DeepMind > [!Note] > This model card is for the new versions of the Gemma 4 family optimized with Quantization-Aware Training (QAT), which allows preserving similar quality to bfloat16 while dramatically reducing the memory requirements to load the model. > Four versions of the QAT checkpoints are available: > * **Unquantized QAT checkpoints** (Q4_0): Half-precision weights extracted from the QAT pipeline, ideal for custom downstream compilation and research. Available for Gemma 4 E2B, E4B, 12B, 26B A4B, and 31B, and their drafter models. > * **GGUF** (Q4_0): Ready-to-deploy formats for broad ecosystem compatibility. Available for Gemma 4 E2B, E4B, 12B, 26B A4B, and 31B. > * **Mobile-optimized** (wNa8o8): A custom schema engineered explicitly for mobile hardware efficiency. It features targeted 2-bit decoding layers, optimized KV caches, and static activations to maximize VRAM savings. Available for Gemma 4 E2B and E4B. > * **Compressed Tensors** (w4a16): QAT checkpoints serialized in the compressed-tensors format for native, optimized inference with vLLM. Available for Gemma 4 E2B, E4B, 12B ...

Repository: localaiLicense: apache-2.0

gemma-4-e2b-it-qat-mtp
Gemma 4 E2B IT QAT (Google DeepMind) paired with its Multi-Token Prediction (MTP) drafter head for speculative decoding on the llama.cpp backend. The Q4_K_XL target carries the full multimodal (text + image) model; the small `mtp-gemma-4-E2B-it` head predicts several tokens ahead which the target verifies in parallel, accelerating generation with no change to output quality. E2B is a MatFormer "effective 2B" elastic variant, well suited to lightweight and on-device deployments. The drafter is not a standalone chat model: it only runs paired with the target, which is why both are bundled here. It uses the upstream `gemma4-assistant` architecture registered by llama.cpp PR #23398, so it loads on stock llama.cpp without any patch. License: Apache 2.0 | Authors: Google DeepMind (target/drafter checkpoints), Unsloth (GGUF conversion)

Repository: localaiLicense: apache-2.0

gemma-4-e4b-it-qat-mtp
Gemma 4 E4B IT QAT (Google DeepMind) paired with its Multi-Token Prediction (MTP) drafter head for speculative decoding on the llama.cpp backend. The Q4_K_XL target carries the full multimodal (text + image) model; the small `mtp-gemma-4-E4B-it` head predicts several tokens ahead which the target verifies in parallel, accelerating generation with no change to output quality. E4B is a MatFormer "effective 4B" elastic variant, balancing quality and footprint for on-device and edge deployments. The drafter is not a standalone chat model: it only runs paired with the target, which is why both are bundled here. It uses the upstream `gemma4-assistant` architecture registered by llama.cpp PR #23398, so it loads on stock llama.cpp without any patch. License: Apache 2.0 | Authors: Google DeepMind (target/drafter checkpoints), Unsloth (GGUF conversion)

Repository: localaiLicense: apache-2.0

gemma-4-12b-it-qat-mtp
Gemma 4 12B IT QAT (Google DeepMind) paired with its Multi-Token Prediction (MTP) drafter head for speculative decoding on the llama.cpp backend. The Q4_K_XL target carries the full multimodal (text + image) model; the small `mtp-gemma-4-12B-it` head predicts several tokens ahead which the target verifies in parallel, accelerating generation with no change to output quality. As a dense model, Gemma 4 12B is among the sizes that benefit most from MTP, with the llama.cpp PR reporting well over 1.4x decode speedup. The drafter is not a standalone chat model: it only runs paired with the target, which is why both are bundled here. It uses the upstream `gemma4-assistant` architecture registered by llama.cpp PR #23398, so it loads on stock llama.cpp without any patch. License: Apache 2.0 | Authors: Google DeepMind (target/drafter checkpoints), Unsloth (GGUF conversion)

Repository: localaiLicense: apache-2.0

gemma-4-31b-it-qat-mtp
Gemma 4 31B IT QAT (Google DeepMind), the largest dense model in the family, paired with its Multi-Token Prediction (MTP) drafter head for speculative decoding on the llama.cpp backend. The Q4_K_XL target carries the full multimodal (text + image) model; the small `mtp-gemma-4-31B-it` head predicts several tokens ahead which the target verifies in parallel, accelerating generation with no change to output quality. Dense models like 31B are the sizes that benefit most from MTP. The drafter is not a standalone chat model: it only runs paired with the target, which is why both are bundled here. It uses the upstream `gemma4-assistant` architecture registered by llama.cpp PR #23398, so it loads on stock llama.cpp without any patch. License: Apache 2.0 | Authors: Google DeepMind (target/drafter checkpoints), Unsloth (GGUF conversion)

Repository: localaiLicense: apache-2.0

qwen3.6-27b-nvfp4-mtp-vllm-cpp
Qwen3.6-27B NVFP4 on vllm.cpp with MTP speculative decoding enabled. MTP (Multi-Token Prediction) drafts from a head that ships inside the target checkpoint's own mtp.* tensors, so there is no second model to download and no extra weights to manage. The verifier accepts roughly 85% of drafted tokens on prose and 92% on code, worth about 1.5x to 1.6x the decode throughput of the same weights with speculation off, and it holds that lead at concurrency 2, 4 and 8. Same weights and same revision pin as qwen3.6-27b-nvfp4-vllm-cpp; install that entry instead if you would rather not spend the extra memory. The speculative state (a doubled recurrent-state slot plus the draft cache and head) costs roughly 3.6 GB on top of the base footprint. MTP here is depth 1 by construction: the engine refuses num_speculative_tokens above 1 for this method.

Repository: localaiLicense: apache-2.0

qwen3.6-27b-nvfp4-dflash-vllm-cpp
Qwen3.6-27B NVFP4 on vllm.cpp with DFlash block-diffusion speculative decoding: the fastest configuration of this model the engine ships. Where MTP drafts one token at a time, DFlash drafts a whole 16-token block in a single non-autoregressive pass from a separate 3.5 GB drafter, then the target verifies the block in one step. At concurrency 1 that measures 2.9x the throughput of the same weights with speculation off, and at or above vLLM's own DFlash-on decode. Both checkpoints are installed for you: the target as a revision-pinned snapshot, the drafter into models/Qwen3.6-27B-DFlash, which is where the backend looks when speculative_config.model names it. The drafter shares the target's embed_tokens and lm_head, so the two are not independently swappable. Needs a Blackwell-class NVIDIA GPU and roughly 28 GB of weights in total.

Repository: localaiLicense: apache-2.0

qwen3.6-35b-a3b-nvfp4-mtp-vllm-cpp
Qwen3.6-35B-A3B NVFP4 on vllm.cpp with MTP speculative decoding enabled. The draft head ships inside the checkpoint's own mtp.* tensors, so there is no second model to download. On this model the speculative path is token-exact against speculation-off on both the synchronous and asynchronous schedulers. Same weights as qwen3.6-35b-a3b-nvfp4-vllm-cpp; install that entry instead if you would rather not spend the extra memory on speculative state. MTP is depth 1 by construction on this engine.

Repository: localaiLicense: apache-2.0

Page 1