Filter by type:
Filter by tags:
carbon-8b-q4
Carbon-8B is the largest model in Hugging Face's Carbon family of genomic
foundation models. It targets DNA and RNA sequence generation, recovery,
variant-effect prediction, and motif-perturbation analysis with a native
context length of 32,768 hybrid 6-mer DNA tokens.
This default entry uses the Q4_K_M GGUF. A higher-quality Q8_0 build is
available as a variant. Prefix DNA sequences with `` and use uppercase
A, C, G, and T characters in groups of six.
Repository: localaiLicense: apache-2.0
nemotron-3-embed-8b-q4
Nemotron-3-Embed-8B is NVIDIA's larger multilingual text embedding model
for retrieval, semantic search, and RAG. This Q4_K_M GGUF balances retrieval
quality with local resource use and supports 36 languages. Prefix retrieval
queries with `query: ` and documents with `passage: `.
Repository: localaiLicense: openmdw-1.1
qwen3.6-35b-a3b-genesis-hermes-v7-q8-k-p
Qwen3.6-35B-A3B Genesis Hermes V7 in the high-quality Q8_K_P GGUF format,
with the shared F16 multimodal projector. This is the largest non-MTP build
in the published V7 set.
Repository: localaiLicense: apache-2.0
ternary-bonsai-8b-q2-g64
Ternary Bonsai 8B (PrismML), GGUF Q2_0 with group-64 packing (each FP16 scale shared
across 64 weights instead of 128). Slightly larger (~2.31 GB) but matches llama.cpp's
native 64-value Q2_0 block layout. Runs on LocalAI's `bonsai` backend. License: Apache 2.0.
Repository: localaiLicense: apache-2.0
lfm2.5-1.2b-instruct
Try LFM • Docs • LEAP • Discord
# LFM2.5-1.2B-Instruct
LFM2.5 is a new family of hybrid models designed for **on-device deployment**. It builds on the LFM2 architecture with extended pre-training and reinforcement learning.
- **Best-in-class performance**: A 1.2B model rivaling much larger models, bringing high-quality AI to your pocket.
- **Fast edge inference**: 239 tok/s decode on AMD CPU, 82 tok/s on mobile NPU. Runs under 1GB of memory with day-one support for llama.cpp, MLX, and vLLM.
- **Scaled training**: Extended pre-training from 10T to 28T tokens and large-scale multi-stage reinforcement learning.
Find more information about LFM2.5 in our blog post.
## 🗒️ Model Details
LFM2.5-1.2B-Instruct is a general-purpose text-only model with the following features:
...
Repository: localaiLicense: other
qwen3.6-27b-dflash
Qwen3.6-27B (dense) paired with its DFlash block-diffusion drafter for speculative decoding on the llama.cpp backend. DFlash gives its largest speedups on dense targets like this one.
DFlash produces a whole block of draft tokens in a single forward pass and injects the target model's hidden states into the drafter's attention, which keeps the drafter tiny while making drafting GPU-friendly. The Q4_K_M file carries the full Qwen3.6-27B target; the ~1.8 GB Q8_0 drafter (`draft-dflash`) accelerates generation without changing the target's outputs. The drafter is not a standalone chat model: it only runs paired with the target, which is why both are bundled here.
Flash attention is required for DFlash and is enabled in this config. A GPU is recommended.
License: Apache 2.0.
Repository: localaiLicense: apache-2.0
gemma-4-e2b-it-qat-q4_0
Gemma 4 E2B is a multimodal (text + image) instruction-tuned model from Google DeepMind, optimized with Quantization-Aware Training (QAT) to preserve bfloat16-level quality at a fraction of the memory. E2B is a MatFormer "effective 2B" elastic variant: it carries a larger backbone but runs at an effective 2B-parameter footprint, making it well suited to lightweight and on-device deployments. This is the official Google Q4_0 GGUF, shipped with its multimodal projector.
License: Apache 2.0 | Authors: Google DeepMind
Repository: localaiLicense: apache-2.0
gemma-4-26b-a4b-it-qat-q4_0
Gemma 4 26B-A4B is a multimodal (text + image) instruction-tuned Mixture-of-Experts model from Google DeepMind, optimized with Quantization-Aware Training (QAT) to preserve bfloat16-level quality at a fraction of the memory. With 26B total parameters and ~4B active per token, it delivers large-model quality at a much lower inference cost. This is the official Google Q4_0 GGUF, shipped with its multimodal projector.
License: Apache 2.0 | Authors: Google DeepMind
Repository: localaiLicense: apache-2.0
gemma-4-31b-it-qat-q4_0
Gemma 4 31B is the largest dense multimodal (text + image) instruction-tuned model in the Gemma 4 family from Google DeepMind, optimized with Quantization-Aware Training (QAT) to preserve bfloat16-level quality while dramatically reducing the memory required to load the model. This is the official Google Q4_0 GGUF, shipped with its multimodal projector.
License: Apache 2.0 | Authors: Google DeepMind
Repository: localaiLicense: apache-2.0
gemma-4-31b-it-qat-mtp
Gemma 4 31B IT QAT (Google DeepMind), the largest dense model in the family, paired with its Multi-Token Prediction (MTP) drafter head for speculative decoding on the llama.cpp backend. The Q4_K_XL target carries the full multimodal (text + image) model; the small `mtp-gemma-4-31B-it` head predicts several tokens ahead which the target verifies in parallel, accelerating generation with no change to output quality. Dense models like 31B are the sizes that benefit most from MTP.
The drafter is not a standalone chat model: it only runs paired with the target, which is why both are bundled here. It uses the upstream `gemma4-assistant` architecture registered by llama.cpp PR #23398, so it loads on stock llama.cpp without any patch.
License: Apache 2.0 | Authors: Google DeepMind (target/drafter checkpoints), Unsloth (GGUF conversion)
Repository: localaiLicense: apache-2.0
lfm2.5-8b-a1b
Try LFM •
Docs •
LEAP •
Discord
# LFM2.5-8B-A1B
LFM2.5 is a new family of hybrid models designed for on-device deployment. It builds on the LFM2 architecture with extended pre-training and reinforcement learning.
- **On-device personal assistant**: Designed to power real-life applications, chaining tool calls, and following complex instructions on all devices.
- **Compressed performance**: Competitive with much larger dense and MoE models on instruction following and agentic tasks.
- **Unmatched throughput**: Fastest in its size class on both CPU and GPU inference, with day-one support for llama.cpp, MLX, vLLM, and SGLang.
Find more information about LFM2.5-8B-A1B in our blog post.
**AA-Omniscience Index (higher is better) rewards correct answers and penalizes hallucinations. Scores range from -100 to 100. See more results on Artificial Analysis.*
## 🗒️ Model Details
LFM2.5-8B-A1B is a general-purpose text-only model with the following features:
...
Repository: localaiLicense: other
qwopus3.5-9b-coder-mtp
# 🌟 Qwopus3.5-9B-v3.5
## 💡 Model Overview & v3.5 Design
Qwopus3.5-9B-v3.5 is a **data-scaled continuation** of the Qwopus3.5-9B-v3 model.
The training data in v3.5 is expanded to cover a broader range of domains, including mathematics, programming, puzzle-solving, multilingual dialogue, instruction-following, multi-turn interactions, and STEM-related tasks.
Qwopus3.5-9B-v3.5 is a reasoning-enhanced model based on **Qwen3.5-9B**, designed for:
- 🧩 Structured reasoning
- 🔧 Tool-augmented workflows
- 🔁 Multi-step agentic tasks
- ⚡ Token-efficient inference
Compared with Qwopus3.5-9B-v3, **3.5 version does not introduce a new architecture, RL stage, or template redesign**.
This version is trained with approximately **2× more SFT data**.
## 🎯 Motivation & Generalization Insight
The motivation behind v3.5 comes from a simple observation:
> This work is motivated by the hypothesis that scaling high-quality SFT data may further enhance the generalization ability of large language models.
In earlier Qwopus3.5 experiments, structured reasoning was observed to improve both **accuracy and efficiency**:
...
Repository: localaiLicense: apache-2.0
nemotron-3-nano-omni-30b-a3b-reasoning-apex
# Model Overview
### Description:
NVIDIA Nemotron 3 Nano Omni is a multimodal large language model that unifies video, audio, image, and text understanding to support enterprise-grade Q&A, summarization, transcription, and document intelligence workflows. It extends the Nemotron Nano family with integrated video+speech comprehension, Graphical User Interface (GUI), Optical Character Recognition (OCR), and speech transcription capabilities, enabling end-to-end processing of rich enterprise content such as meeting recordings, M&E assets, training videos, and complex business documents. NVIDIA Nemotron 3 Nano Omni was developed by NVIDIA as part of the Nemotron model family.
This model is available for commercial use.
This model was improved using Qwen3-VL-30B-A3B-Instruct, Qwen3.5-122B-A10B, Qwen3.5-397B-A17B, Qwen2.5-VL-72B-Instruct, and gpt-oss-120b. For more information, please see the Training Dataset section below.
### License/Terms of Use
Governing Terms: Use of this model is governed by the NVIDIA Open Model Agreement
### Deployment Geography:
Global
...
Repository: localaiLicense: other
gemma-4-31b-it
Google Gemma 4 31B-IT is the largest dense model in the Gemma 4 family with 31B parameters. It handles text and image input, generating text output, with a 256K context window and support for 140+ languages. Provides the highest quality outputs in the Gemma 4 lineup, well-suited for complex reasoning, summarization, and image understanding tasks.
Repository: localaiLicense: apache-2.0
nanbeige4.1-3b-q8
Nanbeige4.1-3B is built upon Nanbeige4-3B-Base and represents an enhanced iteration of our previous reasoning model, Nanbeige4-3B-Thinking-2511, achieved through further post-training optimization with supervised fine-tuning (SFT) and reinforcement learning (RL). As a highly competitive open-source model at a small parameter scale, Nanbeige4.1-3B illustrates that compact models can simultaneously achieve robust reasoning, preference alignment, and effective agentic behaviors.
Key features:
Strong Reasoning: Capable of solving complex, multi-step problems through sustained and coherent reasoning within a single forward pass, reliably producing correct answers on benchmarks like LiveCodeBench-Pro, IMO-Answer-Bench, and AIME 2026 I.
Robust Preference Alignment: Outperforms same-scale models (e.g., Qwen3-4B-2507, Nanbeige4-3B-2511) and larger models (e.g., Qwen3-30B-A3B, Qwen3-32B) on Arena-Hard-v2 and Multi-Challenge.
Agentic Capability: First general small model to natively support deep-search tasks and sustain complex problem-solving with >500 rounds of tool invocations; excels in benchmarks like xBench-DeepSearch (75), Browse-Comp (39), and others.
Repository: localaiLicense: apache-2.0
nanbeige4.1-3b-q4
Nanbeige4.1-3B is built upon Nanbeige4-3B-Base and represents an enhanced iteration of our previous reasoning model, Nanbeige4-3B-Thinking-2511, achieved through further post-training optimization with supervised fine-tuning (SFT) and reinforcement learning (RL). As a highly competitive open-source model at a small parameter scale, Nanbeige4.1-3B illustrates that compact models can simultaneously achieve robust reasoning, preference alignment, and effective agentic behaviors.
Key features:
Strong Reasoning: Capable of solving complex, multi-step problems through sustained and coherent reasoning within a single forward pass, reliably producing correct answers on benchmarks like LiveCodeBench-Pro, IMO-Answer-Bench, and AIME 2026 I.
Robust Preference Alignment: Outperforms same-scale models (e.g., Qwen3-4B-2507, Nanbeige4-3B-2511) and larger models (e.g., Qwen3-30B-A3B, Qwen3-32B) on Arena-Hard-v2 and Multi-Challenge.
Agentic Capability: First general small model to natively support deep-search tasks and sustain complex problem-solving with >500 rounds of tool invocations; excels in benchmarks like xBench-DeepSearch (75), Browse-Comp (39), and others.
Repository: localaiLicense: apache-2.0
nemo-parakeet-tdt-0.6b
NVIDIA NeMo Parakeet TDT 0.6B v3 is an automatic speech recognition (ASR) model from NVIDIA's NeMo toolkit. Parakeet models are state-of-the-art ASR models trained on large-scale English audio data.
Repository: localaiLicense: cc-by-4.0
longcat-video
LongCat-Video served by LocalAI's dedicated CUDA backend. Generates video
from a text prompt or a start image. The SDPA attention path works without
FlashAttention and is suitable for CUDA 13 ARM64 systems such as DGX Spark.
This is a very large checkpoint (roughly 83 GB in Hugging Face storage) and
requires Linux with an NVIDIA CUDA GPU plus substantial memory and disk.
Repository: localaiLicense: mit
longcat-video-avatar-1.5
LongCat-Video-Avatar-1.5 served by LocalAI's dedicated CUDA backend. Turns
speech plus a prompt into an avatar video, optionally conditioning on a
portrait, and continues across multiple segments for longer audio.
Avatar generation also loads tokenizer, text encoder, and VAE components
from LongCat-Video. Plan for very large downloads and substantial NVIDIA
GPU or unified memory; CPU and macOS execution are unsupported.
Repository: localaiLicense: mit
qwen3.6-35b-a3b-nvfp4-vllm-cpp
Qwen3.6-35B-A3B in NVFP4, served by vllm.cpp. A 35B mixture-of-experts model
with roughly 3B parameters active per token, so it reads like a much larger
model while costing about as much per token as a small one.
This is the engine's gated MoE checkpoint: token-for-token identical to vLLM
over the 315-prompt battery on both the synchronous and asynchronous paths,
at 0.92x to 0.97x vLLM's throughput from concurrency 1 to 32.
The architecture is a gated-delta-net hybrid, so automatic prefix caching is
off by default here where it would be on for a dense model. That is the
engine's own default and this entry does not override it.
Needs a Blackwell-class NVIDIA GPU and roughly 23 GB of weights plus KV cache.
Repository: localaiLicense: apache-2.0
qwen3-0.6b-vllm-cpp
Qwen3-0.6B on vllm.cpp, in bf16. Roughly 1.4 GB of weights, which makes it
the cheapest way to confirm a vllm-cpp install actually serves before
committing disk and memory to one of the large checkpoints.
It runs anywhere the backend does, CPU included, and it is a real chat model
rather than a stub, so tool calling and the thinking split can be exercised
on it too.
Repository: localaiLicense: apache-2.0
Page 1