You can now run DeepSeek-V3.1 on your local device!

Unlocking Local Deployment of DeepSeek-V3.1: A Guide for Advanced Users

In an exciting development for AI enthusiasts and developers, DeepSeek-V3.1, a sophisticated language model, is now accessible for local deployment. Leveraging innovative quantization techniques, it can run efficiently on high-end hardware setups, offering greater control and customization.

Overview

DeepSeek-V3.1 has been optimized to operate on systems equipped with approximately 170GB of RAM. This breakthrough is achieved through dynamic 1-bit GGUFs, which reduce the original model size from a hefty 715GB to just 170GB—an impressive 80% size reduction. This is made possible by intelligently quantizing individual layers, balancing model fidelity with hardware constraints.

Availability

The latest dynamic imatrix GGUFs for DeepSeek-V3.1 are now available for download at Unsloth’s Hugging Face repository. Additionally, a simplified version labeled TQ1_0, optimized for compatibility with Ollama, is provided in a single file, facilitating seamless integration via the ollama run command:

hf.co/unsloth/DeepSeek-V3.1-GGUF:TQ1_0

Technical Details

The quantization process employs a layered approach:

• Key layers, deemed critical, utilize higher bit representations (6-8 bits) to preserve performance.
• Less influential layers are quantized down to reduce overall size.

This meticulous approach was calibrated using over 2–3 million high-quality tokens, ensuring the model maintains accuracy despite aggressive compression.

Operational Guidelines

To maximize the effectiveness of DeepSeek-V3.1, consider the following configurations:

• Enable the appropriate chat template with the --jinja flag.
• Activate advanced thinking modes via enable_thinking = True or thinking = True.
• For compatibility with existing frameworks and to avoid errors like:

terminate called after throwing an instance of 'std::runtime_error' what(): split method must have between 1 and 1 positional arguments and between 0 and 0 keyword arguments at row 3, column 1908

ensure you are using the latest quantization versions, as these issues have been addressed.

Recommended Settings

• Set temperature and sampling parameters: --temp 0.6 --top_p 0.95.
• To offload Mixture of Experts (MoE) layers into RAM and improve performance, include:

-ot ".ffn_.*_exps.=CPU"

• For extended context lengths, utilize KV cache quantization by specifying:

--cache-type-k q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1

Note: To support V quantization features, compile llama.cpp with Flash Attention enabled.

Further Resources

Comprehensive documentation detailing setup procedures and advanced configurations can be found at DeepSeek-V3.1 Documentation. For optimal results, we recommend using Q2_K_XL or Q3_K_XL quantization modes, which have demonstrated excellent performance.

Conclusion

The ability to run DeepSeek-V3.1 locally marks a significant milestone in AI deployment flexibility, empowering users with greater control, security, and customization. By adopting these advanced quantization techniques and configurations, developers can harness the full potential of this powerful language model on their own hardware.

Interested users and developers are encouraged to explore the documentation and experiment with different settings to tailor the model to their specific needs.