DBRX
Databricks · March 2024
● activeOpen Sourcemixture of expertstext
Parameters132B (36B active)
Context Window32K tokens
Why It Matters
Enterprise-grade open MoE model from Databricks — proved that data companies could build competitive LLMs with their proprietary data pipeline expertise.
Description
An enterprise-grade open-source Mixture-of-Experts model from Databricks, with 132 billion total parameters but only 36 billion active at any time (since MoE models only activate the most relevant 'expert' sub-networks for each input). Built leveraging Databricks' proprietary data pipeline expertise, it outperformed LLaMA 2 70B and Mixtral on many benchmarks while being more efficient to run.
Key Innovations
MoE
MoEArchitecture where only a fraction of the model's parameters are active for each input, allowing massive scale with lower compute.
Open Weight
Open WeightModel weights are publicly released but training data/code may not be. Enables fine-tuning but not full reproduction.
External Links
More from Community / Uncensored
WizardLM2023-06 · 13B
Dolphin (Eric Hartford)2023-07 · —
Hermes (Nous Research)2023-09 · —
LLaMA 4 Scout Abliterated2025-05 · 81B (17B active × 16 experts)
DeepSeek R1 Uncensored2025-03 · 671B (37B active)
Bark2023-04 · —
Mistral 7B Uncensored2024-01 · 7B
Qwen 2.5 72B Abliterated2025-01 · 72B
SOLAR 10.7B Uncensored2024-02 · 10.7B
LLaMA 3.1 405B Abliterated2024-08 · 405B