KevinBytes Logo
LLM Treeof Life

WizardLM

Microsoft Research / WizardLM Team · June 2023

● activeOpen Weightdecoder onlytext
Parameters13B
Context Window4K tokens
Variants7B, 13B, 70B

Why It Matters

Pioneered the Evol-Instruct technique for automatically creating harder training data, proving that smarter training recipes could close the gap between open and closed models.

Description

Fine-tuned version of LLaMA using a technique called Evol-Instruct, which automatically generates increasingly complex training instructions to teach the model harder tasks. Became one of the first open-source models that could reliably follow user instructions, and served as the foundation for many community-modified variants.

Notable Milestones

  • ▸Basis for dozens of community-modified uncensored models
  • ▸Demonstrated that small open models could match larger ones with better training data

Key Innovations

Instruction Tuning
Instruction TuningFine-tuning a model on instruction-response pairs so it follows user commands more reliably.
Open Weight
Open WeightModel weights are publicly released but training data/code may not be. Enables fine-tuning but not full reproduction.

Family Tree

Built On

LLaMA

Lineage

LLaMA→WizardLM

External Links

Research Paper
PreviousStarCoder
NextAiroboros