Changes from previous version
Moves from a dense 7B model to a sparse 46.7B MoE using 12.9B active parameters per token after Mistral 7B.
Release Summary
Mixtral 8x7B introduced an Apache 2.0 sparse mixture-of-experts model with 32K context, multilingual support, and strong code performance.
Timeline
December 11, 2023
Mixtral 8x7B released
Mistral publishes its first sparse mixture-of-experts model.