Mixtral of Experts
Mistral AI continues its mission to deliver the best open models to the developer community. Moving forward in AI requires taking new technological turns beyond reusing well-known architectures and training paradigms. Most importantly, it requires making the community benefit from original models to foster new inventions and usages.
TL;DR
- Mixtral 8x7B is a new open-weight sparse mixture of experts (SMoE) model released by Mistral AI under the Apache 2.0 license.
- It outperforms Llama 2 70B on most benchmarks and matches or exceeds GPT-3.5 performance, with 6x faster inference.
- The model has 46.7B total parameters but uses only 12.9B parameters per token, balancing performance with efficiency.
- Capabilities include handling 32k tokens, supporting English, French, Italian, German, and Spanish, strong code generation, and achieving an 8.3 score on MT-Bench when fine-tuned for instruction following.
- Mixtral shows less bias on the BBQ benchmark and more positive sentiments on BOLD compared to Llama 2.
- Mistral AI is also offering Mixtral 8x7B Instruct, optimized for instruction following, and the model can be deployed using an open-source stack with vLLM and Skypilot.