🏠 Homepage | 🏆 Leaderboard | 📜 arXiv | 🤗 HF Dataset | 🤗 HF Paper | 𝕏 Twitter
MixEval is a A dynamic benchmark evaluating LLMs using real-world user queries and benchmarks, achieving a 0.96 model ranking correlation with Chatbot Arena and costs around $0.6 to run using GPT-3.5 as a Judge.
You can find more information and access the MixEval leaderboard here.
This is a fork of the original MixEval repository. The original repository can be found here. I created this fork to make the integration and use of MixEval easier during the training of new models. This Fork includes several improved feature to make usages easier and more flexible. Including:
- Evaluation of Local Models during or post trainig with
transformers - Hugging Face Datasets integration to avoid the need of local files.
- Use of Hugging Face TGI or vLLM to accelerate evaluation and making it more manageable
- Improved markdown outputs and timing for the training
- Fixed pip install for remote or CI Integration.
# Fork with more losely dependencies
pip install git+https://github.com/philschmid/MixEval --upgradeNote: If you want to evaluate models that are not included Take a look here. Zephyr example here.
Remote Hugging Face model with existing config:
# MODEL_PARSER_API=<your openai api key
MODEL_PARSER_API=$(echo $OPENAI_API_KEY) python -m mix_eval.evaluate \
--data_path hf://zeitgeist-ai/mixeval \
--model_name zephyr_7b_beta \
--benchmark mixeval_hard \
--version 2024-06-01 \
--batch_size 20 \
--output_dir results \
--api_parallel_num 20Using vLLM/TGI with hosted or local API:
- start you environment
python -m vllm.entrypoints.openai.api_server --model alignment-handbook/zephyr-7b-dpo-full- run the following command
MODEL_PARSER_API=$(echo $OPENAI_API_KEY) API_URL=http://localhost:8000/v1 python -m mix_eval.evaluate \
--data_path hf://zeitgeist-ai/mixeval \
--model_name local_api \
--model_path alignment-handbook/zephyr-7b-dpo-full \
--benchmark mixeval_hard \
--version 2024-06-01 \
--batch_size 20 \
--output_dir results \
--api_parallel_num 20- Results
| Metric | Score |
| --------------------------- | ------- |
| MBPP | 100.00% |
| OpenBookQA | 62.50% |
| DROP | 47.60% |
| BBH | 43.10% |
| MATH | 38.10% |
| PIQA | 37.50% |
| TriviaQA | 37.30% |
| BoolQ | 35.10% |
| CommonsenseQA | 34.00% |
| GSM8k | 33.60% |
| MMLU | 29.00% |
| HellaSwag | 27.90% |
| AGIEval | 26.80% |
| GPQA | 0.00% |
| ARC | 0.00% |
| SIQA | 0.00% |
| overall score (final score) | 34.85% |
Total time: 398.0534451007843Takes around 5 minutes to evaluate.
Local Hugging Face model from path:
# MODEL_PARSER_API=<your openai api key>
MODEL_PARSER_API=$(echo $OPENAI_API_KEY) python -m mix_eval.evaluate \
--data_path hf://zeitgeist-ai/mixeval \
--model_path my/local/path \
--output_dir results/agi-5 \
--model_name local_chat \
--benchmark mixeval_hard \
--version 2024-06-01 \
--batch_size 20 \
--api_parallel_num 20Remote Hugging Face model without config and defaults
Note: We use the model name local_chat to avoid the need for a config file and load it from the Hugging Face model hub.
# MODEL_PARSER_API=<your openai api key>
MODEL_PARSER_API=$(echo $OPENAI_API_KEY) python -m mix_eval.evaluate \
--data_path hf://zeitgeist-ai/mixeval \
--model_path alignment-handbook/zephyr-7b-sft-full \
--output_dir results/handbook-zephyr \
--model_name local_chat \
--benchmark mixeval_hard \
--version 2024-06-01 \
--batch_size 20 \
--api_parallel_num 20