| --- |
| datasets: HuggingFaceFW/fineweb-edu |
| widget: |
| - example_title: Example interaction |
| text: During photosynthesis in green plants |
| inference: |
| parameters: |
| repetition_penalty: 1.3 |
| language: |
| - en |
| library_name: transformers |
| license: mit |
| --- |
| |
| # Model Card for gpt2-124M-edu-fineweb-10B |
|
|
| A 124M parameter GPT2 model trained with the 10B fineweb-edu dataset using [https://github.com/karpathy/llm.c](https://github.com/karpathy/llm.c) |
|
|
| Training took 20 hours on a single 4090 GPU (limited to 350W), giving the following graphs: |
|
|
|  |
|
|
| ## Training |
|
|
| The training parameters where: |
| ``` |
| ./train_gpt2cu \ |
| -i "dev/data/edu_fineweb10B/edu_fineweb_train_*.bin" \ |
| -j "dev/data/edu_fineweb10B/edu_fineweb_val_*.bin" \ |
| -o log124M \ |
| -e "d12" \ |
| -b 56 -t 1024 \ |
| -d 458752 \ |
| -r 1 \ |
| -z 1 \ |
| -c 0.1 \ |
| -l 0.002 \ |
| -q 0.0 \ |
| -u 700 \ |
| -n 5000 \ |
| -v 250 -s 20000 \ |
| -h 1 |
| ``` |
|
|
| The model has had no further finetuning. |
|
|
| ## Evaluation |
|
|
| Evals using [Eleuther AI Harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/b281b0921b636bc36ad05c0b0b0763bd6dd43463) |
| as described in the [open_llm_leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) |
| and comparing with those published for [openai-community/gpt2](https://huggingface.co/openai-community/gpt2) |
|
|
|  |
|
|
| | Eval Test | Score | |
| | --------- | ----- | |
| | arc_challenge (25 shot) | 24.49 | |
| | gsm8k (5 shot) | 0.08 | |
| | hellaswag (10 shot) | 32.64 | |
| | mmlu (5 shot) | 26.06 | |
| | truthfulqa (0 shot) | 42.45 | |
| | winogrande (5 shot) | 52.17 | |
| | **Overall Score** | **29.65** | |
