| --- |
| language: |
| - sv |
| license: apache-2.0 |
| library_name: transformers |
| tags: |
| - unsloth |
| datasets: |
| - neph1/bellman-7b-finetune |
| - neph1/codefeedback-swedish |
| base_model: |
| - mistralai/Mistral-Nemo-Instruct-2407 |
| --- |
| |
| # Model Card for Bellman |
|
|
| This version of bellman is finetuned from Mistral-Nemo-Instruct-2407. |
| It's a rank 128 qlora trained for about 1 epoch. |
| It's finetuned for prompt question answering, based on a dataset created from |
| Swedish wikipedia, with a lot of Sweden-centric questions. |
| New from previous versions is questions from a translated code-feedback dataset, as well as a number of stories. |
|
|
| Consider this a work in progress as I adjust the training for this new model size. Will provide a few updates to the model. |
|
|
| Quality seems to be dependent on "initial seed". Sometimes it's just amazing, and sometimes quite bad. If you're having trouble getting good results, try lowering temperature. |
|
|
|
|
| For GGUFs, please look to: <br> https://huggingface.co/mradermacher/Mistral-Nemo-Instruct-bellman-12b-GGUF and <br> |
| https://huggingface.co/mradermacher/Mistral-Nemo-Instruct-bellman-12b-i1-GGUF |
|
|
|  |
|
|
| [](https://ko-fi.com/T6T3S8VXY) |
|
|
| ## Model Details |
|
|
| Rank: 128 |
|
|
| Trained with Unsloth on a 3090 |
|
|
| Differences from base model: |
|
|
| The base model is pretty good at Swedish already, but my 'vibe check' says this finetune is slightly improved. There are less weird wordings. |
|
|
| Bellman is trained on fairly short answers and tends to be less verbose. |
|
|
| Output example (Mistral-Nemo-Instruct-bellman-12b.i1-Q4_K_M.gguf): |
|
|
| User: "Hej!" |
|
|
| Bellman: "Hej! Vad kan jag hjälpa dig med?" |
|
|
| User: "Berätta lite om Sverige" |
|
|
| Bellman: "Sverige är ett land i norra Europa med en rik historia. Det består av 21 län och har en yta på drygt 450,000 kvadratkilometer. Stockholm är Sveriges huvudstad." |
|
|
| User: "När grundades det?" |
|
|
| Bellman: "Det är inte exakt känt när Sverige grundades. Det var först under 1300-talet att Sverige började ge sig uttryck i internationella affärer." |
|
|
| User: "Vad hände 1523?" |
|
|
| Bellman: "År 1523 blev Gustav Vasa kung över Sverige och inledde reformationen i landet. Han var också den förste monarken att bära titeln "Sveriges Konung"." |
|
|
| User: "Hur länge levde han?" |
|
|
| Bellman: "Gustav Vasa levde från år 1496 till 1560." |
|
|
| ### Training Parameters |
| per_device_train_batch_size = 2,<br> |
| gradient_accumulation_steps = 64,<br> |
| num_train_epochs=3,<br> |
| warmup_steps = 5,<br> |
| learning_rate = 1e-4,<br> |
| logging_steps = 15,<br> |
| optim = "adamw_8bit",<br> |
| weight_decay = 0.01,<br> |
| lr_scheduler_type = "linear",<br> |
| seed = 3407,<br> |
| per_device_eval_batch_size = 2,<br> |
| evaluation_strategy="steps",<br> |
| eval_accumulation_steps = 64,<br> |
| eval_steps = 15,<br> |
| eval_delay = 0,<br> |
| save_strategy="steps",<br> |
| save_steps=50,<br> |
| |
| ### Model Description |
|
|
|
|
| - **Developed by:** Me |
| - **Funded by:** Me |
| - **Model type:** Instruct |
| - **Language(s) (NLP):** Swedish |
| - **License:** Apache 2 License |
| - **Finetuned from model:** Mistral-Nemo-Instruct-2407 |
|
|
| ## Model Card Contact |
|
|
| rickard@mindemia.com |
