Instructions to use Envoid/Dendrite-L3-10B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Envoid/Dendrite-L3-10B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Envoid/Dendrite-L3-10B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Envoid/Dendrite-L3-10B")
model = AutoModelForCausalLM.from_pretrained("Envoid/Dendrite-L3-10B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use Envoid/Dendrite-L3-10B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Envoid/Dendrite-L3-10B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Envoid/Dendrite-L3-10B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Envoid/Dendrite-L3-10B

SGLang

How to use Envoid/Dendrite-L3-10B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Envoid/Dendrite-L3-10B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Envoid/Dendrite-L3-10B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Envoid/Dendrite-L3-10B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Envoid/Dendrite-L3-10B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Envoid/Dendrite-L3-10B with Docker Model Runner:
```
docker model run hf.co/Envoid/Dendrite-L3-10B
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

This model is experimental and thus results cannot be gauranteed.

Dendrite-L3-10B

In a similar vein to Libra-19B this model was created by taking all of the layers of one model and stacking along with them the first number of layers (8 in this case) from a donor model but in the reverse order.

In this case the base model used was Poppy_Porpoise-DADA-8B and the donor model used was Llama-3-8B-Instruct-DADA

It was then finetuned for 10 epochs on the Dendrite dataset at a low learning rate to repair the disorder and integrate the donor layers.

The following mergekit config was used:

slices:
  - sources:
    - model: ./Poppy_Porpoise-DADA-8B
      layer_range: [0, 32]
  - sources:
    - model: ./Llama-3-8B-Instruct-DADA
      layer_range: [7, 8]
  - sources:
    - model: ./Llama-3-8B-Instruct-DADA
      layer_range: [6, 7]
  - sources:
    - model: ./Llama-3-8B-Instruct-DADA
      layer_range: [5, 6]
  - sources:
    - model: ./Llama-3-8B-Instruct-DADA
      layer_range: [4, 5]
  - sources:
    - model: ./Llama-3-8B-Instruct-DADA
      layer_range: [3, 4]
  - sources:
    - model: ./Llama-3-8B-Instruct-DADA
      layer_range: [2, 3]
  - sources:
    - model: ./Llama-3-8B-Instruct-DADA
      layer_range: [1, 2]
  - sources:
    - model: ./Llama-3-8B-Instruct-DADA
      layer_range: [0, 1]
merge_method: passthrough
dtype: float16

Unlike in the case of Libra-19B this models moral alignment seems very much intact.

In order to get the best results from this model you should uncheck "skip special tokens" on your front-end and add "<|eot_id|>" to your custom stopping strings.

It has been tested with a number of different Llama-3 prompt templates and seems to work well.

It regained its base assistant personality during the retraining process, however, using assistant style prompt templates and assistant cards in SillyTavern gives it fairly interesting replies.

It has been tested in RP, assistant and creative writing use cases and at a quick glance seems to work well.

Training was done using qlora-pipe

exl2 RPCAL care of Qaunt Cartel

GGUFs care of Quant Cartel

Downloads last month: 1

Safetensors

Model size

10B params

Tensor type

BF16

Model tree for Envoid/Dendrite-L3-10B

Quantizations

2 models