Failure to reproduce QA Format response from the README
The current README (https://huggingface.co/microsoft/phi-1_5/blob/914c8fb3c681ebe3cacbe3c748858a572283ddde/README.md) poses the QA format.
Trying to reproduce the response, I get nowhere close to what the README says (see output below). What am I missing?
# With transformers==4.36.2 and tokenizers==0.15.0
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL_NAME = "microsoft/phi-1_5"
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForCausalLM.from_pretrained(MODEL_NAME)
generation = model.generate(
**tokenizer(
"Write a detailed analogy between mathematics and a lighthouse.\n\nAnswer:",
return_tensors="pt",
),
max_length=30,
do_sample=True,
)
print(tokenizer.batch_decode(generation, skip_special_tokens=True))
Running this prints:
['Write a detailed analogy between mathematics and a lighthouse.\n\nAnswer:\n\n\n\n\n\n\n\n\n\n']
model = AutoModelForCausalLM.from_pretrained(MODEL_NAME)
At this point transformers 4.36.2 should print tons of warnings about keys mismatch: you are using built-in version of phi from 4.36.2 which is not compatible with weights in this phi-1.5 repo.
Either force transformers to load the code from this repo, or use repo with compatible version (see https://github.com/huggingface/transformers/issues/28416 for example) or do torch.load to load weights manually and poke them with a pointy stick until they become suitable to be loaded with load_state_dict
Hello @jamesbraza !
We just pushed a fix to the config.json and it should work now. However, as per the remark on the model card:
If you are using transformers<4.37.0, always load the model with trust_remote_code=True to prevent side-effects.
Best regards,
Gustavo.