Error: "Some weights of the model checkpoint were not used"
#4
by
EvokerKing
- opened
Code:
from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModelForMaskedLM.from_pretrained("bert-base-uncased")
Error:
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']
- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
EvokerKing
changed discussion status to
closed
EvokerKing
changed discussion status to
open
Same issue
Hello! This is a warning, not an error. It tells you that by loading the bert-base-uncased checkpoint in the BertForMaskedLM architecture, you're dropping two weights: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias'].
These are the weights used for next-sentence prediction, which aren't necessary for Masked Language Modeling.
If you're only interested in doing masked language modeling, then you can safely disregard this warning.
