Description
This model is a distilled version of the BERT base model. It was introduced in this paper. The code for the distillation process can be found here. This model is cased: it does make a difference between english and English.
How to use
embeddings = DistilBertEmbeddings.pretrained("distilbert_base_cased", "en") \
.setInputCols("sentence", "token") \
.setOutputCol("embeddings")
nlp_pipeline = Pipeline(stages=[document_assembler, sentence_detector, tokenizer, embeddings])
val embeddings = DistilBertEmbeddings.pretrained("distilbert_base_cased", "en")
.setInputCols("sentence", "token")
.setOutputCol("embeddings")
val pipeline = new Pipeline().setStages(Array(document_assembler, sentence_detector, tokenizer, embeddings))
import nlu
nlu.load("en.embed.distilbert").predict("""Put your text here.""")
Model Information
| Model Name: | distilbert_base_cased |
| Compatibility: | Spark NLP 3.1.0+ |
| License: | Open Source |
| Edition: | Official |
| Input Labels: | [token, sentence] |
| Output Labels: | [embeddings] |
| Language: | en |
| Case sensitive: | true |
Data Source
https://huggingface.co/distilbert-base-cased
Benchmarking
When fine-tuned on downstream tasks, this model achieves the following results:
Glue test results:
| Task | MNLI | QQP | QNLI | SST-2 | CoLA | STS-B | MRPC | RTE |
|:----:|:----:|:----:|:----:|:-----:|:----:|:-----:|:----:|:----:|
| | 81.5 | 87.8 | 88.2 | 90.4 | 47.2 | 85.5 | 85.6 | 60.6 |