Description
The DeBERTa model was proposed in [[https://arxiv.org/abs/2006.03654 DeBERTa: Decoding-enhanced BERT with Disentangled Attention]] by Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen It is based on Google’s BERT model released in 2018 and Facebook’s RoBERTa model released in 2019. Compared to RoBERTa-Large, a DeBERTa model trained on half of the training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9% (90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%).
Predicted Entities
How to use
embeddings = DeBertaEmbeddings.pretrained("deberta_v3_large", "en") \
.setInputCols("sentence", "token") \
.setOutputCol("embeddings")
val embeddings = DeBertaEmbeddings.pretrained("deberta_v3_large", "en")
.setInputCols("sentence", "token")
.setOutputCol("embeddings")
import nlu
nlu.load("en.embed.deberta_v3_large").predict("""Put your text here.""")
Model Information
| Model Name: | deberta_v3_large |
| Compatibility: | Spark NLP 3.4.2+ |
| License: | Open Source |
| Edition: | Official |
| Input Labels: | [token, sentence] |
| Output Labels: | [embeddings] |
| Language: | en |
| Size: | 1.0 GB |
| Case sensitive: | true |
| Max sentence length: | 128 |
References
https://huggingface.co/microsoft/deberta-v3-large
Benchmarking
#### Fine-tuning on NLU tasks
dev results on SQuAD 2.0 and MNLI tasks.
| Model |Vocabulary(K)|Backbone #Params(M)| SQuAD 2.0(F1/EM) | MNLI-m/mm(ACC)|
|-------------------|----------|-------------------|-----------|----------|
| RoBERTa-large |50 |304 | 89.4/86.5 | 90.2 |
| XLNet-large |32 |- | 90.6/87.9 | 90.8 |
| DeBERTa-large |50 |- | 90.7/88.0 | 91.3 |
| **DeBERTa-v3-large**|128|304 | **91.5/89.0**| **91.8/91.9**|
PREVIOUSDeBERTa base model