DistilRoBERTa Base Sentence Embeddings(sent_distilroberta_base)

Description

This model is a distilled version of the RoBERTa-base model. It follows the same training procedure as DistilBERT.

The code for the distillation process can be found here. This model is case-sensitive: it makes a difference between english and English.

The model has 6 layers, 768 dimensions, and 12 heads, totalizing 82M parameters (compared to 125M parameters for RoBERTa-base). On average DistilRoBERTa is twice as fast as Roberta-base.

Predicted Entities

Live Demo Open in Colab Download Copy S3 URI

How to use

embeddings = RoBertaSentenceEmbeddings.pretrained("sent_distilroberta_base", "en") \
      .setInputCols("sentence") \
      .setOutputCol("embeddings")

val embeddings = RoBertaSentenceEmbeddings.pretrained("sent_distilroberta_base", "en")
      .setInputCols("sentence")
      .setOutputCol("embeddings")

import nlu
nlu.load("en.embed_sentence.distil_roberta.distilled_base").predict("""Put your text here.""")

Model Information

Data Source

https://huggingface.co/distilroberta-base

Benchmarking

When fine-tuned on downstream tasks, this model achieves the following results:

Glue test results:

| Task | MNLI | QQP  | QNLI | SST-2 | CoLA | STS-B | MRPC | RTE  |
|:----:|:----:|:----:|:----:|:-----:|:----:|:-----:|:----:|:----:|
|      | 84.0 | 89.4 | 90.8 | 92.5  | 59.3 | 88.3  | 86.6 | 67.9 |