Contains classes for RoBertaEmbeddings.

Module Contents#



Creates word embeddings using RoBERTa.

class RoBertaEmbeddings(classname='com.johnsnowlabs.nlp.embeddings.RoBertaEmbeddings', java_model=None)[source]#

Creates word embeddings using RoBERTa.

The RoBERTa model was proposed in RoBERTa: A Robustly Optimized BERT Pretraining Approach by Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov. It is based on Google’s BERT model released in 2018.

It builds on BERT and modifies key hyperparameters, removing the next-sentence pretraining objective and training with much larger mini-batches and learning rates.

Pretrained models can be loaded with pretrained() of the companion object:

>>> embeddings = RoBertaEmbeddings.pretrained() \
...     .setInputCols(["document", "token"]) \
...     .setOutputCol("embeddings")

The default model is "roberta_base", if no name is provided. For available pretrained models please see the Models Hub.

For extended examples of usage, see the Examples. To see which models are compatible and how to import them see Import Transformers into Spark NLP 🚀.

Input Annotation types

Output Annotation type




Size of every batch, by default 8


Number of embedding dimensions, by default 768


Whether to ignore case in tokens for embeddings matching, by default True


Max sentence length to process, by default 128


ConfigProto from tensorflow, serialized into byte array.


  • RoBERTa has the same architecture as BERT, but uses a byte-level BPE as a tokenizer (same as GPT-2) and uses a different pretraining scheme.

  • RoBERTa doesn’t have token_type_ids, you don’t need to indicate which token belongs to which segment. Just separate your segments with the separation token tokenizer.sep_token (or </s>)


RoBERTa: A Robustly Optimized BERT Pretraining Approach

Paper Abstract:

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

Source of the original code: RoBERTa: A Robustly Optimized BERT Pretraining Approach on GitHub.


>>> import sparknlp
>>> from sparknlp.base import *
>>> from sparknlp.annotator import *
>>> from import Pipeline
>>> documentAssembler = DocumentAssembler() \
...     .setInputCol("text") \
...     .setOutputCol("document")
>>> tokenizer = Tokenizer() \
...     .setInputCols(["document"]) \
...     .setOutputCol("token")
>>> embeddings = RoBertaEmbeddings.pretrained() \
...     .setInputCols(["document", "token"]) \
...     .setOutputCol("embeddings") \
...     .setCaseSensitive(True)
>>> embeddingsFinisher = EmbeddingsFinisher() \
...     .setInputCols(["embeddings"]) \
...     .setOutputCols("finished_embeddings") \
...     .setOutputAsVector(True) \
...     .setCleanAnnotations(False)
>>> pipeline = Pipeline() \
...     .setStages([
...       documentAssembler,
...       tokenizer,
...       embeddings,
...       embeddingsFinisher
...     ])
>>> data = spark.createDataFrame([["This is a sentence."]]).toDF("text")
>>> result =
>>> result.selectExpr("explode(finished_embeddings) as result").show(5, 80)
|                                                                          result|

Sets configProto from tensorflow, serialized into byte array.


ConfigProto from tensorflow, serialized into byte array

static loadSavedModel(folder, spark_session, use_openvino=False)[source]#

Loads a locally saved model.


Folder of the saved model


The current SparkSession

use_openvino: bool

Use OpenVINO backend


The restored model

static pretrained(name='roberta_base', lang='en', remote_loc=None)[source]#

Downloads and loads a pretrained model.

namestr, optional

Name of the pretrained model, by default “roberta_base”

langstr, optional

Language of the pretrained model, by default “en”

remote_locstr, optional

Optional remote address of the resource, by default None. Will use Spark NLPs repositories otherwise.


The restored model