sparknlp.annotator.embeddings.roberta_sentence_embeddings#

Classes#

class RoBertaSentenceEmbeddings(classname='com.johnsnowlabs.nlp.embeddings.RoBertaSentenceEmbeddings', java_model=None)[source]#

Sentence-level embeddings using RoBERTa. The RoBERTa model was proposed in RoBERTa: A Robustly Optimized BERT Pretraining Approach by Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov. It is based on Google’s BERT model released in 2018. It builds on BERT and modifies key hyperparameters, removing the next-sentence pretraining objective and training with much larger mini-batches and learning rates. Pretrained models can be loaded with pretrained of the companion object:

Pretrained models can be loaded with pretrained() of the companion object:

>>> embeddings = RoBertaSentenceEmbeddings.pretrained() \
...    .setInputCols(["sentence"]) \
...    .setOutputCol("sentence_embeddings")

The default model is "sent_roberta_base", if no name is provided.

For available pretrained models please see the Models Hub.

Input Annotation types	Output Annotation type
DOCUMENT	SENTENCE_EMBEDDINGS

Parameters:

batchSize

Size of every batch, by default 8

caseSensitive

Whether to ignore case in tokens for embeddings matching, by default False

dimension

Number of embedding dimensions, by default 768

maxSentenceLength

Max sentence length to process, by default 128

configProtoBytes

ConfigProto from tensorflow, serialized into byte array.

References

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Paper abstract:

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

Examples

>>> import sparknlp
>>> from sparknlp.base import *
>>> from sparknlp.annotator import *
>>> from pyspark.ml import Pipeline
>>> documentAssembler = DocumentAssembler() \
...     .setInputCol("text") \
...     .setOutputCol("document")
>>> sentence = SentenceDetector() \
...     .setInputCols(["document"]) \
...     .setOutputCol("sentence")
>>> embeddings = RoBertaSentenceEmbeddings.pretrained() \
...     .setInputCols(["sentence"]) \
...     .setOutputCol("sentence_embeddings")
>>> embeddingsFinisher = EmbeddingsFinisher() \
...     .setInputCols(["sentence_embeddings"]) \
...     .setOutputCols("finished_embeddings") \
...     .setOutputAsVector(True)
>>> pipeline = Pipeline().setStages([
...     documentAssembler,
...     sentence,
...     embeddings,
...     embeddingsFinisher
... ])
>>> data = spark.createDataFrame([["John loves apples. Mary loves oranges. John loves Mary."]]).toDF("text")
>>> result = pipeline.fit(data).transform(data)
>>> result.selectExpr("explode(finished_embeddings) as result").show(5, 80)
+--------------------------------------------------------------------------------+
|                                                                          result|
+--------------------------------------------------------------------------------+
|[-0.8951074481010437,0.13753940165042877,0.3108254075050354,-1.65693199634552...|
|[-0.6180210709571838,-0.12179657071828842,-0.191165953874588,-1.4497021436691...|
|[-0.822715163230896,0.7568016648292542,-0.1165061742067337,-1.59048593044281,...|
+--------------------------------------------------------------------------------+

setConfigProtoBytes(b)[source]#

Sets configProto from tensorflow, serialized into byte array.

Parameters:

bList[int]

ConfigProto from tensorflow, serialized into byte array

static loadSavedModel(folder, spark_session)[source]#

Loads a locally saved model.

Parameters:

folderstr

Folder of the saved model

spark_sessionpyspark.sql.SparkSession

The current SparkSession

Returns:

BertSentenceEmbeddings

The restored model

static pretrained(name='sent_roberta_base', lang='en', remote_loc=None)[source]#

Downloads and loads a pretrained model.

Parameters:

namestr, optional

Name of the pretrained model, by default “sent_roberta_base”

langstr, optional

Language of the pretrained model, by default “en”

remote_locstr, optional

Optional remote address of the resource, by default None. Will use Spark NLPs repositories otherwise.

Returns:

RoBertaSentenceEmbeddings

The restored model