sparknlp.annotator.seq2seq.t5_transformer#

Classes#

class T5Transformer(classname='com.johnsnowlabs.nlp.annotators.seq2seq.T5Transformer', java_model=None)[source]#

T5: the Text-To-Text Transfer Transformer

T5 reconsiders all NLP tasks into a unified text-to-text-format where the input and output are always text strings, in contrast to BERT-style models that can only output either a class label or a span of the input. The text-to-text framework is able to use the same model, loss function, and hyper-parameters on any NLP task, including machine translation, document summarization, question answering, and classification tasks (e.g., sentiment analysis). T5 can even apply to regression tasks by training it to predict the string representation of a number instead of the number itself.

Pretrained models can be loaded with pretrained() of the companion object:

>>> t5 = T5Transformer.pretrained() \
...     .setTask("summarize:") \
...     .setInputCols(["document"]) \
...     .setOutputCol("summaries")

The default model is "t5_small", if no name is provided. For available pretrained models please see the Models Hub.

For extended examples of usage, see the Examples.

Input Annotation types	Output Annotation type
DOCUMENT	DOCUMENT

Parameters:

configProtoBytes

ConfigProto from tensorflow, serialized into byte array.

task

Transformer’s task, e.g. summarize:

minOutputLength

Minimum length of the sequence to be generated

maxOutputLength

Maximum length of output text

doSample

Whether or not to use sampling; use greedy decoding otherwise

temperature

The value used to module the next token probabilities

topK

The number of highest probability vocabulary tokens to keep for top-k-filtering

topP

Top cumulative probability for vocabulary tokens

If set to float < 1, only the most probable tokens with probabilities that add up to topP or higher are kept for generation.

repetitionPenalty

The parameter for repetition penalty. 1.0 means no penalty.

noRepeatNgramSize

If set to int > 0, all ngrams of that size can only occur once

ignoreTokenIds

A list of token ids which are ignored in the decoder’s output

Notes

This is a very computationally expensive module especially on larger sequence. The use of an accelerator such as GPU is recommended.

References

• Exploring Transfer Learning with T5: the Text-To-Text Transfer Transformer

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

• google-research/text-to-text-transfer-transformer

Paper Abstract:

Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new Colossal Clean Crawled Corpus, we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.

Examples

>>> import sparknlp
>>> from sparknlp.base import *
>>> from sparknlp.annotator import *
>>> from pyspark.ml import Pipeline
>>> documentAssembler = DocumentAssembler() \
...     .setInputCol("text") \
...     .setOutputCol("documents")
>>> t5 = T5Transformer.pretrained("t5_small") \
...     .setTask("summarize:") \
...     .setInputCols(["documents"]) \
...     .setMaxOutputLength(200) \
...     .setOutputCol("summaries")
>>> pipeline = Pipeline().setStages([documentAssembler, t5])
>>> data = spark.createDataFrame([[
...     "Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a " +
...     "downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness" +
...     " of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this " +
...     "paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework " +
...     "that converts all text-based language problems into a text-to-text format. Our systematic study compares " +
...     "pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens " +
...     "of language understanding tasks. By combining the insights from our exploration with scale and our new " +
...     "Colossal Clean Crawled Corpus, we achieve state-of-the-art results on many benchmarks covering " +
...     "summarization, question answering, text classification, and more. To facilitate future work on transfer " +
...     "learning for NLP, we release our data set, pre-trained models, and code."
... ]]).toDF("text")
>>> result = pipeline.fit(data).transform(data)
>>> result.select("summaries.result").show(truncate=False)
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|result                                                                                                                                                                                                        |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|[transfer learning has emerged as a powerful technique in natural language processing (NLP) the effectiveness of transfer learning has given rise to a diversity of approaches, methodologies, and practice .]|
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

setIgnoreTokenIds(value)[source]#

A list of token ids which are ignored in the decoder’s output.

Parameters:

valueList[int]

The words to be filtered out

setConfigProtoBytes(b)[source]#

Sets configProto from tensorflow, serialized into byte array.

Parameters:

bList[int]

ConfigProto from tensorflow, serialized into byte array

setTask(value)[source]#

Sets the transformer’s task, e.g. summarize:.

Parameters:

valuestr

The transformer’s task

setMinOutputLength(value)[source]#

Sets minimum length of the sequence to be generated.

Parameters:

valueint

Minimum length of the sequence to be generated

setMaxOutputLength(value)[source]#

Sets maximum length of output text.

Parameters:

valueint

Maximum length of output text

setStopAtEos(b)[source]#

Stop text generation when the end-of-sentence token is encountered.

Parameters:

bbool

whether to stop at end-of-sentence token or not

setMaxNewTokens(value)[source]#

Sets the maximum number of new tokens to be generated

Parameters:

valueint

the maximum number of new tokens to be generated

setDoSample(value)[source]#

Sets whether or not to use sampling, use greedy decoding otherwise.

Parameters:

valuebool

Whether or not to use sampling; use greedy decoding otherwise

setTemperature(value)[source]#

Sets the value used to module the next token probabilities.

Parameters:

valuefloat

The value used to module the next token probabilities

setTopK(value)[source]#

Sets the number of highest probability vocabulary tokens to keep for top-k-filtering.

Parameters:

valueint

Number of highest probability vocabulary tokens to keep

setTopP(value)[source]#

Sets the top cumulative probability for vocabulary tokens.

If set to float < 1, only the most probable tokens with probabilities that add up to topP or higher are kept for generation.

Parameters:

valuefloat

Cumulative probability for vocabulary tokens

setRepetitionPenalty(value)[source]#

Sets the parameter for repetition penalty. 1.0 means no penalty.

Parameters:

valuefloat

The repetition penalty

References

See Ctrl: A Conditional Transformer Language Model For Controllable Generation for more details.

setNoRepeatNgramSize(value)[source]#

Sets size of n-grams that can only occur once.

If set to int > 0, all ngrams of that size can only occur once.

Parameters:

valueint

N-gram size can only occur once

setUseCache(value)[source]#

Cache internal state of the model to improve performance

Parameters:

valuebool

Whether or not to use cache

static loadSavedModel(folder, spark_session)[source]#

Loads a locally saved model.

Parameters:

folderstr

Folder of the saved model

spark_sessionpyspark.sql.SparkSession

The current SparkSession

Returns:

T5Transformer

The restored model

static pretrained(name='t5_small', lang='en', remote_loc=None)[source]#

Downloads and loads a pretrained model.

Parameters:

namestr, optional

Name of the pretrained model, by default “t5_small”

langstr, optional

Language of the pretrained model, by default “en”

remote_locstr, optional

Optional remote address of the resource, by default None. Will use Spark NLPs repositories otherwise.

Returns:

T5Transformer

The restored model