`sparknlp.annotator.seq2seq.m2m100_transformer`#

Contains classes for the M2M100Transformer.

Module Contents#

Classes#

M2M100Transformer

M2M100 : multilingual translation model

class M2M100Transformer(classname='com.johnsnowlabs.nlp.annotators.seq2seq.M2M100Transformer', java_model=None)[source]#

M2M100 : multilingual translation model

M2M100 is a multilingual encoder-decoder (seq-to-seq) model trained for Many-to-Many multilingual translation.

The model can directly translate between the 9,900 directions of 100 languages.

Pretrained models can be loaded with pretrained() of the companion object:

>>> m2m100 = M2M100Transformer.pretrained() \
...     .setInputCols(["document"]) \
...     .setOutputCol("generation")

The default model is "m2m100_418M", if no name is provided. For available pretrained models please see the Models Hub.

Input Annotation types	Output Annotation type
`DOCUMENT`	`DOCUMENT`

Parameters:

configProtoBytes

ConfigProto from tensorflow, serialized into byte array.

minOutputLength

Minimum length of the sequence to be generated, by default 0

maxOutputLength

Maximum length of output text, by default 20

doSample

Whether or not to use sampling; use greedy decoding otherwise, by default False

temperature

The value used to module the next token probabilities, by default 1.0

topK

The number of highest probability vocabulary tokens to keep for top-k-filtering, by default 50

topP

Top cumulative probability for vocabulary tokens, by default 1.0

If set to float < 1, only the most probable tokens with probabilities that add up to topP or higher are kept for generation.

repetitionPenalty

The parameter for repetition penalty, 1.0 means no penalty. , by default 1.0

noRepeatNgramSize

If set to int > 0, all ngrams of that size can only occur once, by default 0

ignoreTokenIds

A list of token ids which are ignored in the decoder’s output, by default []

srcLang

Source Language (Default: en)

tgtLang

Target Language (Default: fr)

Languages Covered

—–

Afrikaans (af), Amharic (am), Arabic (ar), Asturian (ast), Azerbaijani (az), Bashkir (ba),

Belarusian (be), Bulgarian (bg), Bengali (bn), Breton (br), Bosnian (bs), Catalan; Valencian

(ca), Cebuano (ceb), Czech (cs), Welsh (cy), Danish (da), German (de), Greeek (el), English

(en), Spanish (es), Estonian (et), Persian (fa), Fulah (ff), Finnish (fi), French (fr),

Western Frisian (fy), Irish (ga), Gaelic; Scottish Gaelic (gd), Galician (gl), Gujarati (gu),

Hausa (ha), Hebrew (he), Hindi (hi), Croatian (hr), Haitian; Haitian Creole (ht), Hungarian

(hu), Armenian (hy), Indonesian (id), Igbo (ig), Iloko (ilo), Icelandic (is), Italian (it),

Japanese (ja), Javanese (jv), Georgian (ka), Kazakh (kk), Central Khmer (km), Kannada (kn),

Korean (ko), Luxembourgish; Letzeburgesch (lb), Ganda (lg), Lingala (ln), Lao (lo), Lithuanian

(lt), Latvian (lv), Malagasy (mg), Macedonian (mk), Malayalam (ml), Mongolian (mn), Marathi

(mr), Malay (ms), Burmese (my), Nepali (ne), Dutch; Flemish (nl), Norwegian (no), Northern

Sotho (ns), Occitan (post 1500) (oc), Oriya (or), Panjabi; Punjabi (pa), Polish (pl), Pushto;

Pashto (ps), Portuguese (pt), Romanian; Moldavian; Moldovan (ro), Russian (ru), Sindhi (sd),

Sinhala; Sinhalese (si), Slovak (sk), Slovenian (sl), Somali (so), Albanian (sq), Serbian

(sr), Swati (ss), Sundanese (su), Swedish (sv), Swahili (sw), Tamil (ta), Thai (th), Tagalog

(tl), Tswana (tn), Turkish (tr), Ukrainian (uk), Urdu (ur), Uzbek (uz), Vietnamese (vi), Wolof

(wo), Xhosa (xh), Yiddish (yi), Yoruba (yo), Chinese (zh), Zulu (zu)

References

Paper Abstract:

Existing work in translation demonstrated the potential of massively multilingual machine translation by training

a single model able to translate between any pair of languages. However, much of this work is English-Centric by training only on data which was translated from or to English. While this is supported by large sources of training data, it does not reflect translation needs worldwide. In this work, we create a true Many-to-Many multilingual translation model that can translate directly between any pair of 100 languages. We build and open source a training dataset that covers thousands of language directions with supervised data, created through large-scale mining. Then, we explore how to effectively increase model capacity through a combination of dense scaling and language-specific sparse parameters to create high quality models. Our focus on non-English-Centric models brings gains of more than 10 BLEU when directly translating between non-English directions while performing competitively to the best single systems of WMT. We open-source our scripts so that others may reproduce the data, evaluation, and final M2M-100 model.*

Examples

>>> import sparknlp
>>> from sparknlp.base import *
>>> from sparknlp.annotator import *
>>> from pyspark.ml import Pipeline
>>> documentAssembler = DocumentAssembler() \
...     .setInputCol("text") \
...     .setOutputCol("documents")
>>> m2m100 = M2M100Transformer.pretrained("m2m100_418M") \
...     .setInputCols(["documents"]) \
...     .setMaxOutputLength(50) \
...     .setOutputCol("generation") \
...     .setSrcLang("en") \
...     .setTgtLang("fr")
>>> pipeline = Pipeline().setStages([documentAssembler, m2m100])
>>> data = spark.createDataFrame([["生活就像一盒巧克力。"]]).toDF("text")
>>> result = pipeline.fit(data).transform(data)
>>> result.select("summaries.generation").show(truncate=False)
+-------------------------------------------------------------------------------------------+
|result                                                                                     |
+-------------------------------------------------------------------------------------------+
|[ Life is like a box of chocolate.]                                                        |
+-------------------------------------------------------------------------------------------+

setIgnoreTokenIds(value)[source]#

A list of token ids which are ignored in the decoder’s output.

Parameters:

valueList[int]: The words to be filtered out

setConfigProtoBytes(b)[source]#

Sets configProto from tensorflow, serialized into byte array.

Parameters:

bList[int]: ConfigProto from tensorflow, serialized into byte array

setMinOutputLength(value)[source]#

Sets minimum length of the sequence to be generated.

Parameters:

valueint: Minimum length of the sequence to be generated

setMaxOutputLength(value)[source]#

Sets maximum length of output text.

Parameters:

valueint: Maximum length of output text

setDoSample(value)[source]#

Sets whether or not to use sampling, use greedy decoding otherwise.

Parameters:

valuebool: Whether or not to use sampling; use greedy decoding otherwise

setTemperature(value)[source]#

Sets the value used to module the next token probabilities.

Parameters:

valuefloat: The value used to module the next token probabilities

setTopK(value)[source]#

Sets the number of highest probability vocabulary tokens to keep for top-k-filtering.

Parameters:

valueint: Number of highest probability vocabulary tokens to keep

setTopP(value)[source]#

Sets the top cumulative probability for vocabulary tokens.

If set to float < 1, only the most probable tokens with probabilities that add up to topP or higher are kept for generation.

Parameters:

valuefloat: Cumulative probability for vocabulary tokens

setRepetitionPenalty(value)[source]#

Sets the parameter for repetition penalty. 1.0 means no penalty.

Parameters:

valuefloat: The repetition penalty

References

See Ctrl: A Conditional Transformer Language Model For Controllable Generation for more details.

setNoRepeatNgramSize(value)[source]#

Sets size of n-grams that can only occur once.

If set to int > 0, all ngrams of that size can only occur once.

Parameters:

valueint: N-gram size can only occur once

setBeamSize(value)[source]#

Sets the number of beam size for beam search, by default 4.

Parameters:

valueint: Number of beam size for beam search

setSrcLang(value)[source]#

Sets source language.

Parameters:

valuestr: Source language

setTgtLang(value)[source]#

Sets target language.

Parameters:

valuestr: Target language

static loadSavedModel(folder, spark_session, use_openvino=False)[source]#

Loads a locally saved model.

Parameters:

folderstr: Folder of the saved model
spark_sessionpyspark.sql.SparkSession: The current SparkSession

Returns:

M2M100Transformer: The restored model

static pretrained(name='m2m100_418M', lang='xx', remote_loc=None)[source]#

Downloads and loads a pretrained model.

Parameters:

namestr, optional: Name of the pretrained model, by default “m2m100_418M”
langstr, optional: Language of the pretrained model, by default “en”
remote_locstr, optional: Optional remote address of the resource, by default None. Will use Spark NLPs repositories otherwise.

Returns:

M2M100Transformer: The restored model

sparknlp.annotator.seq2seq.m2m100_transformer#

Module Contents#

Classes#

`sparknlp.annotator.seq2seq.m2m100_transformer`#