sparknlp.annotator.ld_dl.language_detector_dl#

Classes#

class LanguageDetectorDL(classname='com.johnsnowlabs.nlp.annotators.ld.dl.LanguageDetectorDL', java_model=None)[source]#

Language Identification and Detection by using CNN and RNN architectures in TensorFlow, or supervised fastText models such as GlotLID on ONNX.

LanguageDetectorDL is an annotator that detects the language of documents or sentences depending on the inputCols. The CNN models are trained on large datasets such as Wikipedia and Tatoeba. Depending on the language (how similar the characters are), they work best with text longer than 140 characters. Their output is a language code in Wiki Code style.

The fastText models run on ONNX and turn text into features exactly as fastText does. GlotLID v4 ("ld_glotlid_v4", or "ld_glotlid_v4_int8" at a quarter of the size) labels 2,060 languages and scripts as an ISO 639-3 code plus an ISO 15924 script, such as eng_Latn or cmn_Hani, and has zxx_Code, zxx_Zmth and zxx_Zsym for code, mathematics and symbols. For these models the metadata also holds confidence, the score the label was judged on, and scriptPooling, minLetters and topK shape the result. Each sentence goes to the model whole, so split long documents into sentences first. To import other fastText models, see the import notebook.

Pretrained models can be loaded with pretrained() of the companion object:

>>> languageDetector = LanguageDetectorDL.pretrained() \
...     .setInputCols(["sentence"]) \
...     .setOutputCol("language")

The default model is "ld_wiki_tatoeba_cnn_21", default language is "xx" (meaning multi-lingual), if no values are provided.

For available pretrained models please see the Models Hub.

For extended examples of usage, see the Examples.

Input Annotation types	Output Annotation type
DOCUMENT	LANGUAGE

Parameters:

configProtoBytes

ConfigProto from tensorflow, serialized into byte array.

threshold

The minimum threshold for the final result otheriwse it will be either neutral or the value set in thresholdLabel, by default 0.5

thresholdLabel

In case the score is less than threshold, what should be the label, by default Unknown

coalesceSentences

If sets to true the output of all sentences will be averaged to one output instead of one output per sentence, by default True.

languages

The languages used to trained the model

topK

Number of language scores kept in the metadata, the highest ones; 0 keeps all of them, by default 0. ONNX (fastText) models only.

scriptPooling

Whether the score of a non-Latin top label is the sum of all labels in its script, by default False. ONNX (fastText) models only.

minLetters

When the top label is in Latin script (_Latn), text with fewer letters than this gets thresholdLabel; 0 turns it off, by default 0. ONNX (fastText) models only.

Examples

>>> import sparknlp
>>> from sparknlp.base import *
>>> from sparknlp.annotator import *
>>> from pyspark.ml import Pipeline
>>> documentAssembler = DocumentAssembler() \
...     .setInputCol("text") \
...     .setOutputCol("document")
>>> languageDetector = LanguageDetectorDL.pretrained() \
...     .setInputCols("document") \
...     .setOutputCol("language")
>>> pipeline = Pipeline() \
...     .setStages([
...       documentAssembler,
...       languageDetector
...     ])
>>> data = spark.createDataFrame([
...     ["Spark NLP is an open-source text processing library for advanced natural language processing for the Python, Java and Scala programming languages."],
...     ["Spark NLP est une bibliothèque de traitement de texte open source pour le traitement avancé du langage naturel pour les langages de programmation Python, Java et Scala."],
...     ["Spark NLP ist eine Open-Source-Textverarbeitungsbibliothek für fortgeschrittene natürliche Sprachverarbeitung für die Programmiersprachen Python, Java und Scala."]
... ]).toDF("text")
>>> result = pipeline.fit(data).transform(data)
>>> result.select("language.result").show(truncate=False)
+------+
|result|
+------+
|[en]  |
|[fr]  |
|[de]  |
+------+

name = 'LanguageDetectorDL'[source]#

inputAnnotatorTypes[source]#

outputAnnotatorType = 'language'[source]#

configProtoBytes[source]#

threshold[source]#

thresholdLabel[source]#

coalesceSentences[source]#

languages[source]#

topK[source]#

scriptPooling[source]#

minLetters[source]#

setConfigProtoBytes(b)[source]#

Sets configProto from tensorflow, serialized into byte array.

Parameters:

bList[int]

ConfigProto from tensorflow, serialized into byte array

setThreshold(v)[source]#

Sets the minimum threshold for the final result otherwise it will be either neutral or the value set in thresholdLabel, by default 0.5.

Parameters:

vfloat

Minimum threshold for the final result

setThresholdLabel(p)[source]#

Sets what should be the label in case the score is less than threshold, by default Unknown.

Parameters:

pstr

The replacement label.

setCoalesceSentences(value)[source]#

Sets if the output of all sentences will be averaged to one output instead of one output per sentence, by default True.

Parameters:

valuebool

If the output of all sentences will be averaged to one output

setTopK(value)[source]#

Sets the number of language scores kept in the metadata, the highest ones; 0 keeps all of them, by default 0. ONNX (fastText) models only.

Parameters:

valueint

Number of scores to keep

setScriptPooling(value)[source]#

Sets whether the score of a non-Latin top label is the sum of all labels in its script, so that close varieties such as Mandarin (cmn_Hani) and Cantonese (yue_Hani) do not split it, by default False. The label is still the top one, so the pooled score says how sure the script is rather than the language. ONNX (fastText) models only.

Parameters:

valuebool

Whether to pool same-script scores

setMinLetters(value)[source]#

Sets the minimum number of letters for a Latin-script top label (_Latn); shorter text gets thresholdLabel. 0 turns it off, by default 0. ONNX (fastText) models only.

Parameters:

valueint

Minimum number of letters

static loadSavedModel(folder, spark_session)[source]#

Loads a locally exported model: a TensorFlow CNN export, or a supervised fastText model such as GlotLID exported to ONNX (model.onnx with assets/language.txt, assets/vocab.json and assets/config.json).

Parameters:

folderstr

Folder of the exported model

spark_sessionpyspark.sql.SparkSession

The current SparkSession

Returns:

LanguageDetectorDL

The restored model

static pretrained(name='ld_wiki_tatoeba_cnn_21', lang='xx', remote_loc=None)[source]#

Downloads and loads a pretrained model.

Parameters:

namestr, optional

Name of the pretrained model, by default “ld_wiki_tatoeba_cnn_21”

langstr, optional

Language of the pretrained model, by default “xx”

remote_locstr, optional

Optional remote address of the resource, by default None. Will use Spark NLPs repositories otherwise.

Returns:

LanguageDetectorDL

The restored model