Packages

package dl

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. All

Type Members

  1. class LanguageDetectorDL extends AnnotatorModel[LanguageDetectorDL] with HasSimpleAnnotate[LanguageDetectorDL] with WriteTensorflowModel with WriteOnnxModel with HasEngine

    Language Identification and Detection by using CNN and RNN architectures in TensorFlow, or supervised fastText models such as GlotLID on ONNX.

    Language Identification and Detection by using CNN and RNN architectures in TensorFlow, or supervised fastText models such as GlotLID on ONNX.

    LanguageDetectorDL is an annotator that detects the language of documents or sentences depending on the inputCols. The CNN models are trained on large datasets such as Wikipedia and Tatoeba. Depending on the language (how similar the characters are), they work best with text longer than 140 characters. Their output is a language code in Wiki Code style.

    The fastText models run on ONNX and turn text into features exactly as fastText does. GlotLID v4 ("ld_glotlid_v4", or "ld_glotlid_v4_int8" at a quarter of the size) labels 2,060 languages and scripts as an ISO 639-3 code plus an ISO 15924 script, such as eng_Latn or cmn_Hani, and has zxx_Code, zxx_Zmth and zxx_Zsym for code, mathematics and symbols. For these models the metadata also holds confidence, the score the label was judged on, and three parameters shape the result: scriptPooling, minLetters and topK. Each sentence goes to the model whole, so split long documents into sentences first. To import other fastText models, see the import notebook.

    Pretrained models can be loaded with pretrained of the companion object:

    Val languageDetector = LanguageDetectorDL.pretrained()
      .setInputCols("sentence")
      .setOutputCol("language")

    The default model is "ld_wiki_tatoeba_cnn_21", default language is "xx" (meaning multi-lingual), if no values are provided. For available pretrained models please see the Models Hub.

    For extended examples of usage, see the Examples, the GlotLID example And the LanguageDetectorDLTestSpec.

    Example

    import spark.implicits._
    import com.johnsnowlabs.nlp.base.DocumentAssembler
    import com.johnsnowlabs.nlp.annotators.ld.dl.LanguageDetectorDL
    import org.apache.spark.ml.Pipeline
    
    val documentAssembler = new DocumentAssembler()
      .setInputCol("text")
      .setOutputCol("document")
    
    val languageDetector = LanguageDetectorDL.pretrained()
      .setInputCols("document")
      .setOutputCol("language")
    
    val pipeline = new Pipeline()
      .setStages(Array(
        documentAssembler,
        languageDetector
      ))
    
    val data = Seq(
      "Spark NLP is an open-source text processing library for advanced natural language processing for the Python, Java and Scala programming languages.",
      "Spark NLP est une bibliothèque de traitement de texte open source pour le traitement avancé du langage naturel pour les langages de programmation Python, Java et Scala.",
      "Spark NLP ist eine Open-Source-Textverarbeitungsbibliothek für fortgeschrittene natürliche Sprachverarbeitung für die Programmiersprachen Python, Java und Scala."
    ).toDF("text")
    val result = pipeline.fit(data).transform(data)
    
    result.select("language.result").show(false)
    +------+
    |result|
    +------+
    |[en]  |
    |[fr]  |
    |[de]  |
    +------+
  2. trait ReadLanguageDetectorDLTensorflowModel extends ReadTensorflowModel with ReadOnnxModel
  3. trait ReadablePretrainedLanguageDetectorDLModel extends ParamsAndFeaturesReadable[LanguageDetectorDL] with HasPretrained[LanguageDetectorDL]

Value Members

  1. object LanguageDetectorDL extends ReadablePretrainedLanguageDetectorDLModel with ReadLanguageDetectorDLTensorflowModel with Serializable

    This is the companion object of LanguageDetectorDL.

    This is the companion object of LanguageDetectorDL. Please refer to that class for the documentation.

Ungrouped