package dl
- Alphabetic
- Public
- All
Type Members
-
class
LanguageDetectorDL extends AnnotatorModel[LanguageDetectorDL] with HasSimpleAnnotate[LanguageDetectorDL] with WriteTensorflowModel with WriteOnnxModel with HasEngine
Language Identification and Detection by using CNN and RNN architectures in TensorFlow, or supervised fastText models such as GlotLID on ONNX.
Language Identification and Detection by using CNN and RNN architectures in TensorFlow, or supervised fastText models such as GlotLID on ONNX.
LanguageDetectorDLis an annotator that detects the language of documents or sentences depending on the inputCols. The CNN models are trained on large datasets such as Wikipedia and Tatoeba. Depending on the language (how similar the characters are), they work best with text longer than 140 characters. Their output is a language code in Wiki Code style.The fastText models run on ONNX and turn text into features exactly as fastText does. GlotLID v4 (
"ld_glotlid_v4", or"ld_glotlid_v4_int8"at a quarter of the size) labels 2,060 languages and scripts as an ISO 639-3 code plus an ISO 15924 script, such aseng_Latnorcmn_Hani, and haszxx_Code,zxx_Zmthandzxx_Zsymfor code, mathematics and symbols. For these models the metadata also holdsconfidence, the score the label was judged on, and three parameters shape the result:scriptPooling,minLettersandtopK. Each sentence goes to the model whole, so split long documents into sentences first. To import other fastText models, see the import notebook.Pretrained models can be loaded with
pretrainedof the companion object:Val languageDetector = LanguageDetectorDL.pretrained() .setInputCols("sentence") .setOutputCol("language")
The default model is
"ld_wiki_tatoeba_cnn_21", default language is"xx"(meaning multi-lingual), if no values are provided. For available pretrained models please see the Models Hub.For extended examples of usage, see the Examples, the GlotLID example And the LanguageDetectorDLTestSpec.
Example
import spark.implicits._ import com.johnsnowlabs.nlp.base.DocumentAssembler import com.johnsnowlabs.nlp.annotators.ld.dl.LanguageDetectorDL import org.apache.spark.ml.Pipeline val documentAssembler = new DocumentAssembler() .setInputCol("text") .setOutputCol("document") val languageDetector = LanguageDetectorDL.pretrained() .setInputCols("document") .setOutputCol("language") val pipeline = new Pipeline() .setStages(Array( documentAssembler, languageDetector )) val data = Seq( "Spark NLP is an open-source text processing library for advanced natural language processing for the Python, Java and Scala programming languages.", "Spark NLP est une bibliothèque de traitement de texte open source pour le traitement avancé du langage naturel pour les langages de programmation Python, Java et Scala.", "Spark NLP ist eine Open-Source-Textverarbeitungsbibliothek für fortgeschrittene natürliche Sprachverarbeitung für die Programmiersprachen Python, Java und Scala." ).toDF("text") val result = pipeline.fit(data).transform(data) result.select("language.result").show(false) +------+ |result| +------+ |[en] | |[fr] | |[de] | +------+
- trait ReadLanguageDetectorDLTensorflowModel extends ReadTensorflowModel with ReadOnnxModel
- trait ReadablePretrainedLanguageDetectorDLModel extends ParamsAndFeaturesReadable[LanguageDetectorDL] with HasPretrained[LanguageDetectorDL]
Value Members
-
object
LanguageDetectorDL extends ReadablePretrainedLanguageDetectorDLModel with ReadLanguageDetectorDLTensorflowModel with Serializable
This is the companion object of LanguageDetectorDL.
This is the companion object of LanguageDetectorDL. Please refer to that class for the documentation.