Language Detection & Identification Pipeline - 43 Languages

Description

Language detection and identification is the task of automatically detecting the language(s) present in a document based on the content of the document. LanguageDetectorDL is an annotator that detects the language of documents or sentences depending on the inputCols. In addition, LanguageDetetorDL can accurately detect language from documents with mixed languages by coalescing sentences and select the best candidate.

We have designed and developed Deep Learning models using CNN architectures in TensorFlow/Keras. The models are trained on large datasets such as Wikipedia and Tatoeba with high accuracy evaluated on the Europarl dataset. The output is a language code in Wiki Code style: https://en.wikipedia.org/wiki/List_of_Wikipedias.

This pipeline can detect the following languages:

Predicted Entities

Arabic, Belarusian, Bulgarian, Czech, Danish, German, Greek, English, Esperanto, Spanish, Estonian, Persian, Finnish, French, Hebrew, Hindi, Hungarian, Interlingua, Indonesian, Icelandic, Italian, Japanese, Korean, Latin, Lithuanian, Latvian, Macedonian, Marathi, Dutch, Polish, Portuguese, Romanian, Russian, Slovak, Slovenian, Serbian, Swedish, Tagalog, Turkish, Tatar, Ukrainian, Vietnamese, Chinese.

Live Demo Open in Colab Download Copy S3 URI

How to use

from sparknlp.pretrained import PretrainedPipeline

pipeline = PretrainedPipeline("detect_language_43", lang = "xx")

pipeline.annotate("French author who helped pioneer the science-fiction genre.")

import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline

val pipeline = new PretrainedPipeline("detect_language_43", lang = "xx")

pipeline.annotate("French author who helped pioneer the science-fiction genre.")

import nlu

text = ["French author who helped pioneer the science-fiction genre."]
lang_df = nlu.load("xx.classify.lang.43").predict(text)
lang_df

Results

{'document': ['French author who helped pioneer the science-fiction genre.'],
'language': ['en']}

Model Information

Included Models

• DocumentAssembler
• SentenceDetectorDLModel
• LanguageDetectorDL