Amharic Lemmatizer

Description

This model uses context and language knowledge to assign all forms and inflections of a word to a single root. This enables the pipeline to treat the past and present tense of a verb, for example, as the same word instead of two completely different words. The lemmatizer takes into consideration the context surrounding a word to determine which root is correct when the word form alone is ambiguous.

Live Demo Open in Colab Download Copy S3 URI

How to use

document_assembler = DocumentAssembler() \
    .setInputCol("text") \
    .setOutputCol("document")

tokenizer = Tokenizer()\
        .setInputCols("document")\
        .setOutputCol("token")

lemmatizer = LemmatizerModel.pretrained("lemma", "am") \
        .setInputCols(["token"]) \
        .setOutputCol("lemma")

nlp_pipeline = Pipeline(stages=[document_assembler, tokenize, lemmatizer])
light_pipeline = LightPipeline(nlp_pipeline.fit(spark.createDataFrame([[""]]).toDF("text")))
results = light_pipeline.fullAnnotate(["መጽሐፉን መጽሐፍ ኡ ን አስያዛት አስያዝ ኧ ኣት ።"])

val document_assembler = DocumentAssembler()
    .setInputCol("text")
    .setOutputCol("document")

val tokenizer = Tokenizer()
        .setInputCols("document")
        .setOutputCol("token")

val lemmatizer = LemmatizerModel.pretrained("lemma", "am")
        .setInputCols("token")
        .setOutputCol("lemma")

val pipeline = new Pipeline().setStages(Array(document_assembler, tokenizer, lemmatizer))
val data = Seq("መጽሐፉን መጽሐፍ ኡ ን አስያዛት አስያዝ ኧ ኣት ።").toDF("text")
val result = pipeline.fit(data).transform(data)

import nlu

text = ["መጽሐፉን መጽሐፍ ኡ ን አስያዛት አስያዝ ኧ ኣት ።"]
lemma_df = nlu.load('am.lemma').predict(text, output_level = "document")
lemma_df.lemma.values[0]

Results

{'lemma': [Annotation(token, 0, 4, _, {'sentence': '0'}),
  Annotation(token, 6, 9, መጽሐፍ, {'sentence': '0'}),
  Annotation(token, 11, 11, ኡ, {'sentence': '0'}),
  Annotation(token, 13, 13, ን, {'sentence': '0'}),
  Annotation(token, 15, 19, _, {'sentence': '0'}),
  Annotation(token, 21, 24, አስያዝ, {'sentence': '0'}),
  Annotation(token, 26, 26, ኧ, {'sentence': '0'}),
  Annotation(token, 28, 29, ኣት, {'sentence': '0'}),
  Annotation(token, 31, 31, ።, {'sentence': '0'})]}

Model Information

Data Source

The model was trained on the Universal Dependencies version 2.7.

Reference:

Binyam Ephrem Seyoum ,Yusuke Miyao and Baye Yimam Mekonnen.2018.Universal Dependencies for Amharic. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pp. 2216–2222, Miyazaki, Japan: European Language Resources Association (ELRA)