Description
This model uses context and language knowledge to assign all forms and inflections of a word to a single root. This enables the pipeline to treat the past and present tense of a verb, for example, as the same word instead of two completely different words. The lemmatizer takes into consideration the context surrounding a word to determine which root is correct when the word form alone is ambiguous.
Live Demo Open in Colab Download Copy S3 URI
How to use
document_assembler = DocumentAssembler() \
.setInputCol("text") \
.setOutputCol("document")
word_segmenter = WordSegmenterModel.pretrained('wordseg_gsd_ud', 'ja')\
.setInputCols("document")\
.setOutputCol("token")
lemmatizer = LemmatizerModel.pretrained("lemma", "ja") \
.setInputCols(["token"]) \
.setOutputCol("lemma")
nlp_pipeline = Pipeline(stages=[document_assembler, word_segmenter , lemmatizer])
light_pipeline = LightPipeline(nlp_pipeline.fit(spark.createDataFrame([[""]]).toDF("text")))
results = light_pipeline.fullAnnotate(["これに不快感を示す住民はいましたが,現在,表立って反対や抗議の声を挙げている住民はいないようです。"])
val document_assembler = DocumentAssembler()
.setInputCol("text")
.setOutputCol("document")
val word_segmenter = WordSegmenterModel.pretrained('wordseg_gsd_ud', 'ja')
.setInputCols("document")
.setOutputCol("token")
val lemmatizer = LemmatizerModel.pretrained("lemma", "ja")
.setInputCols("token")
.setOutputCol("lemma")
val pipeline = new Pipeline().setStages(Array(document_assembler, word_segmenter , lemmatizer))
val data = Seq("これに不快感を示す住民はいましたが,現在,表立って反対や抗議の声を挙げている住民はいないようです。").toDF("text")
val result = pipeline.fit(data).transform(data)
import nlu
text = ["これに不快感を示す住民はいましたが,現在,表立って反対や抗議の声を挙げている住民はいないようです。"]
lemma_df = nlu.load('ja.lemma').predict(text, output_level = "document")
lemma_df.lemma.values[0]
Results
{'lemma': [Annotation(token, 0, 1, これ, {'sentence': '0'}),
Annotation(token, 2, 2, にる, {'sentence': '0'}),
Annotation(token, 3, 4, 不快, {'sentence': '0'}),
Annotation(token, 5, 5, 感, {'sentence': '0'}),
Annotation(token, 6, 6, を, {'sentence': '0'}),
Annotation(token, 7, 8, 示す, {'sentence': '0'}),
Annotation(token, 9, 10, 住民, {'sentence': '0'}),
Annotation(token, 11, 11, はる, {'sentence': '0'}),
Annotation(token, 12, 12, いる, {'sentence': '0'}),
Annotation(token, 13, 14, まする, {'sentence': '0'}),
Annotation(token, 15, 15, たる, {'sentence': '0'}),
Annotation(token, 16, 16, がる, {'sentence': '0'}),
Annotation(token, 17, 17, ,, {'sentence': '0'}),
Annotation(token, 18, 19, 現在, {'sentence': '0'}),
Annotation(token, 20, 20, ,, {'sentence': '0'}),
Annotation(token, 21, 23, 表立つ, {'sentence': '0'}),
Annotation(token, 24, 24, てる, {'sentence': '0'}),
Annotation(token, 25, 26, 反対, {'sentence': '0'}),
Annotation(token, 27, 27, やる, {'sentence': '0'}),
Annotation(token, 28, 29, 抗議, {'sentence': '0'}),
Annotation(token, 30, 30, のる, {'sentence': '0'}),
Annotation(token, 31, 31, 声, {'sentence': '0'}),
Annotation(token, 32, 32, を, {'sentence': '0'}),
Annotation(token, 33, 34, 挙げる, {'sentence': '0'}),
Annotation(token, 35, 35, てる, {'sentence': '0'}),
Annotation(token, 36, 37, いる, {'sentence': '0'}),
Annotation(token, 38, 39, 住民, {'sentence': '0'}),
Annotation(token, 40, 40, はる, {'sentence': '0'}),
Annotation(token, 41, 41, いる, {'sentence': '0'}),
Annotation(token, 42, 43, なぐ, {'sentence': '0'}),
Annotation(token, 44, 45, よう, {'sentence': '0'}),
Annotation(token, 46, 47, です, {'sentence': '0'}),
Annotation(token, 48, 48, 。, {'sentence': '0'})]}
Model Information
| Model Name: | lemma |
| Compatibility: | Spark NLP 2.7.0+ |
| Edition: | Official |
| Input Labels: | [document] |
| Output Labels: | [token] |
| Language: | ja |
Data Source
The model was trained using the universal dependencies data set version 2 and the IPADIC dictionary from Mecab.
References:
- Asahara, M., Kanayama, H., Tanaka, T., Miyao, Y., Uematsu, S., Mori, S., Matsumoto, Y., Omura, M., & Murawaki, Y. (2018). Universal Dependencies Version 2 for Japanese. In LREC-2018.