Part of Speech for Japanese

Description

This model annotates the part of speech of tokens in a text. The parts of speech annotated include PRON (pronoun), CCONJ (coordinating conjunction), and 13 others. The part of speech model is useful for extracting the grammatical structure of a piece of text automatically.

Live Demo Open in Colab Download Copy S3 URI

How to use

document_assembler = DocumentAssembler() \
.setInputCol("text") \
.setOutputCol("document")

sentence_detector = SentenceDetector()\
.setInputCols(["document"])\
.setOutputCol("sentence")

word_segmenter = WordSegmenterModel.pretrained("wordseg_gsd_ud", "ja")\
.setInputCols(["sentence"])\
.setOutputCol("token")

pos = PerceptronModel.pretrained("pos_ud_gsd", "ja") \
.setInputCols(["document", "token"]) \
.setOutputCol("pos")

pipeline = Pipeline(stages=[
document_assembler,
sentence_detector,
word_segmenter,
posTagger
])

example = spark.createDataFrame([['院長と話をしたところ、腰痛治療も得意なようです。']], ["text"])

result = pipeline.fit(example).transform(example)

val document_assembler = DocumentAssembler()
.setInputCol("text")
.setOutputCol("document")

val sentence_detector = SentenceDetector()
.setInputCols("document")
.setOutputCol("sentence")

val word_segmenter = WordSegmenterModel.pretrained("wordseg_gsd_ud", "ja")
.setInputCols("sentence")
.setOutputCol("token")

val pos = PerceptronModel.pretrained("pos_ud_gsd", "ja")
.setInputCols(Array("document", "token"))
.setOutputCol("pos")

val pipeline = new Pipeline().setStages(Array(document_assembler, sentence_detector, word_segmenter, pos))

val data = Seq("院長と話をしたところ、腰痛治療も得意なようです。").toDF("text")
val result = pipeline.fit(data).transform(data)
import nlu

text = ["""5月13日に放送されるフジテレビ系「僕らの音楽」にて、福原美穂とAIという豪華共演が決定した。"""]
pos_df = nlu.load('ja.pos.ud_gsd').predict(text, output_level='token')
pos_df

Results

+------+-----+
|token |pos  |
+------+-----+
|院長  |NOUN |
|と    |ADP  |
|話    |NOUN |
|を    |ADP  |
|し    |VERB |
|た    |AUX  |
|ところ|NOUN |
|、    |PUNCT|
|腰痛  |NOUN |
|治療  |NOUN |
|も    |ADP  |
|得意  |ADJ  |
|な    |AUX  |
|よう  |AUX  |
|です  |AUX  |
|。    |PUNCT|
+------+-----+

Model Information

Model Name: pos_ud_gsd
Compatibility: Spark NLP 2.7.0+
Edition: Official
Input Labels: [sentence, token]
Output Labels: [pos]
Language: ja

Data Source

The model was trained on the Universal Dependencies, curated by Google.

Reference:

Asahara, M., Kanayama, H., Tanaka, T., Miyao, Y., Uematsu, S., Mori, S., Matsumoto, Y., Omura, M., & Murawaki, Y. (2018). Universal Dependencies Version 2 for Japanese. In LREC-2018.

Benchmarking

| pos_tag      | precision | recall | f1-score | support |
|--------------|-----------|--------|----------|---------|
| ADJ          | 0.90      | 0.78   | 0.84     | 350     |
| ADP          | 0.98      | 0.99   | 0.99     | 2804    |
| ADV          | 0.87      | 0.65   | 0.74     | 220     |
| AUX          | 0.95      | 0.98   | 0.96     | 1768    |
| CCONJ        | 0.97      | 0.93   | 0.95     | 42      |
| DET          | 1.00      | 1.00   | 1.00     | 66      |
| INTJ         | 0.00      | 0.00   | 0.00     | 1       |
| NOUN         | 0.93      | 0.98   | 0.95     | 3692    |
| NUM          | 0.99      | 0.98   | 0.99     | 251     |
| PART         | 0.96      | 0.83   | 0.89     | 128     |
| PRON         | 0.97      | 0.94   | 0.95     | 101     |
| PROPN        | 0.92      | 0.70   | 0.79     | 313     |
| PUNCT        | 1.00      | 1.00   | 1.00     | 1294    |
| SCONJ        | 0.97      | 0.94   | 0.96     | 682     |
| SYM          | 0.99      | 1.00   | 0.99     | 67      |
| VERB         | 0.96      | 0.92   | 0.94     | 1255    |
| accuracy     | 0.96      | 13034  |          |         |
| macro avg    | 0.90      | 0.85   | 0.87     | 13034   |
| weighted avg | 0.96      | 0.96   | 0.95     | 13034   |