sparknlp.annotator.er.entity_ruler#

Classes#

class EntityRulerApproach[source]#

Fits an Annotator to match exact strings or regex patterns provided in a file against a Document and assigns them an named entity. The definitions can contain any number of named entities.

There are multiple ways and formats to set the extraction resource. It is possible to set it either as a “JSON”, “JSONL” or “CSV” file. A path to the file needs to be provided to setPatternsResource. The file format needs to be set as the “format” field in the option parameter map and depending on the file type, additional parameters might need to be set.

If the file is in a JSON format, then the rule definitions need to be given in a list with the fields “id”, “label” and “patterns”:

 [
    {
      "id": "person-regex",
      "label": "PERSON",
      "patterns": ["\w+\s\w+", "\w+-\w+"]
    },
    {
      "id": "locations-words",
      "label": "LOCATION",
      "patterns": ["Winterfell"]
    }
]

The same fields also apply to a file in the JSONL format:

{"id": "names-with-j", "label": "PERSON", "patterns": ["Jon", "John", "John Snow"]}
{"id": "names-with-s", "label": "PERSON", "patterns": ["Stark", "Snow"]}
{"id": "names-with-e", "label": "PERSON", "patterns": ["Eddard", "Eddard Stark"]}

In order to use a CSV file, an additional parameter “delimiter” needs to be set. In this case, the delimiter might be set by using .setPatternsResource("patterns.csv", ReadAs.TEXT, {"format": "csv", "delimiter": "|")}):

PERSON|Jon
PERSON|John
PERSON|John Snow
LOCATION|Winterfell

Input Annotation types	Output Annotation type
DOCUMENT, TOKEN	CHUNK

Parameters:

patternsResource

Resource in JSON or CSV format to map entities to patterns

useStorage

Whether to use RocksDB storage to serialize patterns

Examples

>>> import sparknlp
>>> from sparknlp.base import *
>>> from sparknlp.annotator import *
>>> from sparknlp.common import *
>>> from pyspark.ml import Pipeline

In this example, the entities file as the form of:

PERSON|Jon
PERSON|John
PERSON|John Snow
LOCATION|Winterfell

where each line represents an entity and the associated string delimited by “|”.

>>> documentAssembler = DocumentAssembler() \
...     .setInputCol("text") \
...     .setOutputCol("document")
>>> tokenizer = Tokenizer() \
...     .setInputCols(["document"]) \
...     .setOutputCol("token")
>>> entityRuler = EntityRulerApproach() \
...     .setInputCols(["document", "token"]) \
...     .setOutputCol("entities") \
...     .setPatternsResource(
...       "patterns.csv",
...       ReadAs.TEXT,
...       {"format": "csv", "delimiter": "\\|"}
...     )
>>> pipeline = Pipeline().setStages([
...     documentAssembler,
...     tokenizer,
...     entityRuler
... ])
>>> data = spark.createDataFrame([["Jon Snow wants to be lord of Winterfell."]]).toDF("text")
>>> result = pipeline.fit(data).transform(data)
>>> result.selectExpr("explode(entities)").show(truncate=False)
+--------------------------------------------------------------------+
|col                                                                 |
+--------------------------------------------------------------------+
|[chunk, 0, 2, Jon, [entity -> PERSON, sentence -> 0], []]           |
|[chunk, 29, 38, Winterfell, [entity -> LOCATION, sentence -> 0], []]|
+--------------------------------------------------------------------+

setPatternsResource(path, read_as=ReadAs.TEXT, options={'format': 'JSON'})[source]#

Sets Resource in JSON or CSV format to map entities to patterns.

Parameters:

pathstr

Path to the resource

read_asstr, optional

How to interpret the resource, by default ReadAs.TEXT

optionsdict, optional

Options for parsing the resource, by default {“format”: “JSON”}

setUseStorage(value)[source]#

Sets whether to use RocksDB storage to serialize patterns.

Parameters:

valuebool

Whether to use RocksDB storage to serialize patterns.

setSentenceMatch(value)[source]#

Sets whether to find match at sentence level.

Parameters:

valuebool

True: sentence level. False: token level

setAlphabetResource(path)[source]#

Alphabet Resource (a simple plain text with all language characters)

Parameters:

pathstr

Path to the resource

class EntityRulerModel(classname='com.johnsnowlabs.nlp.annotators.er.EntityRulerModel', java_model=None)[source]#

Instantiated model of the EntityRulerApproach. For usage and examples see the documentation of the main class.

Input Annotation types	Output Annotation type
DOCUMENT, TOKEN	CHUNK