`sparknlp.annotator.sentence.sentence_detector_sat`#

Contains classes for the SaT (Segment any Text) sentence detector.

Module Contents#

Classes#

SentenceDetectorSaTModel

Sentence detector based on the wtpsplit / SaT (Segment any Text) transformer models.

class SentenceDetectorSaTModel(classname='com.johnsnowlabs.nlp.annotators.sbd.sat.SentenceDetectorSaTModel', java_model=None)[source]#

Sentence detector based on the wtpsplit / SaT (Segment any Text) transformer models.

SaT is a per-token boundary detector built on an XLM-R backbone: for every sub-word token the model predicts a sentence-boundary probability. The document is tokenized with SentencePiece, sliced into overlapping windows (so documents longer than 512 tokens are supported), run through ONNX, and the per-token probabilities are merged and projected back onto characters to produce the final sentence spans.

This annotator only supports models exported to ONNX with an XLM-R SentencePiece tokenizer (e.g. segment-any-text/sat-12l-sm and segment-any-text/sat-12l). A locally exported model can be loaded with loadSavedModel(), and pretrained models with pretrained().

Input Annotation types	Output Annotation type
`DOCUMENT`	`DOCUMENT`

Parameters:

threshold: Boundary probability threshold; a boundary is placed once a character’s probability is >= threshold (Default: 0.25 for sat-12l-sm). Ignored when minSentenceLength or maxSentenceLength is set.
blockSize: Number of real sub-word tokens per ONNX window, max 510 for XLM-R (Default: 510).
stride: Number of tokens to advance between consecutive overlapping windows (Default: 256).
satBatchSize: Number of windows to send to ONNX in a single forward pass (Default: 8).
weighting: Window-overlap weighting strategy, "hat" or "uniform" (Default: “hat”).
trimWhitespace: Whether to strip leading/trailing whitespace from each detected sentence (Default: True).
explodeSentences: Whether to split each detected sentence into its own Dataset row (Default: True). Each sentence is always a separate annotation; this flag only controls the row layout (when True the output column is exploded so every sentence lands on its own row), mirroring SentenceDetectorDLModel.
minSentenceLength: Minimum sentence length in characters, 0 = no minimum (Default: 0). When this or maxSentenceLength is set, the model switches to length-constrained (Viterbi) segmentation and threshold is ignored.
maxSentenceLength: Maximum sentence length in characters, 0 = no maximum (Default: 0). See minSentenceLength.

Examples

>>> import sparknlp
>>> from sparknlp.base import *
>>> from sparknlp.annotator import *
>>> from pyspark.ml import Pipeline
>>> documentAssembler = DocumentAssembler() \
...     .setInputCol("text") \
...     .setOutputCol("document")
>>> sentenceDetector = SentenceDetectorSaTModel.pretrained() \
...     .setInputCols(["document"]) \
...     .setOutputCol("sentence")
>>> pipeline = Pipeline().setStages([documentAssembler, sentenceDetector])
>>> data = spark.createDataFrame([["This is a sentence. This is another one."]]).toDF("text")
>>> result = pipeline.fit(data).transform(data)
>>> result.selectExpr("explode(sentence.result) as sentence").show(truncate=False)
+----------------------+
|sentence              |
+----------------------+
|This is a sentence.   |
|This is another one.  |
+----------------------+

name = 'SentenceDetectorSaTModel'[source]#

inputAnnotatorTypes[source]#

outputAnnotatorType = 'document'[source]#

threshold[source]#

blockSize[source]#

stride[source]#

satBatchSize[source]#

weighting[source]#

trimWhitespace[source]#

explodeSentences[source]#

minSentenceLength[source]#

maxSentenceLength[source]#

setThreshold(value)[source]#

Sets the boundary probability threshold (Default: 0.25).

Parameters:

valuefloat: Boundary probability threshold

getThreshold()[source]#: Gets the boundary probability threshold.

setBlockSize(value)[source]#

Sets the number of real sub-word tokens per ONNX window (max 510).

Parameters:

valueint: Real sub-word tokens per window

getBlockSize()[source]#: Gets the number of real sub-word tokens per ONNX window.

setStride(value)[source]#

Sets the token stride between consecutive overlapping windows.

Parameters:

valueint: Token stride between overlapping windows

getStride()[source]#: Gets the token stride between overlapping windows.

setSatBatchSize(value)[source]#

Sets the number of windows per ONNX forward pass.

Parameters:

valueint: Number of windows per ONNX batch

getSatBatchSize()[source]#: Gets the number of windows per ONNX batch.

setWeighting(value)[source]#

Sets the window-overlap weighting strategy, "hat" or "uniform".

Parameters:

valuestr: Overlap weighting strategy

getWeighting()[source]#: Gets the window-overlap weighting strategy.

setTrimWhitespace(value)[source]#

Sets whether to strip leading/trailing whitespace from each sentence.

Parameters:

valuebool: Whether to trim whitespace

getTrimWhitespace()[source]#: Gets whether whitespace is trimmed from sentence boundaries.

setExplodeSentences(value)[source]#

Sets whether to split each sentence into its own Dataset row (Default: True).

Parameters:

valuebool: Whether to explode sentences into separate rows

getExplodeSentences()[source]#: Gets whether sentences are exploded into separate rows.

setMinSentenceLength(value)[source]#

Sets the minimum sentence length in characters (0 = no minimum).

Setting this (or maxSentenceLength) switches to length-constrained segmentation and disables threshold.

Parameters:

valueint: Minimum sentence length in characters

getMinSentenceLength()[source]#: Gets the minimum sentence length in characters.

setMaxSentenceLength(value)[source]#

Sets the maximum sentence length in characters (0 = no maximum).

Setting this (or minSentenceLength) switches to length-constrained segmentation and disables threshold.

Parameters:

valueint: Maximum sentence length in characters

getMaxSentenceLength()[source]#: Gets the maximum sentence length in characters.

static loadSavedModel(folder, spark_session)[source]#

Loads a locally exported SaT ONNX model.

Parameters:

folderstr: Folder of the saved model (containing model.onnx and assets/sentencepiece.bpe.model)
spark_sessionpyspark.sql.SparkSession: The current SparkSession

Returns:

SentenceDetectorSaTModel: The restored model

static pretrained(name='sat_12l_sm', lang='xx', remote_loc=None)[source]#

Downloads and loads a pretrained model.

Parameters:

namestr, optional: Name of the pretrained model, by default “sat_12l_sm”
langstr, optional: Language of the pretrained model, by default “xx”
remote_locstr, optional: Optional remote address of the resource, by default None. Will use Spark NLPs repositories otherwise.

Returns:

SentenceDetectorSaTModel: The restored model

sparknlp.annotator.sentence.sentence_detector_sat#

Module Contents#

Classes#

`sparknlp.annotator.sentence.sentence_detector_sat`#