sparknlp.annotator.seq2seq.summarization#

Classes#

class Summarization[source]#

High-level, task-oriented document summarization.

Summarization is a zero-configuration estimator: state what you want (summary length, style, focus) and the annotator decides how to produce it (model selection, prompting, generation settings, long-document handling). No prompt writing or model choice is required:

>>> summarizer = Summarization() \
...     .setInputCols(["document"]) \
...     .setOutputCol("summary")

fit() downloads the default (or user-overridden) pretrained model and returns a SummarizationModel.

Three methods are supported, each with an automatically selected default model:

• llm (default): an instruction-tuned GGUF LLM run with llama.cpp; the annotator owns the summarization prompt, system prompt, safe generation defaults and reasoning-mode suppression.

• encoder_decoder: a specialized abstractive summarization model (DistilBART fine-tuned on XSum).

• extractive: selects the most central sentences from the original document using sentence embeddings, position-augmented centrality and MMR redundancy control.

Documents longer than the model context are chunked at sentence boundaries, summarized per chunk, and the intermediate summaries are combined and summarized again (see setLongDocumentStrategy).

Input Annotation types	Output Annotation type
DOCUMENT	DOCUMENT

Examples

>>> import sparknlp
>>> from sparknlp.base import *
>>> from sparknlp.annotator import *
>>> from pyspark.ml import Pipeline
>>> documentAssembler = DocumentAssembler() \
...     .setInputCol("text") \
...     .setOutputCol("document")
>>> summarizer = Summarization() \
...     .setInputCols(["document"]) \
...     .setOutputCol("summary") \
...     .setMethod("extractive") \
...     .setMaxSummaryLength(100)
>>> pipeline = Pipeline().setStages([documentAssembler, summarizer])
>>> data = spark.createDataFrame([["Long document text ..."]]).toDF("text")
>>> result = pipeline.fit(data).transform(data)
>>> result.select("summary.result").show(truncate=False)

name = 'Summarization'[source]#

inputAnnotatorTypes[source]#

outputAnnotatorType = 'document'[source]#

class SummarizationModel(classname='com.johnsnowlabs.nlp.annotators.seq2seq.SummarizationModel', java_model=None)[source]#

Fitted model produced by Summarization.

Orchestrates the resolved summarization delegate: prompt building, long-document chunking, delegate inference, output cleanup and transparency metadata (method, model, engine, token estimates, chunk count) on each output annotation.

Saving this model persists the delegate (model weights included), so fitted pipelines reload without network access.

Input Annotation types	Output Annotation type
DOCUMENT	DOCUMENT

name = 'SummarizationModel'[source]#

inputAnnotatorTypes[source]#

outputAnnotatorType = 'document'[source]#

resolvedModel[source]#

getResolvedModel()[source]#

Gets the pretrained model name resolved at fit time.

Returns:

str

Resolved pretrained model name

close()[source]#

Frees the llama.cpp native resources held by the llm delegate.

No-op for the encoder_decoder and extractive methods.