Packages

class Summarization extends AnnotatorApproach[SummarizationModel] with SummarizationParams

High-level, task-oriented document summarization.

Summarization is a zero-configuration estimator: users state what they want (summary length, style, focus) and the annotator decides how to produce it (model selection, prompting, generation settings, long-document handling). No prompt writing or model choice is required:

val summarizer = new Summarization()
  .setInputCols("document")
  .setOutputCol("summary")

val model = pipeline.fit(data) // default model resolves and downloads here

Three methods are supported, each with an automatically selected default model:

  • llm (default): an instruction-tuned GGUF LLM run with llama.cpp (AutoGGUFModel); the annotator owns the summarization prompt, system prompt, safe generation defaults and reasoning-mode suppression.
  • encoder_decoder: a specialized abstractive summarization model (BartTransformer, DistilBART fine-tuned on XSum).
  • extractive: selects the most central sentences from the original document using sentence embeddings, position-augmented centrality and MMR redundancy control; the output contains only text from the source document.

Documents longer than the model context are handled automatically (see SummarizationParams.longDocumentStrategy): the document is split at sentence boundaries into overlapping chunks, each chunk is summarized, and the intermediate summaries are combined and summarized again.

fit() downloads the default (or user-overridden) pretrained model and returns a SummarizationModel; saved fitted pipelines embed the resolved model and load offline.

Example

import com.johnsnowlabs.nlp.base.DocumentAssembler
import com.johnsnowlabs.nlp.annotators.seq2seq.Summarization
import org.apache.spark.ml.Pipeline

val documentAssembler = new DocumentAssembler().setInputCol("text").setOutputCol("document")

val summarizer = new Summarization()
  .setInputCols("document")
  .setOutputCol("summary")
  .setMethod("extractive")
  .setMaxSummaryLength(100)

val pipeline = new Pipeline().setStages(Array(documentAssembler, summarizer))
val result = pipeline.fit(data).transform(data)
result.select("summary.result").show(false)
Linear Supertypes
SummarizationParams, AnnotatorApproach[SummarizationModel], CanBeLazy, DefaultParamsWritable, MLWritable, HasOutputAnnotatorType, HasOutputAnnotationCol, HasInputAnnotationCols, Estimator[SummarizationModel], PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Grouped
  2. Alphabetic
  3. By Inheritance
Inherited
  1. Summarization
  2. SummarizationParams
  3. AnnotatorApproach
  4. CanBeLazy
  5. DefaultParamsWritable
  6. MLWritable
  7. HasOutputAnnotatorType
  8. HasOutputAnnotationCol
  9. HasInputAnnotationCols
  10. Estimator
  11. PipelineStage
  12. Logging
  13. Params
  14. Serializable
  15. Serializable
  16. Identifiable
  17. AnyRef
  18. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new Summarization()
  2. new Summarization(uid: String)

    uid

    required uid for storing annotator to disk

Type Members

  1. type AnnotatorType = String
    Definition Classes
    HasOutputAnnotatorType

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def $[T](param: Param[T]): T
    Attributes
    protected
    Definition Classes
    Params
  4. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  5. def _fit(dataset: Dataset[_], recursiveStages: Option[PipelineModel]): SummarizationModel
    Attributes
    protected
    Definition Classes
    AnnotatorApproach
  6. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  7. def beforeTraining(spark: SparkSession): Unit
    Definition Classes
    AnnotatorApproach
  8. final def checkSchema(schema: StructType, inputAnnotatorType: String): Boolean
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  9. val chunkOverlap: IntParam

    Number of sentences repeated between consecutive chunks (Default: 1).

    Number of sentences repeated between consecutive chunks (Default: 1).

    Definition Classes
    SummarizationParams
  10. val chunkSize: IntParam

    Chunk size in approximate tokens used for hierarchical summarization (Default: 0 = derive automatically from the model's context limit).

    Chunk size in approximate tokens used for hierarchical summarization (Default: 0 = derive automatically from the model's context limit).

    Definition Classes
    SummarizationParams
  11. final def clear(param: Param[_]): Summarization.this.type
    Definition Classes
    Params
  12. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  13. final def copy(extra: ParamMap): Estimator[SummarizationModel]
    Definition Classes
    AnnotatorApproach → Estimator → PipelineStage → Params
  14. def copyValues[T <: Params](to: T, extra: ParamMap): T
    Attributes
    protected
    Definition Classes
    Params
  15. final def defaultCopy[T <: Params](extra: ParamMap): T
    Attributes
    protected
    Definition Classes
    Params
  16. val description: String
    Definition Classes
    Summarization → AnnotatorApproach
  17. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  18. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  19. def explainParam(param: Param[_]): String
    Definition Classes
    Params
  20. def explainParams(): String
    Definition Classes
    Params
  21. final val extraInputCols: StringArrayParam
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  22. final def extractParamMap(): ParamMap
    Definition Classes
    Params
  23. final def extractParamMap(extra: ParamMap): ParamMap
    Definition Classes
    Params
  24. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  25. final def fit(dataset: Dataset[_]): SummarizationModel
    Definition Classes
    AnnotatorApproach → Estimator
  26. def fit(dataset: Dataset[_], paramMaps: Seq[ParamMap]): Seq[SummarizationModel]
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" )
  27. def fit(dataset: Dataset[_], paramMap: ParamMap): SummarizationModel
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" )
  28. def fit(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): SummarizationModel
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" ) @varargs()
  29. val focus: Param[String]

    Optional free-text focus hint included in the LLM prompt, e.g.

    Optional free-text focus hint included in the LLM prompt, e.g. "main findings" (Default: ""). Only applies to the llm method.

    Definition Classes
    SummarizationParams
  30. final def get[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  31. def getChunkOverlap: Int

    Definition Classes
    SummarizationParams
  32. def getChunkSize: Int

    Definition Classes
    SummarizationParams
  33. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  34. final def getDefault[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  35. def getFocus: String

    Definition Classes
    SummarizationParams
  36. def getGpuLayers: Int

    Definition Classes
    SummarizationParams
  37. def getInputCols: Array[String]

    returns

    input annotations columns currently used

    Definition Classes
    HasInputAnnotationCols
  38. def getLazyAnnotator: Boolean
    Definition Classes
    CanBeLazy
  39. def getLongDocumentStrategy: String

    Definition Classes
    SummarizationParams
  40. def getMaxSummaryLength: Int

    Definition Classes
    SummarizationParams
  41. def getMethod: String

    Definition Classes
    SummarizationParams
  42. def getMinSummaryLength: Int

    Definition Classes
    SummarizationParams
  43. def getMmrLambda: Float

    Definition Classes
    SummarizationParams
  44. def getModel: String

    Definition Classes
    SummarizationParams
  45. def getNoRepeatNgramSize: Int

    Definition Classes
    SummarizationParams
  46. def getNumBeams: Int

    Definition Classes
    SummarizationParams
  47. final def getOrDefault[T](param: Param[T]): T
    Definition Classes
    Params
  48. final def getOutputCol: String

    Gets annotation column name going to generate

    Gets annotation column name going to generate

    Definition Classes
    HasOutputAnnotationCol
  49. def getParam(paramName: String): Param[Any]
    Definition Classes
    Params
  50. def getPositionBias: Float

    Definition Classes
    SummarizationParams
  51. def getSummaryStyle: String

    Definition Classes
    SummarizationParams
  52. def getTemperature: Float

    Definition Classes
    SummarizationParams
  53. def getTopP: Float

    Definition Classes
    SummarizationParams
  54. val gpuLayers: IntParam

    Number of model layers offloaded to the GPU for the llm method (Default: 99 = offload all).

    Number of model layers offloaded to the GPU for the llm method (Default: 99 = offload all). Set to 0 on CPU-only clusters. Ignored by the other methods.

    Definition Classes
    SummarizationParams
  55. final def hasDefault[T](param: Param[T]): Boolean
    Definition Classes
    Params
  56. def hasParam(paramName: String): Boolean
    Definition Classes
    Params
  57. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  58. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  59. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  60. val inputAnnotatorTypes: Array[String]

    Input Annotator Type: DOCUMENT

    Input Annotator Type: DOCUMENT

    Definition Classes
    Summarization → HasInputAnnotationCols
  61. final val inputCols: StringArrayParam

    columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified

    columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified

    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  62. final def isDefined(param: Param[_]): Boolean
    Definition Classes
    Params
  63. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  64. final def isSet(param: Param[_]): Boolean
    Definition Classes
    Params
  65. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  66. val lazyAnnotator: BooleanParam
    Definition Classes
    CanBeLazy
  67. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  68. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  69. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  70. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  71. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  72. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  73. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  74. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  75. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  76. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  77. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  78. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  79. val longDocumentStrategy: Param[String]

    Strategy for documents longer than the model context: auto, truncate or hierarchical (Default: auto).

    Strategy for documents longer than the model context: auto, truncate or hierarchical (Default: auto). auto summarizes directly when the document fits and falls back to hierarchical chunk-then-combine summarization when it does not. Since hierarchical also summarizes a fitting document in a single pass, auto and hierarchical currently behave identically; only truncate differs (it cuts the document to the context budget).

    Definition Classes
    SummarizationParams
  80. val maxSummaryLength: IntParam

    Target maximum summary length in words/tokens, approximate (Default: 250).

    Target maximum summary length in words/tokens, approximate (Default: 250).

    Definition Classes
    SummarizationParams
  81. val method: Param[String]

    Summarization method: llm, encoder_decoder or extractive (Default: llm).

    Summarization method: llm, encoder_decoder or extractive (Default: llm).

    • llm: an instruction-tuned GGUF LLM (via AutoGGUFModel / llama.cpp) prompted for summarization.
    • encoder_decoder: a specialized abstractive summarization model (DistilBART).
    • extractive: selects the most central sentences of the original document (embedding-based centrality with position prior and MMR redundancy control).
    Definition Classes
    SummarizationParams
  82. val minSummaryLength: IntParam

    Minimum summary length in words/tokens, generative methods only (Default: 20).

    Minimum summary length in words/tokens, generative methods only (Default: 20).

    Definition Classes
    SummarizationParams
  83. val mmrLambda: FloatParam

    MMR trade-off between relevance and redundancy in extractive selection, higher = more relevance-driven (Default: 0.7).

    MMR trade-off between relevance and redundancy in extractive selection, higher = more relevance-driven (Default: 0.7). Only applies to the extractive method.

    Definition Classes
    SummarizationParams
  84. val model: Param[String]

    Optional pretrained model name overriding the method's default model (Default: "" = use the method's default).

    Optional pretrained model name overriding the method's default model (Default: "" = use the method's default).

    Definition Classes
    SummarizationParams
  85. def msgHelper(schema: StructType): String
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  86. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  87. val noRepeatNgramSize: IntParam

    Size of n-grams that may not repeat in the generated summary (Default: 3).

    Size of n-grams that may not repeat in the generated summary (Default: 3). Only applies to the encoder_decoder method; 0 disables the constraint.

    Definition Classes
    SummarizationParams
  88. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  89. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  90. val numBeams: IntParam

    Number of beams for beam search (Default: 4).

    Number of beams for beam search (Default: 4). Only applies to the encoder_decoder method.

    Definition Classes
    SummarizationParams
  91. def onTrained(model: SummarizationModel, spark: SparkSession): Unit
    Definition Classes
    AnnotatorApproach
  92. val optionalInputAnnotatorTypes: Array[String]
    Definition Classes
    HasInputAnnotationCols
  93. val outputAnnotatorType: String

    Output Annotator Type: DOCUMENT

    Output Annotator Type: DOCUMENT

    Definition Classes
    Summarization → HasOutputAnnotatorType
  94. final val outputCol: Param[String]
    Attributes
    protected
    Definition Classes
    HasOutputAnnotationCol
  95. lazy val params: Array[Param[_]]
    Definition Classes
    Params
  96. val positionBias: FloatParam

    Weight of the lead-position prior in extractive sentence ranking (Default: 0.3).

    Weight of the lead-position prior in extractive sentence ranking (Default: 0.3). Only applies to the extractive method.

    Definition Classes
    SummarizationParams
  97. def save(path: String): Unit
    Definition Classes
    MLWritable
    Annotations
    @Since( "1.6.0" ) @throws( ... )
  98. final def set(paramPair: ParamPair[_]): Summarization.this.type
    Attributes
    protected
    Definition Classes
    Params
  99. final def set(param: String, value: Any): Summarization.this.type
    Attributes
    protected
    Definition Classes
    Params
  100. final def set[T](param: Param[T], value: T): Summarization.this.type
    Definition Classes
    Params
  101. def setChunkOverlap(value: Int): Summarization.this.type

    Definition Classes
    SummarizationParams
  102. def setChunkSize(value: Int): Summarization.this.type

    Definition Classes
    SummarizationParams
  103. final def setDefault(paramPairs: ParamPair[_]*): Summarization.this.type
    Attributes
    protected
    Definition Classes
    Params
  104. final def setDefault[T](param: Param[T], value: T): Summarization.this.type
    Attributes
    protected[org.apache.spark.ml]
    Definition Classes
    Params
  105. def setExtraInputCols(value: Array[String]): Summarization.this.type
    Definition Classes
    HasInputAnnotationCols
  106. def setFocus(value: String): Summarization.this.type

    Definition Classes
    SummarizationParams
  107. def setGpuLayers(value: Int): Summarization.this.type

    Definition Classes
    SummarizationParams
  108. final def setInputCols(value: String*): Summarization.this.type
    Definition Classes
    HasInputAnnotationCols
  109. def setInputCols(value: Array[String]): Summarization.this.type

    Overrides required annotators column if different than default

    Overrides required annotators column if different than default

    Definition Classes
    HasInputAnnotationCols
  110. def setLazyAnnotator(value: Boolean): Summarization.this.type
    Definition Classes
    CanBeLazy
  111. def setLongDocumentStrategy(value: String): Summarization.this.type

    Definition Classes
    SummarizationParams
  112. def setMaxSummaryLength(value: Int): Summarization.this.type

    Definition Classes
    SummarizationParams
  113. def setMethod(value: String): Summarization.this.type

    Definition Classes
    SummarizationParams
  114. def setMinSummaryLength(value: Int): Summarization.this.type

    Definition Classes
    SummarizationParams
  115. def setMmrLambda(value: Float): Summarization.this.type

    Definition Classes
    SummarizationParams
  116. def setModel(value: String): Summarization.this.type

    Definition Classes
    SummarizationParams
  117. def setNoRepeatNgramSize(value: Int): Summarization.this.type

    Definition Classes
    SummarizationParams
  118. def setNumBeams(value: Int): Summarization.this.type

    Definition Classes
    SummarizationParams
  119. final def setOutputCol(value: String): Summarization.this.type

    Overrides annotation column name when transforming

    Overrides annotation column name when transforming

    Definition Classes
    HasOutputAnnotationCol
  120. def setPositionBias(value: Float): Summarization.this.type

    Definition Classes
    SummarizationParams
  121. def setSummaryStyle(value: String): Summarization.this.type

    Definition Classes
    SummarizationParams
  122. def setTemperature(value: Float): Summarization.this.type

    Definition Classes
    SummarizationParams
  123. def setTopP(value: Float): Summarization.this.type

    Definition Classes
    SummarizationParams
  124. val summaryStyle: Param[String]

    Summary style used to build the LLM prompt: concise, detailed or bullets (Default: concise).

    Summary style used to build the LLM prompt: concise, detailed or bullets (Default: concise). Only applies to the llm method.

    Definition Classes
    SummarizationParams
  125. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  126. val temperature: FloatParam

    Generation temperature (Default: 0.2).

    Generation temperature (Default: 0.2). Applies to the llm method.

    Definition Classes
    SummarizationParams
  127. def toString(): String
    Definition Classes
    Identifiable → AnyRef → Any
  128. val topP: FloatParam

    Top-p (nucleus) sampling (Default: 0.9).

    Top-p (nucleus) sampling (Default: 0.9). Applies to the llm method.

    Definition Classes
    SummarizationParams
  129. def train(dataset: Dataset[_], recursivePipeline: Option[PipelineModel]): SummarizationModel

    Resolves the summarization method to a configured SummarizationModel, downloading the default pretrained delegate when no explicit model name was set.

    Resolves the summarization method to a configured SummarizationModel, downloading the default pretrained delegate when no explicit model name was set.

    Definition Classes
    Summarization → AnnotatorApproach
  130. final def transformSchema(schema: StructType): StructType

    requirement for pipeline transformation validation.

    requirement for pipeline transformation validation. It is called on fit()

    Definition Classes
    AnnotatorApproach → PipelineStage
  131. def transformSchema(schema: StructType, logging: Boolean): StructType
    Attributes
    protected
    Definition Classes
    PipelineStage
    Annotations
    @DeveloperApi()
  132. val uid: String
    Definition Classes
    Summarization → Identifiable
  133. def validate(schema: StructType): Boolean

    takes a Dataset and checks to see if all the required annotation types are present.

    takes a Dataset and checks to see if all the required annotation types are present.

    schema

    to be validated

    returns

    True if all the required types are present, else false

    Attributes
    protected
    Definition Classes
    AnnotatorApproach
  134. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  135. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  136. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  137. def write: MLWriter
    Definition Classes
    DefaultParamsWritable → MLWritable

Inherited from SummarizationParams

Inherited from AnnotatorApproach[SummarizationModel]

Inherited from CanBeLazy

Inherited from DefaultParamsWritable

Inherited from MLWritable

Inherited from HasOutputAnnotatorType

Inherited from HasOutputAnnotationCol

Inherited from HasInputAnnotationCols

Inherited from Estimator[SummarizationModel]

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Parameters

Annotator types

Required input and expected output annotator types

Members

Parameter setters

Parameter getters