class SummarizationModel extends Model[SummarizationModel] with RawAnnotator[SummarizationModel] with CanBeLazy with SummarizationParams
Fitted model produced by Summarization.
Orchestrates the resolved summarization delegate at the DataFrame level: prompt building, long-document chunking (hierarchical map/reduce summarization), delegate inference, output cleanup and transparency metadata. The delegate is one of:
- AutoGGUFModel for method
llm, - BartTransformer for method
encoder_decoder, - MPNetEmbeddings (+ rule-based sentence detection and a PacSum-style ranker) for method
extractive.
Saving this model persists the delegate (model weights included) under the model path, so fitted pipelines reload without network access.
Notes:
- The summarization
methodis bound at fit time: this model only holds the delegate of the method it was fitted with. Changingmethodon a fitted model fails at transform with an explanatory error; fit a new Summarization stage instead. All other task parameters (lengths, style, focus, strategy, ...) can be changed freely after fit. - As a DataFrame-level orchestrator this annotator is not supported in LightPipeline.
transformconfigures the shared delegate (input/output columns, generation settings) in place, so a single model instance is not safe for concurrenttransformcalls; use one instance per concurrent caller.- Transforms cache the input row ids and the computed summaries for the lifetime of the
returned DataFrame (generative inference intermediates are released eagerly). These caches
back the returned DataFrame and cannot be unpersisted through this API; they are freed
when the SparkSession ends or via
spark.catalog.clearCache()(which clears all cached data). The encoder_decoder method pins inference to a single partition (the BART backend is not thread-safe); the llm and extractive methods keep the input partitioning. - For the
llmmethod the document text is embedded into the prompt, so a document containing instructions can influence its own summary (prompt injection); the system prompt reduces but does not eliminate this.
- Grouped
- Alphabetic
- By Inheritance
- SummarizationModel
- SummarizationParams
- CanBeLazy
- RawAnnotator
- HasOutputAnnotationCol
- HasInputAnnotationCols
- HasOutputAnnotatorType
- ParamsAndFeaturesWritable
- HasFeatures
- DefaultParamsWritable
- MLWritable
- Model
- Transformer
- PipelineStage
- Logging
- Params
- Serializable
- Serializable
- Identifiable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
Type Members
-
type
AnnotatorType = String
- Definition Classes
- HasOutputAnnotatorType
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
$[T](param: Param[T]): T
- Attributes
- protected
- Definition Classes
- Params
-
def
$$[T](feature: StructFeature[T]): T
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[K, V](feature: MapFeature[K, V]): Map[K, V]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[T](feature: SetFeature[T]): Set[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[T](feature: ArrayFeature[T]): Array[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
final
def
checkSchema(schema: StructType, inputAnnotatorType: String): Boolean
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
-
val
chunkOverlap: IntParam
Number of sentences repeated between consecutive chunks (Default:
1).Number of sentences repeated between consecutive chunks (Default:
1).- Definition Classes
- SummarizationParams
-
val
chunkSize: IntParam
Chunk size in approximate tokens used for hierarchical summarization (Default:
0= derive automatically from the model's context limit).Chunk size in approximate tokens used for hierarchical summarization (Default:
0= derive automatically from the model's context limit).- Definition Classes
- SummarizationParams
-
final
def
clear(param: Param[_]): SummarizationModel.this.type
- Definition Classes
- Params
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
close(): Unit
Closes the llama.cpp backend if the llm delegate is loaded, freeing native resources.
-
def
copy(extra: ParamMap): SummarizationModel
requirement for annotators copies
requirement for annotators copies
- Definition Classes
- SummarizationModel → RawAnnotator → Model → Transformer → PipelineStage → Params
-
def
copyValues[T <: Params](to: T, extra: ParamMap): T
- Attributes
- protected
- Definition Classes
- Params
-
final
def
defaultCopy[T <: Params](extra: ParamMap): T
- Attributes
- protected
- Definition Classes
- Params
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
explainParam(param: Param[_]): String
- Definition Classes
- Params
-
def
explainParams(): String
- Definition Classes
- Params
-
final
val
extraInputCols: StringArrayParam
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
-
def
extraValidate(structType: StructType): Boolean
- Attributes
- protected
- Definition Classes
- SummarizationModel → RawAnnotator
-
def
extraValidateMsg: String
Override for additional custom schema checks
Override for additional custom schema checks
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
final
def
extractParamMap(): ParamMap
- Definition Classes
- Params
-
final
def
extractParamMap(extra: ParamMap): ParamMap
- Definition Classes
- Params
-
val
features: ArrayBuffer[Feature[_, _, _]]
- Definition Classes
- HasFeatures
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
val
focus: Param[String]
Optional free-text focus hint included in the LLM prompt, e.g.
Optional free-text focus hint included in the LLM prompt, e.g. "main findings" (Default:
""). Only applies to thellmmethod.- Definition Classes
- SummarizationParams
-
def
get[T](feature: StructFeature[T]): Option[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[K, V](feature: MapFeature[K, V]): Option[Map[K, V]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[T](feature: SetFeature[T]): Option[Set[T]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[T](feature: ArrayFeature[T]): Option[Array[T]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
get[T](param: Param[T]): Option[T]
- Definition Classes
- Params
-
def
getChunkOverlap: Int
- Definition Classes
- SummarizationParams
-
def
getChunkSize: Int
- Definition Classes
- SummarizationParams
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
final
def
getDefault[T](param: Param[T]): Option[T]
- Definition Classes
- Params
- def getEmbeddingsDelegate: MPNetEmbeddings
-
def
getFocus: String
- Definition Classes
- SummarizationParams
-
def
getGpuLayers: Int
- Definition Classes
- SummarizationParams
-
def
getInputCols: Array[String]
- returns
input annotations columns currently used
- Definition Classes
- HasInputAnnotationCols
-
def
getLazyAnnotator: Boolean
- Definition Classes
- CanBeLazy
- def getLlmDelegate: AutoGGUFModel
-
def
getLongDocumentStrategy: String
- Definition Classes
- SummarizationParams
-
def
getMaxSummaryLength: Int
- Definition Classes
- SummarizationParams
-
def
getMethod: String
- Definition Classes
- SummarizationParams
-
def
getMinSummaryLength: Int
- Definition Classes
- SummarizationParams
-
def
getMmrLambda: Float
- Definition Classes
- SummarizationParams
-
def
getModel: String
- Definition Classes
- SummarizationParams
-
def
getNoRepeatNgramSize: Int
- Definition Classes
- SummarizationParams
-
def
getNumBeams: Int
- Definition Classes
- SummarizationParams
-
final
def
getOrDefault[T](param: Param[T]): T
- Definition Classes
- Params
-
final
def
getOutputCol: String
Gets annotation column name going to generate
Gets annotation column name going to generate
- Definition Classes
- HasOutputAnnotationCol
-
def
getParam(paramName: String): Param[Any]
- Definition Classes
- Params
-
def
getPositionBias: Float
- Definition Classes
- SummarizationParams
- def getResolvedModel: String
- def getSeq2SeqDelegate: BartTransformer
-
def
getSummaryStyle: String
- Definition Classes
- SummarizationParams
-
def
getTemperature: Float
- Definition Classes
- SummarizationParams
-
def
getTopP: Float
- Definition Classes
- SummarizationParams
-
val
gpuLayers: IntParam
Number of model layers offloaded to the GPU for the
llmmethod (Default:99= offload all).Number of model layers offloaded to the GPU for the
llmmethod (Default:99= offload all). Set to0on CPU-only clusters. Ignored by the other methods.- Definition Classes
- SummarizationParams
-
final
def
hasDefault[T](param: Param[T]): Boolean
- Definition Classes
- Params
-
def
hasParam(paramName: String): Boolean
- Definition Classes
- Params
-
def
hasParent: Boolean
- Definition Classes
- Model
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
val
inputAnnotatorTypes: Array[String]
Input Annotator Type: DOCUMENT
Input Annotator Type: DOCUMENT
- Definition Classes
- SummarizationModel → HasInputAnnotationCols
-
final
val
inputCols: StringArrayParam
columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified
columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
-
final
def
isDefined(param: Param[_]): Boolean
- Definition Classes
- Params
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
isSet(param: Param[_]): Boolean
- Definition Classes
- Params
-
def
isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
val
lazyAnnotator: BooleanParam
- Definition Classes
- CanBeLazy
-
def
log: Logger
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logName: String
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
val
longDocumentStrategy: Param[String]
Strategy for documents longer than the model context:
auto,truncateorhierarchical(Default:auto).Strategy for documents longer than the model context:
auto,truncateorhierarchical(Default:auto).autosummarizes directly when the document fits and falls back to hierarchical chunk-then-combine summarization when it does not. Sincehierarchicalalso summarizes a fitting document in a single pass,autoandhierarchicalcurrently behave identically; onlytruncatediffers (it cuts the document to the context budget).- Definition Classes
- SummarizationParams
-
val
maxSummaryLength: IntParam
Target maximum summary length in words/tokens, approximate (Default:
250).Target maximum summary length in words/tokens, approximate (Default:
250).- Definition Classes
- SummarizationParams
-
val
method: Param[String]
Summarization method:
llm,encoder_decoderorextractive(Default:llm).Summarization method:
llm,encoder_decoderorextractive(Default:llm).llm: an instruction-tuned GGUF LLM (via AutoGGUFModel / llama.cpp) prompted for summarization.encoder_decoder: a specialized abstractive summarization model (DistilBART).extractive: selects the most central sentences of the original document (embedding-based centrality with position prior and MMR redundancy control).
- Definition Classes
- SummarizationParams
-
val
minSummaryLength: IntParam
Minimum summary length in words/tokens, generative methods only (Default:
20).Minimum summary length in words/tokens, generative methods only (Default:
20).- Definition Classes
- SummarizationParams
-
val
mmrLambda: FloatParam
MMR trade-off between relevance and redundancy in extractive selection, higher = more relevance-driven (Default:
0.7).MMR trade-off between relevance and redundancy in extractive selection, higher = more relevance-driven (Default:
0.7). Only applies to theextractivemethod.- Definition Classes
- SummarizationParams
-
val
model: Param[String]
Optional pretrained model name overriding the method's default model (Default:
""= use the method's default).Optional pretrained model name overriding the method's default model (Default:
""= use the method's default).- Definition Classes
- SummarizationParams
-
def
msgHelper(schema: StructType): String
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
val
noRepeatNgramSize: IntParam
Size of n-grams that may not repeat in the generated summary (Default:
3).Size of n-grams that may not repeat in the generated summary (Default:
3). Only applies to theencoder_decodermethod;0disables the constraint.- Definition Classes
- SummarizationParams
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
val
numBeams: IntParam
Number of beams for beam search (Default:
4).Number of beams for beam search (Default:
4). Only applies to theencoder_decodermethod.- Definition Classes
- SummarizationParams
-
def
onWrite(path: String, spark: SparkSession): Unit
- Definition Classes
- SummarizationModel → ParamsAndFeaturesWritable
-
val
optionalInputAnnotatorTypes: Array[String]
- Definition Classes
- HasInputAnnotationCols
-
val
outputAnnotatorType: String
Output Annotator Type: DOCUMENT
Output Annotator Type: DOCUMENT
- Definition Classes
- SummarizationModel → HasOutputAnnotatorType
-
final
val
outputCol: Param[String]
- Attributes
- protected
- Definition Classes
- HasOutputAnnotationCol
-
lazy val
params: Array[Param[_]]
- Definition Classes
- Params
-
var
parent: Estimator[SummarizationModel]
- Definition Classes
- Model
-
val
positionBias: FloatParam
Weight of the lead-position prior in extractive sentence ranking (Default:
0.3).Weight of the lead-position prior in extractive sentence ranking (Default:
0.3). Only applies to theextractivemethod.- Definition Classes
- SummarizationParams
-
val
resolvedModel: Param[String]
Name of the pretrained model resolved at fit time (for transparency metadata).
-
def
save(path: String): Unit
- Definition Classes
- MLWritable
- Annotations
- @Since( "1.6.0" ) @throws( ... )
-
def
set[T](feature: StructFeature[T], value: T): SummarizationModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[K, V](feature: MapFeature[K, V], value: Map[K, V]): SummarizationModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[T](feature: SetFeature[T], value: Set[T]): SummarizationModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[T](feature: ArrayFeature[T], value: Array[T]): SummarizationModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
set(paramPair: ParamPair[_]): SummarizationModel.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
set(param: String, value: Any): SummarizationModel.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
set[T](param: Param[T], value: T): SummarizationModel.this.type
- Definition Classes
- Params
-
def
setChunkOverlap(value: Int): SummarizationModel.this.type
- Definition Classes
- SummarizationParams
-
def
setChunkSize(value: Int): SummarizationModel.this.type
- Definition Classes
- SummarizationParams
-
def
setDefault[T](feature: StructFeature[T], value: () ⇒ T): SummarizationModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[K, V](feature: MapFeature[K, V], value: () ⇒ Map[K, V]): SummarizationModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[T](feature: SetFeature[T], value: () ⇒ Set[T]): SummarizationModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[T](feature: ArrayFeature[T], value: () ⇒ Array[T]): SummarizationModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
setDefault(paramPairs: ParamPair[_]*): SummarizationModel.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
setDefault[T](param: Param[T], value: T): SummarizationModel.this.type
- Attributes
- protected[org.apache.spark.ml]
- Definition Classes
- Params
- def setEmbeddingsDelegate(value: MPNetEmbeddings): SummarizationModel.this.type
-
def
setExtraInputCols(value: Array[String]): SummarizationModel.this.type
- Definition Classes
- HasInputAnnotationCols
-
def
setFocus(value: String): SummarizationModel.this.type
- Definition Classes
- SummarizationParams
-
def
setGpuLayers(value: Int): SummarizationModel.this.type
- Definition Classes
- SummarizationParams
-
final
def
setInputCols(value: String*): SummarizationModel.this.type
- Definition Classes
- HasInputAnnotationCols
-
def
setInputCols(value: Array[String]): SummarizationModel.this.type
Overrides required annotators column if different than default
Overrides required annotators column if different than default
- Definition Classes
- HasInputAnnotationCols
-
def
setLazyAnnotator(value: Boolean): SummarizationModel.this.type
- Definition Classes
- CanBeLazy
- def setLlmDelegate(value: AutoGGUFModel): SummarizationModel.this.type
-
def
setLongDocumentStrategy(value: String): SummarizationModel.this.type
- Definition Classes
- SummarizationParams
-
def
setMaxSummaryLength(value: Int): SummarizationModel.this.type
- Definition Classes
- SummarizationParams
-
def
setMethod(value: String): SummarizationModel.this.type
- Definition Classes
- SummarizationParams
-
def
setMinSummaryLength(value: Int): SummarizationModel.this.type
- Definition Classes
- SummarizationParams
-
def
setMmrLambda(value: Float): SummarizationModel.this.type
- Definition Classes
- SummarizationParams
-
def
setModel(value: String): SummarizationModel.this.type
- Definition Classes
- SummarizationParams
-
def
setNoRepeatNgramSize(value: Int): SummarizationModel.this.type
- Definition Classes
- SummarizationParams
-
def
setNumBeams(value: Int): SummarizationModel.this.type
- Definition Classes
- SummarizationParams
-
final
def
setOutputCol(value: String): SummarizationModel.this.type
Overrides annotation column name when transforming
Overrides annotation column name when transforming
- Definition Classes
- HasOutputAnnotationCol
-
def
setParent(parent: Estimator[SummarizationModel]): SummarizationModel
- Definition Classes
- Model
-
def
setPositionBias(value: Float): SummarizationModel.this.type
- Definition Classes
- SummarizationParams
- def setResolvedModel(value: String): SummarizationModel.this.type
- def setSeq2SeqDelegate(value: BartTransformer): SummarizationModel.this.type
-
def
setSummaryStyle(value: String): SummarizationModel.this.type
- Definition Classes
- SummarizationParams
-
def
setTemperature(value: Float): SummarizationModel.this.type
- Definition Classes
- SummarizationParams
-
def
setTopP(value: Float): SummarizationModel.this.type
- Definition Classes
- SummarizationParams
-
val
summaryStyle: Param[String]
Summary style used to build the LLM prompt:
concise,detailedorbullets(Default:concise).Summary style used to build the LLM prompt:
concise,detailedorbullets(Default:concise). Only applies to thellmmethod.- Definition Classes
- SummarizationParams
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
val
temperature: FloatParam
Generation temperature (Default:
0.2).Generation temperature (Default:
0.2). Applies to thellmmethod.- Definition Classes
- SummarizationParams
-
def
toString(): String
- Definition Classes
- Identifiable → AnyRef → Any
-
val
topP: FloatParam
Top-p (nucleus) sampling (Default:
0.9).Top-p (nucleus) sampling (Default:
0.9). Applies to thellmmethod.- Definition Classes
- SummarizationParams
-
def
transform(dataset: Dataset[_]): DataFrame
- Definition Classes
- SummarizationModel → Transformer
-
def
transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame
- Definition Classes
- Transformer
- Annotations
- @Since( "2.0.0" )
-
def
transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame
- Definition Classes
- Transformer
- Annotations
- @Since( "2.0.0" ) @varargs()
-
final
def
transformSchema(schema: StructType): StructType
requirement for pipeline transformation validation.
requirement for pipeline transformation validation. It is called on fit()
- Definition Classes
- RawAnnotator → PipelineStage
-
def
transformSchema(schema: StructType, logging: Boolean): StructType
- Attributes
- protected
- Definition Classes
- PipelineStage
- Annotations
- @DeveloperApi()
-
val
uid: String
- Definition Classes
- SummarizationModel → Identifiable
-
def
validate(schema: StructType): Boolean
takes a Dataset and checks to see if all the required annotation types are present.
takes a Dataset and checks to see if all the required annotation types are present.
- schema
to be validated
- returns
True if all the required types are present, else false
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
wrapColumnMetadata(col: Column): Column
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
def
write: MLWriter
- Definition Classes
- ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable
Inherited from SummarizationParams
Inherited from CanBeLazy
Inherited from RawAnnotator[SummarizationModel]
Inherited from HasOutputAnnotationCol
Inherited from HasInputAnnotationCols
Inherited from HasOutputAnnotatorType
Inherited from ParamsAndFeaturesWritable
Inherited from HasFeatures
Inherited from DefaultParamsWritable
Inherited from MLWritable
Inherited from Model[SummarizationModel]
Inherited from Transformer
Inherited from PipelineStage
Inherited from Logging
Inherited from Params
Inherited from Serializable
Inherited from Serializable
Inherited from Identifiable
Inherited from AnyRef
Inherited from Any
Parameters
Annotator types
Required input and expected output annotator types