VisionEncoderDecoderForImageCaptioning

Companion object VisionEncoderDecoderForImageCaptioning

class VisionEncoderDecoderForImageCaptioning extends AnnotatorModel[VisionEncoderDecoderForImageCaptioning] with HasBatchedAnnotateImage[VisionEncoderDecoderForImageCaptioning] with HasImageFeatureProperties with WriteTensorflowModel with WriteOnnxModel with HasEngine with HasRescaleFactor with HasGeneratorProperties

VisionEncoderDecoder model that converts images into text captions. It allows for the use of pretrained vision auto-encoding models, such as ViT, BEiT, or DeiT as the encoder, in combination with pretrained language models, like RoBERTa, GPT2, or BERT as the decoder.

Pretrained models can be loaded with pretrained of the companion object:

val imageClassifier = VisionEncoderDecoderForImageCaptioning.pretrained()
  .setInputCols("image_assembler")
  .setOutputCol("caption")

The default model is "image_captioning_vit_gpt2", if no name is provided.

For available pretrained models please see the Models Hub.

Models from the HuggingFace 🤗 Transformers library are also compatible with Spark NLP 🚀. To see which models are compatible and how to import them see https://github.com/JohnSnowLabs/spark-nlp/discussions/5669 and to see more extended examples, see VisionEncoderDecoderTestSpec.

Note:

This is a very computationally expensive module especially on larger batch sizes. The use of an accelerator such as GPU is recommended.

Example

import com.johnsnowlabs.nlp.annotator._
import com.johnsnowlabs.nlp.ImageAssembler
import org.apache.spark.ml.Pipeline

val imageDF: DataFrame = spark.read
  .format("image")
  .option("dropInvalid", value = true)
  .load("src/test/resources/image/")

val imageAssembler = new ImageAssembler()
  .setInputCol("image")
  .setOutputCol("image_assembler")

val imageCaptioning = VisionEncoderDecoderForImageCaptioning
  .pretrained()
  .setBeamSize(2)
  .setDoSample(false)
  .setInputCols("image_assembler")
  .setOutputCol("caption")

val pipeline = new Pipeline().setStages(Array(imageAssembler, imageCaptioning))
val pipelineDF = pipeline.fit(imageDF).transform(imageDF)

pipelineDF
  .selectExpr("reverse(split(image.origin, '/'))[0] as image_name", "caption.result")
  .show(truncate = false)

+-----------------+---------------------------------------------------------+
|image_name       |result                                                   |
+-----------------+---------------------------------------------------------+
|palace.JPEG      |[a large room filled with furniture and a large window]  |
|egyptian_cat.jpeg|[a cat laying on a couch next to another cat]            |
|hippopotamus.JPEG|[a brown bear in a body of water]                        |
|hen.JPEG         |[a flock of chickens standing next to each other]        |
|ostrich.JPEG     |[a large bird standing on top of a lush green field]     |
|junco.JPEG       |[a small bird standing on a wet ground]                  |
|bluetick.jpg     |[a small dog standing on a wooden floor]                 |
|chihuahua.jpg    |[a small brown dog wearing a blue sweater]               |
|tractor.JPEG     |[a man is standing in a field with a tractor]            |
|ox.JPEG          |[a large brown cow standing on top of a lush green field]|
+-----------------+---------------------------------------------------------+

Linear Supertypes

HasGeneratorProperties, HasRescaleFactor, HasEngine, WriteOnnxModel, WriteTensorflowModel, HasImageFeatureProperties, HasBatchedAnnotateImage[VisionEncoderDecoderForImageCaptioning], AnnotatorModel[VisionEncoderDecoderForImageCaptioning], CanBeLazy, RawAnnotator[VisionEncoderDecoderForImageCaptioning], HasOutputAnnotationCol, HasInputAnnotationCols, HasOutputAnnotatorType, ParamsAndFeaturesWritable, HasFeatures, DefaultParamsWritable, MLWritable, Model[VisionEncoderDecoderForImageCaptioning], Transformer, PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any

Ordering

Grouped
Alphabetic
By Inheritance

Inherited

VisionEncoderDecoderForImageCaptioning
HasGeneratorProperties
HasRescaleFactor
HasEngine
WriteOnnxModel
WriteTensorflowModel
HasImageFeatureProperties
HasBatchedAnnotateImage
AnnotatorModel
CanBeLazy
RawAnnotator
HasOutputAnnotationCol
HasInputAnnotationCols
HasOutputAnnotatorType
ParamsAndFeaturesWritable
HasFeatures
DefaultParamsWritable
MLWritable
Model
Transformer
PipelineStage
Logging
Params
Serializable
Serializable
Identifiable
AnyRef
Any

Hide All
Show All

Visibility

Public
All

Instance Constructors

new VisionEncoderDecoderForImageCaptioning()
Annotator reference id.
Annotator reference id. Used to identify elements in metadata or to refer to this annotator type
new VisionEncoderDecoderForImageCaptioning(uid: String)
uid
required uid for storing annotator to disk

Type Members

type AnnotationContent = Seq[Row]
internal types to show Rows as a relevant StructType Should be deleted once Spark releases UserDefinedTypes to @developerAPI
internal types to show Rows as a relevant StructType Should be deleted once Spark releases UserDefinedTypes to @developerAPI

Attributes
protected
Definition Classes
AnnotatorModel
type AnnotatorType = String

Definition Classes
HasOutputAnnotatorType

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def $[T](param: Param[T]): T

Attributes
protected
Definition Classes
Params
def $$[T](feature: StructFeature[T]): T

Attributes
protected
Definition Classes
HasFeatures
def $$[K, V](feature: MapFeature[K, V]): Map[K, V]

Attributes
protected
Definition Classes
HasFeatures
def $$[T](feature: SetFeature[T]): Set[T]

Attributes
protected
Definition Classes
HasFeatures
def $$[T](feature: ArrayFeature[T]): Array[T]

Attributes
protected
Definition Classes
HasFeatures
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def _transform(dataset: Dataset[_], recursivePipeline: Option[PipelineModel]): DataFrame

Attributes
protected
Definition Classes
AnnotatorModel
def afterAnnotate(dataset: DataFrame): DataFrame

Attributes
protected
Definition Classes
AnnotatorModel
final def asInstanceOf[T0]: T0

Definition Classes
Any
def batchAnnotate(batchedAnnotations: Seq[Array[AnnotationImage]]): Seq[Seq[Annotation]]
Takes a document and annotations and produces new annotations of this annotator's annotation type
Takes a document and annotations and produces new annotations of this annotator's annotation type
batchedAnnotations
Annotations that correspond to inputAnnotationCols generated by previous annotators if any
returns
any number of annotations processed for every input annotation. Not necessary one to one relationship

Definition Classes
VisionEncoderDecoderForImageCaptioning → HasBatchedAnnotateImage
def batchProcess(rows: Iterator[_]): Iterator[Row]

Definition Classes
HasBatchedAnnotateImage
val batchSize: IntParam
Size of every batch (Default depends on model).
Size of every batch (Default depends on model).

Definition Classes
HasBatchedAnnotateImage
val beamSize: IntParam
Beam size for the beam search algorithm (Default: 4)
Beam size for the beam search algorithm (Default: 4)

Definition Classes
HasGeneratorProperties
def beforeAnnotate(dataset: Dataset[_]): Dataset[_]

Attributes
protected
Definition Classes
AnnotatorModel
final def checkSchema(schema: StructType, inputAnnotatorType: String): Boolean

Attributes
protected
Definition Classes
HasInputAnnotationCols
final def clear(param: Param[_]): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
Params
def clone(): AnyRef

Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws( ... ) @native()
val configProtoBytes: IntArrayParam
ConfigProto from tensorflow, serialized into byte array.
ConfigProto from tensorflow, serialized into byte array. Get with config_proto.SerializeToString()
def copy(extra: ParamMap): VisionEncoderDecoderForImageCaptioning
requirement for annotators copies
requirement for annotators copies

Definition Classes
RawAnnotator → Model → Transformer → PipelineStage → Params
def copyValues[T <: Params](to: T, extra: ParamMap): T

Attributes
protected
Definition Classes
Params
final def defaultCopy[T <: Params](extra: ParamMap): T

Attributes
protected
Definition Classes
Params
val doNormalize: BooleanParam
Whether or not to normalize the input with mean and standard deviation
Whether or not to normalize the input with mean and standard deviation

Definition Classes
HasImageFeatureProperties
val doRescale: BooleanParam
Whether to rescale the image values by rescaleFactor.
Whether to rescale the image values by rescaleFactor.

Definition Classes
HasRescaleFactor
val doResize: BooleanParam
Whether to resize the input to a certain size
Whether to resize the input to a certain size

Definition Classes
HasImageFeatureProperties
val doSample: BooleanParam
Whether or not to use sampling, use greedy decoding otherwise (Default: false)
Whether or not to use sampling, use greedy decoding otherwise (Default: false)

Definition Classes
HasGeneratorProperties
val engine: Param[String]
This param is set internally once via loadSavedModel.
This param is set internally once via loadSavedModel. That's why there is no setter

Definition Classes
HasEngine
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def explainParam(param: Param[_]): String

Definition Classes
Params
def explainParams(): String

Definition Classes
Params
def extraValidate(structType: StructType): Boolean

Attributes
protected
Definition Classes
RawAnnotator
def extraValidateMsg: String
Override for additional custom schema checks
Override for additional custom schema checks

Attributes
protected
Definition Classes
RawAnnotator
final def extractParamMap(): ParamMap

Definition Classes
Params
final def extractParamMap(extra: ParamMap): ParamMap

Definition Classes
Params
val featureExtractorType: Param[String]
Name of model's architecture for feature extraction
Name of model's architecture for feature extraction

Definition Classes
HasImageFeatureProperties
val features: ArrayBuffer[Feature[_, _, _]]

Definition Classes
HasFeatures
def finalize(): Unit

Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
val generationConfig: StructFeature[GenerationConfig]

Attributes
protected[nlp]
def get[T](feature: StructFeature[T]): Option[T]

Attributes
protected
Definition Classes
HasFeatures
def get[K, V](feature: MapFeature[K, V]): Option[Map[K, V]]

Attributes
protected
Definition Classes
HasFeatures
def get[T](feature: SetFeature[T]): Option[Set[T]]

Attributes
protected
Definition Classes
HasFeatures
def get[T](feature: ArrayFeature[T]): Option[Array[T]]

Attributes
protected
Definition Classes
HasFeatures
final def get[T](param: Param[T]): Option[T]

Definition Classes
Params
def getBatchSize: Int
Size of every batch.
Size of every batch.

Definition Classes
HasBatchedAnnotateImage
def getBeamSize: Int

Definition Classes
HasGeneratorProperties
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
Annotations
@native()
def getConfigProtoBytes: Option[Array[Byte]]
ConfigProto from tensorflow, serialized into byte array.
ConfigProto from tensorflow, serialized into byte array. Get with config_proto.SerializeToString()
final def getDefault[T](param: Param[T]): Option[T]

Definition Classes
Params
def getDoNormalize: Boolean

Definition Classes
HasImageFeatureProperties
def getDoRescale: Boolean

Definition Classes
HasRescaleFactor
def getDoResize: Boolean

Definition Classes
HasImageFeatureProperties
def getDoSample: Boolean

Definition Classes
HasGeneratorProperties
def getEngine: String

Definition Classes
HasEngine
def getFeatureExtractorType: String

Definition Classes
HasImageFeatureProperties
def getGenerationConfig: GenerationConfig

Attributes
protected[nlp]
def getImageMean: Array[Double]

Definition Classes
HasImageFeatureProperties
def getImageStd: Array[Double]

Definition Classes
HasImageFeatureProperties
def getInputCols: Array[String]
returns
input annotations columns currently used

Definition Classes
HasInputAnnotationCols
def getLazyAnnotator: Boolean

Definition Classes
CanBeLazy
def getMaxOutputLength: Int

Definition Classes
HasGeneratorProperties
def getMerges: Map[(String, String), Int]

Attributes
protected[nlp]
def getMinOutputLength: Int

Definition Classes
HasGeneratorProperties
def getModelIfNotSet: VisionEncoderDecoder
def getNReturnSequences: Int

Definition Classes
HasGeneratorProperties
def getNoRepeatNgramSize: Int

Definition Classes
HasGeneratorProperties
final def getOrDefault[T](param: Param[T]): T

Definition Classes
Params
final def getOutputCol: String
Gets annotation column name going to generate
Gets annotation column name going to generate

Definition Classes
HasOutputAnnotationCol
def getParam(paramName: String): Param[Any]

Definition Classes
Params
def getRandomSeed: Option[Long]

Definition Classes
HasGeneratorProperties
def getRepetitionPenalty: Double

Definition Classes
HasGeneratorProperties
def getResample: Int

Definition Classes
HasImageFeatureProperties
def getRescaleFactor: Double

Definition Classes
HasRescaleFactor
def getSignatures: Option[Map[String, String]]
def getSize: Int

Definition Classes
HasImageFeatureProperties
def getStopTokenIds: Array[Int]

Definition Classes
HasGeneratorProperties
def getTask: Option[String]

Definition Classes
HasGeneratorProperties
def getTemperature: Double

Definition Classes
HasGeneratorProperties
def getTopK: Int

Definition Classes
HasGeneratorProperties
def getTopP: Double

Definition Classes
HasGeneratorProperties
def getVocabulary: Map[String, Int]

Attributes
protected[nlp]
final def hasDefault[T](param: Param[T]): Boolean

Definition Classes
Params
def hasParam(paramName: String): Boolean

Definition Classes
Params
def hasParent: Boolean

Definition Classes
Model
def hashCode(): Int

Definition Classes
AnyRef → Any
Annotations
@native()
val imageMean: DoubleArrayParam
The sequence of means for each channel, to be used when normalizing images
The sequence of means for each channel, to be used when normalizing images

Definition Classes
HasImageFeatureProperties
val imageStd: DoubleArrayParam
The sequence of standard deviations for each channel, to be used when normalizing images
The sequence of standard deviations for each channel, to be used when normalizing images

Definition Classes
HasImageFeatureProperties
def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

Attributes
protected
Definition Classes
Logging
def initializeLogIfNecessary(isInterpreter: Boolean): Unit

Attributes
protected
Definition Classes
Logging
val inputAnnotatorTypes: Array[AnnotatorType]
Input annotator type : IMAGE
Input annotator type : IMAGE

Definition Classes
VisionEncoderDecoderForImageCaptioning → HasInputAnnotationCols
final val inputCols: StringArrayParam
columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified
columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified

Attributes
protected
Definition Classes
HasInputAnnotationCols
final def isDefined(param: Param[_]): Boolean

Definition Classes
Params
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
final def isSet(param: Param[_]): Boolean

Definition Classes
Params
def isTraceEnabled(): Boolean

Attributes
protected
Definition Classes
Logging
val lazyAnnotator: BooleanParam

Definition Classes
CanBeLazy
def log: Logger

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logName: String

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
val maxInputLength: IntParam
max length of the input sequence (Default: 0)
max length of the input sequence (Default: 0)

Definition Classes
HasGeneratorProperties
val maxOutputLength: IntParam
Maximum length of the sequence to be generated (Default: 20)
Maximum length of the sequence to be generated (Default: 20)

Definition Classes
HasGeneratorProperties
val merges: MapFeature[(String, String), Int]
Holding merges.txt for BPE Tokenization
Holding merges.txt for BPE Tokenization

Attributes
protected[nlp]
val minOutputLength: IntParam
Minimum length of the sequence to be generated (Default: 0)
Minimum length of the sequence to be generated (Default: 0)

Definition Classes
HasGeneratorProperties
def msgHelper(schema: StructType): String

Attributes
protected
Definition Classes
HasInputAnnotationCols
val nReturnSequences: IntParam
The number of sequences to return from the beam search.
The number of sequences to return from the beam search.

Definition Classes
HasGeneratorProperties
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
val noRepeatNgramSize: IntParam
If set to int > 0, all ngrams of that size can only occur once (Default: 0)
If set to int > 0, all ngrams of that size can only occur once (Default: 0)

Definition Classes
HasGeneratorProperties
final def notify(): Unit

Definition Classes
AnyRef
Annotations
@native()
final def notifyAll(): Unit

Definition Classes
AnyRef
Annotations
@native()
def onWrite(path: String, spark: SparkSession): Unit

Definition Classes
VisionEncoderDecoderForImageCaptioning → ParamsAndFeaturesWritable
val optionalInputAnnotatorTypes: Array[String]

Definition Classes
HasInputAnnotationCols
val outputAnnotatorType: AnnotatorType
Output annotator type : CATEGORY
Output annotator type : CATEGORY

Definition Classes
VisionEncoderDecoderForImageCaptioning → HasOutputAnnotatorType
final val outputCol: Param[String]

Attributes
protected
Definition Classes
HasOutputAnnotationCol
lazy val params: Array[Param[_]]

Definition Classes
Params
var parent: Estimator[VisionEncoderDecoderForImageCaptioning]

Definition Classes
Model
val randomSeed: Option[Long]
Optional Random seed for the model.
Optional Random seed for the model. Needs to be of type Int.

Definition Classes
HasGeneratorProperties
val repetitionPenalty: DoubleParam
The parameter for repetition penalty (Default: 1.0).
The parameter for repetition penalty (Default: 1.0). 1.0 means no penalty. See this paper for more details.

Definition Classes
HasGeneratorProperties
val resample: IntParam
An optional resampling filter.
An optional resampling filter. This can be one of PIL.Image.NEAREST, PIL.Image.BOX, PIL.Image.BILINEAR, PIL.Image.HAMMING, PIL.Image.BICUBIC or PIL.Image.LANCZOS. Only has an effect if do_resize is set to True

Definition Classes
HasImageFeatureProperties
val rescaleFactor: DoubleParam
Factor to scale the image values (Default: 1 / 255.0).
Factor to scale the image values (Default: 1 / 255.0).

Definition Classes
HasRescaleFactor
def save(path: String): Unit

Definition Classes
MLWritable
Annotations
@Since( "1.6.0" ) @throws( ... )
def set[T](feature: StructFeature[T], value: T): VisionEncoderDecoderForImageCaptioning.this.type

Attributes
protected
Definition Classes
HasFeatures
def set[K, V](feature: MapFeature[K, V], value: Map[K, V]): VisionEncoderDecoderForImageCaptioning.this.type

Attributes
protected
Definition Classes
HasFeatures
def set[T](feature: SetFeature[T], value: Set[T]): VisionEncoderDecoderForImageCaptioning.this.type

Attributes
protected
Definition Classes
HasFeatures
def set[T](feature: ArrayFeature[T], value: Array[T]): VisionEncoderDecoderForImageCaptioning.this.type

Attributes
protected
Definition Classes
HasFeatures
final def set(paramPair: ParamPair[_]): VisionEncoderDecoderForImageCaptioning.this.type

Attributes
protected
Definition Classes
Params
final def set(param: String, value: Any): VisionEncoderDecoderForImageCaptioning.this.type

Attributes
protected
Definition Classes
Params
final def set[T](param: Param[T], value: T): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
Params
def setBatchSize(size: Int): VisionEncoderDecoderForImageCaptioning.this.type
Size of every batch.
Size of every batch.

Definition Classes
HasBatchedAnnotateImage
def setBeamSize(beamNum: Int): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasGeneratorProperties
def setConfigProtoBytes(bytes: Array[Int]): VisionEncoderDecoderForImageCaptioning.this.type
ConfigProto from tensorflow, serialized into byte array.
ConfigProto from tensorflow, serialized into byte array. Get with config_proto.SerializeToString()
def setDefault[T](feature: StructFeature[T], value: () ⇒ T): VisionEncoderDecoderForImageCaptioning.this.type

Attributes
protected
Definition Classes
HasFeatures
def setDefault[K, V](feature: MapFeature[K, V], value: () ⇒ Map[K, V]): VisionEncoderDecoderForImageCaptioning.this.type

Attributes
protected
Definition Classes
HasFeatures
def setDefault[T](feature: SetFeature[T], value: () ⇒ Set[T]): VisionEncoderDecoderForImageCaptioning.this.type

Attributes
protected
Definition Classes
HasFeatures
def setDefault[T](feature: ArrayFeature[T], value: () ⇒ Array[T]): VisionEncoderDecoderForImageCaptioning.this.type

Attributes
protected
Definition Classes
HasFeatures
final def setDefault(paramPairs: ParamPair[_]*): VisionEncoderDecoderForImageCaptioning.this.type

Attributes
protected
Definition Classes
Params
final def setDefault[T](param: Param[T], value: T): VisionEncoderDecoderForImageCaptioning.this.type

Attributes
protected[org.apache.spark.ml]
Definition Classes
Params
def setDoNormalize(value: Boolean): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasImageFeatureProperties
def setDoRescale(value: Boolean): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasRescaleFactor
def setDoResize(value: Boolean): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasImageFeatureProperties
def setDoSample(value: Boolean): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasGeneratorProperties
def setFeatureExtractorType(value: String): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasImageFeatureProperties
def setGenerationConfig(value: GenerationConfig): VisionEncoderDecoderForImageCaptioning.this.type

Attributes
protected[nlp]
def setImageMean(value: Array[Double]): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasImageFeatureProperties
def setImageStd(value: Array[Double]): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasImageFeatureProperties
final def setInputCols(value: String*): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasInputAnnotationCols
def setInputCols(value: Array[String]): VisionEncoderDecoderForImageCaptioning.this.type
Overrides required annotators column if different than default
Overrides required annotators column if different than default

Definition Classes
HasInputAnnotationCols
def setLazyAnnotator(value: Boolean): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
CanBeLazy
def setMaxInputLength(value: Int): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasGeneratorProperties
def setMaxOutputLength(value: Int): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasGeneratorProperties
def setMerges(value: Map[(String, String), Int]): VisionEncoderDecoderForImageCaptioning.this.type

Attributes
protected[nlp]
def setMinOutputLength(value: Int): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasGeneratorProperties
def setModelIfNotSet(spark: SparkSession, tensorflowWrapper: Option[TensorflowWrapper], onnxWrapper: Option[EncoderDecoderWithoutPastWrappers], preprocessor: Preprocessor): VisionEncoderDecoderForImageCaptioning.this.type
def setNReturnSequences(beamNum: Int): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasGeneratorProperties
def setNoRepeatNgramSize(value: Int): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasGeneratorProperties
final def setOutputCol(value: String): VisionEncoderDecoderForImageCaptioning.this.type
Overrides annotation column name when transforming
Overrides annotation column name when transforming

Definition Classes
HasOutputAnnotationCol
def setParent(parent: Estimator[VisionEncoderDecoderForImageCaptioning]): VisionEncoderDecoderForImageCaptioning

Definition Classes
Model
def setRandomSeed(value: Long): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasGeneratorProperties
def setRepetitionPenalty(value: Double): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasGeneratorProperties
def setResample(value: Int): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasImageFeatureProperties
def setRescaleFactor(value: Double): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasRescaleFactor
def setSignatures(value: Map[String, String]): VisionEncoderDecoderForImageCaptioning.this.type
def setSize(value: Int): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasImageFeatureProperties
def setStopTokenIds(value: Array[Int]): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasGeneratorProperties
def setTask(value: String): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasGeneratorProperties
def setTemperature(value: Double): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasGeneratorProperties
def setTopK(value: Int): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasGeneratorProperties
def setTopP(value: Double): VisionEncoderDecoderForImageCaptioning.this.type

Definition Classes
HasGeneratorProperties
def setVocabulary(value: Map[String, Int]): VisionEncoderDecoderForImageCaptioning.this.type

Attributes
protected[nlp]
val signatures: MapFeature[String, String]
It contains TF model signatures for the laded saved model
val size: IntParam
Resize the input to the given size.
Resize the input to the given size. If a tuple is provided, it should be (width, height). If only an integer is provided, then the input will be resized to (size, size). Only has an effect if do_resize is set to True.

Definition Classes
HasImageFeatureProperties
val stopTokenIds: IntArrayParam
Stop tokens to terminate the generation
Stop tokens to terminate the generation

Definition Classes
HasGeneratorProperties
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
val task: Param[String]
Set transformer task, e.g.
Set transformer task, e.g. "summarize:" (Default: "").

Definition Classes
HasGeneratorProperties
val temperature: DoubleParam
The value used to module the next token probabilities (Default: 1.0)
The value used to module the next token probabilities (Default: 1.0)

Definition Classes
HasGeneratorProperties
def toString(): String

Definition Classes
Identifiable → AnyRef → Any
val topK: IntParam
The number of highest probability vocabulary tokens to keep for top-k-filtering (Default: 50)
The number of highest probability vocabulary tokens to keep for top-k-filtering (Default: 50)

Definition Classes
HasGeneratorProperties
val topP: DoubleParam
If set to float < 1.0, only the most probable tokens with probabilities that add up to topP or higher are kept for generation (Default: 1.0)
If set to float < 1.0, only the most probable tokens with probabilities that add up to topP or higher are kept for generation (Default: 1.0)

Definition Classes
HasGeneratorProperties
final def transform(dataset: Dataset[_]): DataFrame
Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content
Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content
dataset
Dataset[Row]

Definition Classes
AnnotatorModel → Transformer
def transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame

Definition Classes
Transformer
Annotations
@Since( "2.0.0" )
def transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame

Definition Classes
Transformer
Annotations
@Since( "2.0.0" ) @varargs()
final def transformSchema(schema: StructType): StructType
requirement for pipeline transformation validation.
requirement for pipeline transformation validation. It is called on fit()

Definition Classes
RawAnnotator → PipelineStage
def transformSchema(schema: StructType, logging: Boolean): StructType

Attributes
protected
Definition Classes
PipelineStage
Annotations
@DeveloperApi()
val uid: String

Definition Classes
VisionEncoderDecoderForImageCaptioning → Identifiable
def validate(schema: StructType): Boolean
takes a Dataset and checks to see if all the required annotation types are present.
takes a Dataset and checks to see if all the required annotation types are present.
schema
to be validated
returns
True if all the required types are present, else false

Attributes
protected
Definition Classes
RawAnnotator
val vocabulary: MapFeature[String, Int]
Vocabulary used to encode the words to ids with bpeTokenizer.encode
Vocabulary used to encode the words to ids with bpeTokenizer.encode

Attributes
protected[nlp]
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... ) @native()
def wrapColumnMetadata(col: Column): Column

Attributes
protected
Definition Classes
RawAnnotator
def write: MLWriter

Definition Classes
ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable
def writeOnnxModel(path: String, spark: SparkSession, onnxWrapper: OnnxWrapper, suffix: String, fileName: String): Unit

Definition Classes
WriteOnnxModel
def writeOnnxModels(path: String, spark: SparkSession, onnxWrappersWithNames: Seq[(OnnxWrapper, String)], suffix: String): Unit

Definition Classes
WriteOnnxModel
def writeTensorflowHub(path: String, tfPath: String, spark: SparkSession, suffix: String = "_use"): Unit

Definition Classes
WriteTensorflowModel
def writeTensorflowModel(path: String, spark: SparkSession, tensorflow: TensorflowWrapper, suffix: String, filename: String, configProtoBytes: Option[Array[Byte]] = None): Unit

Definition Classes
WriteTensorflowModel
def writeTensorflowModelV2(path: String, spark: SparkSession, tensorflow: TensorflowWrapper, suffix: String, filename: String, configProtoBytes: Option[Array[Byte]] = None, savedSignatures: Option[Map[String, String]] = None): Unit

Definition Classes
WriteTensorflowModel

Inherited from HasGeneratorProperties

Inherited from HasRescaleFactor

Inherited from HasEngine

Inherited from WriteOnnxModel

Inherited from WriteTensorflowModel

Inherited from HasImageFeatureProperties

Inherited from HasBatchedAnnotateImage[VisionEncoderDecoderForImageCaptioning]

Inherited from AnnotatorModel[VisionEncoderDecoderForImageCaptioning]

Inherited from CanBeLazy

Inherited from RawAnnotator[VisionEncoderDecoderForImageCaptioning]

Inherited from HasOutputAnnotationCol

Inherited from HasInputAnnotationCols

Inherited from HasOutputAnnotatorType

Inherited from ParamsAndFeaturesWritable

Inherited from HasFeatures

Inherited from DefaultParamsWritable

Inherited from MLWritable

Inherited from Model[VisionEncoderDecoderForImageCaptioning]

Inherited from Transformer

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Parameters

A list of (hyper-)parameter keys this annotator can take. Users can set and get the parameter values through setters and getters, respectively.

Packages

VisionEncoderDecoderForImageCaptioning

Companion object VisionEncoderDecoderForImageCaptioning

Example

Instance Constructors

Type Members

Value Members

Inherited from HasGeneratorProperties

Inherited from HasRescaleFactor

Inherited from HasEngine

Inherited from WriteOnnxModel

Inherited from WriteTensorflowModel

Inherited from HasImageFeatureProperties

Inherited from HasBatchedAnnotateImage[VisionEncoderDecoderForImageCaptioning]

Inherited from AnnotatorModel[VisionEncoderDecoderForImageCaptioning]

Inherited from CanBeLazy

Inherited from RawAnnotator[VisionEncoderDecoderForImageCaptioning]

Inherited from HasOutputAnnotationCol

Inherited from HasInputAnnotationCols

Inherited from HasOutputAnnotatorType

Inherited from ParamsAndFeaturesWritable

Inherited from HasFeatures

Inherited from DefaultParamsWritable

Inherited from MLWritable

Inherited from Model[VisionEncoderDecoderForImageCaptioning]

Inherited from Transformer

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Parameters

Annotator types

Members

Parameter setters

Parameter getters

Packages

VisionEncoderDecoderForImageCaptioning 

Companion object VisionEncoderDecoderForImageCaptioning

Example

Instance Constructors

Type Members

Value Members

Inherited from HasGeneratorProperties

Inherited from HasRescaleFactor

Inherited from HasEngine

Inherited from WriteOnnxModel

Inherited from WriteTensorflowModel

Inherited from HasImageFeatureProperties

Inherited from HasBatchedAnnotateImage[VisionEncoderDecoderForImageCaptioning]

Inherited from AnnotatorModel[VisionEncoderDecoderForImageCaptioning]

Inherited from CanBeLazy

Inherited from RawAnnotator[VisionEncoderDecoderForImageCaptioning]

Inherited from HasOutputAnnotationCol

Inherited from HasInputAnnotationCols

Inherited from HasOutputAnnotatorType

Inherited from ParamsAndFeaturesWritable

Inherited from HasFeatures

Inherited from DefaultParamsWritable

Inherited from MLWritable

Inherited from Model[VisionEncoderDecoderForImageCaptioning]

Inherited from Transformer

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Parameters

Annotator types

Members

Parameter setters

Parameter getters

VisionEncoderDecoderForImageCaptioning