com.johnsnowlabs.nlp.annotators.cv
BLIPForQuestionAnswering
Companion object BLIPForQuestionAnswering
class BLIPForQuestionAnswering extends AnnotatorModel[BLIPForQuestionAnswering] with HasBatchedAnnotateImage[BLIPForQuestionAnswering] with HasImageFeatureProperties with WriteTensorflowModel with HasEngine
BLIPForQuestionAnswering can load BLIP models for visual question answering. The model consists of a vision encoder, a text encoder as well as a text decoder. The vision encoder will encode the input image, the text encoder will encode the input question together with the encoding of the image, and the text decoder will output the answer to the question.
Pretrained models can be loaded with pretrained
of the companion object:
val visualQAClassifier = BLIPForQuestionAnswering.pretrained() .setInputCols("image_assembler") .setOutputCol("answer")
The default model is "blip_vqa_base"
, if no name is provided.
For available pretrained models please see the Models Hub.
Models from the HuggingFace 🤗 Transformers library are also compatible with Spark NLP 🚀. To see which models are compatible and how to import them see https://github.com/JohnSnowLabs/spark-nlp/discussions/5669 and to see more extended examples, see https://github.com/JohnSnowLabs/spark-nlp/blob/master/src/test/scala/com/johnsnowlabs/nlp/annotators/cv/BLIPForQuestionAnsweringTest.scala.
Example
import spark.implicits._ import com.johnsnowlabs.nlp.base._ import com.johnsnowlabs.nlp.annotator._ import org.apache.spark.ml.Pipeline val imageDF: DataFrame = ResourceHelper.spark.read .format("image") .option("dropInvalid", value = true) .load(imageFolder) val testDF: DataFrame = imageDF.withColumn("text", lit("What's this picture about?")) val imageAssembler: ImageAssembler = new ImageAssembler() .setInputCol("image") .setOutputCol("image_assembler") val visualQAClassifier = BLIPForQuestionAnswering.pretrained() .setInputCols("image_assembler") .setOutputCol("answer") val pipeline = new Pipeline().setStages(Array( imageAssembler, visualQAClassifier )) val result = pipeline.fit(testDF).transform(testDF) result.select("image_assembler.origin", "answer.result").show(false) +--------------------------------------+------+ |origin |result| +--------------------------------------+------+ |[file:///content/images/cat_image.jpg]|[cats]| +--------------------------------------+------+
- See also
CLIPForZeroShotClassification for Zero Shot Image Classifier
Annotators Main Page for a list of transformer based classifiers
- Grouped
- Alphabetic
- By Inheritance
- BLIPForQuestionAnswering
- HasEngine
- WriteTensorflowModel
- HasImageFeatureProperties
- HasBatchedAnnotateImage
- AnnotatorModel
- CanBeLazy
- RawAnnotator
- HasOutputAnnotationCol
- HasInputAnnotationCols
- HasOutputAnnotatorType
- ParamsAndFeaturesWritable
- HasFeatures
- DefaultParamsWritable
- MLWritable
- Model
- Transformer
- PipelineStage
- Logging
- Params
- Serializable
- Serializable
- Identifiable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Parameters
A list of (hyper-)parameter keys this annotator can take. Users can set and get the parameter values through setters and getters, respectively.
-
val
batchSize: IntParam
Size of every batch (Default depends on model).
Size of every batch (Default depends on model).
- Definition Classes
- HasBatchedAnnotateImage
-
val
configProtoBytes: IntArrayParam
ConfigProto from tensorflow, serialized into byte array.
ConfigProto from tensorflow, serialized into byte array. Get with config_proto.SerializeToString()
-
val
doNormalize: BooleanParam
Whether or not to normalize the input with mean and standard deviation
Whether or not to normalize the input with mean and standard deviation
- Definition Classes
- HasImageFeatureProperties
-
val
doResize: BooleanParam
Whether to resize the input to a certain size
Whether to resize the input to a certain size
- Definition Classes
- HasImageFeatureProperties
-
val
engine: Param[String]
This param is set internally once via loadSavedModel.
This param is set internally once via loadSavedModel. That's why there is no setter
- Definition Classes
- HasEngine
-
val
featureExtractorType: Param[String]
Name of model's architecture for feature extraction
Name of model's architecture for feature extraction
- Definition Classes
- HasImageFeatureProperties
-
val
imageMean: DoubleArrayParam
The sequence of means for each channel, to be used when normalizing images
The sequence of means for each channel, to be used when normalizing images
- Definition Classes
- HasImageFeatureProperties
-
val
imageStd: DoubleArrayParam
The sequence of standard deviations for each channel, to be used when normalizing images
The sequence of standard deviations for each channel, to be used when normalizing images
- Definition Classes
- HasImageFeatureProperties
-
val
maxSentenceLength: IntParam
Max sentence length to process (Default:
512
) -
val
resample: IntParam
An optional resampling filter.
An optional resampling filter. This can be one of PIL.Image.NEAREST, PIL.Image.BOX, PIL.Image.BILINEAR, PIL.Image.HAMMING, PIL.Image.BICUBIC or PIL.Image.LANCZOS. Only has an effect if do_resize is set to True
- Definition Classes
- HasImageFeatureProperties
-
val
signatures: MapFeature[String, String]
It contains TF model signatures for the laded saved model
-
val
size: IntParam
Resize the input to the given size.
Resize the input to the given size. If a tuple is provided, it should be (width, height). If only an integer is provided, then the input will be resized to (size, size). Only has an effect if do_resize is set to True.
- Definition Classes
- HasImageFeatureProperties
-
val
vocabulary: MapFeature[String, Int]
Vocabulary used to encode the words to ids with WordPieceEncoder
Members
-
type
AnnotatorType = String
- Definition Classes
- HasOutputAnnotatorType
-
def
batchAnnotate(batchedAnnotations: Seq[Array[AnnotationImage]]): Seq[Seq[Annotation]]
takes a document and annotations and produces new annotations of this annotator's annotation type
takes a document and annotations and produces new annotations of this annotator's annotation type
- batchedAnnotations
Annotations in batches that correspond to inputAnnotationCols generated by previous annotators if any
- returns
any number of annotations processed for every batch of input annotations. Not necessary one to one relationship
- Definition Classes
- BLIPForQuestionAnswering → HasBatchedAnnotateImage
-
def
batchProcess(rows: Iterator[_]): Iterator[Row]
- Definition Classes
- HasBatchedAnnotateImage
-
final
def
clear(param: Param[_]): BLIPForQuestionAnswering.this.type
- Definition Classes
- Params
-
def
copy(extra: ParamMap): BLIPForQuestionAnswering
requirement for annotators copies
requirement for annotators copies
- Definition Classes
- RawAnnotator → Model → Transformer → PipelineStage → Params
-
def
explainParam(param: Param[_]): String
- Definition Classes
- Params
-
def
explainParams(): String
- Definition Classes
- Params
-
final
def
extractParamMap(): ParamMap
- Definition Classes
- Params
-
final
def
extractParamMap(extra: ParamMap): ParamMap
- Definition Classes
- Params
-
val
features: ArrayBuffer[Feature[_, _, _]]
- Definition Classes
- HasFeatures
-
final
def
get[T](param: Param[T]): Option[T]
- Definition Classes
- Params
-
final
def
getDefault[T](param: Param[T]): Option[T]
- Definition Classes
- Params
-
def
getInputCols: Array[String]
- returns
input annotations columns currently used
- Definition Classes
- HasInputAnnotationCols
-
def
getLazyAnnotator: Boolean
- Definition Classes
- CanBeLazy
-
final
def
getOrDefault[T](param: Param[T]): T
- Definition Classes
- Params
-
final
def
getOutputCol: String
Gets annotation column name going to generate
Gets annotation column name going to generate
- Definition Classes
- HasOutputAnnotationCol
-
def
getParam(paramName: String): Param[Any]
- Definition Classes
- Params
-
final
def
hasDefault[T](param: Param[T]): Boolean
- Definition Classes
- Params
-
def
hasParam(paramName: String): Boolean
- Definition Classes
- Params
-
def
hasParent: Boolean
- Definition Classes
- Model
-
val
inputAnnotatorTypes: Array[AnnotatorType]
Annotator reference id.
Annotator reference id. Used to identify elements in metadata or to refer to this annotator type
- Definition Classes
- BLIPForQuestionAnswering → HasInputAnnotationCols
-
final
def
isDefined(param: Param[_]): Boolean
- Definition Classes
- Params
-
final
def
isSet(param: Param[_]): Boolean
- Definition Classes
- Params
-
val
lazyAnnotator: BooleanParam
- Definition Classes
- CanBeLazy
-
def
onWrite(path: String, spark: SparkSession): Unit
- Definition Classes
- BLIPForQuestionAnswering → ParamsAndFeaturesWritable
-
val
optionalInputAnnotatorTypes: Array[String]
- Definition Classes
- HasInputAnnotationCols
-
val
outputAnnotatorType: AnnotatorType
- Definition Classes
- BLIPForQuestionAnswering → HasOutputAnnotatorType
-
lazy val
params: Array[Param[_]]
- Definition Classes
- Params
-
var
parent: Estimator[BLIPForQuestionAnswering]
- Definition Classes
- Model
-
def
save(path: String): Unit
- Definition Classes
- MLWritable
- Annotations
- @Since( "1.6.0" ) @throws( ... )
-
final
def
set[T](param: Param[T], value: T): BLIPForQuestionAnswering.this.type
- Definition Classes
- Params
-
final
def
setInputCols(value: String*): BLIPForQuestionAnswering.this.type
- Definition Classes
- HasInputAnnotationCols
-
def
setInputCols(value: Array[String]): BLIPForQuestionAnswering.this.type
Overrides required annotators column if different than default
Overrides required annotators column if different than default
- Definition Classes
- HasInputAnnotationCols
-
def
setLazyAnnotator(value: Boolean): BLIPForQuestionAnswering.this.type
- Definition Classes
- CanBeLazy
-
final
def
setOutputCol(value: String): BLIPForQuestionAnswering.this.type
Overrides annotation column name when transforming
Overrides annotation column name when transforming
- Definition Classes
- HasOutputAnnotationCol
-
def
setParent(parent: Estimator[BLIPForQuestionAnswering]): BLIPForQuestionAnswering
- Definition Classes
- Model
-
def
toString(): String
- Definition Classes
- Identifiable → AnyRef → Any
-
final
def
transform(dataset: Dataset[_]): DataFrame
Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content
Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content
- dataset
Dataset[Row]
- Definition Classes
- AnnotatorModel → Transformer
-
def
transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame
- Definition Classes
- Transformer
- Annotations
- @Since( "2.0.0" )
-
def
transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame
- Definition Classes
- Transformer
- Annotations
- @Since( "2.0.0" ) @varargs()
-
final
def
transformSchema(schema: StructType): StructType
requirement for pipeline transformation validation.
requirement for pipeline transformation validation. It is called on fit()
- Definition Classes
- RawAnnotator → PipelineStage
-
val
uid: String
- Definition Classes
- BLIPForQuestionAnswering → Identifiable
-
def
write: MLWriter
- Definition Classes
- ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable
-
def
writeTensorflowHub(path: String, tfPath: String, spark: SparkSession, suffix: String = "_use"): Unit
- Definition Classes
- WriteTensorflowModel
-
def
writeTensorflowModel(path: String, spark: SparkSession, tensorflow: TensorflowWrapper, suffix: String, filename: String, configProtoBytes: Option[Array[Byte]] = None): Unit
- Definition Classes
- WriteTensorflowModel
-
def
writeTensorflowModelV2(path: String, spark: SparkSession, tensorflow: TensorflowWrapper, suffix: String, filename: String, configProtoBytes: Option[Array[Byte]] = None, savedSignatures: Option[Map[String, String]] = None): Unit
- Definition Classes
- WriteTensorflowModel
Parameter setters
-
def
setBatchSize(size: Int): BLIPForQuestionAnswering.this.type
Size of every batch.
Size of every batch.
- Definition Classes
- HasBatchedAnnotateImage
-
def
setConfigProtoBytes(bytes: Array[Int]): BLIPForQuestionAnswering.this.type
ConfigProto from tensorflow, serialized into byte array.
ConfigProto from tensorflow, serialized into byte array. Get with config_proto.SerializeToString()
-
def
setDoNormalize(value: Boolean): BLIPForQuestionAnswering.this.type
- Definition Classes
- HasImageFeatureProperties
-
def
setDoResize(value: Boolean): BLIPForQuestionAnswering.this.type
- Definition Classes
- HasImageFeatureProperties
-
def
setFeatureExtractorType(value: String): BLIPForQuestionAnswering.this.type
- Definition Classes
- HasImageFeatureProperties
-
def
setImageMean(value: Array[Double]): BLIPForQuestionAnswering.this.type
- Definition Classes
- HasImageFeatureProperties
-
def
setImageStd(value: Array[Double]): BLIPForQuestionAnswering.this.type
- Definition Classes
- HasImageFeatureProperties
- def setMaxSentenceLength(value: Int): BLIPForQuestionAnswering.this.type
- def setModelIfNotSet(spark: SparkSession, preprocessor: Preprocessor, tensorflow: TensorflowWrapper): BLIPForQuestionAnswering.this.type
-
def
setResample(value: Int): BLIPForQuestionAnswering.this.type
- Definition Classes
- HasImageFeatureProperties
- def setSignatures(value: Map[String, String]): BLIPForQuestionAnswering.this.type
-
def
setSize(value: Int): BLIPForQuestionAnswering.this.type
- Definition Classes
- HasImageFeatureProperties
- def setVocabulary(value: Map[String, Int]): BLIPForQuestionAnswering.this.type
Parameter getters
-
def
getBatchSize: Int
Size of every batch.
Size of every batch.
- Definition Classes
- HasBatchedAnnotateImage
-
def
getConfigProtoBytes: Option[Array[Byte]]
ConfigProto from tensorflow, serialized into byte array.
ConfigProto from tensorflow, serialized into byte array. Get with config_proto.SerializeToString()
-
def
getDoNormalize: Boolean
- Definition Classes
- HasImageFeatureProperties
-
def
getDoResize: Boolean
- Definition Classes
- HasImageFeatureProperties
-
def
getEngine: String
- Definition Classes
- HasEngine
-
def
getFeatureExtractorType: String
- Definition Classes
- HasImageFeatureProperties
-
def
getImageMean: Array[Double]
- Definition Classes
- HasImageFeatureProperties
-
def
getImageStd: Array[Double]
- Definition Classes
- HasImageFeatureProperties
- def getMaxSentenceLength: Int
- def getModelIfNotSet: BLIPClassifier
-
def
getResample: Int
- Definition Classes
- HasImageFeatureProperties
- def getSignatures: Option[Map[String, String]]
-
def
getSize: Int
- Definition Classes
- HasImageFeatureProperties