class CLIPForZeroShotClassification extends AnnotatorModel[CLIPForZeroShotClassification] with HasBatchedAnnotateImage[CLIPForZeroShotClassification] with HasImageFeatureProperties with WriteTensorflowModel with WriteOnnxModel with WriteOpenvinoModel with HasEngine with HasRescaleFactor

Zero Shot Image Classifier based on CLIP.

CLIP (Contrastive Language-Image Pre-Training) is a neural network that was trained on image and text pairs. It has the ability to predict images without training on any hard-coded labels. This makes it very flexible, as labels can be provided during inference. This is similar to the zero-shot capabilities of the GPT-2 and 3 models.

Pretrained models can be loaded with pretrained of the companion object:

val imageClassifier = CLIPForZeroShotClassification.pretrained()
  .setInputCols("image_assembler")
  .setOutputCol("label")

The default model is "zero_shot_classifier_clip_vit_base_patch32", if no name is provided.

For available pretrained models please see the Models Hub.

Models from the HuggingFace 🤗 Transformers library are also compatible with Spark NLP 🚀. To see which models are compatible and how to import them see https://github.com/JohnSnowLabs/spark-nlp/discussions/5669 and to see more extended examples, see CLIPForZeroShotClassificationTestSpec.

Example

import com.johnsnowlabs.nlp.ImageAssembler
import com.johnsnowlabs.nlp.annotator._
import org.apache.spark.ml.Pipeline

val imageDF = ResourceHelper.spark.read
  .format("image")
  .option("dropInvalid", value = true)
  .load("src/test/resources/image/")

val imageAssembler: ImageAssembler = new ImageAssembler()
  .setInputCol("image")
  .setOutputCol("image_assembler")

val candidateLabels = Array(
  "a photo of a bird",
  "a photo of a cat",
  "a photo of a dog",
  "a photo of a hen",
  "a photo of a hippo",
  "a photo of a room",
  "a photo of a tractor",
  "a photo of an ostrich",
  "a photo of an ox")

val imageClassifier = CLIPForZeroShotClassification
  .pretrained()
  .setInputCols("image_assembler")
  .setOutputCol("label")
  .setCandidateLabels(candidateLabels)

val pipeline =
  new Pipeline().setStages(Array(imageAssembler, imageClassifier)).fit(imageDF).transform(imageDF)

pipeline
  .selectExpr("reverse(split(image.origin, '/'))[0] as image_name", "label.result")
  .show(truncate = false)
+-----------------+-----------------------+
|image_name       |result                 |
+-----------------+-----------------------+
|palace.JPEG      |[a photo of a room]    |
|egyptian_cat.jpeg|[a photo of a cat]     |
|hippopotamus.JPEG|[a photo of a hippo]   |
|hen.JPEG         |[a photo of a hen]     |
|ostrich.JPEG     |[a photo of an ostrich]|
|junco.JPEG       |[a photo of a bird]    |
|bluetick.jpg     |[a photo of a dog]     |
|chihuahua.jpg    |[a photo of a dog]     |
|tractor.JPEG     |[a photo of a tractor] |
|ox.JPEG          |[a photo of an ox]     |
+-----------------+-----------------------+
Linear Supertypes
HasRescaleFactor, HasEngine, WriteOpenvinoModel, WriteOnnxModel, WriteTensorflowModel, HasImageFeatureProperties, HasBatchedAnnotateImage[CLIPForZeroShotClassification], AnnotatorModel[CLIPForZeroShotClassification], CanBeLazy, RawAnnotator[CLIPForZeroShotClassification], HasOutputAnnotationCol, HasInputAnnotationCols, HasOutputAnnotatorType, ParamsAndFeaturesWritable, HasFeatures, DefaultParamsWritable, MLWritable, Model[CLIPForZeroShotClassification], Transformer, PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Grouped
  2. Alphabetic
  3. By Inheritance
Inherited
  1. CLIPForZeroShotClassification
  2. HasRescaleFactor
  3. HasEngine
  4. WriteOpenvinoModel
  5. WriteOnnxModel
  6. WriteTensorflowModel
  7. HasImageFeatureProperties
  8. HasBatchedAnnotateImage
  9. AnnotatorModel
  10. CanBeLazy
  11. RawAnnotator
  12. HasOutputAnnotationCol
  13. HasInputAnnotationCols
  14. HasOutputAnnotatorType
  15. ParamsAndFeaturesWritable
  16. HasFeatures
  17. DefaultParamsWritable
  18. MLWritable
  19. Model
  20. Transformer
  21. PipelineStage
  22. Logging
  23. Params
  24. Serializable
  25. Serializable
  26. Identifiable
  27. AnyRef
  28. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Parameters

A list of (hyper-)parameter keys this annotator can take. Users can set and get the parameter values through setters and getters, respectively.

  1. val batchSize: IntParam

    Size of every batch (Default depends on model).

    Size of every batch (Default depends on model).

    Definition Classes
    HasBatchedAnnotateImage
  2. val candidateLabels: StringArrayParam

    Candidate labels for classification, you can set candidateLabels dynamically during the runtime

  3. val doNormalize: BooleanParam

    Whether or not to normalize the input with mean and standard deviation

    Whether or not to normalize the input with mean and standard deviation

    Definition Classes
    HasImageFeatureProperties
  4. val doRescale: BooleanParam

    Whether to rescale the image values by rescaleFactor.

    Whether to rescale the image values by rescaleFactor.

    Definition Classes
    HasRescaleFactor
  5. val doResize: BooleanParam

    Whether to resize the input to a certain size

    Whether to resize the input to a certain size

    Definition Classes
    HasImageFeatureProperties
  6. val engine: Param[String]

    This param is set internally once via loadSavedModel.

    This param is set internally once via loadSavedModel. That's why there is no setter

    Definition Classes
    HasEngine
  7. val featureExtractorType: Param[String]

    Name of model's architecture for feature extraction

    Name of model's architecture for feature extraction

    Definition Classes
    HasImageFeatureProperties
  8. val imageMean: DoubleArrayParam

    The sequence of means for each channel, to be used when normalizing images

    The sequence of means for each channel, to be used when normalizing images

    Definition Classes
    HasImageFeatureProperties
  9. val imageStd: DoubleArrayParam

    The sequence of standard deviations for each channel, to be used when normalizing images

    The sequence of standard deviations for each channel, to be used when normalizing images

    Definition Classes
    HasImageFeatureProperties
  10. val resample: IntParam

    An optional resampling filter.

    An optional resampling filter. This can be one of PIL.Image.NEAREST, PIL.Image.BOX, PIL.Image.BILINEAR, PIL.Image.HAMMING, PIL.Image.BICUBIC or PIL.Image.LANCZOS. Only has an effect if do_resize is set to True

    Definition Classes
    HasImageFeatureProperties
  11. val rescaleFactor: DoubleParam

    Factor to scale the image values (Default: 1 / 255.0).

    Factor to scale the image values (Default: 1 / 255.0).

    Definition Classes
    HasRescaleFactor
  12. val size: IntParam

    Resize the input to the given size.

    Resize the input to the given size. If a tuple is provided, it should be (width, height). If only an integer is provided, then the input will be resized to (size, size). Only has an effect if do_resize is set to True.

    Definition Classes
    HasImageFeatureProperties

Annotator types

Required input and expected output annotator types

  1. val inputAnnotatorTypes: Array[AnnotatorType]

    Input annotator type : IMAGE

    Input annotator type : IMAGE

    Definition Classes
    CLIPForZeroShotClassification → HasInputAnnotationCols
  2. val outputAnnotatorType: AnnotatorType

    Output annotator type : CATEGORY

    Output annotator type : CATEGORY

    Definition Classes
    CLIPForZeroShotClassification → HasOutputAnnotatorType

Members

  1. type AnnotatorType = String
    Definition Classes
    HasOutputAnnotatorType
  1. def batchAnnotate(batchedAnnotations: Seq[Array[AnnotationImage]]): Seq[Seq[Annotation]]

    Takes a document and annotations and produces new annotations of this annotator's annotation type

    Takes a document and annotations and produces new annotations of this annotator's annotation type

    batchedAnnotations

    Annotations that correspond to inputAnnotationCols generated by previous annotators if any

    returns

    any number of annotations processed for every input annotation. Not necessary one to one relationship

    Definition Classes
    CLIPForZeroShotClassification → HasBatchedAnnotateImage
  2. def batchProcess(rows: Iterator[_]): Iterator[Row]
    Definition Classes
    HasBatchedAnnotateImage
  3. final def clear(param: Param[_]): CLIPForZeroShotClassification.this.type
    Definition Classes
    Params
  4. def copy(extra: ParamMap): CLIPForZeroShotClassification

    requirement for annotators copies

    requirement for annotators copies

    Definition Classes
    RawAnnotator → Model → Transformer → PipelineStage → Params
  5. def explainParam(param: Param[_]): String
    Definition Classes
    Params
  6. def explainParams(): String
    Definition Classes
    Params
  7. final def extractParamMap(): ParamMap
    Definition Classes
    Params
  8. final def extractParamMap(extra: ParamMap): ParamMap
    Definition Classes
    Params
  9. val features: ArrayBuffer[Feature[_, _, _]]
    Definition Classes
    HasFeatures
  10. final def get[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  11. final def getDefault[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  12. def getInputCols: Array[String]

    returns

    input annotations columns currently used

    Definition Classes
    HasInputAnnotationCols
  13. def getLazyAnnotator: Boolean
    Definition Classes
    CanBeLazy
  14. final def getOrDefault[T](param: Param[T]): T
    Definition Classes
    Params
  15. final def getOutputCol: String

    Gets annotation column name going to generate

    Gets annotation column name going to generate

    Definition Classes
    HasOutputAnnotationCol
  16. def getParam(paramName: String): Param[Any]
    Definition Classes
    Params
  17. final def hasDefault[T](param: Param[T]): Boolean
    Definition Classes
    Params
  18. def hasParam(paramName: String): Boolean
    Definition Classes
    Params
  19. def hasParent: Boolean
    Definition Classes
    Model
  20. final def isDefined(param: Param[_]): Boolean
    Definition Classes
    Params
  21. final def isSet(param: Param[_]): Boolean
    Definition Classes
    Params
  22. val lazyAnnotator: BooleanParam
    Definition Classes
    CanBeLazy
  23. def onWrite(path: String, spark: SparkSession): Unit
    Definition Classes
    CLIPForZeroShotClassification → ParamsAndFeaturesWritable
  24. val optionalInputAnnotatorTypes: Array[String]
    Definition Classes
    HasInputAnnotationCols
  25. lazy val params: Array[Param[_]]
    Definition Classes
    Params
  26. var parent: Estimator[CLIPForZeroShotClassification]
    Definition Classes
    Model
  27. def save(path: String): Unit
    Definition Classes
    MLWritable
    Annotations
    @Since( "1.6.0" ) @throws( ... )
  28. final def set[T](param: Param[T], value: T): CLIPForZeroShotClassification.this.type
    Definition Classes
    Params
  29. final def setInputCols(value: String*): CLIPForZeroShotClassification.this.type
    Definition Classes
    HasInputAnnotationCols
  30. def setInputCols(value: Array[String]): CLIPForZeroShotClassification.this.type

    Overrides required annotators column if different than default

    Overrides required annotators column if different than default

    Definition Classes
    HasInputAnnotationCols
  31. def setLazyAnnotator(value: Boolean): CLIPForZeroShotClassification.this.type
    Definition Classes
    CanBeLazy
  32. final def setOutputCol(value: String): CLIPForZeroShotClassification.this.type

    Overrides annotation column name when transforming

    Overrides annotation column name when transforming

    Definition Classes
    HasOutputAnnotationCol
  33. def setParent(parent: Estimator[CLIPForZeroShotClassification]): CLIPForZeroShotClassification
    Definition Classes
    Model
  34. def toString(): String
    Definition Classes
    Identifiable → AnyRef → Any
  35. final def transform(dataset: Dataset[_]): DataFrame

    Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content

    Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content

    dataset

    Dataset[Row]

    Definition Classes
    AnnotatorModel → Transformer
  36. def transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame
    Definition Classes
    Transformer
    Annotations
    @Since( "2.0.0" )
  37. def transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame
    Definition Classes
    Transformer
    Annotations
    @Since( "2.0.0" ) @varargs()
  38. final def transformSchema(schema: StructType): StructType

    requirement for pipeline transformation validation.

    requirement for pipeline transformation validation. It is called on fit()

    Definition Classes
    RawAnnotator → PipelineStage
  39. val uid: String
    Definition Classes
    CLIPForZeroShotClassification → Identifiable
  40. def write: MLWriter
    Definition Classes
    ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable
  41. def writeOnnxModel(path: String, spark: SparkSession, onnxWrapper: OnnxWrapper, suffix: String, fileName: String): Unit
    Definition Classes
    WriteOnnxModel
  42. def writeOnnxModels(path: String, spark: SparkSession, onnxWrappersWithNames: Seq[(OnnxWrapper, String)], suffix: String): Unit
    Definition Classes
    WriteOnnxModel
  43. def writeOpenvinoModel(path: String, spark: SparkSession, openvinoWrapper: OpenvinoWrapper, suffix: String, fileName: String): Unit
    Definition Classes
    WriteOpenvinoModel
  44. def writeOpenvinoModels(path: String, spark: SparkSession, ovWrappersWithNames: Seq[(OpenvinoWrapper, String)], suffix: String): Unit
    Definition Classes
    WriteOpenvinoModel
  45. def writeTensorflowHub(path: String, tfPath: String, spark: SparkSession, suffix: String = "_use"): Unit
    Definition Classes
    WriteTensorflowModel
  46. def writeTensorflowModel(path: String, spark: SparkSession, tensorflow: TensorflowWrapper, suffix: String, filename: String, configProtoBytes: Option[Array[Byte]] = None): Unit
    Definition Classes
    WriteTensorflowModel
  47. def writeTensorflowModelV2(path: String, spark: SparkSession, tensorflow: TensorflowWrapper, suffix: String, filename: String, configProtoBytes: Option[Array[Byte]] = None, savedSignatures: Option[Map[String, String]] = None): Unit
    Definition Classes
    WriteTensorflowModel

Parameter setters

  1. def setBatchSize(size: Int): CLIPForZeroShotClassification.this.type

    Size of every batch.

    Size of every batch.

    Definition Classes
    HasBatchedAnnotateImage
  2. def setCandidateLabels(value: Array[String]): CLIPForZeroShotClassification.this.type

  3. def setDoNormalize(value: Boolean): CLIPForZeroShotClassification.this.type

    Definition Classes
    HasImageFeatureProperties
  4. def setDoRescale(value: Boolean): CLIPForZeroShotClassification.this.type

    Definition Classes
    HasRescaleFactor
  5. def setDoResize(value: Boolean): CLIPForZeroShotClassification.this.type

    Definition Classes
    HasImageFeatureProperties
  6. def setFeatureExtractorType(value: String): CLIPForZeroShotClassification.this.type

    Definition Classes
    HasImageFeatureProperties
  7. def setImageMean(value: Array[Double]): CLIPForZeroShotClassification.this.type

    Definition Classes
    HasImageFeatureProperties
  8. def setImageStd(value: Array[Double]): CLIPForZeroShotClassification.this.type

    Definition Classes
    HasImageFeatureProperties
  9. def setModelIfNotSet(spark: SparkSession, tensorflow: Option[TensorflowWrapper], onnx: Option[OnnxWrapper], openvinoWrapper: Option[OpenvinoWrapper], preprocessor: Preprocessor): CLIPForZeroShotClassification.this.type

  10. def setResample(value: Int): CLIPForZeroShotClassification.this.type

    Definition Classes
    HasImageFeatureProperties
  11. def setRescaleFactor(value: Double): CLIPForZeroShotClassification.this.type

    Definition Classes
    HasRescaleFactor
  12. def setSize(value: Int): CLIPForZeroShotClassification.this.type

    Definition Classes
    HasImageFeatureProperties

Parameter getters

  1. def getBatchSize: Int

    Size of every batch.

    Size of every batch.

    Definition Classes
    HasBatchedAnnotateImage
  2. def getCandidateLabels: Array[String]

  3. def getDoNormalize: Boolean

    Definition Classes
    HasImageFeatureProperties
  4. def getDoRescale: Boolean

    Definition Classes
    HasRescaleFactor
  5. def getDoResize: Boolean

    Definition Classes
    HasImageFeatureProperties
  6. def getEngine: String

    Definition Classes
    HasEngine
  7. def getFeatureExtractorType: String

    Definition Classes
    HasImageFeatureProperties
  8. def getImageMean: Array[Double]

    Definition Classes
    HasImageFeatureProperties
  9. def getImageStd: Array[Double]

    Definition Classes
    HasImageFeatureProperties
  10. def getModelIfNotSet: CLIP

  11. def getResample: Int

    Definition Classes
    HasImageFeatureProperties
  12. def getRescaleFactor: Double

    Definition Classes
    HasRescaleFactor
  13. def getSize: Int

    Definition Classes
    HasImageFeatureProperties