com.johnsnowlabs.nlp.annotators.cv
DolphinForDocumentParsing
Companion object DolphinForDocumentParsing
class DolphinForDocumentParsing extends AnnotatorModel[DolphinForDocumentParsing] with HasBatchedAnnotateImage[DolphinForDocumentParsing] with HasImageFeatureProperties with HasRescaleFactor with WriteOnnxModel with HasEngine
Document image parsing with ByteDance Dolphin.
Dolphin follows an analyze-then-parse design: stage 1 generates the page's layout elements in reading order, then each element is cropped and parsed independently in stage 2, batched by type. Tables come back as HTML, formulas as LaTeX, text as markdown.
The task prompt is the decoder prefix; there is no chat template. Dolphin 1.5 and 1.0 load through the same path. The release is recognised from its stage-1 output, which decides the box geometry and whether formulas and code get their own prompts (1.5 only).
Example
val imageAssembler = new ImageAssembler() .setInputCol("image") .setOutputCol("image_assembler") val dolphin = DolphinForDocumentParsing.pretrained() .setInputCols("image_assembler") .setOutputCol("elements") .setParsingMode("page") new Pipeline().setStages(Array(imageAssembler, dolphin))
- Grouped
- Alphabetic
- By Inheritance
- DolphinForDocumentParsing
- HasEngine
- WriteOnnxModel
- HasRescaleFactor
- HasImageFeatureProperties
- HasBatchedAnnotateImage
- AnnotatorModel
- CanBeLazy
- RawAnnotator
- HasOutputAnnotationCol
- HasInputAnnotationCols
- HasOutputAnnotatorType
- ParamsAndFeaturesWritable
- HasFeatures
- DefaultParamsWritable
- MLWritable
- Model
- Transformer
- PipelineStage
- Logging
- Params
- Serializable
- Serializable
- Identifiable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
Type Members
-
type
AnnotationContent = Seq[Row]
internal types to show Rows as a relevant StructType Should be deleted once Spark releases UserDefinedTypes to @developerAPI
internal types to show Rows as a relevant StructType Should be deleted once Spark releases UserDefinedTypes to @developerAPI
- Attributes
- protected
- Definition Classes
- AnnotatorModel
-
type
AnnotatorType = String
- Definition Classes
- HasOutputAnnotatorType
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
$[T](param: Param[T]): T
- Attributes
- protected
- Definition Classes
- Params
-
def
$$[T](feature: StructFeature[T]): T
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[K, V](feature: MapFeature[K, V]): Map[K, V]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[T](feature: SetFeature[T]): Set[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[T](feature: ArrayFeature[T]): Array[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
_transform(dataset: Dataset[_], recursivePipeline: Option[PipelineModel]): DataFrame
- Attributes
- protected
- Definition Classes
- AnnotatorModel
-
val
adjustBoxEdges: BooleanParam
Dolphin 1.0 only: its boxes are widened off ink as its reference does.
Dolphin 1.0 only: its boxes are widened off ink as its reference does. 1.5 boxes are used as generated.
-
def
afterAnnotate(dataset: DataFrame): DataFrame
- Attributes
- protected
- Definition Classes
- AnnotatorModel
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
batchAnnotate(batchedAnnotations: Seq[Array[AnnotationImage]]): Seq[Seq[Annotation]]
batchProcesszips rows with results, so this must return exactly oneSeqper input row and in the same order — a shorter result silently truncates the DataFrame.batchProcesszips rows with results, so this must return exactly oneSeqper input row and in the same order — a shorter result silently truncates the DataFrame.Note
AnnotationImage.textis deliberately ignored:Reader2Imagedefaults itspromptTemplatetoqwen2vl-chatand would otherwise inject chat markup that Dolphin would happily treat as the task prompt. The task comes fromparsingMode.- batchedAnnotations
Annotations in batches that correspond to inputAnnotationCols generated by previous annotators if any
- returns
any number of annotations processed for every batch of input annotations. Not necessary one to one relationship
- Definition Classes
- DolphinForDocumentParsing → HasBatchedAnnotateImage
-
def
batchProcess(rows: Iterator[_]): Iterator[Row]
- Definition Classes
- HasBatchedAnnotateImage
-
val
batchSize: IntParam
Size of every batch (Default depends on model).
Size of every batch (Default depends on model).
- Definition Classes
- HasBatchedAnnotateImage
-
def
beforeAnnotate(dataset: Dataset[_]): Dataset[_]
- Attributes
- protected
- Definition Classes
- AnnotatorModel
-
final
def
checkSchema(schema: StructType, inputAnnotatorType: String): Boolean
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
-
final
def
clear(param: Param[_]): DolphinForDocumentParsing.this.type
- Definition Classes
- Params
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
copy(extra: ParamMap): DolphinForDocumentParsing
requirement for annotators copies
requirement for annotators copies
- Definition Classes
- RawAnnotator → Model → Transformer → PipelineStage → Params
-
def
copyValues[T <: Params](to: T, extra: ParamMap): T
- Attributes
- protected
- Definition Classes
- Params
- val customPrompt: Param[String]
-
final
def
defaultCopy[T <: Params](extra: ParamMap): T
- Attributes
- protected
- Definition Classes
- Params
-
val
doNormalize: BooleanParam
Whether or not to normalize the input with mean and standard deviation
Whether or not to normalize the input with mean and standard deviation
- Definition Classes
- HasImageFeatureProperties
-
val
doRescale: BooleanParam
Whether to rescale the image values by rescaleFactor.
Whether to rescale the image values by rescaleFactor.
- Definition Classes
- HasRescaleFactor
-
val
doResize: BooleanParam
Whether to resize the input to a certain size
Whether to resize the input to a certain size
- Definition Classes
- HasImageFeatureProperties
-
val
elementBatchSize: IntParam
Stage-2 crops decoded per forward pass.
Stage-2 crops decoded per forward pass. Each row's KV cache grows with
maxOutputLength, so memory scales with this value. -
val
engine: Param[String]
This param is set internally once via loadSavedModel.
This param is set internally once via loadSavedModel. That's why there is no setter
- Definition Classes
- HasEngine
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
explainParam(param: Param[_]): String
- Definition Classes
- Params
-
def
explainParams(): String
- Definition Classes
- Params
-
final
val
extraInputCols: StringArrayParam
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
-
def
extraValidate(structType: StructType): Boolean
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
def
extraValidateMsg: String
Override for additional custom schema checks
Override for additional custom schema checks
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
final
def
extractParamMap(): ParamMap
- Definition Classes
- Params
-
final
def
extractParamMap(extra: ParamMap): ParamMap
- Definition Classes
- Params
-
val
featureExtractorType: Param[String]
Name of model's architecture for feature extraction
Name of model's architecture for feature extraction
- Definition Classes
- HasImageFeatureProperties
-
val
features: ArrayBuffer[Feature[_, _, _]]
- Definition Classes
- HasFeatures
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
val
generationConfig: StructFeature[GenerationConfig]
- Attributes
- protected[nlp]
-
def
get[T](feature: StructFeature[T]): Option[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[K, V](feature: MapFeature[K, V]): Option[Map[K, V]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[T](feature: SetFeature[T]): Option[Set[T]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[T](feature: ArrayFeature[T]): Option[Array[T]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
get[T](param: Param[T]): Option[T]
- Definition Classes
- Params
- def getAdjustBoxEdges: Boolean
-
def
getBatchSize: Int
Size of every batch.
Size of every batch.
- Definition Classes
- HasBatchedAnnotateImage
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def getCustomPrompt: String
-
final
def
getDefault[T](param: Param[T]): Option[T]
- Definition Classes
- Params
-
def
getDoNormalize: Boolean
- Definition Classes
- HasImageFeatureProperties
-
def
getDoRescale: Boolean
- Definition Classes
- HasRescaleFactor
-
def
getDoResize: Boolean
- Definition Classes
- HasImageFeatureProperties
- def getElementBatchSize: Int
-
def
getEngine: String
- Definition Classes
- HasEngine
-
def
getFeatureExtractorType: String
- Definition Classes
- HasImageFeatureProperties
- def getGenerationConfig: GenerationConfig
-
def
getImageMean: Array[Double]
- Definition Classes
- HasImageFeatureProperties
-
def
getImageStd: Array[Double]
- Definition Classes
- HasImageFeatureProperties
-
def
getInputCols: Array[String]
- returns
input annotations columns currently used
- Definition Classes
- HasInputAnnotationCols
- def getLayoutMaxOutputLength: Int
-
def
getLazyAnnotator: Boolean
- Definition Classes
- CanBeLazy
- def getMaxOutputLength: Int
- def getMinElementSize: Int
- def getModelIfNotSet: Dolphin
-
final
def
getOrDefault[T](param: Param[T]): T
- Definition Classes
- Params
-
final
def
getOutputCol: String
Gets annotation column name going to generate
Gets annotation column name going to generate
- Definition Classes
- HasOutputAnnotationCol
- def getOutputFormat: String
-
def
getParam(paramName: String): Param[Any]
- Definition Classes
- Params
- def getParsingMode: String
- def getRepetitionPenalty: Double
-
def
getResample: Int
- Definition Classes
- HasImageFeatureProperties
-
def
getRescaleFactor: Double
- Definition Classes
- HasRescaleFactor
-
def
getSize: Int
- Definition Classes
- HasImageFeatureProperties
-
final
def
hasDefault[T](param: Param[T]): Boolean
- Definition Classes
- Params
-
def
hasParam(paramName: String): Boolean
- Definition Classes
- Params
-
def
hasParent: Boolean
- Definition Classes
- Model
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
val
imageMean: DoubleArrayParam
The sequence of means for each channel, to be used when normalizing images
The sequence of means for each channel, to be used when normalizing images
- Definition Classes
- HasImageFeatureProperties
-
val
imageStd: DoubleArrayParam
The sequence of standard deviations for each channel, to be used when normalizing images
The sequence of standard deviations for each channel, to be used when normalizing images
- Definition Classes
- HasImageFeatureProperties
-
def
initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
val
inputAnnotatorTypes: Array[AnnotatorType]
Annotator reference id.
Annotator reference id. Used to identify elements in metadata or to refer to this annotator type
- Definition Classes
- DolphinForDocumentParsing → HasInputAnnotationCols
-
final
val
inputCols: StringArrayParam
columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified
columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
-
final
def
isDefined(param: Param[_]): Boolean
- Definition Classes
- Params
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
isSet(param: Param[_]): Boolean
- Definition Classes
- Params
-
def
isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- val layoutMaxOutputLength: IntParam
-
val
lazyAnnotator: BooleanParam
- Definition Classes
- CanBeLazy
-
def
log: Logger
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logName: String
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
val
maxOutputLength: IntParam
Generation stops at the decoder's 4096-position window, prompt included, whatever the cap.
-
val
merges: MapFeature[(String, String), Int]
- Attributes
- protected[nlp]
-
val
minElementSize: IntParam
Page mode drops elements whose crop is this small, as upstream does.
-
def
msgHelper(schema: StructType): String
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
def
onWrite(path: String, spark: SparkSession): Unit
- Definition Classes
- DolphinForDocumentParsing → ParamsAndFeaturesWritable
-
val
optionalInputAnnotatorTypes: Array[String]
- Definition Classes
- HasInputAnnotationCols
-
val
outputAnnotatorType: AnnotatorType
- Definition Classes
- DolphinForDocumentParsing → HasOutputAnnotatorType
-
final
val
outputCol: Param[String]
- Attributes
- protected
- Definition Classes
- HasOutputAnnotationCol
- val outputFormat: Param[String]
-
lazy val
params: Array[Param[_]]
- Definition Classes
- Params
-
var
parent: Estimator[DolphinForDocumentParsing]
- Definition Classes
- Model
-
val
parsingMode: Param[String]
What the input image is:
What the input image is:
page(default): a whole page. Stage 1 + stage 2.layout: a whole page; stage 1 only.table/text/formula/code: an already cropped region; stage 2 only.formulaandcodeare Dolphin 1.5 prompts.
-
val
repetitionPenalty: DoubleParam
1.0 (off) matches Dolphin 1.5's reference pipeline; Dolphin 1.0's used 1.1.
-
val
resample: IntParam
An optional resampling filter.
An optional resampling filter. This can be one of PIL.Image.NEAREST, PIL.Image.BOX, PIL.Image.BILINEAR, PIL.Image.HAMMING, PIL.Image.BICUBIC or PIL.Image.LANCZOS. Only has an effect if do_resize is set to True
- Definition Classes
- HasImageFeatureProperties
-
val
rescaleFactor: DoubleParam
Factor to scale the image values (Default:
1 / 255.0).Factor to scale the image values (Default:
1 / 255.0).- Definition Classes
- HasRescaleFactor
-
def
save(path: String): Unit
- Definition Classes
- MLWritable
- Annotations
- @Since( "1.6.0" ) @throws( ... )
-
def
set[T](feature: StructFeature[T], value: T): DolphinForDocumentParsing.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[K, V](feature: MapFeature[K, V], value: Map[K, V]): DolphinForDocumentParsing.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[T](feature: SetFeature[T], value: Set[T]): DolphinForDocumentParsing.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[T](feature: ArrayFeature[T], value: Array[T]): DolphinForDocumentParsing.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
set(paramPair: ParamPair[_]): DolphinForDocumentParsing.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
set(param: String, value: Any): DolphinForDocumentParsing.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
set[T](param: Param[T], value: T): DolphinForDocumentParsing.this.type
- Definition Classes
- Params
- def setAdjustBoxEdges(value: Boolean): DolphinForDocumentParsing.this.type
-
def
setBatchSize(size: Int): DolphinForDocumentParsing.this.type
Size of every batch.
Size of every batch.
- Definition Classes
- HasBatchedAnnotateImage
- def setCustomPrompt(value: String): DolphinForDocumentParsing.this.type
-
def
setDefault[T](feature: StructFeature[T], value: () ⇒ T): DolphinForDocumentParsing.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[K, V](feature: MapFeature[K, V], value: () ⇒ Map[K, V]): DolphinForDocumentParsing.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[T](feature: SetFeature[T], value: () ⇒ Set[T]): DolphinForDocumentParsing.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[T](feature: ArrayFeature[T], value: () ⇒ Array[T]): DolphinForDocumentParsing.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
setDefault(paramPairs: ParamPair[_]*): DolphinForDocumentParsing.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
setDefault[T](param: Param[T], value: T): DolphinForDocumentParsing.this.type
- Attributes
- protected[org.apache.spark.ml]
- Definition Classes
- Params
-
def
setDoNormalize(value: Boolean): DolphinForDocumentParsing.this.type
- Definition Classes
- HasImageFeatureProperties
-
def
setDoRescale(value: Boolean): DolphinForDocumentParsing.this.type
- Definition Classes
- HasRescaleFactor
-
def
setDoResize(value: Boolean): DolphinForDocumentParsing.this.type
- Definition Classes
- HasImageFeatureProperties
- def setElementBatchSize(value: Int): DolphinForDocumentParsing.this.type
-
def
setExtraInputCols(value: Array[String]): DolphinForDocumentParsing.this.type
- Definition Classes
- HasInputAnnotationCols
-
def
setFeatureExtractorType(value: String): DolphinForDocumentParsing.this.type
- Definition Classes
- HasImageFeatureProperties
- def setGenerationConfig(value: GenerationConfig): DolphinForDocumentParsing.this.type
-
def
setImageMean(value: Array[Double]): DolphinForDocumentParsing.this.type
- Definition Classes
- HasImageFeatureProperties
-
def
setImageStd(value: Array[Double]): DolphinForDocumentParsing.this.type
- Definition Classes
- HasImageFeatureProperties
-
final
def
setInputCols(value: String*): DolphinForDocumentParsing.this.type
- Definition Classes
- HasInputAnnotationCols
-
def
setInputCols(value: Array[String]): DolphinForDocumentParsing.this.type
Overrides required annotators column if different than default
Overrides required annotators column if different than default
- Definition Classes
- HasInputAnnotationCols
- def setLayoutMaxOutputLength(value: Int): DolphinForDocumentParsing.this.type
-
def
setLazyAnnotator(value: Boolean): DolphinForDocumentParsing.this.type
- Definition Classes
- CanBeLazy
- def setMaxOutputLength(value: Int): DolphinForDocumentParsing.this.type
- def setMerges(value: Map[(String, String), Int]): DolphinForDocumentParsing.this.type
- def setMinElementSize(value: Int): DolphinForDocumentParsing.this.type
- def setModelIfNotSet(spark: SparkSession, onnxWrappers: Option[EncoderDecoderWrappers], preprocessor: Preprocessor): DolphinForDocumentParsing.this.type
-
final
def
setOutputCol(value: String): DolphinForDocumentParsing.this.type
Overrides annotation column name when transforming
Overrides annotation column name when transforming
- Definition Classes
- HasOutputAnnotationCol
- def setOutputFormat(value: String): DolphinForDocumentParsing.this.type
-
def
setParent(parent: Estimator[DolphinForDocumentParsing]): DolphinForDocumentParsing
- Definition Classes
- Model
- def setParsingMode(value: String): DolphinForDocumentParsing.this.type
- def setRepetitionPenalty(value: Double): DolphinForDocumentParsing.this.type
-
def
setResample(value: Int): DolphinForDocumentParsing.this.type
- Definition Classes
- HasImageFeatureProperties
-
def
setRescaleFactor(value: Double): DolphinForDocumentParsing.this.type
- Definition Classes
- HasRescaleFactor
-
def
setSize(value: Int): DolphinForDocumentParsing.this.type
- Definition Classes
- HasImageFeatureProperties
- def setVocabulary(value: Map[String, Int]): DolphinForDocumentParsing.this.type
-
val
size: IntParam
Resize the input to the given size.
Resize the input to the given size. If a tuple is provided, it should be (width, height). If only an integer is provided, then the input will be resized to (size, size). Only has an effect if do_resize is set to True.
- Definition Classes
- HasImageFeatureProperties
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- Identifiable → AnyRef → Any
-
final
def
transform(dataset: Dataset[_]): DataFrame
Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content
Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content
- dataset
Dataset[Row]
- Definition Classes
- AnnotatorModel → Transformer
-
def
transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame
- Definition Classes
- Transformer
- Annotations
- @Since( "2.0.0" )
-
def
transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame
- Definition Classes
- Transformer
- Annotations
- @Since( "2.0.0" ) @varargs()
-
final
def
transformSchema(schema: StructType): StructType
requirement for pipeline transformation validation.
requirement for pipeline transformation validation. It is called on fit()
- Definition Classes
- RawAnnotator → PipelineStage
-
def
transformSchema(schema: StructType, logging: Boolean): StructType
- Attributes
- protected
- Definition Classes
- PipelineStage
- Annotations
- @DeveloperApi()
-
val
uid: String
- Definition Classes
- DolphinForDocumentParsing → Identifiable
-
def
validate(schema: StructType): Boolean
takes a Dataset and checks to see if all the required annotation types are present.
takes a Dataset and checks to see if all the required annotation types are present.
- schema
to be validated
- returns
True if all the required types are present, else false
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
val
vocabulary: MapFeature[String, Int]
- Attributes
- protected[nlp]
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
wrapColumnMetadata(col: Column): Column
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
def
write: MLWriter
- Definition Classes
- ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable
-
def
writeOnnxModel(path: String, spark: SparkSession, onnxWrapper: OnnxWrapper, suffix: String, fileName: String): Unit
- Definition Classes
- WriteOnnxModel
-
def
writeOnnxModels(path: String, spark: SparkSession, onnxWrappersWithNames: Seq[(OnnxWrapper, String)], suffix: String): Unit
- Definition Classes
- WriteOnnxModel