class Cleaner extends MarianTransformer
- Grouped
- Alphabetic
- By Inheritance
- Cleaner
- MarianTransformer
- HasProtectedParams
- HasEngine
- WriteSentencePieceModel
- WriteOnnxModel
- WriteTensorflowModel
- HasBatchedAnnotate
- AnnotatorModel
- CanBeLazy
- RawAnnotator
- HasOutputAnnotationCol
- HasInputAnnotationCols
- HasOutputAnnotatorType
- ParamsAndFeaturesWritable
- HasFeatures
- DefaultParamsWritable
- MLWritable
- Model
- Transformer
- PipelineStage
- Logging
- Params
- Serializable
- Serializable
- Identifiable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
Type Members
-
implicit
class
ProtectedParam[T] extends Param[T]
- Definition Classes
- HasProtectedParams
-
type
AnnotationContent = Seq[Row]
internal types to show Rows as a relevant StructType Should be deleted once Spark releases UserDefinedTypes to @developerAPI
internal types to show Rows as a relevant StructType Should be deleted once Spark releases UserDefinedTypes to @developerAPI
- Attributes
- protected
- Definition Classes
- AnnotatorModel
-
type
AnnotatorType = String
- Definition Classes
- HasOutputAnnotatorType
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
$[T](param: Param[T]): T
- Attributes
- protected
- Definition Classes
- Params
-
def
$$[T](feature: StructFeature[T]): T
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[K, V](feature: MapFeature[K, V]): Map[K, V]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[T](feature: SetFeature[T]): Set[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[T](feature: ArrayFeature[T]): Array[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
_transform(dataset: Dataset[_], recursivePipeline: Option[PipelineModel]): DataFrame
- Attributes
- protected
- Definition Classes
- AnnotatorModel
-
def
afterAnnotate(dataset: DataFrame): DataFrame
- Attributes
- protected
- Definition Classes
- AnnotatorModel
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
batchAnnotate(batchedAnnotations: Seq[Array[Annotation]]): Seq[Seq[Annotation]]
takes a document and annotations and produces new annotations of this annotator's annotation type
takes a document and annotations and produces new annotations of this annotator's annotation type
- batchedAnnotations
Annotations that correspond to inputAnnotationCols generated by previous annotators if any
- returns
any number of annotations processed for every input annotation. Not necessary one to one relationship
- Definition Classes
- Cleaner → MarianTransformer → HasBatchedAnnotate
-
def
batchProcess(rows: Iterator[_]): Iterator[Row]
- Definition Classes
- HasBatchedAnnotate
-
val
batchSize: IntParam
Size of every batch (Default depends on model).
Size of every batch (Default depends on model).
- Definition Classes
- HasBatchedAnnotate
-
def
beforeAnnotate(dataset: Dataset[_]): Dataset[_]
- Attributes
- protected
- Definition Classes
- AnnotatorModel
- def buildAnnotation(transformation: (String) ⇒ String)(annotation: Annotation): Annotation
- val bullets: Param[Boolean]
-
final
def
checkSchema(schema: StructType, inputAnnotatorType: String): Boolean
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
- val cleanPostfixPattern: Param[String]
- val cleanPrefixPattern: Param[String]
-
val
cleanerMode: Param[String]
cleanerMode can take the following values:
cleanerMode can take the following values:
bytes_string_to_string
: Converts a string representation of a byte string (e.g., containing escape sequences) to an Annotation structure using the specified encoding.
-
final
def
clear(param: Param[_]): Cleaner.this.type
- Definition Classes
- Params
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
val
configProtoBytes: IntArrayParam
ConfigProto from tensorflow, serialized into byte array.
ConfigProto from tensorflow, serialized into byte array. Get with config_proto.SerializeToString()
- Definition Classes
- MarianTransformer
-
def
copy(extra: ParamMap): MarianTransformer
requirement for annotators copies
requirement for annotators copies
- Definition Classes
- RawAnnotator → Model → Transformer → PipelineStage → Params
-
def
copyValues[T <: Params](to: T, extra: ParamMap): T
- Attributes
- protected
- Definition Classes
- Params
- val dashes: Param[Boolean]
-
final
def
defaultCopy[T <: Params](extra: ParamMap): T
- Attributes
- protected
- Definition Classes
- Params
-
val
doSample: BooleanParam
Whether or not to use sampling, use greedy decoding otherwise (Default:
false
)Whether or not to use sampling, use greedy decoding otherwise (Default:
false
)- Definition Classes
- MarianTransformer
- val encoding: Param[String]
-
val
engine: Param[String]
This param is set internally once via loadSavedModel.
This param is set internally once via loadSavedModel. That's why there is no setter
- Definition Classes
- HasEngine
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
explainParam(param: Param[_]): String
- Definition Classes
- Params
-
def
explainParams(): String
- Definition Classes
- Params
-
def
extraValidate(structType: StructType): Boolean
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
def
extraValidateMsg: String
Override for additional custom schema checks
Override for additional custom schema checks
- Attributes
- protected
- Definition Classes
- RawAnnotator
- val extraWhitespace: Param[Boolean]
-
final
def
extractParamMap(): ParamMap
- Definition Classes
- Params
-
final
def
extractParamMap(extra: ParamMap): ParamMap
- Definition Classes
- Params
-
val
features: ArrayBuffer[Feature[_, _, _]]
- Definition Classes
- HasFeatures
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
def
get[T](feature: StructFeature[T]): Option[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[K, V](feature: MapFeature[K, V]): Option[Map[K, V]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[T](feature: SetFeature[T]): Option[Set[T]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[T](feature: ArrayFeature[T]): Option[Array[T]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
get[T](param: Param[T]): Option[T]
- Definition Classes
- Params
-
def
getBatchSize: Int
Size of every batch.
Size of every batch.
- Definition Classes
- HasBatchedAnnotate
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
getConfigProtoBytes: Option[Array[Byte]]
- Definition Classes
- MarianTransformer
-
final
def
getDefault[T](param: Param[T]): Option[T]
- Definition Classes
- Params
-
def
getDoSample: Boolean
- Definition Classes
- MarianTransformer
-
def
getEngine: String
- Definition Classes
- HasEngine
-
def
getIgnoreTokenIds: Array[Int]
- Definition Classes
- MarianTransformer
-
def
getInputCols: Array[String]
- returns
input annotations columns currently used
- Definition Classes
- HasInputAnnotationCols
-
def
getLangId: String
- Definition Classes
- MarianTransformer
-
def
getLazyAnnotator: Boolean
- Definition Classes
- CanBeLazy
-
def
getMaxInputLength: Int
- Definition Classes
- MarianTransformer
-
def
getMaxOutputLength: Int
- Definition Classes
- MarianTransformer
-
def
getModelIfNotSet: MarianEncoderDecoder
- Definition Classes
- MarianTransformer
-
def
getNoRepeatNgramSize: Int
- Definition Classes
- MarianTransformer
-
final
def
getOrDefault[T](param: Param[T]): T
- Definition Classes
- Params
-
final
def
getOutputCol: String
Gets annotation column name going to generate
Gets annotation column name going to generate
- Definition Classes
- HasOutputAnnotationCol
-
def
getParam(paramName: String): Param[Any]
- Definition Classes
- Params
-
def
getRandomSeed: Option[Long]
- Definition Classes
- MarianTransformer
-
def
getRepetitionPenalty: Double
- Definition Classes
- MarianTransformer
-
def
getSignatures: Option[Map[String, String]]
- Definition Classes
- MarianTransformer
-
def
getTemperature: Double
- Definition Classes
- MarianTransformer
-
def
getTopK: Int
- Definition Classes
- MarianTransformer
-
def
getTopP: Double
- Definition Classes
- MarianTransformer
-
def
getVocabulary: Array[String]
do not remove or replace with $(vocabulary) due to a bug in some models
do not remove or replace with $(vocabulary) due to a bug in some models
- Definition Classes
- MarianTransformer
-
final
def
hasDefault[T](param: Param[T]): Boolean
- Definition Classes
- Params
-
def
hasParam(paramName: String): Boolean
- Definition Classes
- Params
-
def
hasParent: Boolean
- Definition Classes
- Model
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- val ignoreCase: Param[Boolean]
-
var
ignoreTokenIds: IntArrayParam
A list of token ids which are ignored in the decoder's output
A list of token ids which are ignored in the decoder's output
- Definition Classes
- MarianTransformer
-
def
initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
val
inputAnnotatorTypes: Array[String]
Input Annotator Type: DOCUMENT
Input Annotator Type: DOCUMENT
- Definition Classes
- MarianTransformer → HasInputAnnotationCols
-
final
val
inputCols: StringArrayParam
columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified
columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
-
final
def
isDefined(param: Param[_]): Boolean
- Definition Classes
- Params
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
isSet(param: Param[_]): Boolean
- Definition Classes
- Params
-
def
isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
var
langId: Param[String]
A string representing the target language in the form of >>id<< (id = valid target language ID) (Default:
""
)A string representing the target language in the form of >>id<< (id = valid target language ID) (Default:
""
)langId is only needed if the model generates multi-lingual target language texts. For instance, for a 'en-fr' model this param is not required to be set.
- Definition Classes
- MarianTransformer
-
val
lazyAnnotator: BooleanParam
- Definition Classes
- CanBeLazy
-
def
log: Logger
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logName: String
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- val lowercase: Param[Boolean]
-
val
maxInputLength: IntParam
Controls the maximum length for encoder inputs (source language texts) (Default:
40
)Controls the maximum length for encoder inputs (source language texts) (Default:
40
)- Definition Classes
- MarianTransformer
-
val
maxOutputLength: IntParam
Controls the maximum length for decoder outputs (target language texts) (Default:
40
)Controls the maximum length for decoder outputs (target language texts) (Default:
40
)- Definition Classes
- MarianTransformer
-
def
msgHelper(schema: StructType): String
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
val
noRepeatNgramSize: IntParam
If set to int >
0
, all ngrams of that size can only occur once (Default:0
)If set to int >
0
, all ngrams of that size can only occur once (Default:0
)- Definition Classes
- MarianTransformer
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
def
onWrite(path: String, spark: SparkSession): Unit
- Definition Classes
- MarianTransformer → ParamsAndFeaturesWritable
-
val
optionalInputAnnotatorTypes: Array[String]
- Definition Classes
- HasInputAnnotationCols
-
val
outputAnnotatorType: AnnotatorType
Annotator reference id.
Annotator reference id. Used to identify elements in metadata or to refer to this annotator type
- Definition Classes
- Cleaner → MarianTransformer → HasOutputAnnotatorType
-
final
val
outputCol: Param[String]
- Attributes
- protected
- Definition Classes
- HasOutputAnnotationCol
-
lazy val
params: Array[Param[_]]
- Definition Classes
- Params
-
var
parent: Estimator[MarianTransformer]
- Definition Classes
- Model
-
var
randomSeed: Option[Long]
Optional Random seed for the model.
Optional Random seed for the model. Needs to be of type
Long
.- Definition Classes
- MarianTransformer
-
val
repetitionPenalty: DoubleParam
The parameter for repetition penalty (Default:
1.0
).The parameter for repetition penalty (Default:
1.0
).1.0
means no penalty. See this paper for more details.- Definition Classes
- MarianTransformer
-
def
save(path: String): Unit
- Definition Classes
- MLWritable
- Annotations
- @Since( "1.6.0" ) @throws( ... )
-
def
set[T](param: ProtectedParam[T], value: T): Cleaner.this.type
Sets the value for a protected Param.
Sets the value for a protected Param.
If the parameter was already set, it will not be set again. Default values do not count as a set value and can be overridden.
- T
Type of the parameter
- param
Protected parameter to set
- value
Value for the parameter
- returns
This object
- Definition Classes
- HasProtectedParams
-
def
set[T](feature: StructFeature[T], value: T): Cleaner.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[K, V](feature: MapFeature[K, V], value: Map[K, V]): Cleaner.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[T](feature: SetFeature[T], value: Set[T]): Cleaner.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[T](feature: ArrayFeature[T], value: Array[T]): Cleaner.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
set(paramPair: ParamPair[_]): Cleaner.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
set(param: String, value: Any): Cleaner.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
set[T](param: Param[T], value: T): Cleaner.this.type
- Definition Classes
- Params
-
def
setBatchSize(size: Int): Cleaner.this.type
Size of every batch.
Size of every batch.
- Definition Classes
- HasBatchedAnnotate
- def setBullets(value: Boolean): Cleaner.this.type
- def setCleanPostfixPattern(value: String): Cleaner.this.type
- def setCleanPrefixPattern(value: String): Cleaner.this.type
- def setCleanerMode(value: String): Cleaner.this.type
-
def
setConfigProtoBytes(bytes: Array[Int]): Cleaner.this.type
- Definition Classes
- MarianTransformer
- def setDashes(value: Boolean): Cleaner.this.type
-
def
setDefault[T](feature: StructFeature[T], value: () ⇒ T): Cleaner.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[K, V](feature: MapFeature[K, V], value: () ⇒ Map[K, V]): Cleaner.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[T](feature: SetFeature[T], value: () ⇒ Set[T]): Cleaner.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[T](feature: ArrayFeature[T], value: () ⇒ Array[T]): Cleaner.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
setDefault(paramPairs: ParamPair[_]*): Cleaner.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
setDefault[T](param: Param[T], value: T): Cleaner.this.type
- Attributes
- protected[org.apache.spark.ml]
- Definition Classes
- Params
-
def
setDoSample(value: Boolean): Cleaner.this.type
- Definition Classes
- MarianTransformer
- def setEncoding(value: String): Cleaner.this.type
- def setExtraWhitespace(value: Boolean): Cleaner.this.type
- def setIgnoreCase(value: Boolean): Cleaner.this.type
-
def
setIgnoreTokenIds(tokenIds: Array[Int]): Cleaner.this.type
- Definition Classes
- MarianTransformer
-
final
def
setInputCols(value: String*): Cleaner.this.type
- Definition Classes
- HasInputAnnotationCols
-
def
setInputCols(value: Array[String]): Cleaner.this.type
Overrides required annotators column if different than default
Overrides required annotators column if different than default
- Definition Classes
- HasInputAnnotationCols
-
def
setLangId(lang: String): Cleaner.this.type
- Definition Classes
- MarianTransformer
-
def
setLazyAnnotator(value: Boolean): Cleaner.this.type
- Definition Classes
- CanBeLazy
- def setLowercase(value: Boolean): Cleaner.this.type
-
def
setMaxInputLength(value: Int): Cleaner.this.type
- Definition Classes
- MarianTransformer
-
def
setMaxOutputLength(value: Int): Cleaner.this.type
- Definition Classes
- MarianTransformer
-
def
setModelIfNotSet(spark: SparkSession, encoder: OnnxWrapper, decoder: OnnxWrapper, sppSrc: SentencePieceWrapper, sppTrg: SentencePieceWrapper): Cleaner.this.type
- Definition Classes
- MarianTransformer
-
def
setModelIfNotSet(spark: SparkSession, tensorflow: TensorflowWrapper, sppSrc: SentencePieceWrapper, sppTrg: SentencePieceWrapper): Cleaner.this.type
- Definition Classes
- MarianTransformer
-
def
setNoRepeatNgramSize(value: Int): Cleaner.this.type
- Definition Classes
- MarianTransformer
-
final
def
setOutputCol(value: String): Cleaner.this.type
Overrides annotation column name when transforming
Overrides annotation column name when transforming
- Definition Classes
- HasOutputAnnotationCol
-
def
setParent(parent: Estimator[MarianTransformer]): MarianTransformer
- Definition Classes
- Model
-
def
setRandomSeed(value: Long): Cleaner.this.type
- Definition Classes
- MarianTransformer
-
def
setRepetitionPenalty(value: Double): Cleaner.this.type
- Definition Classes
- MarianTransformer
-
def
setSignatures(value: Map[String, String]): Cleaner.this.type
- Definition Classes
- MarianTransformer
- def setStrip(value: Boolean): Cleaner.this.type
-
def
setTemperature(value: Double): Cleaner.this.type
- Definition Classes
- MarianTransformer
-
def
setTopK(value: Int): Cleaner.this.type
- Definition Classes
- MarianTransformer
-
def
setTopP(value: Double): Cleaner.this.type
- Definition Classes
- MarianTransformer
- def setTrailingPunctuation(value: Boolean): Cleaner.this.type
-
def
setVocabulary(value: Array[String]): Cleaner.this.type
- Definition Classes
- MarianTransformer
-
val
signatures: MapFeature[String, String]
It contains TF model signatures for the laded saved model
It contains TF model signatures for the laded saved model
- Definition Classes
- MarianTransformer
- val strip: Param[Boolean]
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
val
temperature: DoubleParam
The value used to module the next token probabilities (Default:
1.0
)The value used to module the next token probabilities (Default:
1.0
)- Definition Classes
- MarianTransformer
-
def
toString(): String
- Definition Classes
- Identifiable → AnyRef → Any
-
val
topK: IntParam
The number of highest probability vocabulary tokens to keep for top-k-filtering (Default:
50
)The number of highest probability vocabulary tokens to keep for top-k-filtering (Default:
50
)- Definition Classes
- MarianTransformer
-
val
topP: DoubleParam
If set to float <
1.0
, only the most probable tokens with probabilities that add up totopP
or higher are kept for generation (Default:1.0
)If set to float <
1.0
, only the most probable tokens with probabilities that add up totopP
or higher are kept for generation (Default:1.0
)- Definition Classes
- MarianTransformer
- val trailingPunctuation: Param[Boolean]
-
final
def
transform(dataset: Dataset[_]): DataFrame
Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content
Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content
- dataset
Dataset[Row]
- Definition Classes
- AnnotatorModel → Transformer
-
def
transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame
- Definition Classes
- Transformer
- Annotations
- @Since( "2.0.0" )
-
def
transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame
- Definition Classes
- Transformer
- Annotations
- @Since( "2.0.0" ) @varargs()
-
final
def
transformSchema(schema: StructType): StructType
requirement for pipeline transformation validation.
requirement for pipeline transformation validation. It is called on fit()
- Definition Classes
- RawAnnotator → PipelineStage
-
def
transformSchema(schema: StructType, logging: Boolean): StructType
- Attributes
- protected
- Definition Classes
- PipelineStage
- Annotations
- @DeveloperApi()
-
val
uid: String
- Definition Classes
- Cleaner → MarianTransformer → Identifiable
-
def
validate(schema: StructType): Boolean
takes a Dataset and checks to see if all the required annotation types are present.
takes a Dataset and checks to see if all the required annotation types are present.
- schema
to be validated
- returns
True if all the required types are present, else false
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
val
vocabulary: StringArrayParam
Vocabulary used to encode and decode piece tokens generated by SentencePiece.
Vocabulary used to encode and decode piece tokens generated by SentencePiece. This will be set once the model is created and cannot be changed afterwards
- Definition Classes
- MarianTransformer
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
wrapColumnMetadata(col: Column): Column
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
def
write: MLWriter
- Definition Classes
- ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable
-
def
writeOnnxModel(path: String, spark: SparkSession, onnxWrapper: OnnxWrapper, suffix: String, fileName: String): Unit
- Definition Classes
- WriteOnnxModel
-
def
writeOnnxModels(path: String, spark: SparkSession, onnxWrappersWithNames: Seq[(OnnxWrapper, String)], suffix: String): Unit
- Definition Classes
- WriteOnnxModel
-
def
writeSentencePieceModel(path: String, spark: SparkSession, spp: SentencePieceWrapper, suffix: String, filename: String): Unit
- Definition Classes
- WriteSentencePieceModel
-
def
writeTensorflowHub(path: String, tfPath: String, spark: SparkSession, suffix: String = "_use"): Unit
- Definition Classes
- WriteTensorflowModel
-
def
writeTensorflowModel(path: String, spark: SparkSession, tensorflow: TensorflowWrapper, suffix: String, filename: String, configProtoBytes: Option[Array[Byte]] = None): Unit
- Definition Classes
- WriteTensorflowModel
-
def
writeTensorflowModelV2(path: String, spark: SparkSession, tensorflow: TensorflowWrapper, suffix: String, filename: String, configProtoBytes: Option[Array[Byte]] = None, savedSignatures: Option[Map[String, String]] = None): Unit
- Definition Classes
- WriteTensorflowModel
Inherited from MarianTransformer
Inherited from HasProtectedParams
Inherited from HasEngine
Inherited from WriteSentencePieceModel
Inherited from WriteOnnxModel
Inherited from WriteTensorflowModel
Inherited from HasBatchedAnnotate[MarianTransformer]
Inherited from AnnotatorModel[MarianTransformer]
Inherited from CanBeLazy
Inherited from RawAnnotator[MarianTransformer]
Inherited from HasOutputAnnotationCol
Inherited from HasInputAnnotationCols
Inherited from HasOutputAnnotatorType
Inherited from ParamsAndFeaturesWritable
Inherited from HasFeatures
Inherited from DefaultParamsWritable
Inherited from MLWritable
Inherited from Model[MarianTransformer]
Inherited from Transformer
Inherited from PipelineStage
Inherited from Logging
Inherited from Params
Inherited from Serializable
Inherited from Serializable
Inherited from Identifiable
Inherited from AnyRef
Inherited from Any
setParam *
Parameters
A list of (hyper-)parameter keys this annotator can take. Users can set and get the parameter values through setters and getters, respectively.
Annotator types
Required input and expected output annotator types