com.johnsnowlabs.nlp.annotators.spell.symmetric
SymmetricDeleteModel
Companion object SymmetricDeleteModel
class SymmetricDeleteModel extends AnnotatorModel[SymmetricDeleteModel] with HasSimpleAnnotate[SymmetricDeleteModel] with SymmetricDeleteParams
Symmetric Delete spelling correction algorithm.
The Symmetric Delete spelling correction algorithm reduces the complexity of edit candidate generation and dictionary lookup for a given Damerau-Levenshtein distance. It is six orders of magnitude faster (than the standard approach with deletes + transposes + replaces + inserts) and language independent.
Inspired by SymSpell.
Pretrained models can be loaded with pretrained
of the companion object:
val spell = SymmetricDeleteModel.pretrained() .setInputCols("token") .setOutputCol("spell")
The default model is "spellcheck_sd"
, if no name is provided. For available pretrained
models please see the Models Hub.
See SymmetricDeleteModelTestSpec for further reference.
Example
import spark.implicits._ import com.johnsnowlabs.nlp.base.DocumentAssembler import com.johnsnowlabs.nlp.annotators.Tokenizer import com.johnsnowlabs.nlp.annotators.spell.symmetric.SymmetricDeleteModel import org.apache.spark.ml.Pipeline val documentAssembler = new DocumentAssembler() .setInputCol("text") .setOutputCol("document") val tokenizer = new Tokenizer() .setInputCols("document") .setOutputCol("token") val spellChecker = SymmetricDeleteModel.pretrained() .setInputCols("token") .setOutputCol("spell") val pipeline = new Pipeline().setStages(Array( documentAssembler, tokenizer, spellChecker )) val data = Seq("spmetimes i wrrite wordz erong.").toDF("text") val result = pipeline.fit(data).transform(data) result.select("spell.result").show(false) +--------------------------------------+ |result | +--------------------------------------+ |[sometimes, i, write, words, wrong, .]| +--------------------------------------+
- See also
NorvigSweetingModel for an alternative approach to spell checking
ContextSpellCheckerModel for a DL based approach
- Grouped
- Alphabetic
- By Inheritance
- SymmetricDeleteModel
- SymmetricDeleteParams
- HasSimpleAnnotate
- AnnotatorModel
- CanBeLazy
- RawAnnotator
- HasOutputAnnotationCol
- HasInputAnnotationCols
- HasOutputAnnotatorType
- ParamsAndFeaturesWritable
- HasFeatures
- DefaultParamsWritable
- MLWritable
- Model
- Transformer
- PipelineStage
- Logging
- Params
- Serializable
- Serializable
- Identifiable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Type Members
-
type
AnnotationContent = Seq[Row]
internal types to show Rows as a relevant StructType Should be deleted once Spark releases UserDefinedTypes to @developerAPI
internal types to show Rows as a relevant StructType Should be deleted once Spark releases UserDefinedTypes to @developerAPI
- Attributes
- protected
- Definition Classes
- AnnotatorModel
-
type
AnnotatorType = String
- Definition Classes
- HasOutputAnnotatorType
- case class SuggestedWord(correction: String, frequency: Long, distance: Int, score: Double) extends Product with Serializable
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
$[T](param: Param[T]): T
- Attributes
- protected
- Definition Classes
- Params
-
def
$$[T](feature: StructFeature[T]): T
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[K, V](feature: MapFeature[K, V]): Map[K, V]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[T](feature: SetFeature[T]): Set[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[T](feature: ArrayFeature[T]): Array[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
_transform(dataset: Dataset[_], recursivePipeline: Option[PipelineModel]): DataFrame
- Attributes
- protected
- Definition Classes
- AnnotatorModel
-
def
afterAnnotate(dataset: DataFrame): DataFrame
- Attributes
- protected
- Definition Classes
- AnnotatorModel
-
def
annotate(annotations: Seq[Annotation]): Seq[Annotation]
takes a document and annotations and produces new annotations of this annotator's annotation type
takes a document and annotations and produces new annotations of this annotator's annotation type
- annotations
Annotations that correspond to inputAnnotationCols generated by previous annotators if any
- returns
any number of annotations processed for every input annotation. Not necessary one to one relationship
- Definition Classes
- SymmetricDeleteModel → HasSimpleAnnotate
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
beforeAnnotate(dataset: Dataset[_]): Dataset[_]
- Attributes
- protected
- Definition Classes
- AnnotatorModel
-
final
def
checkSchema(schema: StructType, inputAnnotatorType: String): Boolean
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
- def checkSpellWord(originalWord: String): (String, Double)
-
final
def
clear(param: Param[_]): SymmetricDeleteModel.this.type
- Definition Classes
- Params
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
copy(extra: ParamMap): SymmetricDeleteModel
requirement for annotators copies
requirement for annotators copies
- Definition Classes
- RawAnnotator → Model → Transformer → PipelineStage → Params
-
def
copyValues[T <: Params](to: T, extra: ParamMap): T
- Attributes
- protected
- Definition Classes
- Params
-
final
def
defaultCopy[T <: Params](extra: ParamMap): T
- Attributes
- protected
- Definition Classes
- Params
-
val
deletesThreshold: IntParam
Minimum frequency of corrections a word needs to have to be considered from training.
Minimum frequency of corrections a word needs to have to be considered from training. Increase if training set is LARGE (Default:
0
).- Definition Classes
- SymmetricDeleteParams
-
val
derivedWords: MapFeature[String, (List[String], Long)]
- Attributes
- protected
-
def
dfAnnotate: UserDefinedFunction
Wraps annotate to happen inside SparkSQL user defined functions in order to act with org.apache.spark.sql.Column
Wraps annotate to happen inside SparkSQL user defined functions in order to act with org.apache.spark.sql.Column
- returns
udf function to be applied to inputCols using this annotator's annotate function as part of ML transformation
- Definition Classes
- HasSimpleAnnotate
-
val
dictionary: MapFeature[String, Long]
- Attributes
- protected
-
val
dupsLimit: IntParam
Maximum duplicate of characters in a word to consider (Default:
2
).Maximum duplicate of characters in a word to consider (Default:
2
).- Definition Classes
- SymmetricDeleteParams
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
explainParam(param: Param[_]): String
- Definition Classes
- Params
-
def
explainParams(): String
- Definition Classes
- Params
-
def
extraValidate(structType: StructType): Boolean
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
def
extraValidateMsg: String
Override for additional custom schema checks
Override for additional custom schema checks
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
final
def
extractParamMap(): ParamMap
- Definition Classes
- Params
-
final
def
extractParamMap(extra: ParamMap): ParamMap
- Definition Classes
- Params
-
val
features: ArrayBuffer[Feature[_, _, _]]
- Definition Classes
- HasFeatures
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
val
frequencyThreshold: IntParam
Minimum frequency of words to be considered from training.
Minimum frequency of words to be considered from training. Increase if training set is LARGE (Default:
0
).- Definition Classes
- SymmetricDeleteParams
-
def
get[T](feature: StructFeature[T]): Option[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[K, V](feature: MapFeature[K, V]): Option[Map[K, V]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[T](feature: SetFeature[T]): Option[Set[T]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[T](feature: ArrayFeature[T]): Option[Array[T]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
get[T](param: Param[T]): Option[T]
- Definition Classes
- Params
- def getCaseWordType(word: String): Char
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
final
def
getDefault[T](param: Param[T]): Option[T]
- Definition Classes
- Params
-
def
getDeletesThreshold: Int
Minimum frequency of corrections a word needs to have to be considered from training.
Minimum frequency of corrections a word needs to have to be considered from training. Increase if training set is LARGE (Default:
0
).- Definition Classes
- SymmetricDeleteParams
- def getDictionarySuggestions(word: String): Option[SuggestedWord]
-
def
getDupsLimit: Int
Maximum duplicate of characters in a word to consider (Default:
2
).Maximum duplicate of characters in a word to consider (Default:
2
).- Definition Classes
- SymmetricDeleteParams
-
def
getFrequencyThreshold: Int
Minimum frequency of words to be considered from training.
Minimum frequency of words to be considered from training. Increase if training set is LARGE (Default:
0
).- Definition Classes
- SymmetricDeleteParams
-
def
getInputCols: Array[String]
- returns
input annotations columns currently used
- Definition Classes
- HasInputAnnotationCols
-
def
getLazyAnnotator: Boolean
- Definition Classes
- CanBeLazy
-
def
getMaxEditDistance: Int
Max edit distance characters to derive strings from a word
Max edit distance characters to derive strings from a word
- Definition Classes
- SymmetricDeleteParams
-
final
def
getOrDefault[T](param: Param[T]): T
- Definition Classes
- Params
-
final
def
getOutputCol: String
Gets annotation column name going to generate
Gets annotation column name going to generate
- Definition Classes
- HasOutputAnnotationCol
-
def
getParam(paramName: String): Param[Any]
- Definition Classes
- Params
- def getScoreFrequency(word: String): Double
-
def
getSuggestedCorrections(word: String): Option[SuggestedWord]
Return list of suggested corrections for potentially incorrectly spelled word
- def getSymmetricSuggestions(word: String): Option[SuggestedWord]
-
final
def
hasDefault[T](param: Param[T]): Boolean
- Definition Classes
- Params
-
def
hasParam(paramName: String): Boolean
- Definition Classes
- Params
-
def
hasParent: Boolean
- Definition Classes
- Model
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
val
inputAnnotatorTypes: Array[AnnotatorType]
Input annotator type: TOKEN
Input annotator type: TOKEN
- Definition Classes
- SymmetricDeleteModel → HasInputAnnotationCols
-
final
val
inputCols: StringArrayParam
columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified
columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
-
final
def
isDefined(param: Param[_]): Boolean
- Definition Classes
- Params
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def isNoisyWord(word: String): Boolean
-
final
def
isSet(param: Param[_]): Boolean
- Definition Classes
- Params
-
def
isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
val
lazyAnnotator: BooleanParam
- Definition Classes
- CanBeLazy
-
def
log: Logger
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logName: String
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
val
longestWordLength: IntParam
Length of longest word in corpus
Length of longest word in corpus
- Definition Classes
- SymmetricDeleteParams
-
val
maxEditDistance: IntParam
Max edit distance characters to derive strings from a word (Default:
3
)Max edit distance characters to derive strings from a word (Default:
3
)- Definition Classes
- SymmetricDeleteParams
-
val
maxFrequency: LongParam
Maximum frequency of a word in the corpus
Maximum frequency of a word in the corpus
- Definition Classes
- SymmetricDeleteParams
-
val
minFrequency: LongParam
Minimum frequency of a word in the corpus
Minimum frequency of a word in the corpus
- Definition Classes
- SymmetricDeleteParams
-
def
msgHelper(schema: StructType): String
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def normalizeFrequencyValue(value: Long): Double
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
def
onWrite(path: String, spark: SparkSession): Unit
- Attributes
- protected
- Definition Classes
- ParamsAndFeaturesWritable
-
val
optionalInputAnnotatorTypes: Array[String]
- Definition Classes
- HasInputAnnotationCols
-
val
outputAnnotatorType: AnnotatorType
Output annotator type: TOKEN
Output annotator type: TOKEN
- Definition Classes
- SymmetricDeleteModel → HasOutputAnnotatorType
-
final
val
outputCol: Param[String]
- Attributes
- protected
- Definition Classes
- HasOutputAnnotationCol
-
lazy val
params: Array[Param[_]]
- Definition Classes
- Params
-
var
parent: Estimator[SymmetricDeleteModel]
- Definition Classes
- Model
-
def
save(path: String): Unit
- Definition Classes
- MLWritable
- Annotations
- @Since( "1.6.0" ) @throws( ... )
-
def
set[T](feature: StructFeature[T], value: T): SymmetricDeleteModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[K, V](feature: MapFeature[K, V], value: Map[K, V]): SymmetricDeleteModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[T](feature: SetFeature[T], value: Set[T]): SymmetricDeleteModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[T](feature: ArrayFeature[T], value: Array[T]): SymmetricDeleteModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
set(paramPair: ParamPair[_]): SymmetricDeleteModel.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
set(param: String, value: Any): SymmetricDeleteModel.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
set[T](param: Param[T], value: T): SymmetricDeleteModel.this.type
- Definition Classes
- Params
-
def
setDefault[T](feature: StructFeature[T], value: () ⇒ T): SymmetricDeleteModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[K, V](feature: MapFeature[K, V], value: () ⇒ Map[K, V]): SymmetricDeleteModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[T](feature: SetFeature[T], value: () ⇒ Set[T]): SymmetricDeleteModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[T](feature: ArrayFeature[T], value: () ⇒ Array[T]): SymmetricDeleteModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
setDefault(paramPairs: ParamPair[_]*): SymmetricDeleteModel.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
setDefault[T](param: Param[T], value: T): SymmetricDeleteModel.this.type
- Attributes
- protected[org.apache.spark.ml]
- Definition Classes
- Params
-
def
setDeletesThreshold(value: Int): SymmetricDeleteModel.this.type
Minimum frequency of corrections a word needs to have to be considered from training.
Minimum frequency of corrections a word needs to have to be considered from training. Increase if training set is LARGE (Default:
0
).- Definition Classes
- SymmetricDeleteParams
- def setDerivedWords(value: Map[String, (List[String], Long)]): SymmetricDeleteModel.this.type
- def setDictionary(value: Map[String, Long]): SymmetricDeleteModel.this.type
-
def
setDupsLimit(value: Int): SymmetricDeleteModel.this.type
Maximum duplicate of characters in a word to consider (Default:
2
)Maximum duplicate of characters in a word to consider (Default:
2
)- Definition Classes
- SymmetricDeleteParams
-
def
setFrequencyThreshold(value: Int): SymmetricDeleteModel.this.type
Minimum frequency of words to be considered from training.
Minimum frequency of words to be considered from training. Increase if training set is LARGE (Default:
0
)- Definition Classes
- SymmetricDeleteParams
-
final
def
setInputCols(value: String*): SymmetricDeleteModel.this.type
- Definition Classes
- HasInputAnnotationCols
-
def
setInputCols(value: Array[String]): SymmetricDeleteModel.this.type
Overrides required annotators column if different than default
Overrides required annotators column if different than default
- Definition Classes
- HasInputAnnotationCols
-
def
setLazyAnnotator(value: Boolean): SymmetricDeleteModel.this.type
- Definition Classes
- CanBeLazy
-
def
setLongestWordLength(value: Int): SymmetricDeleteModel.this.type
Length of longest word in corpus
Length of longest word in corpus
- Definition Classes
- SymmetricDeleteParams
-
def
setMaxEditDistance(value: Int): SymmetricDeleteModel.this.type
Max edit distance characters to derive strings from a word
Max edit distance characters to derive strings from a word
- Definition Classes
- SymmetricDeleteParams
-
def
setMaxFrequency(value: Long): SymmetricDeleteModel.this.type
Maximum frequency of a word in the corpus
Maximum frequency of a word in the corpus
- Definition Classes
- SymmetricDeleteParams
-
def
setMinFrequency(value: Long): SymmetricDeleteModel.this.type
Minimum frequency of a word in the corpus
Minimum frequency of a word in the corpus
- Definition Classes
- SymmetricDeleteParams
-
final
def
setOutputCol(value: String): SymmetricDeleteModel.this.type
Overrides annotation column name when transforming
Overrides annotation column name when transforming
- Definition Classes
- HasOutputAnnotationCol
-
def
setParent(parent: Estimator[SymmetricDeleteModel]): SymmetricDeleteModel
- Definition Classes
- Model
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- Identifiable → AnyRef → Any
-
final
def
transform(dataset: Dataset[_]): DataFrame
Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content
Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content
- dataset
Dataset[Row]
- Definition Classes
- AnnotatorModel → Transformer
-
def
transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame
- Definition Classes
- Transformer
- Annotations
- @Since( "2.0.0" )
-
def
transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame
- Definition Classes
- Transformer
- Annotations
- @Since( "2.0.0" ) @varargs()
-
final
def
transformSchema(schema: StructType): StructType
requirement for pipeline transformation validation.
requirement for pipeline transformation validation. It is called on fit()
- Definition Classes
- RawAnnotator → PipelineStage
-
def
transformSchema(schema: StructType, logging: Boolean): StructType
- Attributes
- protected
- Definition Classes
- PipelineStage
- Annotations
- @DeveloperApi()
- def transformToOriginalCaseType(caseType: Char, word: String): String
-
val
uid: String
- Definition Classes
- SymmetricDeleteModel → Identifiable
-
def
validate(schema: StructType): Boolean
takes a Dataset and checks to see if all the required annotation types are present.
takes a Dataset and checks to see if all the required annotation types are present.
- schema
to be validated
- returns
True if all the required types are present, else false
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
wrapColumnMetadata(col: Column): Column
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
def
write: MLWriter
- Definition Classes
- ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable
Inherited from SymmetricDeleteParams
Inherited from HasSimpleAnnotate[SymmetricDeleteModel]
Inherited from AnnotatorModel[SymmetricDeleteModel]
Inherited from CanBeLazy
Inherited from RawAnnotator[SymmetricDeleteModel]
Inherited from HasOutputAnnotationCol
Inherited from HasInputAnnotationCols
Inherited from HasOutputAnnotatorType
Inherited from ParamsAndFeaturesWritable
Inherited from HasFeatures
Inherited from DefaultParamsWritable
Inherited from MLWritable
Inherited from Model[SymmetricDeleteModel]
Inherited from Transformer
Inherited from PipelineStage
Inherited from Logging
Inherited from Params
Inherited from Serializable
Inherited from Serializable
Inherited from Identifiable
Inherited from AnyRef
Inherited from Any
Parameters
A list of (hyper-)parameter keys this annotator can take. Users can set and get the parameter values through setters and getters, respectively.
Annotator types
Required input and expected output annotator types