Packages

class NorvigSweetingApproach extends AnnotatorApproach[NorvigSweetingModel] with NorvigSweetingParams

Trains annotator, that retrieves tokens and makes corrections automatically if not found in an English dictionary, based on the algorithm by Peter Norvig.

The algorithm is based on a Bayesian approach to spell checking: Given the word we look in the provided dictionary to choose the word with the highest probability to be the correct one.

A dictionary of correct spellings must be provided with setDictionary either in the form of a text file or directly as an ExternalResource, where each word is parsed by a regex pattern.

Inspired by the spell checker by Peter Norvig: How to Write a Spelling Corrector.

For instantiated/pretrained models, see NorvigSweetingModel.

For extended examples of usage, see the NorvigSweetingTestSpec.

Example

In this example, the dictionary "words.txt" has the form of

...
gummy
gummic
gummier
gummiest
gummiferous
...

This dictionary is then set to be the basis of the spell checker.

import com.johnsnowlabs.nlp.base.DocumentAssembler
import com.johnsnowlabs.nlp.annotators.Tokenizer
import com.johnsnowlabs.nlp.annotators.spell.norvig.NorvigSweetingApproach
import org.apache.spark.ml.Pipeline

val documentAssembler = new DocumentAssembler()
  .setInputCol("text")
  .setOutputCol("document")

val tokenizer = new Tokenizer()
  .setInputCols("document")
  .setOutputCol("token")

val spellChecker = new NorvigSweetingApproach()
  .setInputCols("token")
  .setOutputCol("spell")
  .setDictionary("src/test/resources/spell/words.txt")

val pipeline = new Pipeline().setStages(Array(
  documentAssembler,
  tokenizer,
  spellChecker
))

val pipelineModel = pipeline.fit(trainingData)
See also

SymmetricDeleteApproach for an alternative approach to spell checking

ContextSpellCheckerApproach for a DL based approach

Linear Supertypes
NorvigSweetingParams, AnnotatorApproach[NorvigSweetingModel], CanBeLazy, DefaultParamsWritable, MLWritable, HasOutputAnnotatorType, HasOutputAnnotationCol, HasInputAnnotationCols, Estimator[NorvigSweetingModel], PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Grouped
  2. Alphabetic
  3. By Inheritance
Inherited
  1. NorvigSweetingApproach
  2. NorvigSweetingParams
  3. AnnotatorApproach
  4. CanBeLazy
  5. DefaultParamsWritable
  6. MLWritable
  7. HasOutputAnnotatorType
  8. HasOutputAnnotationCol
  9. HasInputAnnotationCols
  10. Estimator
  11. PipelineStage
  12. Logging
  13. Params
  14. Serializable
  15. Serializable
  16. Identifiable
  17. AnyRef
  18. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new NorvigSweetingApproach()
  2. new NorvigSweetingApproach(uid: String)

Type Members

  1. type AnnotatorType = String
    Definition Classes
    HasOutputAnnotatorType

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def $[T](param: Param[T]): T
    Attributes
    protected
    Definition Classes
    Params
  4. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  5. def _fit(dataset: Dataset[_], recursiveStages: Option[PipelineModel]): NorvigSweetingModel
    Attributes
    protected
    Definition Classes
    AnnotatorApproach
  6. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  7. def beforeTraining(spark: SparkSession): Unit
    Definition Classes
    AnnotatorApproach
  8. val caseSensitive: BooleanParam

    Sensitivity on spell checking (Default: true).

    Sensitivity on spell checking (Default: true). Might affect accuracy

    Definition Classes
    NorvigSweetingParams
  9. final def checkSchema(schema: StructType, inputAnnotatorType: String): Boolean
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  10. final def clear(param: Param[_]): NorvigSweetingApproach.this.type
    Definition Classes
    Params
  11. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  12. final def copy(extra: ParamMap): Estimator[NorvigSweetingModel]
    Definition Classes
    AnnotatorApproach → Estimator → PipelineStage → Params
  13. def copyValues[T <: Params](to: T, extra: ParamMap): T
    Attributes
    protected
    Definition Classes
    Params
  14. final def defaultCopy[T <: Params](extra: ParamMap): T
    Attributes
    protected
    Definition Classes
    Params
  15. val description: String

    Spell checking algorithm inspired on Norvig model

    Spell checking algorithm inspired on Norvig model

    Definition Classes
    NorvigSweetingApproach → AnnotatorApproach
  16. val dictionary: ExternalResourceParam

    External dictionary to be used, which needs "tokenPattern" (Default: \S+) for parsing the resource.

    External dictionary to be used, which needs "tokenPattern" (Default: \S+) for parsing the resource.

    Example

    ...
gummy
gummic
gummier
gummiest
gummiferous
...
  17. val doubleVariants: BooleanParam

    Increase search at cost of performance (Default: false).

    Increase search at cost of performance (Default: false). Enables extra check for word combinations, More accuracy at performance

    Definition Classes
    NorvigSweetingParams
  18. val dupsLimit: IntParam

    Maximum duplicate of characters in a word to consider (Default: 2).

    Maximum duplicate of characters in a word to consider (Default: 2). Maximum duplicate of characters to account for.

    Definition Classes
    NorvigSweetingParams
  19. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  20. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  21. def explainParam(param: Param[_]): String
    Definition Classes
    Params
  22. def explainParams(): String
    Definition Classes
    Params
  23. final def extractParamMap(): ParamMap
    Definition Classes
    Params
  24. final def extractParamMap(extra: ParamMap): ParamMap
    Definition Classes
    Params
  25. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  26. final def fit(dataset: Dataset[_]): NorvigSweetingModel
    Definition Classes
    AnnotatorApproach → Estimator
  27. def fit(dataset: Dataset[_], paramMaps: Seq[ParamMap]): Seq[NorvigSweetingModel]
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" )
  28. def fit(dataset: Dataset[_], paramMap: ParamMap): NorvigSweetingModel
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" )
  29. def fit(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): NorvigSweetingModel
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" ) @varargs()
  30. val frequencyPriority: BooleanParam

    Applies frequency over hamming in intersections (Default: true).

    Applies frequency over hamming in intersections (Default: true). When false hamming takes priority

    Definition Classes
    NorvigSweetingParams
  31. final def get[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  32. def getCaseSensitive: Boolean

    Sensitivity on spell checking (Default: true).

    Sensitivity on spell checking (Default: true). Might affect accuracy

    Definition Classes
    NorvigSweetingParams
  33. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  34. final def getDefault[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  35. def getDoubleVariants: Boolean

    Increase search at cost of performance (Default: false).

    Increase search at cost of performance (Default: false). Enables extra check for word combinations

    Definition Classes
    NorvigSweetingParams
  36. def getDupsLimit: Int

    Maximum duplicate of characters in a word to consider (Default: 2).

    Maximum duplicate of characters in a word to consider (Default: 2). Maximum duplicate of characters to account for.

    Definition Classes
    NorvigSweetingParams
  37. def getFrequencyPriority: Boolean

    Applies frequency over hamming in intersections (Default: true).

    Applies frequency over hamming in intersections (Default: true). When false hamming takes priority

    Definition Classes
    NorvigSweetingParams
  38. def getInputCols: Array[String]

    returns

    input annotations columns currently used

    Definition Classes
    HasInputAnnotationCols
  39. def getIntersections: Int

    Hamming intersections to attempt (Default: 10).

    Hamming intersections to attempt (Default: 10).

    Definition Classes
    NorvigSweetingParams
  40. def getLazyAnnotator: Boolean
    Definition Classes
    CanBeLazy
  41. final def getOrDefault[T](param: Param[T]): T
    Definition Classes
    Params
  42. final def getOutputCol: String

    Gets annotation column name going to generate

    Gets annotation column name going to generate

    Definition Classes
    HasOutputAnnotationCol
  43. def getParam(paramName: String): Param[Any]
    Definition Classes
    Params
  44. def getReductLimit: Int

    Word reduction limit (Default: 3).

    Word reduction limit (Default: 3).

    Definition Classes
    NorvigSweetingParams
  45. def getShortCircuit: Boolean

    Increase performance at cost of accuracy (Default: false).

    Increase performance at cost of accuracy (Default: false). Faster but less accurate mode

    Definition Classes
    NorvigSweetingParams
  46. def getVowelSwapLimit: Int

    Vowel swap attempts (Default: 6).

    Vowel swap attempts (Default: 6).

    Definition Classes
    NorvigSweetingParams
  47. def getWordSizeIgnore: Int

    Minimum size of word before ignoring (Default: 3).

    Minimum size of word before ignoring (Default: 3). Minimum size of word before moving on.

    Definition Classes
    NorvigSweetingParams
  48. final def hasDefault[T](param: Param[T]): Boolean
    Definition Classes
    Params
  49. def hasParam(paramName: String): Boolean
    Definition Classes
    Params
  50. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  51. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  52. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  53. val inputAnnotatorTypes: Array[AnnotatorType]

    Input annotator type : TOKEN

    Input annotator type : TOKEN

    Definition Classes
    NorvigSweetingApproach → HasInputAnnotationCols
  54. final val inputCols: StringArrayParam

    columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified

    columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified

    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  55. val intersections: IntParam

    Hamming intersections to attempt (Default: 10).

    Hamming intersections to attempt (Default: 10).

    Definition Classes
    NorvigSweetingParams
  56. final def isDefined(param: Param[_]): Boolean
    Definition Classes
    Params
  57. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  58. final def isSet(param: Param[_]): Boolean
    Definition Classes
    Params
  59. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  60. val lazyAnnotator: BooleanParam
    Definition Classes
    CanBeLazy
  61. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  62. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  63. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  64. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  65. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  66. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  67. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  68. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  69. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  70. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  71. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  72. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  73. def msgHelper(schema: StructType): String
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  74. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  75. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  76. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  77. def onTrained(model: NorvigSweetingModel, spark: SparkSession): Unit
    Definition Classes
    AnnotatorApproach
  78. val optionalInputAnnotatorTypes: Array[String]
    Definition Classes
    HasInputAnnotationCols
  79. val outputAnnotatorType: AnnotatorType

    Output annotator type : TOKEN

    Output annotator type : TOKEN

    Definition Classes
    NorvigSweetingApproach → HasOutputAnnotatorType
  80. final val outputCol: Param[String]
    Attributes
    protected
    Definition Classes
    HasOutputAnnotationCol
  81. lazy val params: Array[Param[_]]
    Definition Classes
    Params
  82. val reductLimit: IntParam

    Word reduction limit (Default: 3).

    Word reduction limit (Default: 3).

    Definition Classes
    NorvigSweetingParams
  83. def save(path: String): Unit
    Definition Classes
    MLWritable
    Annotations
    @Since( "1.6.0" ) @throws( ... )
  84. final def set(paramPair: ParamPair[_]): NorvigSweetingApproach.this.type
    Attributes
    protected
    Definition Classes
    Params
  85. final def set(param: String, value: Any): NorvigSweetingApproach.this.type
    Attributes
    protected
    Definition Classes
    Params
  86. final def set[T](param: Param[T], value: T): NorvigSweetingApproach.this.type
    Definition Classes
    Params
  87. def setCaseSensitive(value: Boolean): NorvigSweetingApproach.this.type

    Sensitivity on spell checking (Default: true).

    Sensitivity on spell checking (Default: true). Might affect accuracy

    Definition Classes
    NorvigSweetingParams
  88. final def setDefault(paramPairs: ParamPair[_]*): NorvigSweetingApproach.this.type
    Attributes
    protected
    Definition Classes
    Params
  89. final def setDefault[T](param: Param[T], value: T): NorvigSweetingApproach.this.type
    Attributes
    protected[org.apache.spark.ml]
    Definition Classes
    Params
  90. def setDictionary(path: String, tokenPattern: String = "\\S+", readAs: Format = ReadAs.TEXT, options: Map[String, String] = Map("format" -> "text")): NorvigSweetingApproach.this.type

    Path to file with properly spelled words, tokenPattern is the regex pattern to identify them in text, readAs can be ReadAs.TEXT or ReadAs.SPARK, with options passed to Spark reader if the latter is set.

    Path to file with properly spelled words, tokenPattern is the regex pattern to identify them in text, readAs can be ReadAs.TEXT or ReadAs.SPARK, with options passed to Spark reader if the latter is set. Dictionary needs tokenPattern regex for separating words.

  91. def setDictionary(value: ExternalResource): NorvigSweetingApproach.this.type

    External dictionary already in the form of ExternalResource, for which the Map member options has an entry defined for "tokenPattern".

    External dictionary already in the form of ExternalResource, for which the Map member options has an entry defined for "tokenPattern".

    Example

    val resource = ExternalResource(
  "src/test/resources/spell/words.txt",
  ReadAs.TEXT,
  Map("tokenPattern" -> "\\S+")
)
val spellChecker = new NorvigSweetingApproach()
  .setInputCols("token")
  .setOutputCol("spell")
  .setDictionary(resource)
  92. def setDoubleVariants(value: Boolean): NorvigSweetingApproach.this.type

    Increase search at cost of performance (Default: false).

    Increase search at cost of performance (Default: false). Enables extra check for word combinations

    Definition Classes
    NorvigSweetingParams
  93. def setDupsLimit(value: Int): NorvigSweetingApproach.this.type

    Maximum duplicate of characters in a word to consider (Default: 2).

    Maximum duplicate of characters in a word to consider (Default: 2). Maximum duplicate of characters to account for. Defaults to 2.

    Definition Classes
    NorvigSweetingParams
  94. def setFrequencyPriority(value: Boolean): NorvigSweetingApproach.this.type

    Applies frequency over hamming in intersections (Default: true).

    Applies frequency over hamming in intersections (Default: true). When false hamming takes priority

    Definition Classes
    NorvigSweetingParams
  95. final def setInputCols(value: String*): NorvigSweetingApproach.this.type
    Definition Classes
    HasInputAnnotationCols
  96. def setInputCols(value: Array[String]): NorvigSweetingApproach.this.type

    Overrides required annotators column if different than default

    Overrides required annotators column if different than default

    Definition Classes
    HasInputAnnotationCols
  97. def setIntersections(value: Int): NorvigSweetingApproach.this.type

    Hamming intersections to attempt (Default: 10).

    Hamming intersections to attempt (Default: 10).

    Definition Classes
    NorvigSweetingParams
  98. def setLazyAnnotator(value: Boolean): NorvigSweetingApproach.this.type
    Definition Classes
    CanBeLazy
  99. final def setOutputCol(value: String): NorvigSweetingApproach.this.type

    Overrides annotation column name when transforming

    Overrides annotation column name when transforming

    Definition Classes
    HasOutputAnnotationCol
  100. def setReductLimit(value: Int): NorvigSweetingApproach.this.type

    Word reduction limit (Default: 3).

    Word reduction limit (Default: 3).

    Definition Classes
    NorvigSweetingParams
  101. def setShortCircuit(value: Boolean): NorvigSweetingApproach.this.type

    Increase performance at cost of accuracy (Default: false).

    Increase performance at cost of accuracy (Default: false). Faster but less accurate mode

    Definition Classes
    NorvigSweetingParams
  102. def setVowelSwapLimit(value: Int): NorvigSweetingApproach.this.type

    Vowel swap attempts (Default: 6).

    Vowel swap attempts (Default: 6).

    Definition Classes
    NorvigSweetingParams
  103. def setWordSizeIgnore(value: Int): NorvigSweetingApproach.this.type

    Minimum size of word before ignoring (Default: 3).

    Minimum size of word before ignoring (Default: 3). Minimum size of word before moving on.

    Definition Classes
    NorvigSweetingParams
  104. val shortCircuit: BooleanParam

    Increase performance at cost of accuracy (Default: false).

    Increase performance at cost of accuracy (Default: false). Faster but less accurate mode

    Definition Classes
    NorvigSweetingParams
  105. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  106. def toString(): String
    Definition Classes
    Identifiable → AnyRef → Any
  107. def train(dataset: Dataset[_], recursivePipeline: Option[PipelineModel]): NorvigSweetingModel
    Definition Classes
    NorvigSweetingApproach → AnnotatorApproach
  108. final def transformSchema(schema: StructType): StructType

    requirement for pipeline transformation validation.

    requirement for pipeline transformation validation. It is called on fit()

    Definition Classes
    AnnotatorApproach → PipelineStage
  109. def transformSchema(schema: StructType, logging: Boolean): StructType
    Attributes
    protected
    Definition Classes
    PipelineStage
    Annotations
    @DeveloperApi()
  110. val uid: String
    Definition Classes
    NorvigSweetingApproach → Identifiable
  111. def validate(schema: StructType): Boolean

    takes a Dataset and checks to see if all the required annotation types are present.

    takes a Dataset and checks to see if all the required annotation types are present.

    schema

    to be validated

    returns

    True if all the required types are present, else false

    Attributes
    protected
    Definition Classes
    AnnotatorApproach
  112. val vowelSwapLimit: IntParam

    Vowel swap attempts (Default: 6).

    Vowel swap attempts (Default: 6).

    Definition Classes
    NorvigSweetingParams
  113. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  114. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  115. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  116. val wordSizeIgnore: IntParam

    Minimum size of word before ignoring (Default: 3).

    Minimum size of word before ignoring (Default: 3). Minimum size of word before moving on.

    Definition Classes
    NorvigSweetingParams
  117. def write: MLWriter
    Definition Classes
    DefaultParamsWritable → MLWritable

Inherited from NorvigSweetingParams

Inherited from AnnotatorApproach[NorvigSweetingModel]

Inherited from CanBeLazy

Inherited from DefaultParamsWritable

Inherited from MLWritable

Inherited from HasOutputAnnotatorType

Inherited from HasOutputAnnotationCol

Inherited from HasInputAnnotationCols

Inherited from Estimator[NorvigSweetingModel]

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Parameters

A list of (hyper-)parameter keys this annotator can take. Users can set and get the parameter values through setters and getters, respectively.

Annotator types

Required input and expected output annotator types

Members

Parameter setters

Parameter getters