class PerceptronModel extends AnnotatorModel[PerceptronModel] with HasSimpleAnnotate[PerceptronModel] with PerceptronPredictionUtils
Averaged Perceptron model to tag words part-of-speech. Sets a POS tag to each word within a sentence.
This is the instantiated model of the PerceptronApproach. For training your own model, please see the documentation of that class.
Pretrained models can be loaded with pretrained
of the companion object:
val posTagger = PerceptronModel.pretrained() .setInputCols("document", "token") .setOutputCol("pos")
The default model is "pos_anc"
, if no name is provided.
For available pretrained models please see the Models Hub. Additionally, pretrained pipelines are available for this module, see Pipelines.
For extended examples of usage, see the Examples.
Example
import spark.implicits._ import com.johnsnowlabs.nlp.base.DocumentAssembler import com.johnsnowlabs.nlp.annotators.Tokenizer import com.johnsnowlabs.nlp.annotators.pos.perceptron.PerceptronModel import org.apache.spark.ml.Pipeline val documentAssembler = new DocumentAssembler() .setInputCol("text") .setOutputCol("document") val tokenizer = new Tokenizer() .setInputCols("document") .setOutputCol("token") val posTagger = PerceptronModel.pretrained() .setInputCols("document", "token") .setOutputCol("pos") val pipeline = new Pipeline().setStages(Array( documentAssembler, tokenizer, posTagger )) val data = Seq("Peter Pipers employees are picking pecks of pickled peppers").toDF("text") val result = pipeline.fit(data).transform(data) result.selectExpr("explode(pos) as pos").show(false) +-------------------------------------------+ |pos | +-------------------------------------------+ |[pos, 0, 4, NNP, [word -> Peter], []] | |[pos, 6, 11, NNP, [word -> Pipers], []] | |[pos, 13, 21, NNS, [word -> employees], []]| |[pos, 23, 25, VBP, [word -> are], []] | |[pos, 27, 33, VBG, [word -> picking], []] | |[pos, 35, 39, NNS, [word -> pecks], []] | |[pos, 41, 42, IN, [word -> of], []] | |[pos, 44, 50, JJ, [word -> pickled], []] | |[pos, 52, 58, NNS, [word -> peppers], []] | +-------------------------------------------+
- Grouped
- Alphabetic
- By Inheritance
- PerceptronModel
- PerceptronPredictionUtils
- PerceptronUtils
- HasSimpleAnnotate
- AnnotatorModel
- CanBeLazy
- RawAnnotator
- HasOutputAnnotationCol
- HasInputAnnotationCols
- HasOutputAnnotatorType
- ParamsAndFeaturesWritable
- HasFeatures
- DefaultParamsWritable
- MLWritable
- Model
- Transformer
- PipelineStage
- Logging
- Params
- Serializable
- Serializable
- Identifiable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Parameters
A list of (hyper-)parameter keys this annotator can take. Users can set and get the parameter values through setters and getters, respectively.
-
val
model: StructFeature[AveragedPerceptron]
POS model
Annotator types
Required input and expected output annotator types
-
val
inputAnnotatorTypes: Array[AnnotatorType]
Input annotator types : TOKEN, DOCUMENT
Input annotator types : TOKEN, DOCUMENT
- Definition Classes
- PerceptronModel → HasInputAnnotationCols
-
val
outputAnnotatorType: AnnotatorType
Output annotator types : POS
Output annotator types : POS
- Definition Classes
- PerceptronModel → HasOutputAnnotatorType
Members
-
type
AnnotatorType = String
- Definition Classes
- HasOutputAnnotatorType
-
def
annotate(annotations: Seq[Annotation]): Seq[Annotation]
One to one annotation standing from the Tokens perspective, to give each word a corresponding Tag
One to one annotation standing from the Tokens perspective, to give each word a corresponding Tag
- annotations
Annotations that correspond to inputAnnotationCols generated by previous annotators if any
- returns
any number of annotations processed for every input annotation. Not necessary one to one relationship
- Definition Classes
- PerceptronModel → HasSimpleAnnotate
-
final
def
clear(param: Param[_]): PerceptronModel.this.type
- Definition Classes
- Params
-
def
copy(extra: ParamMap): PerceptronModel
requirement for annotators copies
requirement for annotators copies
- Definition Classes
- RawAnnotator → Model → Transformer → PipelineStage → Params
-
def
dfAnnotate: UserDefinedFunction
Wraps annotate to happen inside SparkSQL user defined functions in order to act with org.apache.spark.sql.Column
Wraps annotate to happen inside SparkSQL user defined functions in order to act with org.apache.spark.sql.Column
- returns
udf function to be applied to inputCols using this annotator's annotate function as part of ML transformation
- Definition Classes
- HasSimpleAnnotate
-
def
explainParam(param: Param[_]): String
- Definition Classes
- Params
-
def
explainParams(): String
- Definition Classes
- Params
-
final
def
extractParamMap(): ParamMap
- Definition Classes
- Params
-
final
def
extractParamMap(extra: ParamMap): ParamMap
- Definition Classes
- Params
-
val
features: ArrayBuffer[Feature[_, _, _]]
- Definition Classes
- HasFeatures
-
final
def
get[T](param: Param[T]): Option[T]
- Definition Classes
- Params
-
final
def
getDefault[T](param: Param[T]): Option[T]
- Definition Classes
- Params
-
def
getInputCols: Array[String]
- returns
input annotations columns currently used
- Definition Classes
- HasInputAnnotationCols
-
def
getLazyAnnotator: Boolean
- Definition Classes
- CanBeLazy
-
final
def
getOrDefault[T](param: Param[T]): T
- Definition Classes
- Params
-
final
def
getOutputCol: String
Gets annotation column name going to generate
Gets annotation column name going to generate
- Definition Classes
- HasOutputAnnotationCol
-
def
getParam(paramName: String): Param[Any]
- Definition Classes
- Params
-
final
def
hasDefault[T](param: Param[T]): Boolean
- Definition Classes
- Params
-
def
hasParam(paramName: String): Boolean
- Definition Classes
- Params
-
def
hasParent: Boolean
- Definition Classes
- Model
-
final
def
isDefined(param: Param[_]): Boolean
- Definition Classes
- Params
-
final
def
isSet(param: Param[_]): Boolean
- Definition Classes
- Params
-
val
lazyAnnotator: BooleanParam
- Definition Classes
- CanBeLazy
-
val
optionalInputAnnotatorTypes: Array[String]
- Definition Classes
- HasInputAnnotationCols
-
lazy val
params: Array[Param[_]]
- Definition Classes
- Params
-
var
parent: Estimator[PerceptronModel]
- Definition Classes
- Model
-
def
save(path: String): Unit
- Definition Classes
- MLWritable
- Annotations
- @Since( "1.6.0" ) @throws( ... )
-
final
def
set[T](param: Param[T], value: T): PerceptronModel.this.type
- Definition Classes
- Params
-
final
def
setInputCols(value: String*): PerceptronModel.this.type
- Definition Classes
- HasInputAnnotationCols
-
def
setInputCols(value: Array[String]): PerceptronModel.this.type
Overrides required annotators column if different than default
Overrides required annotators column if different than default
- Definition Classes
- HasInputAnnotationCols
-
def
setLazyAnnotator(value: Boolean): PerceptronModel.this.type
- Definition Classes
- CanBeLazy
-
final
def
setOutputCol(value: String): PerceptronModel.this.type
Overrides annotation column name when transforming
Overrides annotation column name when transforming
- Definition Classes
- HasOutputAnnotationCol
-
def
setParent(parent: Estimator[PerceptronModel]): PerceptronModel
- Definition Classes
- Model
-
def
tag(model: AveragedPerceptron, tokenizedSentences: Array[TokenizedSentence]): Array[TaggedSentence]
Tags a group of sentences into POS tagged sentences The logic here is to create a sentence context, run through every word and evaluate its context Based on how frequent a context appears around a word, such context is given a score which is used to predict Some words are marked as non ambiguous from the beginning
Tags a group of sentences into POS tagged sentences The logic here is to create a sentence context, run through every word and evaluate its context Based on how frequent a context appears around a word, such context is given a score which is used to predict Some words are marked as non ambiguous from the beginning
- tokenizedSentences
Sentence in the form of single word tokens
- returns
A list of sentences which have every word tagged
- Definition Classes
- PerceptronPredictionUtils
-
def
toString(): String
- Definition Classes
- Identifiable → AnyRef → Any
-
final
def
transform(dataset: Dataset[_]): DataFrame
Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content
Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content
- dataset
Dataset[Row]
- Definition Classes
- AnnotatorModel → Transformer
-
def
transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame
- Definition Classes
- Transformer
- Annotations
- @Since( "2.0.0" )
-
def
transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame
- Definition Classes
- Transformer
- Annotations
- @Since( "2.0.0" ) @varargs()
-
final
def
transformSchema(schema: StructType): StructType
requirement for pipeline transformation validation.
requirement for pipeline transformation validation. It is called on fit()
- Definition Classes
- RawAnnotator → PipelineStage
-
val
uid: String
- Definition Classes
- PerceptronModel → Identifiable
-
def
write: MLWriter
- Definition Classes
- ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable
Parameter setters
- def setModel(targetModel: AveragedPerceptron): PerceptronModel.this.type
Parameter getters
- def getModel: AveragedPerceptron