object Benchmark
Throughput and accuracy benchmarking for any fitted Spark NLP PipelineModel, against data
supplied by the caller. See examples/python/benchmarks/ for end-to-end recipes with
recommended public datasets per task.
val report = Benchmark.throughput(pipelineModel, myData) println(report) val accuracy = Benchmark.evaluate(pipelineModel, myGoldData, BenchmarkTask.NER) println(accuracy)
- Alphabetic
- By Inheritance
- Benchmark
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
evaluate(pipelineModel: PipelineModel, goldData: DataFrame, task: BenchmarkTask, textCol: String = "text", labelCol: String = "label", predictedCol: Option[String] = None, predictedLabeledDependencyCol: Option[String] = None, topK: Int = 5): AccuracyReport
Scores
pipelineModel's predictions ongoldDataagainsttask's accuracy metric.Scores
pipelineModel's predictions ongoldDataagainsttask's accuracy metric.The expected shape of
goldDatadepends ontask:NER,POS,WordSegmentation:textCol(raw text) +labelCol(array<string>of gold tags/segment boundaries, aligned by position to the pipeline's own tokenization — forWordSegmentationeach entry is a"begin:end"character-offset string, inclusive on both ends (e.g. a 2-character token starting at 0 is"0:1")Classification,SpellCheck,LanguageDetection:textCol+labelCol(a single gold label string per row)ImageClassification:textCol(the pipeline's image input column, see@parambelow) +labelCol(single gold class label per row). Reportsaccuracyfor the model's top-1 label like every other task, plus an extratop${topK}Accuracymetric.DependencyParsing:textCol+labelCol(array<string>of"headIndex:label"per token)QuestionAnswering:textCol+labelCol, either a single reference answer per row or anarray<string>of every acceptable answer for that question — with an array, each prediction is scored against its best-matching reference, matching the official SQuAD eval script.SpeechRecognition,Translation,Summarization:textCol+labelCol(single reference text per row: expected transcript/translation/summary)
SpeechRecognition's word error rate is case-sensitive, matching jiwer's default.Summarization's ROUGE never stems (matches rouge_score'suse_stemmer=Falsedefault) -- published ROUGE numbers for benchmarks like XSum/CNN-DailyMail typically use Porter stemming, so a real model's score here will read lower than those published figures on stem-sensitive text for reasons that have nothing to do with the model.- textCol
present for documentation of the expected input column; the pipeline's own stages determine which columns are actually read
- predictedCol
which of the pipeline's output columns to score. Optional: by default
evaluatetakes the last column of the type this task expects. Pass this when a pipeline has several stages sharing that type and the default picks the wrong one. Reported back on AccuracyReport.scoredColumns. ForDependencyParsingthis overrides only thedependency-typed column; usepredictedLabeledDependencyColfor thelabeled_dependency-typed one.- predictedLabeledDependencyCol
override for the
labeled_dependency-typed output column, used only byDependencyParsing.- topK
size of the top-k window for
ImageClassification's extratop${topK}Accuracymetric; ignored by every other task.
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
throughput(pipelineModel: PipelineModel, data: DataFrame, textCol: String = "text", warmupRuns: Int = 1, trials: Int = 5): ThroughputReport
Measures how fast
pipelineModelprocessesdata, reporting one rate per type of annotation it produces (tokens/sec, sentences/sec, NER-tokens/sec, etc.Measures how fast
pipelineModelprocessesdata, reporting one rate per type of annotation it produces (tokens/sec, sentences/sec, NER-tokens/sec, etc. — inferred from the pipeline's own output, not configured by the caller).Runs
warmupRunsuntimed passes first (JIT/class-loading/Spark planning overhead), thentrialstimed passes, reporting the mean rate per metric with a 95% confidence interval.datais read once per pass (1 +warmupRuns+trialstimes in total), so if it isn't already cached this persists it for the duration of the call (and unpersists it again afterwards) -- otherwise adatasourced from an expensive or non-deterministic upstream read (a file scan, a.sample()) would have that cost repeated on every pass, contaminating the measured rate and, for a non-deterministic source, varying the row count between trials.- data
input data in whatever shape
pipelineModelexpects (atextcolumn by default)- textCol
present for documentation of the expected input column; the pipeline's own stages determine which columns are actually read
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()