Packages

object Benchmark

Throughput and accuracy benchmarking for any fitted Spark NLP PipelineModel, against data supplied by the caller. See examples/python/benchmarks/ for end-to-end recipes with recommended public datasets per task.

val report = Benchmark.throughput(pipelineModel, myData)
println(report)

val accuracy = Benchmark.evaluate(pipelineModel, myGoldData, BenchmarkTask.NER)
println(accuracy)
Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Benchmark
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def evaluate(pipelineModel: PipelineModel, goldData: DataFrame, task: BenchmarkTask, textCol: String = "text", labelCol: String = "label", predictedCol: Option[String] = None, predictedLabeledDependencyCol: Option[String] = None, topK: Int = 5): AccuracyReport

    Scores pipelineModel's predictions on goldData against task's accuracy metric.

    Scores pipelineModel's predictions on goldData against task's accuracy metric.

    The expected shape of goldData depends on task:

    • NER, POS, WordSegmentation: textCol (raw text) + labelCol (array<string> of gold tags/segment boundaries, aligned by position to the pipeline's own tokenization — for WordSegmentation each entry is a "begin:end" character-offset string, inclusive on both ends (e.g. a 2-character token starting at 0 is "0:1")
    • Classification, SpellCheck, LanguageDetection: textCol + labelCol (a single gold label string per row)
    • ImageClassification: textCol (the pipeline's image input column, see @param below) + labelCol (single gold class label per row). Reports accuracy for the model's top-1 label like every other task, plus an extra top${topK}Accuracy metric.
    • DependencyParsing: textCol + labelCol (array<string> of "headIndex:label" per token)
    • QuestionAnswering: textCol + labelCol, either a single reference answer per row or an array<string> of every acceptable answer for that question — with an array, each prediction is scored against its best-matching reference, matching the official SQuAD eval script.
    • SpeechRecognition, Translation, Summarization: textCol + labelCol (single reference text per row: expected transcript/translation/summary)

    SpeechRecognition's word error rate is case-sensitive, matching jiwer's default.

    Summarization's ROUGE never stems (matches rouge_score's use_stemmer=False default) -- published ROUGE numbers for benchmarks like XSum/CNN-DailyMail typically use Porter stemming, so a real model's score here will read lower than those published figures on stem-sensitive text for reasons that have nothing to do with the model.

    textCol

    present for documentation of the expected input column; the pipeline's own stages determine which columns are actually read

    predictedCol

    which of the pipeline's output columns to score. Optional: by default evaluate takes the last column of the type this task expects. Pass this when a pipeline has several stages sharing that type and the default picks the wrong one. Reported back on AccuracyReport.scoredColumns. For DependencyParsing this overrides only the dependency-typed column; use predictedLabeledDependencyCol for the labeled_dependency-typed one.

    predictedLabeledDependencyCol

    override for the labeled_dependency-typed output column, used only by DependencyParsing.

    topK

    size of the top-k window for ImageClassification's extra top${topK}Accuracy metric; ignored by every other task.

  9. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  13. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  14. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  17. def throughput(pipelineModel: PipelineModel, data: DataFrame, textCol: String = "text", warmupRuns: Int = 1, trials: Int = 5): ThroughputReport

    Measures how fast pipelineModel processes data, reporting one rate per type of annotation it produces (tokens/sec, sentences/sec, NER-tokens/sec, etc.

    Measures how fast pipelineModel processes data, reporting one rate per type of annotation it produces (tokens/sec, sentences/sec, NER-tokens/sec, etc. — inferred from the pipeline's own output, not configured by the caller).

    Runs warmupRuns untimed passes first (JIT/class-loading/Spark planning overhead), then trials timed passes, reporting the mean rate per metric with a 95% confidence interval. data is read once per pass (1 + warmupRuns + trials times in total), so if it isn't already cached this persists it for the duration of the call (and unpersists it again afterwards) -- otherwise a data sourced from an expensive or non-deterministic upstream read (a file scan, a .sample()) would have that cost repeated on every pass, contaminating the measured rate and, for a non-deterministic source, varying the row count between trials.

    data

    input data in whatever shape pipelineModel expects (a text column by default)

    textCol

    present for documentation of the expected input column; the pipeline's own stages determine which columns are actually read

  18. def toString(): String
    Definition Classes
    AnyRef → Any
  19. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  20. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  21. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped