Packages

package benchmark

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. All

Type Members

  1. case class AccuracyReport(task: BenchmarkTask, overall: Map[String, Double], perClass: Map[String, Map[String, Double]] = Map.empty, support: Long = 0L, scoredColumns: Seq[String] = Seq.empty) extends Product with Serializable

    Result of Benchmark.evaluate.

    Result of Benchmark.evaluate. The metric names in overall and perClass depend on the BenchmarkTask: label-accuracy tasks report accuracy/weightedPrecision/ weightedRecall/weightedF1 (support-weighted across labels); NER and word segmentation report entity/segment-level precision/recall/f1 (micro-averaged: pooled counts across every type, matching conlleval/seqeval, via exact span-boundary matching); dependency parsing reports uas/las; the text-similarity tasks report their own metric (bleu, rouge1/rouge2/rougeL, wer, or exactMatch/f1).

    These averaging conventions differ by task and are NOT interchangeable: overall("f1") from a Classification report and from a NER report are computed differently and are not comparable to each other. They can also differ from a metric the same model already reported during its own training-time evaluation (which typically uses per-token tag accuracy for NER, or macro-averaged F1 for classification) -- that is expected, not a discrepancy to reconcile.

    task

    the task this report was scored against

    overall

    corpus-level metric name -> value

    perClass

    per-label/per-entity-type breakdown, metric name -> value; empty for tasks where a per-class breakdown doesn't apply (e.g. the text-similarity tasks)

    support

    number of scored items (tokens, spans, or documents, depending on the task)

    scoredColumns

    which of the pipeline's output columns this score was actually computed from

  2. sealed abstract class BenchmarkTask extends AnyRef

    The task a Benchmark.evaluate call is scoring against, selecting both the expected shape of goldData and which comparison engine computes the score.

  3. case class MetricRate(annotatorType: String, outputColumn: String, totalItems: Long, meanItemsPerSecond: Double, confidenceInterval95: Double) extends Product with Serializable

    Throughput of a single output annotator type, averaged over the timed trials.

    Throughput of a single output annotator type, averaged over the timed trials.

    annotatorType

    the annotatorType of the output column this rate was computed from (e.g. token, named_entity)

    outputColumn

    the name of the output column

    totalItems

    total annotations produced for this column across all timed trials

    meanItemsPerSecond

    mean rate across trials

    confidenceInterval95

    half-width of the 95% confidence interval around meanItemsPerSecond (normal approximation); 0.0 when there is only one trial

  4. case class ThroughputReport(rates: Seq[MetricRate], trialSeconds: Seq[Double]) extends Product with Serializable

    Result of Benchmark.throughput: one MetricRate per annotation type the pipeline produced, plus the raw per-trial elapsed times behind them.

Value Members

  1. object AccuracyReport extends Serializable
  2. object Benchmark

    Throughput and accuracy benchmarking for any fitted Spark NLP PipelineModel, against data supplied by the caller.

    Throughput and accuracy benchmarking for any fitted Spark NLP PipelineModel, against data supplied by the caller. See examples/python/benchmarks/ for end-to-end recipes with recommended public datasets per task.

    val report = Benchmark.throughput(pipelineModel, myData)
    println(report)
    
    val accuracy = Benchmark.evaluate(pipelineModel, myGoldData, BenchmarkTask.NER)
    println(accuracy)
  3. object BenchmarkTask

Ungrouped