package benchmark
- Alphabetic
- Public
- All
Type Members
-
case class
AccuracyReport(task: BenchmarkTask, overall: Map[String, Double], perClass: Map[String, Map[String, Double]] = Map.empty, support: Long = 0L, scoredColumns: Seq[String] = Seq.empty) extends Product with Serializable
Result of Benchmark.evaluate.
Result of Benchmark.evaluate. The metric names in overall and perClass depend on the BenchmarkTask: label-accuracy tasks report
accuracy/weightedPrecision/weightedRecall/weightedF1(support-weighted across labels); NER and word segmentation report entity/segment-levelprecision/recall/f1(micro-averaged: pooled counts across every type, matching conlleval/seqeval, via exact span-boundary matching); dependency parsing reportsuas/las; the text-similarity tasks report their own metric (bleu,rouge1/rouge2/rougeL,wer, orexactMatch/f1).These averaging conventions differ by task and are NOT interchangeable:
overall("f1")from aClassificationreport and from aNERreport are computed differently and are not comparable to each other. They can also differ from a metric the same model already reported during its own training-time evaluation (which typically uses per-token tag accuracy for NER, or macro-averaged F1 for classification) -- that is expected, not a discrepancy to reconcile.- task
the task this report was scored against
- overall
corpus-level metric name -> value
- perClass
per-label/per-entity-type breakdown, metric name -> value; empty for tasks where a per-class breakdown doesn't apply (e.g. the text-similarity tasks)
- support
number of scored items (tokens, spans, or documents, depending on the task)
- scoredColumns
which of the pipeline's output columns this score was actually computed from
-
sealed abstract
class
BenchmarkTask extends AnyRef
The task a Benchmark.evaluate call is scoring against, selecting both the expected shape of
goldDataand which comparison engine computes the score. -
case class
MetricRate(annotatorType: String, outputColumn: String, totalItems: Long, meanItemsPerSecond: Double, confidenceInterval95: Double) extends Product with Serializable
Throughput of a single output annotator type, averaged over the timed trials.
Throughput of a single output annotator type, averaged over the timed trials.
- annotatorType
the
annotatorTypeof the output column this rate was computed from (e.g.token,named_entity)- outputColumn
the name of the output column
- totalItems
total annotations produced for this column across all timed trials
- meanItemsPerSecond
mean rate across trials
- confidenceInterval95
half-width of the 95% confidence interval around meanItemsPerSecond (normal approximation);
0.0when there is only one trial
-
case class
ThroughputReport(rates: Seq[MetricRate], trialSeconds: Seq[Double]) extends Product with Serializable
Result of Benchmark.throughput: one MetricRate per annotation type the pipeline produced, plus the raw per-trial elapsed times behind them.
Value Members
- object AccuracyReport extends Serializable
-
object
Benchmark
Throughput and accuracy benchmarking for any fitted Spark NLP PipelineModel, against data supplied by the caller.
Throughput and accuracy benchmarking for any fitted Spark NLP PipelineModel, against data supplied by the caller. See
examples/python/benchmarks/for end-to-end recipes with recommended public datasets per task.val report = Benchmark.throughput(pipelineModel, myData) println(report) val accuracy = Benchmark.evaluate(pipelineModel, myGoldData, BenchmarkTask.NER) println(accuracy)
- object BenchmarkTask