sparknlp.benchmark#

Classes#

Attributes#

SUPPORTED_TASKS[source]#

MAX_PRINTED_CLASSES = 50[source]#

class MetricRate(annotator_type, output_column, total_items, mean_items_per_second, confidence_interval_95)[source]#

Throughput of a single output annotator type, averaged over the timed trials.

annotator_type[source]#

output_column[source]#

total_items[source]#

mean_items_per_second[source]#

confidence_interval_95[source]#

class ThroughputReport(rates, trial_seconds)[source]#

Result of Benchmark.throughput(): one MetricRate per annotation type the pipeline produced, plus the raw per-trial elapsed times behind them.

rates[source]#

trial_seconds[source]#

class AccuracyReport(task, overall, per_class=None, support=0, scored_columns=None)[source]#

Result of Benchmark.evaluate(). The metric names in overall/per_class depend on the task: label-accuracy tasks report accuracy/weightedPrecision/weightedRecall/ weightedF1 (support-weighted across labels); NER and word segmentation report entity/ segment-level precision/recall/f1 (micro-averaged: pooled counts across every type, matching conlleval/seqeval, via exact span-boundary matching); dependency parsing reports uas/las; the text-similarity tasks report their own metric (bleu, rouge1/rouge2/rougeL, wer, or exact_match/f1).

These averaging conventions differ by task and are NOT interchangeable: overall["f1"] from a classification report and from a NER report are computed differently and are not comparable to each other. They can also differ from a metric the same model already reported during its own training-time evaluation (which typically uses per-token tag accuracy for NER, or macro-averaged F1 for classification) – that is expected, not a discrepancy to reconcile.

scored_columns names which of the pipeline’s output columns the score was computed from.

task[source]#

overall[source]#

per_class[source]#

support = 0[source]#

scored_columns = [][source]#

class Benchmark[source]#

Throughput and accuracy benchmarking for any fitted PipelineModel, against data supplied by the caller.

Examples

>>> from sparknlp.benchmark import Benchmark
>>> report = Benchmark.throughput(pipeline_model, my_data)
>>> print(report)
>>> accuracy = Benchmark.evaluate(pipeline_model, my_gold_data, task="ner")
>>> print(accuracy)

static throughput(pipeline_model, data, text_col='text', warmup_runs=1, trials=5)[source]#

Measures how fast pipeline_model processes data, reporting one rate per type of annotation it produces. Runs warmup_runs untimed passes first, then trials timed passes, reporting the mean rate per metric with a 95% confidence interval.

data is read once per pass (1 + warmup_runs + trials times in total), so if it isn’t already cached this persists it for the duration of the call (and unpersists it again afterwards) – otherwise a data sourced from an expensive or non-deterministic upstream read (a file scan, a .sample()) would have that cost repeated on every pass, contaminating the measured rate and, for a non-deterministic source, varying the row count between trials.

text_col is not read here – the pipeline’s own first stage decides which column it consumes.

static evaluate(pipeline_model, gold_data, task='ner', text_col='text', label_col='label', predicted_col=None, predicted_labeled_dependency_col=None, top_k=5)[source]#

Scores pipeline_model’s predictions on gold_data against task’s accuracy metric.

The expected shape of gold_data depends on task:

• ner, pos, wordsegmentation: text_col (raw text) + label_col (array<string> of gold tags/segment boundaries, aligned by position to the pipeline’s own tokenization – for wordsegmentation each entry is a "begin:end" character-offset string, inclusive on both ends)

• classification, spellcheck, languagedetection: text_col + label_col (a single gold label string per row)

• imageclassification: text_col (the pipeline’s image input column, see note below) + label_col (single gold class label per row). Reports accuracy for the model’s top-1 label like every other task, plus an extra top{top_k}Accuracy metric.

• dependencyparsing: text_col + label_col (array<string> of "headIndex:label" per token)

• questionanswering: text_col + label_col, either a single reference answer per row or an array<string> of every acceptable answer for that question, scored against its best-matching reference like the official SQuAD eval script.

• speechrecognition, translation, summarization: text_col + label_col (single reference text per row)

text_col is not read by evaluate – the pipeline’s own first stage decides which column it consumes, so a mismatched text_col goes undetected.

speechrecognition’s word error rate is case-sensitive, matching jiwer’s default.

summarization’s ROUGE never stems (matches rouge_score’s use_stemmer=False default) – published ROUGE numbers for benchmarks like XSum/ CNN-DailyMail typically use Porter stemming, so a real model’s score here will read lower than those published figures on stem-sensitive text for reasons that have nothing to do with the model.

predicted_col picks which of the pipeline’s output columns to score. Optional: by default evaluate takes the last column of the type this task expects. Pass this when a pipeline has several stages sharing that type and the default picks the wrong one. Reported back on AccuracyReport.scored_columns. For dependencyparsing this overrides only the dependency-typed column; use predicted_labeled_dependency_col for the labeled_dependency-typed one.

top_k sizes the top-k window for imageclassification’s extra top{top_k}Accuracy metric; every other task ignores it.