sparknlp.annotator.cv.dolphin_for_document_parsing#

Classes#

class DolphinForDocumentParsing(classname='com.johnsnowlabs.nlp.annotators.cv.DolphinForDocumentParsing', java_model=None)[source]#

Document image parsing with ByteDance Dolphin.

Dolphin follows an analyze-then-parse design: stage 1 generates the page’s layout elements in reading order, then each element is cropped and parsed independently in stage 2, batched by type. Tables come back as HTML, formulas as LaTeX, text as markdown.

Dolphin 1.5 and 1.0 load the same way. The release is recognised from its stage-1 output, which decides the box geometry and whether formulas and code get their own prompts (1.5 only).

Pretrained models can be loaded with pretrained() of the companion object:

>>> dolphin = DolphinForDocumentParsing.pretrained() \
...     .setInputCols(["image_assembler"]) \
...     .setOutputCol("elements")

The default model is "dolphin_1_5".

Input Annotation types	Output Annotation type
IMAGE	DOCUMENT

Parameters:

parsingMode

What the input image is. page (default) runs both stages on a whole page, layout runs stage 1 only, and table, text, formula or code run stage 2 on an already cropped region. formula and code are Dolphin 1.5 prompts.

outputFormat

elements (default) emits one annotation per document element carrying dolphinLabel, elementType, readingOrder, readingGroup and bboxNormalized metadata, plus bbox in pixels in page and layout modes. bboxNormalized is that box as fractions of the image’s width and height. markdown emits a single assembled document per page, with elementCount and layoutRaw. An image that fails to parse yields one empty annotation with an exception key.

elementBatchSize

Element crops decoded per forward pass, default 4.

maxOutputLength

Maximum tokens generated per element, default 4096. Generation always stops at the decoder’s 4096-position window, prompt included.

layoutMaxOutputLength

Maximum tokens generated for the layout pass, default 4096.

repetitionPenalty

Repetition penalty, default 1.0 (off) as in Dolphin 1.5’s reference pipeline; Dolphin 1.0’s used 1.1.

customPrompt

Overrides the prompt in the table, text, formula and code modes.

adjustBoxEdges

Dolphin 1.0 only: widen its boxes so their edges do not cut through ink, default True. 1.5 boxes are used as generated.

minElementSize

Page mode drops elements whose crop width or height is this many pixels or fewer, default 3, as the reference does.

Notes

The task comes from parsingMode, not from the image annotation’s text field. Reader2Image defaults its promptTemplate to qwen2vl-chat, and that markup would otherwise be fed to Dolphin as a task prompt.

Examples

>>> import sparknlp
>>> from sparknlp.base import *
>>> from sparknlp.annotator import *
>>> from pyspark.ml import Pipeline
>>> imageDF = spark.read.format("image").load(path="pages/")
>>> imageAssembler = ImageAssembler() \
...     .setInputCol("image") \
...     .setOutputCol("image_assembler")
>>> dolphin = DolphinForDocumentParsing.pretrained() \
...     .setInputCols("image_assembler") \
...     .setOutputCol("elements") \
...     .setParsingMode("page")
>>> pipeline = Pipeline().setStages([imageAssembler, dolphin])
>>> result = pipeline.fit(imageDF).transform(imageDF)
>>> result.selectExpr("explode(elements) as e") \
...     .selectExpr("e.metadata['dolphinLabel'] as label", "e.result") \
...     .show(truncate=60)

name = 'DolphinForDocumentParsing'[source]#

inputAnnotatorTypes[source]#

outputAnnotatorType = 'document'[source]#

parsingMode[source]#

outputFormat[source]#

elementBatchSize[source]#

maxOutputLength[source]#

layoutMaxOutputLength[source]#

repetitionPenalty[source]#

customPrompt[source]#

adjustBoxEdges[source]#

minElementSize[source]#

PARSING_MODES = ['page', 'layout', 'table', 'text', 'formula', 'code'][source]#

OUTPUT_FORMATS = ['elements', 'markdown'][source]#

setParsingMode(value)[source]#

Sets the parsing mode: page, layout, table, text, formula or code.

setOutputFormat(value)[source]#

Sets the output format: elements or markdown.

setElementBatchSize(value)[source]#

Sets how many element crops are decoded per forward pass.

setMaxOutputLength(value)[source]#

Sets the maximum tokens generated per element.

setLayoutMaxOutputLength(value)[source]#

Sets the maximum tokens generated for the layout pass.

setRepetitionPenalty(value)[source]#

Sets the repetition penalty.

setCustomPrompt(value)[source]#

Sets a prompt that overrides the default in the single-crop modes.

setAdjustBoxEdges(value)[source]#

Sets whether Dolphin 1.0 boxes are widened away from ink.

setMinElementSize(value)[source]#

Sets the crop size at or below which page mode drops an element.

getParsingMode()[source]#

Gets the parsing mode.

getOutputFormat()[source]#

Gets the output format: elements or markdown.

getElementBatchSize()[source]#

Gets how many element crops are decoded per forward pass.

getMaxOutputLength()[source]#

Gets the maximum tokens generated per element.

getLayoutMaxOutputLength()[source]#

Gets the maximum tokens generated for the layout pass.

getRepetitionPenalty()[source]#

Gets the repetition penalty.

getCustomPrompt()[source]#

Gets the prompt overriding the default in the single-crop modes.

getAdjustBoxEdges()[source]#

Gets whether Dolphin 1.0 boxes are widened away from ink.

getMinElementSize()[source]#

Gets the crop size at or below which page mode drops an element.

static loadSavedModel(folder, spark_session)[source]#

Loads a locally saved model.

Parameters:

folderstr

Folder of the saved model

spark_sessionpyspark.sql.SparkSession

The current SparkSession

Returns:

DolphinForDocumentParsing

The restored model

static pretrained(name='dolphin_1_5', lang='en', remote_loc=None)[source]#

Downloads and loads a pretrained model.

Parameters:

namestr, optional

Name of the pretrained model, by default “dolphin_1_5”

langstr, optional

Language of the pretrained model, by default “en”

remote_locstr, optional

Optional remote address of the resource, by default None. Will use Spark NLPs repositories otherwise.

Returns:

DolphinForDocumentParsing

The restored model