sparknlp.annotator.seq2seq.document_translator#

Classes#

class DocumentTranslator(classname='com.johnsnowlabs.nlp.annotators.seq2seq.DocumentTranslator', java_model=None)[source]#

Reads documents from any supported file type and translates them with a llama.cpp GGUF large-language-model, all in a single Pipeline stage.

Internally it reads the files (PDF, Word, HTML, plain-text, etc.), splits each document into length-bounded sentences with a SentenceDetectorSaTModel, translates every sentence with the GGUF model and merges the translations back into one DOCUMENT annotation per file.

All llama.cpp model and inference parameters are available (see AutoGGUFModel), e.g. setNCtx, setNPredict, setNGpuLayers, setTemperature, setSystemPrompt.

Pretrained models can be loaded with pretrained() of the companion object:

>>> translator = DocumentTranslator.pretrained() \
...     .setContentPath("src/test/resources/reader/html/") \
...     .setContentType("text/html") \
...     .setSrcLang("English") \
...     .setTgtLang("French") \
...     .setOutputCol("translation")

The default model is "qwen3_4b_q8_0_gguf", default language is "en".

Input Annotation types	Output Annotation type
NONE	DOCUMENT

Parameters:

contentPath

Path to the file or directory to read documents from

contentType

MIME content-type hint forwarded to the reader (empty = auto-detect from file extension)

inputCol

DataFrame column holding raw text to parse instead of reading from contentPath

outputAsDocument

Whether to merge all extracted elements into a single DOCUMENT annotation per file

joinString

String used to join extracted elements when outputAsDocument is true

minSentenceLength

Minimum sentence length in characters for the SaT sentence detector (0 = unset)

maxSentenceLength

Maximum sentence length in characters for the SaT sentence detector (0 = unset)

sentenceThreshold

Boundary probability threshold for the SaT sentence detector

srcLang

Source language used to build the translation prompt

tgtLang

Target language used to build the translation prompt

promptTemplate

Per-sentence translation prompt template; {srcLang}, {tgtLang} and {text} are interpolated

batchSize

Number of sentences translated concurrently (llama.cpp parallel decoding slots)

Notes

Translation is computationally expensive; a GPU is recommended. The total context nCtx is split across the batchSize slots, so nCtx / batchSize must cover one sentence’s prompt plus nPredict. Raise setNCtx when raising setBatchSize, setMaxSentenceLength or setNPredict.

Examples

>>> import sparknlp
>>> from sparknlp.base import *
>>> from sparknlp.annotator import *
>>> from pyspark.ml import Pipeline
>>> translator = DocumentTranslator.pretrained() \
...     .setContentType("text/html") \
...     .setContentPath("src/test/resources/reader/html/fake-html.html") \
...     .setMaxSentenceLength(250) \
...     .setSrcLang("English") \
...     .setTgtLang("French") \
...     .setOutputCol("translation")
>>> pipeline = Pipeline().setStages([translator])
>>> data = spark.createDataFrame([[""]]).toDF("text")
>>> result = pipeline.fit(data).transform(data)
>>> result.select("translation.result").show(truncate=False)

name = 'DocumentTranslator'[source]#

outputAnnotatorType = 'document'[source]#

contentPath[source]#

contentType[source]#

inputCol[source]#

outputAsDocument[source]#

joinString[source]#

minSentenceLength[source]#

maxSentenceLength[source]#

sentenceThreshold[source]#

srcLang[source]#

tgtLang[source]#

promptTemplate[source]#

batchSize[source]#

setContentPath(value)[source]#

Sets the path to the file or directory to read documents from.

setContentType(value)[source]#

Sets the MIME content-type hint forwarded to the reader.

setInputCol(value)[source]#

Sets the DataFrame column holding raw text to parse instead of reading from contentPath.

setOutputAsDocument(value)[source]#

Sets whether to merge all extracted elements into a single DOCUMENT annotation per file.

setJoinString(value)[source]#

Sets the string used to join extracted elements when outputAsDocument is true.

setMinSentenceLength(value)[source]#

Sets the minimum sentence length in characters for the SaT sentence detector.

setMaxSentenceLength(value)[source]#

Sets the maximum sentence length in characters for the SaT sentence detector.

setSentenceThreshold(value)[source]#

Sets the boundary probability threshold for the SaT sentence detector.

setSrcLang(value)[source]#

Sets the source language used to build the translation prompt.

setTgtLang(value)[source]#

Sets the target language used to build the translation prompt.

setPromptTemplate(value)[source]#

Sets the per-sentence translation prompt template.

setBatchSize(value)[source]#

Sets the number of sentences translated concurrently (llama.cpp parallel decoding slots).

setNParallel(value)[source]#

Alias for setBatchSize() (number of llama.cpp parallel decoding slots).

static loadSavedModel(path, spark_session)[source]#

Loads a locally saved GGUF model.

Internally this loads an AutoGGUFModel from the given path and wraps it, since the translator is backed by an AutoGGUF llama.cpp model.

Parameters:

pathstr

Path to the gguf model

spark_sessionpyspark.sql.SparkSession

The current SparkSession

Returns:

DocumentTranslator

The restored model

static pretrained(name='qwen3_4b_q8_0_gguf', lang='en', remote_loc=None)[source]#

Downloads and loads a pretrained GGUF model.

Internally this downloads an AutoGGUFModel and wraps it, since the translator is backed by an AutoGGUF llama.cpp model.

Parameters:

namestr, optional

Name of the pretrained model, by default “qwen3_4b_q8_0_gguf”

langstr, optional

Language of the pretrained model, by default “en”

remote_locstr, optional

Optional remote address of the resource, by default None. Will use Spark NLPs repositories otherwise.

Returns:

DocumentTranslator

The restored model

close()[source]#

Closes the llama.cpp model backend freeing resources. The model is reloaded when used again.