Spark NLP - Advanced Settings

SparkNLP Properties

You can change the following Spark NLP configurations via Spark Configuration:

ONNX CUDA native-dependency recovery

Spark NLP first registers the ONNX Runtime CUDA provider normally. If registration succeeds, it does not inspect preload configuration, search the filesystem, call System.load, or retry. If registration fails because a required CUDA shared library cannot be loaded, Spark NLP can preload already-installed libraries inside the executor JVM and retry provider registration once with fresh provider and session-option objects.

• search (default): resolve the packaged provider’s dependency manifest from trusted operator directories, runtime and linker paths, and a bounded search under generic installation roots. Resolution fails on missing or ambiguous candidates, and every group- or world-writable library is rejected.
• explicit: validate the configured complete ordered list of absolute regular files and load those exact canonical files without searching. A group-writable library is accepted only when its owner and group exactly match the authenticated executor identity; use this exception only on a platform with a documented guarantee that the executor group is exclusive to the authenticated executor identity.
• off: preserve the legacy failure path without reading the paths setting, discovery, preload, or retry.

All enabled modes fail closed. Recovery never silently falls back to CPU, does not install or bundle CUDA, does not use Python discovery, and is independent from model warm-up. Native-library paths are runtime Spark configuration and are not persisted in models.

Before any native load, Spark NLP canonicalizes and validates the complete manifest. Each library must be a readable ELF64 shared object whose architecture and uniquely mapped DT_SONAME match the current runtime and required manifest entry. The library and every canonical ancestor must be owned by root or the authenticated POSIX executor identity. Library files must not be world-writable. Search mode rejects group-writable libraries; explicit mode permits group-write only for a file whose owner and group exactly match the authenticated executor identity. Every canonical ancestor remains non-group-writable and non-world-writable. Filesystems without POSIX ownership and permission attributes are rejected. Search symlinks must remain inside their approved canonical root, and directory symlinks are not traversed.

Filesystem work is capped across discovery tiers at 20,000 visited entries, including path-list entries and configured directory canonicalization attempts. The configured preload path-list value and the aggregate runtime-derived path-list text each have an independent 1 MiB cap before trimming or splitting. Generic installation-root and linker-configuration traversal has a maximum depth of 6. The aggregate linker-configuration input across the root file and all recursive includes is capped at 1 MiB. Fixed-path ldconfig -p output is separately capped at 1 MiB, and ldconfig must terminate within 2 seconds after normal completion or forced destruction. Exceeding any limit fails recovery instead of broadening the search.

Example explicit configuration on Linux:

--conf spark.jsl.settings.onnx.cuda.preload.mode=explicit \
--conf spark.jsl.settings.onnx.cuda.preload.paths=/absolute/cuda/libcudart.so.12:/absolute/cuda/libcublasLt.so.12:/absolute/cuda/libcublas.so.12:/absolute/cuda/libcurand.so.10:/absolute/cuda/libcufft.so.11:/absolute/cuda/libcudnn.so.9

The exact SONAME inventory is version-coupled to the ONNX Runtime GPU provider packaged by Spark NLP. Reconcile it against libonnxruntime_providers_cuda.so whenever the ONNX Runtime dependency changes.

How to set Spark NLP Configuration

SparkSession:

You can use .config() during SparkSession creation to set Spark NLP configurations.

from pyspark.sql import SparkSession

spark = SparkSession.builder
    .master("local[*]")
    .config("spark.driver.memory", "16G")
    .config("spark.driver.maxResultSize", "0")
    .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
    .config("spark.kryoserializer.buffer.max", "2000m")
    .config("spark.jsl.settings.pretrained.cache_folder", "sample_data/pretrained")
    .config("spark.jsl.settings.storage.cluster_tmp_dir", "sample_data/storage")
    .config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.12:7.0.0")
    # Apache Spark 4.0.0 (Scala 2.13): com.johnsnowlabs.nlp:spark-nlp-spark400_2.13:7.0.0
    # Apache Spark 4.0.1, 4.1.0, 4.1.1, 4.1.2 (Scala 2.13): com.johnsnowlabs.nlp:spark-nlp_2.13:7.0.0
    .getOrCreate()

spark-shell:

spark-shell \
  --driver-memory 16g \
  --conf spark.driver.maxResultSize=0 \
  --conf spark.serializer=org.apache.spark.serializer.KryoSerializer
  --conf spark.kryoserializer.buffer.max=2000M \
  --conf spark.jsl.settings.pretrained.cache_folder="sample_data/pretrained" \
  --conf spark.jsl.settings.storage.cluster_tmp_dir="sample_data/storage" \
  --packages com.johnsnowlabs.nlp:spark-nlp_2.12:7.0.0

On Apache Spark 4.0.0, use com.johnsnowlabs.nlp:spark-nlp-spark400_2.13:7.0.0; on Apache Spark 4.0.1, 4.1.0, 4.1.1, or 4.1.2, use com.johnsnowlabs.nlp:spark-nlp_2.13:7.0.0.

pyspark:

pyspark \
  --driver-memory 16g \
  --conf spark.driver.maxResultSize=0 \
  --conf spark.serializer=org.apache.spark.serializer.KryoSerializer
  --conf spark.kryoserializer.buffer.max=2000M \
  --conf spark.jsl.settings.pretrained.cache_folder="sample_data/pretrained" \
  --conf spark.jsl.settings.storage.cluster_tmp_dir="sample_data/storage" \
  --packages com.johnsnowlabs.nlp:spark-nlp_2.12:7.0.0

On Apache Spark 4.0.0, use com.johnsnowlabs.nlp:spark-nlp-spark400_2.13:7.0.0; on Apache Spark 4.0.1, 4.1.0, 4.1.1, or 4.1.2, use com.johnsnowlabs.nlp:spark-nlp_2.13:7.0.0.

Databricks:

On a new cluster or existing one you need to add the following to the Advanced Options -> Spark tab:

spark.kryoserializer.buffer.max 2000M
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.jsl.settings.pretrained.cache_folder dbfs:/PATH_TO_CACHE
spark.jsl.settings.storage.cluster_tmp_dir dbfs:/PATH_TO_STORAGE
spark.jsl.settings.annotator.log_folder dbfs:/PATH_TO_LOGS

NOTE: If this is an existing cluster, after adding new configs or changing existing properties you need to restart it.

Additional Configuration for Databricks

When running Email Reader feature sparknlp.read().email("./email-files") on Databricks, it is necessary to include the following Spark configurations to avoid dependency conflicts:

spark.driver.userClassPathFirst true
spark.executor.userClassPathFirst true

These configurations are required because the Databricks runtime environment includes a bundled version of the com.sun.mail:jakarta.mail library, which conflicts with jakarta.activation. By setting these properties, the application ensures that the user-provided libraries take precedence over those bundled in the Databricks environment, resolving the dependency conflict.

Databricks Unity Catalog Volumes and pretrained models

Databricks documents that some JVM-based operations do not support reading from or writing to Unity Catalog Volumes through standard /Volumes/... paths. See the official Databricks guidance here:

Databricks documentation: Work with files on Databricks

Spark NLP pretrained downloads rely on JVM-side file operations for download, move, and unzip. Because of this Databricks limitation, Unity Catalog Volumes are not supported as Spark NLP download/cache targets for spark.jsl.settings.pretrained.cache_folder, spark.jsl.settings.storage.cluster_tmp_dir, or spark.jsl.settings.annotator.log_folder.

For Databricks environments that store pretrained models on a Unity Catalog Volume, the supported workaround is to place the model artifacts on the Volume outside the Spark NLP .pretrained() flow and then load them directly with .load(model_path).

Load a model already stored on a Unity Catalog Volume

from sparknlp.annotator import NerDLModel

model_path = "/Volumes/<catalog>/<schema>/<volume>/cache_pretrained/ner_dl_en_2.4.3_2.4_1584624950746"

ner_model = NerDLModel.load(model_path) \
    .setInputCols(["sentence", "token", "embeddings"]) \
    .setOutputCol("ner")

S3 Integration

Logging:

To configure S3 path for logging while training models. We need to set up AWS credentials as well as an S3 path

spark.conf.set("spark.jsl.settings.annotator.log_folder", "s3://my/s3/path/logs")
spark.conf.set("spark.jsl.settings.aws.credentials.access_key_id", "MY_KEY_ID")
spark.conf.set("spark.jsl.settings.aws.credentials.secret_access_key", "MY_SECRET_ACCESS_KEY")
spark.conf.set("spark.jsl.settings.aws.s3_bucket", "my.bucket")
spark.conf.set("spark.jsl.settings.aws.region", "my-region")

Now you can check the log on your S3 path defined in spark.jsl.settings.annotator.log_folder property. Make sure to use the prefix s3://, otherwise it will use the default configuration.

Tensorflow Graphs:

To reference S3 location for downloading graphs. We need to set up AWS credentials

spark.conf.set("spark.jsl.settings.aws.credentials.access_key_id", "MY_KEY_ID")
spark.conf.set("spark.jsl.settings.aws.credentials.secret_access_key", "MY_SECRET_ACCESS_KEY")
spark.conf.set("spark.jsl.settings.aws.region", "my-region")

MFA Configuration:

In case your AWS account is configured with MFA. You will need first to get temporal credentials and add session token to the configuration as shown in the examples below For logging:

spark.conf.set("spark.jsl.settings.aws.credentials.session_token", "MY_TOKEN")

An example of a bash script that gets temporal AWS credentials can be found here This script requires three arguments:

./aws_tmp_credentials.sh iam_user duration serial_number