sparknlp#

Functions#

Attributes#

annotators[source]#

embeddings[source]#

start(gpu=False, apple_silicon=False, aarch64=False, memory='16G', cache_folder='', log_folder='', cluster_tmp_dir='', params=None, real_time_output=False, output_level=1, skip_sparknlp_maven=False)[source]#

Starts a PySpark instance with default parameters for Spark NLP.

The default parameters would result in the equivalent of:

SparkSession.builder \
    .appName("Spark NLP") \
    .master("local[*]") \
    .config("spark.driver.memory", "16G") \
    .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") \
    .config("spark.kryoserializer.buffer.max", "2000M") \
    .config("spark.driver.maxResultSize", "0") \
    .config("spark.jars.packages", "<resolved Spark NLP Maven coordinate>") \
    .getOrCreate()

Parameters:

gpubool, optional

Whether to enable GPU acceleration (must be set up correctly), by default False

apple_siliconbool, optional

Whether to enable Apple Silicon support for macOS

aarch64bool, optional

Whether to enable Linux Aarch64 support

memorystr, optional

How much memory to allocate for the Spark driver, by default “16G”

cache_folderstr, optional

The location to download and extract pretrained Models and Pipelines. If not set, it will be in the users home directory under cache_pretrained.

log_folderstr, optional

The location to use on a cluster for temporarily files such as unpacking indexes for WordEmbeddings. By default, this locations is the location of hadoop.tmp.dir set via Hadoop configuration for Apache Spark. NOTE: S3 is not supported and it must be local, HDFS, or DBFS.

paramsdict, optional

Custom parameters to set for the Spark configuration, by default None. Set skip_sparknlp_maven to True or "true" to avoid adding the default Spark NLP Maven package when a custom Spark NLP jar is provided.

cluster_tmp_dirstr, optional

The location to save logs from annotators during training. If not set, it will be in the users home directory under annotator_logs.

real_time_outputbool, optional

Whether to read and print JVM output in real time, by default False

output_levelint, optional

Output level for logs, by default 1

skip_sparknlp_mavenbool, optional

Whether to avoid adding the default Spark NLP Maven package. Use this when providing a custom Spark NLP jar with spark.jars.

Returns:

SparkSession

The initiated Spark session.

Notes

The Maven artifact is selected from the installed PySpark version. Spark 3.x uses Scala 2.12. Spark 4.0.0 uses the dedicated spark400 Scala 2.13 artifact. Later Spark 4.x releases use the default Scala 2.13 artifact. Automatic artifact selection does not imply that every release has been validated. Only release versions in major.minor.patch format are accepted.

read(params=None)[source]#

version()[source]#

Returns the current Spark NLP version.

Returns:

str

The current Spark NLP version.