ModernBERT Base ONNX

Description

ModernBERT is a modernized bidirectional encoder-only Transformer model (BERT-style) pre-trained on 2 trillion tokens of English and code data with a native context length of up to 8,192 tokens. ModernBERT leverages recent architectural improvements such as:

• Rotary Positional Embeddings (RoPE) for long-context support.
• Local-Global Alternating Attention for efficiency on long inputs.
• Unpadding and Flash Attention for efficient inference.

ModernBERT’s native long context length makes it ideal for tasks that require processing long documents, such as retrieval, classification, and semantic search within large corpora. The model was trained on a large corpus of text and code, making it suitable for a wide range of downstream tasks, including code retrieval and hybrid (text + code) semantic search.

Live Demo Open in Colab Download Copy S3 URI

How to use

from sparknlp.base import *
from sparknlp.annotator import *
from pyspark.ml import Pipeline

documentAssembler = DocumentAssembler() \
    .setInputCol("text") \
    .setOutputCol("document")

tokenizer = Tokenizer() \
    .setInputCols("document") \
    .setOutputCol("token")

loaded_model = ModernBertEmbeddings \
    .pretrained("modernbert_base_onnx") \
    .setInputCols(["token", "document"]) \
    .setOutputCol("embeddings")

pipeline = Pipeline(stages=[
    documentAssembler,
    tokenizer,
    loaded_model
])

data = spark.createDataFrame([["Covid cases are increasing fast!"]]).toDF("text")

result = pipeline.fit(data).transform(data)
result.select("embeddings.embeddings").show()

import com.johnsnowlabs.nlp.base._
import com.johnsnowlabs.nlp.annotators._
import org.apache.spark.ml.Pipeline

val documentAssembler = new DocumentAssembler()
  .setInputCol("text")
  .setOutputCol("document")

val tokenizer = new Tokenizer()
  .setInputCols("document")
  .setOutputCol("token")

val loadedModel = ModernBertEmbeddings
  .pretrained("modernbert_base_onnx")
  .setInputCols(Array("token", "document"))
  .setOutputCol("embeddings")

val pipeline = new Pipeline().setStages(Array(
  documentAssembler,
  tokenizer,
  loadedModel
))

val data = spark.createDataFrame(Seq("Covid cases are increasing fast!")).toDF("text")

val result = pipeline.fit(data).transform(data)
result.select("embeddings.embeddings").show()

Results

+--------------------+
|          embeddings|
+--------------------+
|[[0.721862, 0.107...|
+--------------------+

Model Information