Edit on GitHub
  • John Snow Labs
  • Jul 31, 2025

Qwen3-4B GGUF (Q8 Quantized) by Qwen

  • qwen3
  • q8
  • quantized
  • conversational
  • en
  • open_source
  • llamacpp
  • gguf

Description

Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts (MoE) models. Built upon extensive training, Qwen3 delivers groundbreaking advancements in reasoning, instruction-following, agent capabilities, and multilingual support

Original model from https://huggingface.co/Qwen/Qwen3-4B

Download Copy S3 URI

How to use

from sparknlp.base import DocumentAssembler
from sparknlp.annotator import AutoGGUFModel
from pyspark.ml import Pipeline

document_assembler = DocumentAssembler()\
    .setInputCol("text")\
    .setOutputCol("document")

auto_gguf_model = AutoGGUFModel.pretrained("qwen3_4b_q8_0_gguf", "en") \
    .setInputCols(["document"]) \
    .setOutputCol("completions") \
    .setBatchSize(4) \
    .setNPredict(-1) \
    .setNGpuLayers(99) \
    .setTemperature(0.4) \
    .setTopK(40) \
    .setTopP(0.9) \
    .setPenalizeNl(True)

pipeline = Pipeline().setStages([
    document_assembler,
    auto_gguf_model
])

data = spark.createDataFrame([
    ["Give me a short introduction to large language model."]
]).toDF("text")

model = pipeline.fit(data)
result = model.transform(data)

result.select("completions").show(truncate=False)


import com.johnsnowlabs.nlp.base.DocumentAssembler
import com.johnsnowlabs.nlp.annotators.auto.gguf.AutoGGUFModel
import org.apache.spark.ml.Pipeline

val documentAssembler = new DocumentAssembler()
  .setInputCol("text")
  .setOutputCol("document")

val autoGGUFModel = AutoGGUFModel.pretrained("qwen3_4b_q8_0_gguf", "en")
  .setInputCols("document")
  .setOutputCol("completions")
  .setBatchSize(4)
  .setNPredict(20)
  .setNGpuLayers(99)
  .setTemperature(0.4f)
  .setTopK(40)
  .setTopP(0.9f)
  .setPenalizeNl(true)

val pipeline = new Pipeline().setStages(Array(
  documentAssembler,
  autoGGUFModel
))

val data = Seq("Give me a short introduction to large language model.").toDF("text")

val model = pipeline.fit(data)
val result = model.transform(data)

result.select("completions").show(false)

Results

Large language models (LLMs) are advanced artificial intelligence systems designed to understand and generate human-like text. Trained on vast amounts of textual data, they can perform tasks such as answering questions, writing stories, coding, summarizing information, and engaging in conversations. These models leverage deep learning techniques to recognize patterns, context, and semantics in language, making them highly versatile tools for various applications, from customer service to creative writing. While powerful, they require careful use to ensure accuracy and ethical considerations.

Model Information

Model Name: qwen3_4b_q8_0_gguf
Compatibility: Spark NLP 6.0.3+
License: Open Source
Edition: Official
Input Labels: [document]
Output Labels: [completions]
Language: en
Size: 4.1 GB