Packages

class LLAMA2Transformer extends AnnotatorModel[LLAMA2Transformer] with HasBatchedAnnotate[LLAMA2Transformer] with ParamsAndFeaturesWritable with WriteOnnxModel with WriteOpenvinoModel with HasGeneratorProperties with WriteSentencePieceModel with HasEngine

Llama 2: Open Foundation and Fine-Tuned Chat Models

The Llama 2 release introduces a family of pretrained and fine-tuned LLMs, ranging in scale from 7B to 70B parameters (7B, 13B, 70B). The pretrained models come with significant improvements over the Llama 1 models, including being trained on 40% more tokens, having a much longer context length (4k tokens 🤯), and using grouped-query attention for fast inference of the 70B model🔥!

However, the most exciting part of this release is the fine-tuned models (Llama 2-Chat), which have been optimized for dialogue applications using Reinforcement Learning from Human Feedback (RLHF). Across a wide range of helpfulness and safety benchmarks, the Llama 2-Chat models perform better than most open models and achieve comparable performance to ChatGPT according to human evaluations.

Pretrained models can be loaded with pretrained of the companion object:

val llama2 = LLAMA2Transformer.pretrained()
  .setInputCols("document")
  .setOutputCol("generation")

The default model is "llama_2_7b_chat_hf_int4", if no name is provided. For available pretrained models please see the Models Hub.

For extended examples of usage, see LLAMA2TestSpec.

References:

Paper Abstract:

In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for dialogue use cases. Our models outperform open-source chat models on most benchmarks we tested, and based on our human evaluations for helpfulness and safety, may be a suitable substitute for closed-source models. We provide a detailed description of our approach to fine-tuning and safety improvements of Llama 2-Chat in order to enable the community to build on our work and contribute to the responsible development of LLMs.

Note:

This is a very computationally expensive module especially on larger sequence. The use of an accelerator such as GPU is recommended.

Example

import spark.implicits._
import com.johnsnowlabs.nlp.base.DocumentAssembler
import com.johnsnowlabs.nlp.annotators.seq2seq.LLAMA2Transformer
import org.apache.spark.ml.Pipeline

val documentAssembler = new DocumentAssembler()
  .setInputCol("text")
  .setOutputCol("documents")

val llama2 = LLAMA2Transformer.pretrained("llama_2_7b_chat_hf_int4")
  .setInputCols(Array("documents"))
  .setMinOutputLength(10)
  .setMaxOutputLength(50)
  .setDoSample(false)
  .setTopK(50)
  .setNoRepeatNgramSize(3)
  .setOutputCol("generation")

val pipeline = new Pipeline().setStages(Array(documentAssembler, llama2))

val data = Seq(
  "My name is Leonardo."
).toDF("text")
val result = pipeline.fit(data).transform(data)

results.select("generation.result").show(truncate = false)
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|result                                                                                                                                                                                              |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|[ My name is Leonardo. I am a man of letters. I have been a man for many years. I was born in the year 1776. I came to the United States in 1776, and I have lived in the United Kingdom since 1776]|
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Linear Supertypes
HasEngine, WriteSentencePieceModel, HasGeneratorProperties, WriteOpenvinoModel, WriteOnnxModel, HasBatchedAnnotate[LLAMA2Transformer], AnnotatorModel[LLAMA2Transformer], CanBeLazy, RawAnnotator[LLAMA2Transformer], HasOutputAnnotationCol, HasInputAnnotationCols, HasOutputAnnotatorType, ParamsAndFeaturesWritable, HasFeatures, DefaultParamsWritable, MLWritable, Model[LLAMA2Transformer], Transformer, PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Grouped
  2. Alphabetic
  3. By Inheritance
Inherited
  1. LLAMA2Transformer
  2. HasEngine
  3. WriteSentencePieceModel
  4. HasGeneratorProperties
  5. WriteOpenvinoModel
  6. WriteOnnxModel
  7. HasBatchedAnnotate
  8. AnnotatorModel
  9. CanBeLazy
  10. RawAnnotator
  11. HasOutputAnnotationCol
  12. HasInputAnnotationCols
  13. HasOutputAnnotatorType
  14. ParamsAndFeaturesWritable
  15. HasFeatures
  16. DefaultParamsWritable
  17. MLWritable
  18. Model
  19. Transformer
  20. PipelineStage
  21. Logging
  22. Params
  23. Serializable
  24. Serializable
  25. Identifiable
  26. AnyRef
  27. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Parameters

A list of (hyper-)parameter keys this annotator can take. Users can set and get the parameter values through setters and getters, respectively.

  1. val batchSize: IntParam

    Size of every batch (Default depends on model).

    Size of every batch (Default depends on model).

    Definition Classes
    HasBatchedAnnotate
  2. val beamSize: IntParam

    Beam size for the beam search algorithm (Default: 4)

    Beam size for the beam search algorithm (Default: 4)

    Definition Classes
    HasGeneratorProperties
  3. val doSample: BooleanParam

    Whether or not to use sampling, use greedy decoding otherwise (Default: false)

    Whether or not to use sampling, use greedy decoding otherwise (Default: false)

    Definition Classes
    HasGeneratorProperties
  4. val engine: Param[String]

    This param is set internally once via loadSavedModel.

    This param is set internally once via loadSavedModel. That's why there is no setter

    Definition Classes
    HasEngine
  5. var ignoreTokenIds: IntArrayParam

    A list of token ids which are ignored in the decoder's output (Default: Array())

  6. val inputAnnotatorTypes: Array[AnnotatorType]

    Input annotator type : DOCUMENT

    Input annotator type : DOCUMENT

    Definition Classes
    LLAMA2Transformer → HasInputAnnotationCols
  7. val maxInputLength: IntParam

    max length of the input sequence (Default: 0)

    max length of the input sequence (Default: 0)

    Definition Classes
    HasGeneratorProperties
  8. val maxOutputLength: IntParam

    Maximum length of the sequence to be generated (Default: 20)

    Maximum length of the sequence to be generated (Default: 20)

    Definition Classes
    HasGeneratorProperties
  9. val minOutputLength: IntParam

    Minimum length of the sequence to be generated (Default: 0)

    Minimum length of the sequence to be generated (Default: 0)

    Definition Classes
    HasGeneratorProperties
  10. val nReturnSequences: IntParam

    The number of sequences to return from the beam search.

    The number of sequences to return from the beam search.

    Definition Classes
    HasGeneratorProperties
  11. val noRepeatNgramSize: IntParam

    If set to int > 0, all ngrams of that size can only occur once (Default: 0)

    If set to int > 0, all ngrams of that size can only occur once (Default: 0)

    Definition Classes
    HasGeneratorProperties
  12. val outputAnnotatorType: String

    Output annotator type : DOCUMENT

    Output annotator type : DOCUMENT

    Definition Classes
    LLAMA2Transformer → HasOutputAnnotatorType
  13. val randomSeed: Option[Long]

    Optional Random seed for the model.

    Optional Random seed for the model. Needs to be of type Int.

    Definition Classes
    HasGeneratorProperties
  14. val repetitionPenalty: DoubleParam

    The parameter for repetition penalty (Default: 1.0).

    The parameter for repetition penalty (Default: 1.0). 1.0 means no penalty. See this paper for more details.

    Definition Classes
    HasGeneratorProperties
  15. val stopTokenIds: IntArrayParam

    Stop tokens to terminate the generation

    Stop tokens to terminate the generation

    Definition Classes
    HasGeneratorProperties
  16. val task: Param[String]

    Set transformer task, e.g.

    Set transformer task, e.g. "summarize:" (Default: "").

    Definition Classes
    HasGeneratorProperties
  17. val temperature: DoubleParam

    The value used to module the next token probabilities (Default: 1.0)

    The value used to module the next token probabilities (Default: 1.0)

    Definition Classes
    HasGeneratorProperties
  18. val topK: IntParam

    The number of highest probability vocabulary tokens to keep for top-k-filtering (Default: 50)

    The number of highest probability vocabulary tokens to keep for top-k-filtering (Default: 50)

    Definition Classes
    HasGeneratorProperties
  19. val topP: DoubleParam

    If set to float < 1.0, only the most probable tokens with probabilities that add up to topP or higher are kept for generation (Default: 1.0)

    If set to float < 1.0, only the most probable tokens with probabilities that add up to topP or higher are kept for generation (Default: 1.0)

    Definition Classes
    HasGeneratorProperties

Members

  1. type AnnotatorType = String
    Definition Classes
    HasOutputAnnotatorType
  1. def batchAnnotate(batchedAnnotations: Seq[Array[Annotation]]): Seq[Seq[Annotation]]

    takes a document and annotations and produces new annotations of this annotator's annotation type

    takes a document and annotations and produces new annotations of this annotator's annotation type

    batchedAnnotations

    Annotations that correspond to inputAnnotationCols generated by previous annotators if any

    returns

    any number of annotations processed for every input annotation. Not necessary one to one relationship

    Definition Classes
    LLAMA2Transformer → HasBatchedAnnotate
  2. def batchProcess(rows: Iterator[_]): Iterator[Row]
    Definition Classes
    HasBatchedAnnotate
  3. final def clear(param: Param[_]): LLAMA2Transformer.this.type
    Definition Classes
    Params
  4. def copy(extra: ParamMap): LLAMA2Transformer

    requirement for annotators copies

    requirement for annotators copies

    Definition Classes
    RawAnnotator → Model → Transformer → PipelineStage → Params
  5. def explainParam(param: Param[_]): String
    Definition Classes
    Params
  6. def explainParams(): String
    Definition Classes
    Params
  7. final def extractParamMap(): ParamMap
    Definition Classes
    Params
  8. final def extractParamMap(extra: ParamMap): ParamMap
    Definition Classes
    Params
  9. val features: ArrayBuffer[Feature[_, _, _]]
    Definition Classes
    HasFeatures
  10. val generationConfig: StructFeature[GenerationConfig]
  11. final def get[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  12. final def getDefault[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  13. def getGenerationConfig: GenerationConfig
  14. def getInputCols: Array[String]

    returns

    input annotations columns currently used

    Definition Classes
    HasInputAnnotationCols
  15. def getLazyAnnotator: Boolean
    Definition Classes
    CanBeLazy
  16. final def getOrDefault[T](param: Param[T]): T
    Definition Classes
    Params
  17. final def getOutputCol: String

    Gets annotation column name going to generate

    Gets annotation column name going to generate

    Definition Classes
    HasOutputAnnotationCol
  18. def getParam(paramName: String): Param[Any]
    Definition Classes
    Params
  19. final def hasDefault[T](param: Param[T]): Boolean
    Definition Classes
    Params
  20. def hasParam(paramName: String): Boolean
    Definition Classes
    Params
  21. def hasParent: Boolean
    Definition Classes
    Model
  22. final def isDefined(param: Param[_]): Boolean
    Definition Classes
    Params
  23. final def isSet(param: Param[_]): Boolean
    Definition Classes
    Params
  24. val lazyAnnotator: BooleanParam
    Definition Classes
    CanBeLazy
  25. def onWrite(path: String, spark: SparkSession): Unit
    Definition Classes
    LLAMA2Transformer → ParamsAndFeaturesWritable
  26. val optionalInputAnnotatorTypes: Array[String]
    Definition Classes
    HasInputAnnotationCols
  27. lazy val params: Array[Param[_]]
    Definition Classes
    Params
  28. var parent: Estimator[LLAMA2Transformer]
    Definition Classes
    Model
  29. def save(path: String): Unit
    Definition Classes
    MLWritable
    Annotations
    @Since( "1.6.0" ) @throws( ... )
  30. final def set[T](param: Param[T], value: T): LLAMA2Transformer.this.type
    Definition Classes
    Params
  31. def setGenerationConfig(value: GenerationConfig): LLAMA2Transformer.this.type
  32. final def setInputCols(value: String*): LLAMA2Transformer.this.type
    Definition Classes
    HasInputAnnotationCols
  33. def setInputCols(value: Array[String]): LLAMA2Transformer.this.type

    Overrides required annotators column if different than default

    Overrides required annotators column if different than default

    Definition Classes
    HasInputAnnotationCols
  34. def setLazyAnnotator(value: Boolean): LLAMA2Transformer.this.type
    Definition Classes
    CanBeLazy
  35. def setMaxInputLength(value: Int): LLAMA2Transformer.this.type
    Definition Classes
    HasGeneratorProperties
  36. final def setOutputCol(value: String): LLAMA2Transformer.this.type

    Overrides annotation column name when transforming

    Overrides annotation column name when transforming

    Definition Classes
    HasOutputAnnotationCol
  37. def setParent(parent: Estimator[LLAMA2Transformer]): LLAMA2Transformer
    Definition Classes
    Model
  38. def toString(): String
    Definition Classes
    Identifiable → AnyRef → Any
  39. final def transform(dataset: Dataset[_]): DataFrame

    Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content

    Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content

    dataset

    Dataset[Row]

    Definition Classes
    AnnotatorModel → Transformer
  40. def transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame
    Definition Classes
    Transformer
    Annotations
    @Since( "2.0.0" )
  41. def transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame
    Definition Classes
    Transformer
    Annotations
    @Since( "2.0.0" ) @varargs()
  42. final def transformSchema(schema: StructType): StructType

    requirement for pipeline transformation validation.

    requirement for pipeline transformation validation. It is called on fit()

    Definition Classes
    RawAnnotator → PipelineStage
  43. val uid: String
    Definition Classes
    LLAMA2Transformer → Identifiable
  44. def write: MLWriter
    Definition Classes
    ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable
  45. def writeOnnxModel(path: String, spark: SparkSession, onnxWrapper: OnnxWrapper, suffix: String, fileName: String): Unit
    Definition Classes
    WriteOnnxModel
  46. def writeOnnxModels(path: String, spark: SparkSession, onnxWrappersWithNames: Seq[(OnnxWrapper, String)], suffix: String): Unit
    Definition Classes
    WriteOnnxModel
  47. def writeOpenvinoModel(path: String, spark: SparkSession, openvinoWrapper: OpenvinoWrapper, suffix: String, fileName: String): Unit
    Definition Classes
    WriteOpenvinoModel
  48. def writeOpenvinoModels(path: String, spark: SparkSession, ovWrappersWithNames: Seq[(OpenvinoWrapper, String)], suffix: String): Unit
    Definition Classes
    WriteOpenvinoModel
  49. def writeSentencePieceModel(path: String, spark: SparkSession, spp: SentencePieceWrapper, suffix: String, filename: String): Unit
    Definition Classes
    WriteSentencePieceModel

Parameter setters

  1. def setBatchSize(size: Int): LLAMA2Transformer.this.type

    Size of every batch.

    Size of every batch.

    Definition Classes
    HasBatchedAnnotate
  2. def setBeamSize(beamNum: Int): LLAMA2Transformer.this.type

    Definition Classes
    HasGeneratorProperties
  3. def setDoSample(value: Boolean): LLAMA2Transformer.this.type

    Definition Classes
    HasGeneratorProperties
  4. def setIgnoreTokenIds(tokenIds: Array[Int]): LLAMA2Transformer.this.type

  5. def setMaxOutputLength(value: Int): LLAMA2Transformer.this.type

    Definition Classes
    HasGeneratorProperties
  6. def setMinOutputLength(value: Int): LLAMA2Transformer.this.type

    Definition Classes
    HasGeneratorProperties
  7. def setModelIfNotSet(spark: SparkSession, onnxWrappers: Option[DecoderWrappers], openvinoWrapper: Option[OpenvinoWrapper], spp: SentencePieceWrapper): LLAMA2Transformer.this.type

  8. def setNReturnSequences(beamNum: Int): LLAMA2Transformer.this.type

    Definition Classes
    HasGeneratorProperties
  9. def setNoRepeatNgramSize(value: Int): LLAMA2Transformer.this.type

    Definition Classes
    HasGeneratorProperties
  10. def setRandomSeed(value: Int): LLAMA2Transformer.this.type

  11. def setRandomSeed(value: Long): LLAMA2Transformer.this.type

    Definition Classes
    HasGeneratorProperties
  12. def setRepetitionPenalty(value: Double): LLAMA2Transformer.this.type

    Definition Classes
    HasGeneratorProperties
  13. def setStopTokenIds(value: Array[Int]): LLAMA2Transformer.this.type

    Definition Classes
    HasGeneratorProperties
  14. def setTask(value: String): LLAMA2Transformer.this.type

    Definition Classes
    HasGeneratorProperties
  15. def setTemperature(value: Double): LLAMA2Transformer.this.type

    Definition Classes
    HasGeneratorProperties
  16. def setTopK(value: Int): LLAMA2Transformer.this.type

    Definition Classes
    HasGeneratorProperties
  17. def setTopP(value: Double): LLAMA2Transformer.this.type

    Definition Classes
    HasGeneratorProperties

Parameter getters

  1. def getBatchSize: Int

    Size of every batch.

    Size of every batch.

    Definition Classes
    HasBatchedAnnotate
  2. def getBeamSize: Int

    Definition Classes
    HasGeneratorProperties
  3. def getDoSample: Boolean

    Definition Classes
    HasGeneratorProperties
  4. def getEngine: String

    Definition Classes
    HasEngine
  5. def getIgnoreTokenIds: Array[Int]

  6. def getMaxOutputLength: Int

    Definition Classes
    HasGeneratorProperties
  7. def getMinOutputLength: Int

    Definition Classes
    HasGeneratorProperties
  8. def getModelIfNotSet: LLAMA2

  9. def getNReturnSequences: Int

    Definition Classes
    HasGeneratorProperties
  10. def getNoRepeatNgramSize: Int

    Definition Classes
    HasGeneratorProperties
  11. def getRandomSeed: Option[Long]

    Definition Classes
    HasGeneratorProperties
  12. def getRepetitionPenalty: Double

    Definition Classes
    HasGeneratorProperties
  13. def getStopTokenIds: Array[Int]

    Definition Classes
    HasGeneratorProperties
  14. def getTask: Option[String]

    Definition Classes
    HasGeneratorProperties
  15. def getTemperature: Double

    Definition Classes
    HasGeneratorProperties
  16. def getTopK: Int

    Definition Classes
    HasGeneratorProperties
  17. def getTopP: Double

    Definition Classes
    HasGeneratorProperties