Packages

package audio

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. All

Type Members

  1. class HubertForCTC extends Wav2Vec2ForCTC

    Hubert Model with a language modeling head on top for Connectionist Temporal Classification (CTC).

    Hubert Model with a language modeling head on top for Connectionist Temporal Classification (CTC). Hubert was proposed in HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units by Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, Abdelrahman Mohamed.

    The annotator takes audio files and transcribes it as text. The audio needs to be provided pre-processed an array of floats.

    Note that this annotator is currently not supported on Apple Silicon processors such as the M1/M2 (Apple Silicon). This is due to the processor not supporting instructions for XLA.

    Pretrained models can be loaded with pretrained of the companion object:

    val speechToText = HubertForCTC.pretrained()
      .setInputCols("audio_assembler")
      .setOutputCol("text")

    The default model is "asr_hubert_large_ls960", if no name is provided.

    For available pretrained models please see the Models Hub.

    To see which models are compatible and how to import them see https://github.com/JohnSnowLabs/spark-nlp/discussions/5669 and to see more extended examples, see HubertForCTCTestSpec.

    References:

    HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

    Paper Abstract:

    Self-supervised approaches for speech representation learning are challenged by three unique problems: (1) there are multiple sound units in each input utterance, (2) there is no lexicon of input sound units during the pre-training phase, and (3) sound units have variable lengths with no explicit segmentation. To deal with these three problems, we propose the Hidden-Unit BERT (HuBERT) approach for self-supervised speech representation learning, which utilizes an offline clustering step to provide aligned target labels for a BERT-like prediction loss. A key ingredient of our approach is applying the prediction loss over the masked regions only, which forces the model to learn a combined acoustic and language model over the continuous inputs. HuBERT relies primarily on the consistency of the unsupervised clustering step rather than the intrinsic quality of the assigned cluster labels. Starting with a simple k-means teacher of 100 clusters, and using two iterations of clustering, the HuBERT model either matches or improves upon the state-of-the-art wav2vec 2.0 performance on the Librispeech (960h) and Libri-light (60,000h) benchmarks with 10min, 1h, 10h, 100h, and 960h fine-tuning subsets. Using a 1B parameter model, HuBERT shows up to 19% and 13% relative WER reduction on the more challenging dev-other and test-other evaluation subsets.

    Example

    import spark.implicits._
    import com.johnsnowlabs.nlp.base._
    import com.johnsnowlabs.nlp.annotators._
    import com.johnsnowlabs.nlp.annotators.audio.HubertForCTC
    import org.apache.spark.ml.Pipeline
    
    val audioAssembler: AudioAssembler = new AudioAssembler()
      .setInputCol("audio_content")
      .setOutputCol("audio_assembler")
    
    val speechToText: HubertForCTC = HubertForCTC
      .pretrained()
      .setInputCols("audio_assembler")
      .setOutputCol("text")
    
    val pipeline: Pipeline = new Pipeline().setStages(Array(audioAssembler, speechToText))
    
    val bufferedSource =
      scala.io.Source.fromFile("src/test/resources/audio/csv/audio_floats.csv")
    
    val rawFloats = bufferedSource
      .getLines()
      .map(_.split(",").head.trim.toFloat)
      .toArray
    bufferedSource.close
    
    val processedAudioFloats = Seq(rawFloats).toDF("audio_content")
    
    val result = pipeline.fit(processedAudioFloats).transform(processedAudioFloats)
    result.select("text.result").show(truncate = false)
    +------------------------------------------------------------------------------------------+
    |result                                                                                    |
    +------------------------------------------------------------------------------------------+
    |[MISTER QUILTER IS THE APOSTLE OF THE MIDLE CLASES AND WE ARE GLAD TO WELCOME HIS GOSPEL ]|
    +------------------------------------------------------------------------------------------+
  2. trait ReadHubertForAudioDLModel extends ReadTensorflowModel
  3. trait ReadWav2Vec2ForAudioDLModel extends ReadTensorflowModel
  4. trait ReadWhisperForCTCDLModel extends ReadTensorflowModel with ReadOnnxModel
  5. trait ReadablePretrainedHubertForAudioModel extends ParamsAndFeaturesReadable[HubertForCTC] with HasPretrained[HubertForCTC]
  6. trait ReadablePretrainedWav2Vec2ForAudioModel extends ParamsAndFeaturesReadable[Wav2Vec2ForCTC] with HasPretrained[Wav2Vec2ForCTC]
  7. trait ReadablePretrainedWhisperForCTCModel extends ParamsAndFeaturesReadable[WhisperForCTC] with HasPretrained[WhisperForCTC]
  8. class Wav2Vec2ForCTC extends AnnotatorModel[Wav2Vec2ForCTC] with HasBatchedAnnotateAudio[Wav2Vec2ForCTC] with HasAudioFeatureProperties with WriteTensorflowModel with HasEngine

    Wav2Vec2 Model with a language modeling head on top for Connectionist Temporal Classification (CTC).

    Wav2Vec2 Model with a language modeling head on top for Connectionist Temporal Classification (CTC). Wav2Vec2 was proposed in wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations by Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael Auli.

    The annotator takes audio files and transcribes it as text. The audio needs to be provided pre-processed an array of floats.

    Note that this annotator is currently not supported on Apple Silicon processors such as the M1/M2 (Apple Silicon). This is due to the processor not supporting instructions for XLA.

    Pretrained models can be loaded with pretrained of the companion object:

    val speechToText = Wav2Vec2ForCTC.pretrained()
      .setInputCols("audio_assembler")
      .setOutputCol("text")

    The default model is "asr_wav2vec2_base_960h", if no name is provided.

    For available pretrained models please see the Models Hub.

    To see which models are compatible and how to import them see https://github.com/JohnSnowLabs/spark-nlp/discussions/5669 and to see more extended examples, see Wav2Vec2ForCTCTestSpec.

    Example

    import spark.implicits._
    import com.johnsnowlabs.nlp.base._
    import com.johnsnowlabs.nlp.annotators._
    import com.johnsnowlabs.nlp.annotators.audio.Wav2Vec2ForCTC
    import org.apache.spark.ml.Pipeline
    
    val audioAssembler: AudioAssembler = new AudioAssembler()
      .setInputCol("audio_content")
      .setOutputCol("audio_assembler")
    
    val speechToText: Wav2Vec2ForCTC = Wav2Vec2ForCTC
      .pretrained()
      .setInputCols("audio_assembler")
      .setOutputCol("text")
    
    val pipeline: Pipeline = new Pipeline().setStages(Array(audioAssembler, speechToText))
    
    val bufferedSource =
      scala.io.Source.fromFile("src/test/resources/audio/csv/audio_floats.csv")
    
    val rawFloats = bufferedSource
      .getLines()
      .map(_.split(",").head.trim.toFloat)
      .toArray
    bufferedSource.close
    
    val processedAudioFloats = Seq(rawFloats).toDF("audio_content")
    
    val result = pipeline.fit(processedAudioFloats).transform(processedAudioFloats)
    result.select("text.result").show(truncate = false)
    +------------------------------------------------------------------------------------------+
    |result                                                                                    |
    +------------------------------------------------------------------------------------------+
    |[MISTER QUILTER IS THE APOSTLE OF THE MIDLE CLASES AND WE ARE GLAD TO WELCOME HIS GOSPEL ]|
    +------------------------------------------------------------------------------------------+
  9. class WhisperForCTC extends AnnotatorModel[WhisperForCTC] with HasBatchedAnnotateAudio[WhisperForCTC] with HasAudioFeatureProperties with WriteTensorflowModel with WriteOnnxModel with HasEngine with HasGeneratorProperties with HasProtectedParams

    Whisper Model with a language modeling head on top for Connectionist Temporal Classification (CTC).

    Whisper Model with a language modeling head on top for Connectionist Temporal Classification (CTC).

    Whisper is an automatic speech recognition (ASR) system trained on 680,000 hours of multilingual and multitask supervised data collected from the web. It transcribe in multiple languages, as well as translate from those languages into English.

    The audio needs to be provided pre-processed an array of floats.

    For multilingual models, the language and the task (transcribe or translate) can be set with setLanguage and setTask.

    Note that at the moment, this annotator only supports greedy search and only Spark Versions 3.4 and up are supported.

    Pretrained models can be loaded with pretrained of the companion object:

    val speechToText = WhisperForCTC.pretrained()
      .setInputCols("audio_assembler")
      .setOutputCol("text")

    The default model is "asr_whisper_tiny_opt", if no name is provided.

    For available pretrained models please see the Models Hub.

    To see which models are compatible and how to import them see https://github.com/JohnSnowLabs/spark-nlp/discussions/5669 and to see more extended examples, see WhisperForCTCTestSpec.

    References:

    Robust Speech Recognition via Large-Scale Weak Supervision

    Paper Abstract:

    We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zero- shot transfer setting without the need for any fine- tuning. When compared to humans, the models approach their accuracy and robustness. We are releasing models and inference code to serve as a foundation for further work on robust speech processing.

    Example

    import spark.implicits._
    import com.johnsnowlabs.nlp.base._
    import com.johnsnowlabs.nlp.annotators._
    import com.johnsnowlabs.nlp.annotators.audio.WhisperForCTC
    import org.apache.spark.ml.Pipeline
    
    val audioAssembler: AudioAssembler = new AudioAssembler()
      .setInputCol("audio_content")
      .setOutputCol("audio_assembler")
    
    val speechToText: WhisperForCTC = WhisperForCTC
      .pretrained()
      .setInputCols("audio_assembler")
      .setOutputCol("text")
    
    val pipeline: Pipeline = new Pipeline().setStages(Array(audioAssembler, speechToText))
    
    val bufferedSource =
      scala.io.Source.fromFile("src/test/resources/audio/txt/librispeech_asr_0.txt")
    
    val rawFloats = bufferedSource
      .getLines()
      .map(_.split(",").head.trim.toFloat)
      .toArray
    bufferedSource.close
    
    val processedAudioFloats = Seq(rawFloats).toDF("audio_content")
    
    val result = pipeline.fit(processedAudioFloats).transform(processedAudioFloats)
    result.select("text.result").show(truncate = false)
    +------------------------------------------------------------------------------------------+
    |result                                                                                    |
    +------------------------------------------------------------------------------------------+
    |[ Mr. Quilter is the apostle of the middle classes and we are glad to welcome his gospel.]|
    +------------------------------------------------------------------------------------------+

Value Members

  1. object HubertForCTC extends ReadablePretrainedHubertForAudioModel with ReadHubertForAudioDLModel with Serializable

    This is the companion object of HubertForCTC.

    This is the companion object of HubertForCTC. Please refer to that class for the documentation.

  2. object Wav2Vec2ForCTC extends ReadablePretrainedWav2Vec2ForAudioModel with ReadWav2Vec2ForAudioDLModel with Serializable

    This is the companion object of Wav2Vec2ForCTC.

    This is the companion object of Wav2Vec2ForCTC. Please refer to that class for the documentation.

  3. object WhisperForCTC extends ReadablePretrainedWhisperForCTCModel with ReadWhisperForCTCDLModel with Serializable

    This is the companion object of WhisperForCTC.

    This is the companion object of WhisperForCTC. Please refer to that class for the documentation.

Ungrouped