Packages

package bpe

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. All

Type Members

  1. class BartTokenizer extends Gpt2Tokenizer
  2. class CLIPTokenizer extends Gpt2Tokenizer
  3. class Gpt2Tokenizer extends BpeTokenizer
  4. class RobertaTokenizer extends Gpt2Tokenizer
  5. case class SpecialToken(content: String, id: Int, singleWord: Boolean = false, lstrip: Boolean = false, rstrip: Boolean = false) extends Product with Serializable
  6. class WhisperTokenDecoder extends Gpt2Tokenizer

    Class used by Whisper model to decode tokens.

    Class used by Whisper model to decode tokens. Does not require merges and is therefore omitted.

    Note that this means this class cannot tokenize strings.

Value Members

  1. object BpeTokenizer

Ungrouped