package bpe
Ordering
- Alphabetic
Visibility
- Public
- All
Type Members
- class BartTokenizer extends Gpt2Tokenizer
- class CLIPTokenizer extends Gpt2Tokenizer
- class Gpt2Tokenizer extends BpeTokenizer
- class RobertaTokenizer extends Gpt2Tokenizer
- case class SpecialToken(content: String, id: Int, singleWord: Boolean = false, lstrip: Boolean = false, rstrip: Boolean = false) extends Product with Serializable
-
class
WhisperTokenDecoder extends Gpt2Tokenizer
Class used by Whisper model to decode tokens.
Class used by Whisper model to decode tokens. Does not require merges and is therefore omitted.
Note that this means this class cannot tokenize strings.
Value Members
- object BpeTokenizer