Tokenizers documentation

Node

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

Node

The Node binding loads a tokenizer.json, encodes text into token ids and decodes ids back into text. It does not build, edit, save or train tokenizers.

Installation

npm install tokenizers

Loading a tokenizer

const { Tokenizer } = require('tokenizers')

const tokenizer = Tokenizer.fromFile('tokenizer.json')

The file is put through the legacy 1.0 → canonical 2.0 upgrade on the way in, so tokenizers already on disk keep loading.

Encoding

encode takes a text, or a pair of texts given as a two-item array, and returns an Encoding. The post-processor joins a pair, for example as [CLS] A [SEP] B [SEP].

const encoding = tokenizer.encode("Hello, y'all! How are you 😁 ?")
const pair = tokenizer.encode(["Hello, y'all!", 'How are you 😁 ?'])
PropertyDescription
idsThe id of each token.
typeIdsThe type id of each token: 0 for the first text of a pair, 1 for the second.
attentionMask1 for each token, 0 for each padding token.

Each property is a Uint32Array rather than a JS Array: an Array costs one napi value per token, which on token-dense input is 13x the encode itself.

encodeBatch encodes a batch of texts and pairs in one call, and returns the encodings in input order. Batch padding pads every item to the longest one.

const batch = tokenizer.encodeBatch(["Hello, y'all!", ["Hello, y'all!", 'How are you 😁 ?']], { padding: {} })

tokenize returns the token of each id, special tokens included. It is decodeTokens(encode(input, options).ids).

const tokens = tokenizer.tokenize("Hello, y'all!")

For a tight loop, encodeBytesInto encodes one text and writes its ids into a Uint32Array you own. That drops two per-call costs: the JS string → UTF-8 copy, and a fresh ArrayBuffer for every call. It returns how many ids it wrote:

const out = new Uint32Array(512)
const n = tokenizer.encodeBytesInto(Buffer.from("Hello, y'all!"), out)
// out.subarray(0, n) holds the ids

Decoding

decode turns ids back into text and skips special tokens unless its second argument is false. decodeTokens gives the token of each id without running the decoder, so a byte-level BPE token reads Ġworld, not world. It keeps special tokens unless its second argument is true.

const text = tokenizer.decode(encoding.ids)
const tokens = tokenizer.decodeTokens(encoding.ids)

Options

Every encode method takes an optional last argument. A field left out keeps the tokenizer’s own behaviour.

const encoding = tokenizer.encode("Hello, y'all!", {
  addSpecialTokens: true,
  encodeSpecialTokens: false,
  padding: { padId: 0, padToken: '[PAD]', length: 16 },
  truncation: { maxLength: 512 },
})
OptionDefaultDescription
addSpecialTokenstrueWhether the post-processor adds its special tokens, such as [CLS] and [SEP].
encodeSpecialTokensfalseWhether a special token written in the text goes through the model (true) or becomes its added-vocabulary id.
paddingconfiguredPadding for this call, replacing the tokenizer’s configured padding. false disables it.
truncationconfiguredTruncation for this call, replacing the tokenizer’s configured truncation. false disables it.

padding

Replaces the configured padding as a whole, so a field left out takes the default below, not the configured value.

FieldDefaultDescription
directionrightWhether pad tokens are appended right or prepended left.
padId0The id of the padding token.
padTypeId0The type id of the padding token.
padToken[PAD]The text of the padding token.
lengthbatch longestPads every encoding to exactly this many tokens.
padToMultipleOfnoneRounds the padded length up to a multiple of this.

truncation

FieldDefaultDescription
maxLengthrequiredThe maximum number of tokens, including special tokens, to keep.
strategylongest_firstWhich sequence of a pair is truncated: longest_first, only_first or only_second.
directionrightWhether to truncate at the end of the sequence or at its beginning.
Update on GitHub