Tokenizers

[DerpLady]

Buying the book, I expected to use my own data for the vocabulary part. Here, stanadrdized tokenizers are used which is rahte ruseless for my use case (I need a model that is specilized in Swedish language for a specific industry). Isnt it a bit misleading to talk about an llm from scratch if we are using a lot of pre-built stuff? E.g vocabulary + weights.

[rasbt]

Unfortunately, building the tokenizer from scratch as well was a bit out of scope, but you could train your own tokenizer using the bonus materials here: https://github.com/rasbt/LLMs-from-scratch/tree/main/ch02/05_bpe-from-scratch

[DerpLady]

Got it, will check the link out - thanks!