CERBERUS/ollama
1
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-13 14:27:00 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
ollama/tokenizer
History
Michael Yang f1373193dc
move tokenizers to separate package (#13825)
2026-02-05 17:44:11 -08:00
..
testdata move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
bytepairencoding.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
bytepairencoding_test.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
sentencepiece.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
sentencepiece_test.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
tokenizer.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
vocabulary.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
vocabulary_test.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
wordpiece.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00
wordpiece_test.go move tokenizers to separate package (#13825) 2026-02-05 17:44:11 -08:00