·
28 commits
to main
since this release
- Update for Hugging Face Tokenizer
- Following Llama tokenizer's manner, add BOS instead of EOD
- No changes for SentencePiece model