Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

has anything been done to like turn common phrases into a single token?

like "can you please" maps to 3895 instead of something like "10 245 87 941"

Or does it not matter since tokenization is already a kind of compression?



You can try cyp but ymmv




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: