Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

"Tokenizer fertility is a metric used to evaluate tokenizer performance and measures a tokenizer’s ability to represent text, calculated by dividing the number of tokens in a text (after tokenizing) by the number of words in that same text"


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: