Show HN: Tinker with Meta’s “tokenizer-free” patcher

A few weeks ago Meta released the Byte Latent Transformer weights and I wanted to have a play around.

If the future is that we tend towards replacing current tokenisation, I wanted to build intuitions around one of the core contribution of Meta’s BLT – entropy-based patching and it’s strengths and weaknesses. No better way of doing that than via tinkering with visualisations in a HF space so thought I’d share!

I’m writing a blog post on an expanded version of this, updates via https://lucalp.dev or https://x.com/lucalp__

Comments URL: https://news.ycombinator.com/item?id=44051774

Points: 1

# Comments: 0