import streamlit as st import json # Set page configuration st.set_page_config( page_title="tokeniser-py Demonstration", page_icon="🔣", layout="wide", ) # Custom CSS for better UI st.markdown(""" """, unsafe_allow_html=True) # Header with logo and title st.markdown("""

tokeniser-py 🔣

Library GitHub (tokeniser-py)

|

Library GitHub (tokeniser-py-lite)

|

HF Dataset (unchunked)

|

GitHub Dataset (chunked)

|

GitHub Imp Files

|

PyPI Package (Main Lib)

|

PyPI Package (Lite Lib)

Learn about language model tokenization

tokeniser-py's custom tokenizer processes text using tokens, which are common sequences of characters found in a set of text. The model learns to understand the statistical relationships between these tokens, and excel at producing the next token in a sequence of tokens. You can use the tool below to understand how a piece of text might be tokenized by a language model, and the total count of tokens in that piece of text.

""", unsafe_allow_html=True) # Initialize tokenizer @st.cache_resource def load_tokenizer(ln="1b", token_ordered=False): try: from tokeniser import Tokeniser # Pass parameters based on selection return Tokeniser(ln=ln, token_ordered=token_ordered) except Exception as e: st.error(f"Error loading tokenizer: {e}") return None # Information about tokenization # st.markdown(""" # """) # st.markdown("") # st.markdown("") st.markdown("###### Model") # Create tabs for different models model_version = st.radio( "", ["Default (1b model unordered)", "1b model ordered", "0.5b model unordered", "0.5b model ordered"], horizontal=True ) # Map selected model version to parameters if model_version == "Default (1b model unordered)": ln_param = "1b" ordered_param = False elif model_version == "1b model ordered": ln_param = "1b" ordered_param = True elif model_version == "0.5b model unordered": ln_param = "0.5b" ordered_param = False else: ln_param = "0.5b" ordered_param = True # Load tokenizer with selected parameters tokenizer = load_tokenizer(ln=ln_param, token_ordered=ordered_param) # Function to generate consistent pastel colors for tokens @st.cache_data def get_token_colors(tokens): # Use hash of token to get consistent colors colors = {} for token in set(tokens): # Generate a pastel color based on the hash of the token hash_val = hash(token) % 360 colors[token] = f"hsl({hash_val}, 80%, 75%)" return colors # Function to display tokens with colors and hover effects def display_colored_tokens(tokens, token_ids, token_colors): html = "" for i, (token, token_id) in enumerate(zip(tokens, token_ids)): # Handle special characters for display if token == '\n': display_token = '\\n' elif token == '\t': display_token = '\\t' else: display_token = token.replace("<", "<").replace(">", ">").replace(" ", " ") html += f'{display_token}' return html # Function to display token IDs def display_token_ids(token_ids): return f'
{json.dumps(token_ids)}
' # Initialize session state for text input if not exists if 'text_input' not in st.session_state: st.session_state.text_input = "Hi I am Tasmay, I am a third year undergraduate at IIT Kharagpur and this is my tokeniser. Please enter your text in this box" st.session_state.text_ind = 0 print(st.session_state.text_ind) st.markdown("###### Enter text to tokenize") # Text input area text_input = st.text_area( "", st.session_state.text_input, height=150, placeholder="Please enter the text to tokenise", # on_change=handle_text_change, ) def clear_text(): st.session_state.text_input = "" def show_example(): examples = [ "Hi I am Tasmay, I am a third year undergraduate at IIT Kharagpur and this is my tokeniser. Please enter your text in this box", "Wop, wop, wop, wop, wop, I'ma do my stuff", "I got loyalty, got royalty inside my DNA", "Sit down, be humble", "We gon' be alright" ] st.session_state.text_ind = (st.session_state.text_ind + 1) % len(examples) st.session_state.text_input = examples[st.session_state.text_ind] # Add CSS for fixed-width buttons that wrap to new line st.markdown(""" """, unsafe_allow_html=True) # Create a horizontal block for buttons button_container = st.container() with button_container: cols = st.columns([1, 1, 10]) with cols[0]: st.button("Clear", on_click=clear_text) with cols[1]: st.button("Show example", on_click=show_example) # Process the text for tokenization if tokenizer: try: tokens, count = tokenizer.tokenise(text_input) token_ids = tokenizer.token_ids(tokens) num_tokens = len(tokens) num_chars = len(text_input) chars_per_token = num_chars / num_tokens if num_tokens > 0 else 0 except Exception as e: st.error(f"Error tokenizing text: {e}") tokens = [] token_ids = [] num_tokens = 0 num_chars = 0 chars_per_token = 0 # Inject custom CSS st.markdown( """ """, unsafe_allow_html=True ) # st.markdown("###### View") # Create view toggle view_option = st.radio( "", ["Text", "Token IDs"], horizontal=True ) # Get token colors if we have tokens token_colors = get_token_colors(tokens) if tokens else {} # Always display the token display, even if empty if view_option == "Text": if tokens: st.markdown(f'
{display_colored_tokens(tokens, token_ids, token_colors)}
', unsafe_allow_html=True) else: st.markdown(f'
No tokens to display
', unsafe_allow_html=True) else: if token_ids: st.markdown(f'
{display_token_ids(token_ids)}
', unsafe_allow_html=True) else: st.markdown(f'
No token IDs to display
', unsafe_allow_html=True) # Always display the stats container, even if empty st.markdown("""
Tokens
{}
Characters
{}
Chars per token
{:.2f}
""".format(num_tokens, num_chars, chars_per_token), unsafe_allow_html=True) # Information box split into multiple markdown elements for better rendering # st.markdown("
", unsafe_allow_html=True) # Section 1: Tokenization Efficiency st.markdown("---") st.markdown("

Tokenization Efficiency

", unsafe_allow_html=True) # Quote block st.markdown("""
A helpful rule of thumb is that one token generally corresponds to ~4 characters of text for common English text. This translates to roughly ¾ of a word (so 100 tokens ~= 75 words).
— OpenAI
""", unsafe_allow_html=True) # Section 2: Our Analysis st.markdown("

Our Analysis

", unsafe_allow_html=True) st.markdown("

We've conducted a thorough analysis of token efficiency of our tokeniser against different tokenizers:

", unsafe_allow_html=True) # Analysis points with enhanced styling st.markdown("""
•
The GPT-2 tokenizer corresponds to approximately 3.9 characters per token
•
English text corpus typically has average word lengths ranging from 4.7 to 5.1 characters, which was observed to be 4.73-4.79 in our dataset
•
Thus for our dataset, traditional tokenizers convert to roughly ⁴⁄₅ of a word (100 tokens ≈ 80 words)
""", unsafe_allow_html=True) # Section 3: tokeniser-py Efficiency st.markdown("

tokeniser-py efficiency

", unsafe_allow_html=True) st.markdown("

Our tokenizer demonstrates different characteristics:

", unsafe_allow_html=True) # Efficiency points with enhanced styling st.markdown("""
•
Average token size of ~2.52 characters** across all token types
•
For alphanumeric tokens only: ~3.97 characters per token
•
This translates to approximately ⁹⁄₁₀ of a word (100 tokens ≈ 90 words)
•
Unlike other tokenizers, we handle spaces (' ') as separate tokens rather than concatenating them with other characters, which affects our total token count
""", unsafe_allow_html=True) # Section 4: Real-world Comparison with completely redesigned styling st.markdown("""

Real-world Comparison

We tested a 28-page blog post across different tokenizers:

1
GPT-4o/GPT-4: ~10.4k tokens
2
GPT-3: ~12.1k tokens
3
tokeniser-py: ~18.8k tokens (including ~8.4k space tokens and ~2.6k other special-char based tokens)
4
tokeniser-py (alphanumeric only): ~7.8k tokens
5
GPT-4/GPT-4o (alphanumeric): ~8k tokens
6
Token corpus size: 131k (tokeniser-py) vs. 100k (GPT-4 multimodal)
""", unsafe_allow_html=True) # Note box with enhanced styling st.markdown("""
Note:

• **2.52 characters is the average (adjusted frequency)-weighted token size i.e. we weigh the token size by their true occurences, obtained after adjusting their observed occurences by their super-tokens' occurences.
• A super-token of a token say 'e' is any token which contains 'e' (like 'ear', 'ears', 'years', etc.). While weighing the token length we find that a smaller tokens have an undue higher weightage due their occurences in super-tokens being added up as well. To adjust this we hierarchially subtract the occurence of a token from its super tokens to get a True frequency.
• Un-adjusted frequency weighting gives an average size of ~2.2 characters per token, and a raw (un-weighted) average results in ~4.6-4.7 chars per token.
• Our tokenization strategy separates non-underscore special characters from alphanumeric tokens.
• We define alphanumeric tokens as any word that doesn't contain special characters (except underscores).
• For OpenAI's tokens, we considered any token containing at least one alphanumeric character (excluding underscores) as an alphanumeric token.
• This difference is due to the different special characters handling methodology followed in both tokeniser.
• The tokeniser's better word representation performance is not only due to technique differences but also because GPT-4 has fewer available tokens (100k vs our 131k) and needs to reserve tokens for multimodal content, further reducing English-specific tokens.
• Additionally, GPT-4's approach of combining special characters with alphanumerical content potentially reduces the availability of relevant alphanumerical tokens. Despite these constraints, GPT-4's tokeniser performs relatively well, though ours provides a valuable research preview into an alternate algorithm.

""", unsafe_allow_html=True) # Section 5: Design Philosophy with enhanced styling st.markdown("

Design Philosophy

", unsafe_allow_html=True) st.markdown("

Our approach prioritizes semantic representation over token count minimization:

", unsafe_allow_html=True) # Philosophy points with enhanced styling st.markdown("""
•
We consciously separate special characters from alphanumeric tokens
•
This provides more available alphanumeric tokens in the vocabulary
•
While this may increase total token count, it improves semantic representation
•
Our design philosophy favors representation quality over token count minimization
•
For example, space (' ') is broken as a separate token in our system compared to being concatenated in standard methods like OpenAI's
•
This approach results in better word representations despite potentially larger token counts
•
While choosing a combination-based tokenizer may reduce token count, our focus on representation offers semantic advantages
•
Combining special tokens with alphanumeric ones adds less semantic value than using pure alphanumeric tokens
""", unsafe_allow_html=True) # Footer link st.markdown("""

Need a programmatic interface for tokenizing text? Check out our tokeniser-py package for Python.

""", unsafe_allow_html=True) # Footer with additional information st.markdown("---") st.markdown("""

About tokeniser-py

A high-performance, fully custom tokeniser built from scratch — no BPE, no existing NLP tokenisation scheme. This tokeniser is based on a unique algorithm developed independently and trained on over 1 billion tokens from the SlimPajama dataset (Val + Test), providing an efficient, interpretable, and extendable tokenisation pipeline.
•
Tokeniser built on a vocabulary of 131,072 tokens
•
Two versions of vocab: 0.5B (Validation-only data) and 1B (Validation + Test data)
•
Token vocab built via a custom algorithm — no Byte Pair Encoding (BPE)
•
Lightweight JSON format for token maps & token count maps
•
Ready for integration into any LLM pre-tokenisation pipeline
[GitHub Repository](https://github.com/Tasmay-Tibrewal/tokeniser-py) | [PyPI Package](https://pypi.org/project/tokeniser-py/) """, unsafe_allow_html=True) import streamlit as st # Add explanation of the library in expandable section with st.expander("Learn more about tokeniser-py"): st.markdown(""" ### 🚀 What This Library Offers - Tokeniser built on a vocabulary of **131,072 tokens** - Two versions of vocab: - `0.5B`: Validation-only data - `1B`: Validation + Test data - Token vocab built via a **custom algorithm** — no Byte Pair Encoding (BPE) - Tokenisation logic includes: - Token lookup from pre-generated token map - Dynamic programming-based segmentation for out-of-vocab tokens - One-hot encoding (NumPy or PyTorch) - Visualisation utilities for tokens and token IDs - Lightweight JSON format for token maps & token count maps - Ready for integration into any LLM pre-tokenisation pipeline """) # Add custom CSS st.markdown(""" """, unsafe_allow_html=True) # Code header and block with simpler HTML st.markdown("""
🛠️ Usage
from tokeniser import Tokeniser

t = Tokeniser()

tokens, count = t.tokenise("Your input text here.")

token_ids = t.token_ids(tokens)
""", unsafe_allow_html=True) st.markdown(""" Use `t.one_hot_tokens(token_ids)` for NumPy-based one-hot encoding, or `op='torch'` for PyTorch. ### 📁 Vocab Files - `ordered_tokenizer_1b_val_test_data.json` — Ordered tokens (1B data) - `unordered_tokenizer_1b_val_test_data.json` — Unordered tokens (1B) - `count_tokenizer_1b_val_test_data.json` — Token counts (1B) - Similar structure for 0.5B val-only version """)