tldt — Too Long, Didn't Tokenize

tldt — Too Long, Didn't Tokenize is an AI text summarization tool that shrinks long documents into their essential sentences before they hit an LLM, cutting token costs for developers and AI engineers. It runs entirely locally as a CLI or Go library with no LLM calls, no API keys, and no token costs, using graph-based extractive algorithms (LexRank, TextRank, an ensemble mode, and a baseline) to select the most representative sentences. It reports up to 97% token reduction and outputs JSON, plain text, or markdown, making it easy to pipe into coding assistants like Claude Code, Cursor, and OpenCode. tldt also fetches URLs and converts HTML to markdown, so crawled pages and API docs can be summarized in place. Because it targets untrusted external content, it adds security preprocessing: prompt-injection detection aligned with the OWASP LLM Top 10, PII and API-key redaction, and Unicode confusables/homoglyph protection. ROUGE evaluation support lets you measure summary quality. Typical uses include compressing transcripts, articles, API documentation, and crawled sites before feeding agents that call AI APIs directly. It is free and open source under the MIT License.

Developers and AI engineers use tldt to locally compress long documents and crawled pages into token-efficient summaries before feeding them to LLMs and coding assistants.