tldt

tldt (Too Long; Didn't Tokenize) solves a practical problem in LLM pipelines: inputs that are too long, too noisy, or too dangerous to pass directly to a model. It uses extractive summarization (LexRank and TextRank) to reduce token count on API payloads, uploaded documents, and crawled web content — no API key or external service required.

Beyond compression, it handles several security concerns that the OWASP LLM Top 10 flags: prompt injection detection, Unicode confusable-character attacks, PII stripping, and accidental API key exposure in text. It also converts HTML to clean Markdown and sanitizes raw text before summarization.

The tool ships as both a standalone CLI and a Go library, making it easy to drop into agents or backend services that call AI APIs directly. Useful for developers building RAG pipelines, document ingestion workflows, or any system where uncontrolled input length and content quality are real operational risks.

Developers building RAG pipelines and agent backends use tldt to shrink and sanitize oversized inputs before they reach an LLM.