What is tldt?
tldt (Too Long; Didn't Tokenize) solves a practical problem in LLM pipelines: inputs that are too long, too noisy, or too dangerous to pass directly to a model. It uses extractive summarization (LexRank and TextRank) to reduce token count on API payloads, uploaded documents, and crawled web content — no API key or external service required.
Beyond compression, it handles several security concerns that the OWASP LLM Top 10 flags: prompt injection detection, Unicode confusable-character attacks, PII stripping, and accidental API key exposure in text. It also converts HTML to clean Markdown and sanitizes raw text before summarization.
The tool ships as both a standalone CLI and a Go library, making it easy to drop into agents or backend services that call AI APIs directly. Useful for developers building RAG pipelines, document ingestion workflows, or any system where uncontrolled input length and content quality are real operational risks.
What is tldt used for?
Developers building RAG pipelines and agent backends use tldt to shrink and sanitize oversized inputs before they reach an LLM.
How much does tldt cost?
Pros and cons
Pros
• Reduces token count with extractive summarization (LexRank and TextRank) — no API keys or LLM calls • Detects prompt injection and Unicode confusable-character attacks • Strips PII and flags accidental API-key exposure in text • Ships as both a standalone CLI and a Go library for easy integration
Cons
• Extractive summarization keeps original sentences rather than rewriting, so it's less fluent than LLM summaries • Go-centric; teams in other stacks integrate via the CLI • Aimed at developers, not non-technical users
Questions about tldt
Does tldt call an LLM or need an API key?
No. It uses local extractive summarization (LexRank/TextRank), so there are no API keys, LLM calls, or token costs.
What security issues does tldt handle?
It addresses several OWASP LLM Top 10 concerns — prompt injection detection, Unicode confusable attacks, PII stripping, and API-key exposure.
How do I integrate tldt?
It ships as a standalone CLI and a Go library, so it drops into agents or backend services that call AI APIs directly.
What problem does tldt solve?
It trims inputs that are too long, noisy, or unsafe before they reach an LLM, cutting token count and reducing operational risk.
What are the best alternatives to tldt?
Reviews
Sign in to rate and review tldt.
Sign in to reviewNo reviews yet. Be the first to review tldt!