Cli Modelarium

Cli Modelarium is an AI developer tool that compares large language models with real statistics from the terminal. Developers provide a prompt, pick the models, and get side-by-side comparisons with performance metrics and cost tracking without setting up any infrastructure. It supports 10 cloud providers — OpenAI, Anthropic, Google, xAI, DeepSeek, Mistral, Groq, OpenRouter, Alibaba/DashScope, and Z.AI — plus local models, and offers group shortcuts like all-flagship, all-budget, and all-fast to run a batch in one command. Where it goes beyond a simple diff is statistical rigor: it runs bootstrap confidence intervals and paired significance tests, includes hallucination detection and LLM-as-judge evaluation panels, and enforces hard spending caps so a comparison run can't blow past a budget. Output is available as JSON and CSV, so results pipe cleanly into other tooling. It runs on Linux, macOS, and Windows and requires Python 3.11+. Installation is a single command, pip install cli-modelarium. The project is open source under the Apache 2.0 license and free to use.

Researchers, developers, and data scientists use it to compare AI language models from the command line with statistically reliable benchmarks.