Cli Modelarium is a command-line interface (CLI) tool designed to compare AI language models with statistical accuracy and reliability. It supports eight cloud service providers, including OpenAI, Anthropic, Google, xAI, DeepSeek, Mistral, Groq, and OpenRouter, in addition to local model installations. Key features include bootstrap confidence intervals, paired significance tests for model comparison, hallucination detection, LLM-as-judge panels, cost tracking with hard caps, and more. Installation is straightforward via pip on Linux, macOS, and Windows systems running Python 3.11+. Ideal for researchers, developers, and data scientists seeking to evaluate AI models rigorously.