tokio-prompt-orchestrator

Rust LLM request orchestrator and drop-in OpenAI-compatible proxy. Requests flow through a bounded five-stage pipeline with deduplication (the same prompt asked twice costs one model call), circuit breakers, retries with jitter, rate limits, a spend cap and a dead-letter queue, in front of Anthropic, OpenAI, llama.cpp or vLLM. Ships as a library, a server binary, REST/SSE, an MCP server and a TUI.