NETRA LLM Gateway & Semantic Cache

NETRA (Sanskrit: नेत्र / Vision / Aperture / Insight) is an enterprise-grade, sovereign AI Gateway and tenant-isolated Semantic Vector Cache. Sitting in the critical path between application microservices and foundation model providers, NETRA delivers predictable latency, multi-provider fast-failover, race-safe atomic admission control, and sub-5ms semantic cache hits.

Key Platform Guarantees¶

• Universal OpenAI Compatibility: Direct drop-in proxy for /v1/chat/completions requiring zero client SDK modifications.
• Two-Tier Semantic Vector Cache: Sub-0.5ms exact hash near-cache (Tier 1) and sub-5ms cosine similarity retrieval (Tier 2) with cryptographic per-tenant boundaries at $0.000000 upstream cost.
• Atomic Admission Control: Single-transaction Redis 8.0 sliding-window rate limiting ensuring 0.00% over-admission under high-concurrency bursts, compliant with standardized IETF RateLimit protocol headers.
• Dynamic Cost Arbitrage: Syntactic difficulty classifier (auto tier) routing simple queries to fast models and complex tasks to frontier reasoning models, achieving up to ~72.4% cost savings.
• Synchronous Upstream Failover & Resilience: Resilience4j circuit breakers (<0.05ms fast-failover), multi-provider fallback chains (fast → primary → fallback), small-model refusal auto-escalation, and zero-glitch SSE streaming failover.
• Disconnect Protection & Instant Reclamation: Instant detection of client socket drops (Broken pipe) on Java 25 Virtual Threads, severing upstream compute in <2ms to halt phantom spend, with automatic rate-limit quota rollback on zero-token aborts.
• Real-Time Financial Telemetry: Exact 6-decimal micro-cost accounting (BigDecimal) in USD with live Forex conversion (USD / INR).

Core Architecture & Technology Stack¶

Pillar	Technology	Role in Ingress Pipeline
Runtime Engine	Java 25 LTS (Virtual Threads / Loom)	High-throughput unbuffered SSE streaming relay & socket cancellation detection
Framework	Spring Boot 4.1.1	Graceful shutdown connection draining, actuator probes & observability
Vector Storage	PostgreSQL 18 & pgvector	Dense vector embeddings (384-dim) with HNSW indexing for sub-5ms cosine caching
Admission Control	Redis 8.0 Alpine (Lua Scripting)	Single-transaction sliding-window rate limiting & atomic budget validation (0.00% over-admission)
Operations Console	React 19 / Vite 6	Real-time SSE audit logging, multi-currency accounting (USD/INR), cache inspector, and modernized sandbox with 1-click router badges (fast/smart/auto) & Stream/JSON toggle
Containerization	Docker Compose & Cloud VPS	Multi-service orchestration with zero-open-port Cloudflare Zero Trust tunnel support

Architectural Topology¶

Canonical Telemetry Contract¶

Every completion response emitted by NETRA strictly carries canonical x-netra-* telemetry headers:

Standardized IETF RateLimit Headers¶

In compliance with draft-ietf-httpapi-ratelimit-headers, all responses carry standardized admission metadata:

Quickstart in 60 Seconds¶

1. Launch Backing Services & Gateway¶

git clone https://github.com/shashankch/netra-llmgateway.git
cd netra-llmgateway
make up

Verify service readiness:

curl -f http://localhost:8080/health
# {"status":"ok","service":"netra-gateway","version":"0.7.0"}

2. Dispatch a Chat Completion¶

curl -i -X POST http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer netra-sk-search-1a2b3c" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "fast",
    "messages": [
      {"role": "user", "content": "What is a message queue?"}
    ]
  }'

3. Launch & Serve Operations Console UI¶

Open the real-time telemetry dashboard, semantic cache inspector, and model playground:

# Docker Compose (included in make up)
http://localhost:3000

# Standalone Development (Vite hot-reload)
make console-dev        # or: cd console && npm install && npm run dev

# Production Build & Preview Server
make console-build
make console-serve      # or: cd console && npm run preview

Explore Technical Documentation¶

• 🏛️ System Architecture — Ingress stages, pipeline flow, and virtual threads execution topology.
• 📋 Architecture Decisions (ADRs) — Architectural decision records, evaluated options, and engineering rationale.
• ⚖️ Gateway Comparison — Architectural tradeoffs against direct vendor APIs and public SaaS aggregators.
• 📡 REST API Reference — Complete OpenAPI 3.1 endpoint schemas, error envelopes, and request bodies.
• 📊 Verification & Benchmark Report — Empirical verification, load benchmarks, 0.00% over-admission guarantees, and routing accuracy results.
• 🚀 Self-Hosting Guide — Turnkey deployment blueprints for Docker, Railway, Render, Fly.io, and Cloud VPS.
• 🗺️ Product Roadmap — Platform engineering milestones and upcoming product releases.
• 🛡️ Security Policy — Vulnerability reporting, security posture, and disclosure guidelines.
• 🤝 Contributing Guidelines — Development workflow, code standards, and verification suites.