Helix

LLM observability: a small SDK wraps multi-provider inference and streams latency, token and error data through Kafka to TimescaleDB, off the response path.

Status: Public repo
Role: Full Stack Developer
Team: Solo

GitHub

The problem

LLM applications need visibility into latency, token usage, and errors, but the logging must not slow down the response the user is waiting for.

Who it is for

Teams building LLM-powered applications.

What I built

An SDK that wraps multi-provider LLM calls (Anthropic, OpenAI, Google, and 300+ models via OpenRouter) and ships latency, token, error, and session metadata through Kafka to an ingestion pipeline backed by PostgreSQL and TimescaleDB. Grafana dashboards show real-time latency, throughput, and error metrics. A Next.js chatbot demonstrates the SDK.

Built with

Frontend: React
Next.js 16
TanStack Query
Tailwind CSS
SSE
Backend: Node.js
Fastify 5
Zod
TypeScript
Data: PostgreSQL
TimescaleDB
Redis
Drizzle
Messaging: Redpanda (Kafka)
LLM Providers: Anthropic
OpenAI
Gemini
OpenRouter
Observability: Prometheus
Grafana
Infra: Docker Compose
AWS EKS
Helm
Turborepo
pnpm

In detail

Models via one OpenRouter key: 300+
Providers behind one client: 4
Added to response path: 0ms
Full-stack Docker boot: 1 cmd

Architecture

A Turborepo pnpm monorepo: three apps and three shared packages.

apps/
  web/         Next.js 16 chat UI (App Router, Turbopack)
  api/         Fastify API gateway: conversation CRUD, SSE streaming
  ingestion/   Kafka consumer + PostgreSQL writer
packages/
  sdk/         Unified LLM client (Anthropic, OpenAI, Gemini, OpenRouter) + redaction
  db/          Drizzle schema + TimescaleDB hypertable SQL
  types/       Shared Zod schemas + inferred TypeScript types
infra/
  docker-compose.yml   one-run local stack
  k8s/                 Helm charts for EKS (one per service)
  grafana/             dashboards + provisioning

Data flows one way to the user and out-of-band to storage:

User -> Next.js (SSE) -> Fastify API Gateway -> SDK Wrapper -> LLM Provider
                                                    |
                                         Kafka: llm.inference.logs
                                                    |
                                       Ingestion Service -> PostgreSQL -> Grafana

Decision

Decoupled write path

The SDK emits one Kafka event per call, fire-and-forget, and ingestion consumes and persists it out-of-band. The gateway never writes inference_logs directly. Kafka is the buffer: an ingestion slowdown grows consumer lag rather than dropping data or stalling replies, and consumers scale independently in one consumer group.

Ingestion flow

Call completes in the SDK
The SDK measures firstTokenMs and latencyMs, builds one versioned KafkaLogPayload envelope (schemaVersion, eventId, emittedAt plus call metadata), and redacts the previews.
Fire-and-forget emit
The payload is emitted to the llm.inference.logs topic with a 2000ms producer timeout. If Kafka is slow or down, the SDK logs to stderr and moves on: the user already has their response.
Validate and redact
apps/ingestion consumes the topic and runs the pipeline: JSON parse, KafkaLogPayloadSchema Zod validation, a second PII redaction layer, then insert via Drizzle.
Persist or dead-letter
The DB write is retried up to 3 times. Validation failures or persistent write failures route to the llm.inference.dlq topic with x-dlq-* headers carrying the original raw bytes.
Never stall
processMessage never throws: every path resolves to a write or a DLQ route, so the consumer offset always advances and the loop never stalls. Grafana reflects new calls within seconds.

Key features

Unified multi-provider SDK

One LLMClient interface over Anthropic, OpenAI, and Gemini direct SDKs plus OpenRouter, which reuses the openai SDK against openrouter.ai/api/v1 for one key and 300+ models. Pure TypeScript, no HTTP layer of its own.

Zero-latency observability

Inference metadata is captured and shipped without touching the response path. The emit is asynchronous and fire-and-forget, so logging can never slow or block the user reply.

TimescaleDB hypertable

inference_logs is a hypertable partitioned on request_at. Time-bucketed dashboard queries (p50/p95/p99 latency, throughput per minute) run orders of magnitude faster than on a plain table, with no change to the query API.

Streaming end to end

SSE from the browser through the gateway to the SDK, which intercepts the stream to measure firstTokenMs. A client disconnect mid-stream aborts the upstream LLM call and emits a cancelled log.

Two-layer PII redaction

Email, phone, SSN, and credit-card patterns are redacted once in the SDK before emit and again in ingestion before the DB write. A raw payload never reaches the database even if one layer is bypassed.

Graceful provider gating

hasProvider(name) lets the gateway refuse a request before hijacking the SSE response, returning 503 provider_not_configured so the client shows a friendly message instead of a silent stream close.

Redis context cache

A trailing 10-message window is read from ctx:{conversation_id} (24h TTL), removing a DB round-trip per message. A cold miss rebuilds the window from redacted DB history.

Conversation lifecycle

List, resume, and archive conversations from the UI. Cancel closes the SSE stream, persists partial output, and sets status='cancelled'. Cancel never deletes: a cancelled conversation stays viewable and posting to it returns 409.

Data model

Four PostgreSQL tables, Drizzle schema as the source of truth:

conversations: one row per chat session, holding provider, model, status.
messages: every user, assistant, and system turn, content always redacted.
inference_logs: one row per LLM API call, a TimescaleDB hypertable.
providers: provider configuration, name, base URL, active flag.

Note

No primary key on the hypertable

TimescaleDB hypertables cannot carry a primary key that excludes the partitioning column, so inference_logs keeps an id with a default but no PK constraint. Idempotency is handled upstream via the payload eventId rather than a database uniqueness guarantee.

Tradeoff

Fire-and-forget tradeoff

Guaranteeing the user response is never blocked means a Kafka outage drops log events rather than queueing them, so observability data can be lost during an incident. Acceptable because observability is non-critical relative to the user reply. A durable disk spool or outbox is the planned improvement.

Outcome

Helix is an LLM observability system with a unified multi-provider SDK, an event-driven ingestion pipeline with Zod validation and a dead-letter queue, a TimescaleDB hypertable tuned for time-series dashboard queries, and Grafana dashboards for latency, throughput, and errors. The central invariant holds throughout: the log write path never blocks the user response. The services are stateless, Kafka absorbs backpressure, and Helm charts cover deployment to AWS EKS.

Challenges

Capturing full inference metadata (latency, first-token time, tokens, status, session IDs) without adding any latency to the user-facing response path
TimescaleDB hypertable on inference_logs, which cannot carry a primary key excluding the partitioning column, so idempotency moves upstream to the Kafka payload eventId
A fire-and-forget log emit with a 2000ms producer timeout, so a broker outage degrades observability instead of blocking the reply
An ingestion processMessage that never throws: every path resolves to a DB write or a DLQ route so the kafkajs consumer offset always advances
Two-layer PII redaction (SDK before emit, ingestion before DB write) so a raw payload never reaches the database even if one layer is bypassed
SSE streamed end to end with SDK stream interception measuring first-token latency, plus client-disconnect abort of the upstream LLM call

What I learned

Decoupling the user response path from the log write path through Kafka lets the ingestion service scale independently in its own consumer group
A hypertable partitioned by request_at makes p50/p95/p99 latency and throughput range scans orders of magnitude faster without changing the query API
Redpanda over Kafka: single binary, no ZooKeeper for local dev, Kafka-compatible API so a production swap is config-only
One unified LLMClient over four providers with hasProvider(name) lets the gateway refuse a request up front with a 503 when a key is missing
Kafka as a buffer turns an ingestion slowdown into growing consumer lag rather than dropped data or stalled user responses
Drizzle schema as the single source of truth with drizzle-kit push, keeping the only hand-written SQL to the idempotent hypertable conversion

Have a role in mind?

Send me a message on X with the role and a link to the posting. My resume has the rest.

Send me a message Book a call

Building a product instead? See how I work with founders.