Why I built Dhara

I wanted to understand how distributed task queues work beyond the abstraction.

I had not actually used a task queue before building Dhara. Instead of starting with an existing implementation, I decided to build one from scratch in Go and PostgreSQL to understand the mechanics underneath it.

That led me to implement task claiming, worker pools, heartbeats, stale task recovery, retries with exponential backoff and full jitter, idempotency, dead-letter handling, transactional enqueueing, and Prometheus-compatible metrics.

The result is both a reusable Go library and a set of pre-built services. A Go application can embed Dhara directly, while the HTTP server and worker binaries provide a ready-to-run deployment when an API boundary is useful.

Architecture

PostgreSQL as the Queue Backend

Dhara uses PostgreSQL as the single source of truth for task state instead of relying on a separate message broker.

Workers claim pending tasks using PostgreSQL’s SELECT ... FOR UPDATE SKIP LOCKED, allowing multiple workers to claim tasks concurrently without claiming the same task twice.

This also keeps task storage and application data in the same database, which enables transactional guarantees that would otherwise require coordination between separate systems.

Library First

The core of Dhara is an embeddable Go library.

Client handles task submission and management, while Worker handles task execution, heartbeats, retries, and stale task recovery. The HTTP API and worker binary are thin layers built on top of these same library components.

This means a Go application can enqueue tasks directly without running an HTTP server.

Transactional Enqueueing

Dhara supports enqueueing a task inside the caller’s existing PostgreSQL transaction.

For example, an application can create an order and enqueue a confirmation email in the same transaction. If the transaction rolls back, the task does not exist. If it commits, both the business data and task are committed together.

Dhara also supports idempotency keys so repeated submissions can safely return an existing task instead of creating duplicates.

Worker Pool and Task Lifecycle

A configurable pool of goroutines claims and executes tasks concurrently.

The lifecycle is persisted in PostgreSQL:

PENDING -> RUNNING -> COMPLETED

Failed tasks are retried with exponential backoff and full jitter. Tasks that exhaust their retry limit move to DEAD and can be manually retried.

Tasks can also be scheduled for later execution, prioritized, cancelled, and queried through the client or HTTP API.

Heartbeats and Reaper

Running tasks periodically update their heartbeat in PostgreSQL.

If a worker crashes or becomes stuck, the reaper detects tasks whose heartbeats have become stale. The task is either requeued for another attempt or moved to the dead-letter state when its retry limit has been exhausted.

This allows the queue to recover from worker failures without leaving tasks permanently stuck in RUNNING.

Observability

Dhara exposes Prometheus-format metrics for task lifecycle and worker state, including:

• tasks enqueued, completed, retried, and dead
• queue size by task status
• total workers
• workers currently processing tasks

The service also uses Go’s slog for structured logging and provides liveness and readiness endpoints.

Benchmarking

I load tested Dhara with k6 while using PostgreSQL as the source of truth for task state.

The test used 20 workers and a handler simulating 50-200ms of I/O work.

The first result was only about 20 tasks/sec, despite an expected throughput of roughly 160 tasks/sec.

The bottleneck turned out to be a worker claim loop that attempted to claim exactly one task per poll interval. After changing the worker to continuously claim while work was available and only wait when the queue was empty, throughput increased to about 148 tasks/sec.

At 100 tasks/sec, Dhara completed all 6,001 tasks with:

• p50: 181ms
• p95: 299ms
• p99: 380ms

At approximately 148 tasks/sec, near worker capacity:

• p50: 1.30s
• p95: 2.03s
• p99: 2.06s

Both runs completed every task with zero task loss.

The benchmark also reinforced why enqueue latency alone is not enough to evaluate a task queue. The HTTP endpoint remained fast even while workers were falling behind. Measuring task completion directly in PostgreSQL exposed the actual bottleneck.

Design Decisions

• PostgreSQL only. No Redis, RabbitMQ, or external broker. PostgreSQL provides durable task state, transactional guarantees, and concurrent task claiming.
• Library first. The core functionality is reusable from Go applications, while the HTTP API and worker are optional service layers.
• Standard library HTTP server. Routing uses Go’s standard net/http instead of a web framework.
• Minimal dependencies. The project avoids ORMs, queue libraries, and unnecessary framework dependencies.
• Transactional enqueueing. Tasks can be committed atomically with the business operation that produced them.
• Correctness before optimization. Failure recovery, retries, idempotency, and graceful shutdown are treated as core queue semantics rather than optional features.

Tech Stack

Status

Dhara’s core task queue functionality is implemented and usable as a Go library or as pre-built HTTP server and worker services.

The project is still evolving, with planned work around richer queue latency metrics, dashboards, cancellation semantics, health checks, validation, and test coverage.

• Source: https://github.com/Md-Talim/dhara
• Stack: Go · PostgreSQL · Docker · Prometheus · slog