Skip to content

Commit 6967597

Browse files
docs(readme): overhaul README with visual assets, Docker quick-start, and storytelling (#5)
Rewrite the README to tell a compelling story about why ProtoMQ exists and what makes it useful for IoT/edge deployments: - New tagline emphasizing MQTT + Protobuf + Zig synergy - Added SVG payload comparison chart (JSON vs Protobuf, 3 scenarios) - Added SVG terminal demo showing the publish/subscribe workflow - Mermaid architecture diagram of the broker internals - Reframed performance benchmarks with context - Docker quick-start with Dockerfile and docker-compose.yml - Condensed Service Discovery and Admin Server sections - Navigation links and FAQ reference - Added .blog/ to .gitignore for draft social media posts Signed-off-by: Gyokhan Kochmarla <gokhan.kocmarli@gmail.com>
1 parent bb95021 commit 6967597

7 files changed

Lines changed: 346 additions & 63 deletions

File tree

.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,3 +36,6 @@ coverage/
3636
# Dependency cache
3737
deps/
3838
vendor/
39+
40+
# Blog drafts (not part of the repo)
41+
.blog/

Dockerfile

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
FROM alpine:3.21 AS builder
2+
3+
# Install Zig 0.15.2 and build dependencies.
4+
RUN apk add --no-cache curl xz && \
5+
curl -L https://ziglang.org/download/0.15.2/zig-x86_64-linux-0.15.2.tar.xz | \
6+
tar -xJ -C /opt && \
7+
mv /opt/zig-x86_64-linux-0.15.2 /opt/zig
8+
ENV PATH="/opt/zig:${PATH}"
9+
10+
WORKDIR /src
11+
COPY build.zig build.zig.zon ./
12+
COPY src/ src/
13+
COPY schemas/ schemas/
14+
15+
RUN zig build -Doptimize=ReleaseSafe -Dtarget=x86_64-linux
16+
17+
# --- Runtime stage ---
18+
FROM alpine:3.21
19+
20+
RUN addgroup -S protomq && adduser -S protomq -G protomq
21+
22+
WORKDIR /opt/protomq
23+
24+
COPY --from=builder /src/zig-out/bin/protomq-server ./bin/protomq-server
25+
COPY --from=builder /src/zig-out/bin/protomq-cli ./bin/protomq-cli
26+
COPY schemas/ ./schemas/
27+
28+
USER protomq
29+
30+
EXPOSE 1883
31+
32+
ENTRYPOINT ["./bin/protomq-server"]

FEATURES.md

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# Features
2+
3+
This document covers ProtoMQ's features in more depth than the README. If you're looking for deployment guides and configuration, check [FAQ.md](FAQ.md).
4+
5+
---
6+
7+
## Service Discovery
8+
9+
ProtoMQ includes a built-in Service Discovery mechanism. Clients can discover available topics and their associated Protobuf schemas — including the full `.proto` source code — by querying the `$SYS/discovery/request` topic.
10+
11+
This lets new clients bootstrap themselves in a single round-trip without any out-of-band configuration or pre-shared schema files.
12+
13+
**Using the CLI:**
14+
15+
```bash
16+
protomq-cli discover --proto-dir schemas
17+
```
18+
19+
Under the hood, the client subscribes to `$SYS/discovery/request` and receives a `ServiceDiscoveryResponse` message containing every registered topic-schema mapping. The response includes the raw `.proto` source so clients can dynamically configure their own decoding logic.
20+
21+
---
22+
23+
## Admin Server
24+
25+
An optional HTTP server for runtime schema management and telemetry. Disabled by default — when the build flag is off, the HTTP code is **completely stripped from the binary** (zero overhead, not just disabled).
26+
27+
### Enabling it
28+
29+
```bash
30+
zig build -Dadmin_server=true run-server
31+
```
32+
33+
### Build comparison
34+
35+
| Build | Memory baseline | Admin API |
36+
|---|---|---|
37+
| `zig build` | ~2.6 MB ||
38+
| `zig build -Dadmin_server=true` | ~4.0 MB ||
39+
40+
The Admin Server runs cooperatively on the same event loop as the MQTT broker — enabling it does **not** degrade per-message MQTT performance.
41+
42+
### Endpoints
43+
44+
All endpoints require `Authorization: Bearer <ADMIN_TOKEN>` (defaults to `admin_secret`, configurable via the `ADMIN_TOKEN` environment variable).
45+
46+
| Method | Path | Description |
47+
|---|---|---|
48+
| `GET` | `/metrics` | Active connections, message throughput, loaded schemas |
49+
| `GET` | `/api/v1/schemas` | Current topic-to-schema mappings |
50+
| `POST` | `/api/v1/schemas` | Register a new `.proto` schema and map it to a topic at runtime |
51+
52+
### Dynamic schema registration
53+
54+
With the Admin Server enabled, you can register schemas at runtime without restarting the broker:
55+
56+
```bash
57+
curl -X POST http://127.0.0.1:8080/api/v1/schemas \
58+
-H "Authorization: Bearer ${ADMIN_TOKEN:-admin_secret}" \
59+
-H "Content-Type: application/json" \
60+
-d '{
61+
"topic": "telemetry/gps",
62+
"message_type": "GpsCoordinate",
63+
"proto_file_content": "syntax = \"proto3\";\nmessage GpsCoordinate { float lat = 1; float lon = 2; }"
64+
}'
65+
```
66+
67+
The schema is parsed in-process and persisted to disk as `schemas/<MessageType>.proto`. The mapping is live immediately — no restart needed.
68+
69+
> **Security note**: The Admin Server binds to `127.0.0.1:8080` only. If you need remote access, use an SSH tunnel or reverse proxy.

README.md

Lines changed: 86 additions & 63 deletions
Original file line numberDiff line numberDiff line change
@@ -2,100 +2,123 @@
22
<h1 align="center">ProtoMQ</h1>
33

44
<p align="center">
5-
<img src="assets/mascot.png" alt="ProtoMQ Mascot" width="300px" />
5+
<img src="assets/mascot.png" alt="ProtoMQ Mascot" width="280px" />
66
<br />
7-
<b>Type-safe, bandwidth-efficient MQTT for the rest of us.</b>
7+
<b>MQTT's simplicity. Protobuf's efficiency. Zig's bare-metal performance.</b>
88
<br />
9-
<i>Stop sending bloated JSON over the wire.</i>
9+
Built for IoT and edge computing.
10+
</p>
11+
12+
<p align="center">
13+
<a href="#quick-start">Quick Start</a> •
14+
<a href="#why-protomq">Why ProtoMQ</a> •
15+
<a href="#performance">Performance</a> •
16+
<a href="FEATURES.md">Features</a> •
17+
<a href="FAQ.md">FAQ</a>
1018
</p>
1119

1220
---
13-
- MQTT v3.1.1 packet parsing (CONNECT, PUBLISH, SUBSCRIBE, etc.)
14-
- Thread-safe Topic Broker with wildcard support (`+`, `#`)
15-
- Custom Protobuf Engine with runtime `.proto` schema parsing
16-
- Topic-based Protobuf schema routing
17-
- Service Discovery & Schema Registry
18-
- CLI with automatic JSON-to-Protobuf encoding
19-
- Structured diagnostic output for Protobuf payloads
2021

21-
### Building
22+
ProtoMQ is an MQTT broker that enforces **Protobuf schemas at the broker level**. All messages on the wire are Protobuf — the broker validates incoming payloads against registered `.proto` schemas and rejects anything that doesn't conform. The bundled CLI can accept JSON and encode it to Protobuf client-side for convenience.
2223

23-
One have to have Zig 0.15.2 or later installed. Please download it from [here](https://ziglang.org/download/).
24+
<p align="center">
25+
<img src="assets/terminal_demo.svg" alt="ProtoMQ terminal demo" width="780px" />
26+
</p>
2427

25-
```bash
26-
# Build server and client
27-
zig build
28+
- **Schema-enforced messaging**`.proto` files define the contract. Malformed payloads get rejected *before* they reach subscribers.
29+
- **Custom Protobuf engine** — written from scratch in Zig. Runtime `.proto` parsing, zero external dependencies.
30+
- **Wildcard topic routing** — full MQTT `+` and `#` wildcard support via a trie-based topic broker.
31+
- **Service Discovery** — clients query `$SYS/discovery/request` to discover topics and download schemas on the fly. No pre-shared `.proto` files needed.
32+
- **Optional Admin HTTP API** — register new schemas and topic mappings at runtime, monitor connections and throughput. Disabled by default, zero overhead when off. See [FEATURES.md](FEATURES.md) for details.
33+
- **Runs in 2.6 MB** — the entire broker with 100 active connections fits in under 3 MB of memory.
2834

29-
# Build and run server
30-
zig build run-server
35+
---
3136

32-
# Build and run client
33-
zig build run-client
37+
### Why ProtoMQ
3438

35-
# Run tests
36-
zig build test
39+
If you've worked with IoT sensor fleets, you've probably been through this: you start with JSON over MQTT because it's easy to debug, every language has a parser, and `mosquitto_sub` lets you eyeball what's going on. It works fine... until you start caring about bandwidth.
3740

38-
# Run all integration tests
39-
./tests/run_all.sh
40-
```
41+
A 12-field sensor reading weighs around 310 bytes in JSON. The same data in Protobuf: 82 bytes. On a cellular-connected device pushing telemetry every 5 seconds, that gap adds up to roughly 1.6 MB/day per device — multiply by a few thousand devices and the data bill starts hurting.
4142

42-
### Limitations
43+
<p align="center">
44+
<img src="assets/payload_comparison.svg" alt="JSON vs Protobuf payload size comparison" width="680px" />
45+
</p>
4346

44-
For the initial release, we support:
45-
- QoS 0 only (at most once delivery)
46-
- No persistent sessions
47-
- No retained messages
48-
- Single-node deployment
47+
But switching to Protobuf usually means code generation per language, keeping stubs in sync across firmware versions, and losing the ability to just read your payloads. ProtoMQ takes a different approach: the broker owns the `.proto` schemas and validates every message against them. The CLI can accept JSON and encode it to Protobuf before publishing, so you get a human-friendly workflow without sacrificing wire efficiency.
4948

50-
### Service Discovery
49+
| | Plain MQTT + JSON | ProtoMQ |
50+
|---|---|---|
51+
| Schema enforcement | None — anything goes | Validated at every `PUBLISH` |
52+
| Payload format | JSON (~170 bytes, 8 fields) | Protobuf (~48 bytes) |
53+
| Client bootstrap | Pre-shared docs | Built-in Service Discovery |
54+
| Code generation | Required per language | CLI encodes JSON → Protobuf for you |
5155

52-
ProtoMQ includes a built-in Service Discovery mechanism. Clients can discover available topics and their associated Protobuf schemas (including the full source code) by querying the `$SYS/discovery/request` topic.
56+
---
5357

54-
**Using the CLI for discovery:**
55-
```bash
56-
# Verify schemas are loaded and available
57-
protomq-cli discover --proto-dir schemas
58-
```
59-
This allows clients to "bootstrap" themselves without needing pre-shared `.proto` files.
58+
### Under the Hood
6059

61-
### Admin Server
60+
ProtoMQ is not a wrapper around an existing broker — it's a ground-up implementation. Here's what makes it tick:
6261

63-
ProtoMQ includes an optional HTTP Admin Server for runtime observability and dynamic schema management without polluting the core MQTT hot-paths.
62+
- **`epoll` / `kqueue` event loop** — single-threaded, no abstraction layer. The network layer talks directly to the OS kernel I/O primitives. On Linux that's `epoll`, on macOS `kqueue`. No libuv, no tokio, no hidden threads.
63+
- **One allocator, full control** — every allocation goes through Zig's `std.mem.Allocator`. No GC, no hidden heap churn, no runtime. You can trace every byte the broker touches.
64+
- **Zero third-party dependencies** — the MQTT parser, TCP connection handler, Protobuf wire format encoder, `.proto` file parser — all written in Zig using only the standard library. `build.zig.zon` has an empty `dependencies` block.
65+
- **Runtime schema registry**`.proto` files are parsed at startup and mapped to MQTT topics. With the Admin Server enabled, you can register new schemas and mappings at runtime over HTTP without restarting the broker.
66+
- **Comptime-generated lookup tables** — the MQTT packet parser uses Zig's `comptime` to build dispatch tables at compile time. No branching, no hash maps — just array indexing.
67+
- **Cross-compilation**`zig build -Dtarget=aarch64-linux` produces a Raspberry Pi binary from a Mac. One command, no toolchain headaches.
6468

65-
- **Dynamic Schema Registration**: Register `.proto` files at runtime via `POST /api/v1/schemas`.
66-
- **Telemetry**: Monitor active connections, message throughput, and schemas via `GET /metrics`.
67-
- **Zero Overhead Footprint**: The Admin Server is disabled by default to preserve the absolute minimum memory footprint for embedded devices. It is strictly conditionally compiled via the `zig build -Dadmin_server=true` flag. Enabling it moderately increases the initial static memory baseline (e.g., from ~2.6 MB to ~4.0 MB) by safely running a parallel HTTP listener, but it executes cooperatively on the same event loop ensuring zero degradation to per-message MQTT performance. When the flag is deactivated, it incurs **zero overhead footprint**.
69+
---
6870

69-
### Performance Results
71+
### Quick Start
7072

71-
ProtoMQ delivers high performance across both high-end and edge hardware:
73+
**Docker:**
7274

73-
| Scenario | Apple M2 Pro | Raspberry Pi 5 |
74-
|----------|--------------|----------------|
75-
| Latency (p99, 100 clients) | 0.44 ms | 0.13 ms |
76-
| Concurrent clients | 10,000 | 10,000 |
77-
| Sustained throughput | 9k msg/s | 9k msg/s |
78-
| Message throughput (small) | 208k msg/s | 147k msg/s |
79-
| Memory (100 clients) | 2.6 MB | 2.5 MB |
75+
```bash
76+
docker compose up
77+
```
8078

81-
Handles 100,000 connection cycles with zero memory leaks and sub-millisecond latency.
79+
The server starts on port `1883` with the schemas from `schemas/`. Connect with any MQTT client.
8280

83-
For detailed methodology and full results, see [ProtoMQ Benchmarking Suite](benchmarks/README.md).
81+
**From source** (requires [Zig 0.15.2+](https://ziglang.org/download/)):
8482

85-
### Contributing
83+
```bash
84+
git clone https://github.com/electricalgorithm/protomq.git
85+
cd protomq
86+
zig build run-server
87+
```
8688

87-
This is currently a learning/development project. Contributions will be welcome after the MVP is complete.
89+
```bash
90+
# In another terminal — publish (CLI encodes JSON to Protobuf for you)
91+
zig build run-client -- publish --topic sensors/temp \
92+
--json '{"device_id":"s-042","temperature":22.5,"humidity":61,"timestamp":1706140800}'
8893

89-
### License
94+
# In another terminal — subscribe
95+
zig build run-client -- subscribe --topic "sensors/#"
96+
```
9097

91-
The project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
98+
---
9299

93-
### Resources
100+
### Performance
94101

95-
- [Zig Documentation](https://ziglang.org/documentation/master/)
96-
- [MQTT v3.1.1 Specification](http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqtt-v3.1.1.html)
97-
- [Protocol Buffers](https://protobuf.dev/)
102+
ProtoMQ handles **208,000 messages/second** on an Apple M2 Pro and **147,000 msg/s** on a Raspberry Pi 5 — with sub-millisecond p99 latency and no memory leaks across 100,000 connection cycles.
103+
104+
| Scenario | Apple M2 Pro | Raspberry Pi 5 |
105+
|----------|--------------|----------------|
106+
| **p99 latency** (100 clients) | 0.44 ms | 0.13 ms |
107+
| **Throughput** (10-byte msgs) | 208k msg/s | 147k msg/s |
108+
| **Throughput** (64 KB msgs) | 39k msg/s | 27k msg/s |
109+
| **Sustained load** (10 min) | 8,981 msg/s | 9,012 msg/s |
110+
| **Memory** (100 connections) | 2.6 MB | 2.5 MB |
111+
| **Connection churn** (100k cycles) | 1,496 conn/s | 1,548 conn/s |
112+
| **Memory leaks** | 0 MB | 0 MB |
113+
114+
All benchmarks run on loopback, `ReleaseSafe` mode, Zig 0.15.2. Methodology and raw results: [`benchmarks/`](benchmarks/README.md).
98115

99116
---
100117

101-
**Note**: This project is under active development. The API and architecture may change significantly.
118+
### Current Limitations
119+
120+
QoS 0 only (at most once delivery), no persistent sessions, no retained messages, single-node deployment. These are scope decisions for the initial release — multi-node and QoS 1/2 are on the roadmap.
121+
122+
### Contributing
123+
124+
Contributions are welcome. If you're interested in MQTT internals, Protobuf wire format, or systems programming in Zig, there's plenty to dig into. See [FEATURES.md](FEATURES.md) for the full feature set and [FAQ.md](FAQ.md) for deployment and configuration guides.

assets/payload_comparison.svg

Lines changed: 71 additions & 0 deletions
Loading

0 commit comments

Comments
 (0)