|
2 | 2 | <h1 align="center">ProtoMQ</h1> |
3 | 3 |
|
4 | 4 | <p align="center"> |
5 | | - <img src="assets/mascot.png" alt="ProtoMQ Mascot" width="300px" /> |
| 5 | + <img src="assets/mascot.png" alt="ProtoMQ Mascot" width="280px" /> |
6 | 6 | <br /> |
7 | | - <b>Type-safe, bandwidth-efficient MQTT for the rest of us.</b> |
| 7 | + <b>MQTT's simplicity. Protobuf's efficiency. Zig's bare-metal performance.</b> |
8 | 8 | <br /> |
9 | | - <i>Stop sending bloated JSON over the wire.</i> |
| 9 | + Built for IoT and edge computing. |
| 10 | +</p> |
| 11 | + |
| 12 | +<p align="center"> |
| 13 | + <a href="#quick-start">Quick Start</a> • |
| 14 | + <a href="#why-protomq">Why ProtoMQ</a> • |
| 15 | + <a href="#performance">Performance</a> • |
| 16 | + <a href="FEATURES.md">Features</a> • |
| 17 | + <a href="FAQ.md">FAQ</a> |
10 | 18 | </p> |
11 | 19 |
|
12 | 20 | --- |
13 | | -- MQTT v3.1.1 packet parsing (CONNECT, PUBLISH, SUBSCRIBE, etc.) |
14 | | -- Thread-safe Topic Broker with wildcard support (`+`, `#`) |
15 | | -- Custom Protobuf Engine with runtime `.proto` schema parsing |
16 | | -- Topic-based Protobuf schema routing |
17 | | -- Service Discovery & Schema Registry |
18 | | -- CLI with automatic JSON-to-Protobuf encoding |
19 | | -- Structured diagnostic output for Protobuf payloads |
20 | 21 |
|
21 | | -### Building |
| 22 | +ProtoMQ is an MQTT broker that enforces **Protobuf schemas at the broker level**. All messages on the wire are Protobuf — the broker validates incoming payloads against registered `.proto` schemas and rejects anything that doesn't conform. The bundled CLI can accept JSON and encode it to Protobuf client-side for convenience. |
22 | 23 |
|
23 | | -One have to have Zig 0.15.2 or later installed. Please download it from [here](https://ziglang.org/download/). |
| 24 | +<p align="center"> |
| 25 | + <img src="assets/terminal_demo.svg" alt="ProtoMQ terminal demo" width="780px" /> |
| 26 | +</p> |
24 | 27 |
|
25 | | -```bash |
26 | | -# Build server and client |
27 | | -zig build |
| 28 | +- **Schema-enforced messaging** — `.proto` files define the contract. Malformed payloads get rejected *before* they reach subscribers. |
| 29 | +- **Custom Protobuf engine** — written from scratch in Zig. Runtime `.proto` parsing, zero external dependencies. |
| 30 | +- **Wildcard topic routing** — full MQTT `+` and `#` wildcard support via a trie-based topic broker. |
| 31 | +- **Service Discovery** — clients query `$SYS/discovery/request` to discover topics and download schemas on the fly. No pre-shared `.proto` files needed. |
| 32 | +- **Optional Admin HTTP API** — register new schemas and topic mappings at runtime, monitor connections and throughput. Disabled by default, zero overhead when off. See [FEATURES.md](FEATURES.md) for details. |
| 33 | +- **Runs in 2.6 MB** — the entire broker with 100 active connections fits in under 3 MB of memory. |
28 | 34 |
|
29 | | -# Build and run server |
30 | | -zig build run-server |
| 35 | +--- |
31 | 36 |
|
32 | | -# Build and run client |
33 | | -zig build run-client |
| 37 | +### Why ProtoMQ |
34 | 38 |
|
35 | | -# Run tests |
36 | | -zig build test |
| 39 | +If you've worked with IoT sensor fleets, you've probably been through this: you start with JSON over MQTT because it's easy to debug, every language has a parser, and `mosquitto_sub` lets you eyeball what's going on. It works fine... until you start caring about bandwidth. |
37 | 40 |
|
38 | | -# Run all integration tests |
39 | | -./tests/run_all.sh |
40 | | -``` |
| 41 | +A 12-field sensor reading weighs around 310 bytes in JSON. The same data in Protobuf: 82 bytes. On a cellular-connected device pushing telemetry every 5 seconds, that gap adds up to roughly 1.6 MB/day per device — multiply by a few thousand devices and the data bill starts hurting. |
41 | 42 |
|
42 | | -### Limitations |
| 43 | +<p align="center"> |
| 44 | + <img src="assets/payload_comparison.svg" alt="JSON vs Protobuf payload size comparison" width="680px" /> |
| 45 | +</p> |
43 | 46 |
|
44 | | -For the initial release, we support: |
45 | | -- QoS 0 only (at most once delivery) |
46 | | -- No persistent sessions |
47 | | -- No retained messages |
48 | | -- Single-node deployment |
| 47 | +But switching to Protobuf usually means code generation per language, keeping stubs in sync across firmware versions, and losing the ability to just read your payloads. ProtoMQ takes a different approach: the broker owns the `.proto` schemas and validates every message against them. The CLI can accept JSON and encode it to Protobuf before publishing, so you get a human-friendly workflow without sacrificing wire efficiency. |
49 | 48 |
|
50 | | -### Service Discovery |
| 49 | +| | Plain MQTT + JSON | ProtoMQ | |
| 50 | +|---|---|---| |
| 51 | +| Schema enforcement | None — anything goes | Validated at every `PUBLISH` | |
| 52 | +| Payload format | JSON (~170 bytes, 8 fields) | Protobuf (~48 bytes) | |
| 53 | +| Client bootstrap | Pre-shared docs | Built-in Service Discovery | |
| 54 | +| Code generation | Required per language | CLI encodes JSON → Protobuf for you | |
51 | 55 |
|
52 | | -ProtoMQ includes a built-in Service Discovery mechanism. Clients can discover available topics and their associated Protobuf schemas (including the full source code) by querying the `$SYS/discovery/request` topic. |
| 56 | +--- |
53 | 57 |
|
54 | | -**Using the CLI for discovery:** |
55 | | -```bash |
56 | | -# Verify schemas are loaded and available |
57 | | -protomq-cli discover --proto-dir schemas |
58 | | -``` |
59 | | -This allows clients to "bootstrap" themselves without needing pre-shared `.proto` files. |
| 58 | +### Under the Hood |
60 | 59 |
|
61 | | -### Admin Server |
| 60 | +ProtoMQ is not a wrapper around an existing broker — it's a ground-up implementation. Here's what makes it tick: |
62 | 61 |
|
63 | | -ProtoMQ includes an optional HTTP Admin Server for runtime observability and dynamic schema management without polluting the core MQTT hot-paths. |
| 62 | +- **`epoll` / `kqueue` event loop** — single-threaded, no abstraction layer. The network layer talks directly to the OS kernel I/O primitives. On Linux that's `epoll`, on macOS `kqueue`. No libuv, no tokio, no hidden threads. |
| 63 | +- **One allocator, full control** — every allocation goes through Zig's `std.mem.Allocator`. No GC, no hidden heap churn, no runtime. You can trace every byte the broker touches. |
| 64 | +- **Zero third-party dependencies** — the MQTT parser, TCP connection handler, Protobuf wire format encoder, `.proto` file parser — all written in Zig using only the standard library. `build.zig.zon` has an empty `dependencies` block. |
| 65 | +- **Runtime schema registry** — `.proto` files are parsed at startup and mapped to MQTT topics. With the Admin Server enabled, you can register new schemas and mappings at runtime over HTTP without restarting the broker. |
| 66 | +- **Comptime-generated lookup tables** — the MQTT packet parser uses Zig's `comptime` to build dispatch tables at compile time. No branching, no hash maps — just array indexing. |
| 67 | +- **Cross-compilation** — `zig build -Dtarget=aarch64-linux` produces a Raspberry Pi binary from a Mac. One command, no toolchain headaches. |
64 | 68 |
|
65 | | -- **Dynamic Schema Registration**: Register `.proto` files at runtime via `POST /api/v1/schemas`. |
66 | | -- **Telemetry**: Monitor active connections, message throughput, and schemas via `GET /metrics`. |
67 | | -- **Zero Overhead Footprint**: The Admin Server is disabled by default to preserve the absolute minimum memory footprint for embedded devices. It is strictly conditionally compiled via the `zig build -Dadmin_server=true` flag. Enabling it moderately increases the initial static memory baseline (e.g., from ~2.6 MB to ~4.0 MB) by safely running a parallel HTTP listener, but it executes cooperatively on the same event loop ensuring zero degradation to per-message MQTT performance. When the flag is deactivated, it incurs **zero overhead footprint**. |
| 69 | +--- |
68 | 70 |
|
69 | | -### Performance Results |
| 71 | +### Quick Start |
70 | 72 |
|
71 | | -ProtoMQ delivers high performance across both high-end and edge hardware: |
| 73 | +**Docker:** |
72 | 74 |
|
73 | | -| Scenario | Apple M2 Pro | Raspberry Pi 5 | |
74 | | -|----------|--------------|----------------| |
75 | | -| Latency (p99, 100 clients) | 0.44 ms | 0.13 ms | |
76 | | -| Concurrent clients | 10,000 | 10,000 | |
77 | | -| Sustained throughput | 9k msg/s | 9k msg/s | |
78 | | -| Message throughput (small) | 208k msg/s | 147k msg/s | |
79 | | -| Memory (100 clients) | 2.6 MB | 2.5 MB | |
| 75 | +```bash |
| 76 | +docker compose up |
| 77 | +``` |
80 | 78 |
|
81 | | -Handles 100,000 connection cycles with zero memory leaks and sub-millisecond latency. |
| 79 | +The server starts on port `1883` with the schemas from `schemas/`. Connect with any MQTT client. |
82 | 80 |
|
83 | | -For detailed methodology and full results, see [ProtoMQ Benchmarking Suite](benchmarks/README.md). |
| 81 | +**From source** (requires [Zig 0.15.2+](https://ziglang.org/download/)): |
84 | 82 |
|
85 | | -### Contributing |
| 83 | +```bash |
| 84 | +git clone https://github.com/electricalgorithm/protomq.git |
| 85 | +cd protomq |
| 86 | +zig build run-server |
| 87 | +``` |
86 | 88 |
|
87 | | -This is currently a learning/development project. Contributions will be welcome after the MVP is complete. |
| 89 | +```bash |
| 90 | +# In another terminal — publish (CLI encodes JSON to Protobuf for you) |
| 91 | +zig build run-client -- publish --topic sensors/temp \ |
| 92 | + --json '{"device_id":"s-042","temperature":22.5,"humidity":61,"timestamp":1706140800}' |
88 | 93 |
|
89 | | -### License |
| 94 | +# In another terminal — subscribe |
| 95 | +zig build run-client -- subscribe --topic "sensors/#" |
| 96 | +``` |
90 | 97 |
|
91 | | -The project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. |
| 98 | +--- |
92 | 99 |
|
93 | | -### Resources |
| 100 | +### Performance |
94 | 101 |
|
95 | | -- [Zig Documentation](https://ziglang.org/documentation/master/) |
96 | | -- [MQTT v3.1.1 Specification](http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqtt-v3.1.1.html) |
97 | | -- [Protocol Buffers](https://protobuf.dev/) |
| 102 | +ProtoMQ handles **208,000 messages/second** on an Apple M2 Pro and **147,000 msg/s** on a Raspberry Pi 5 — with sub-millisecond p99 latency and no memory leaks across 100,000 connection cycles. |
| 103 | + |
| 104 | +| Scenario | Apple M2 Pro | Raspberry Pi 5 | |
| 105 | +|----------|--------------|----------------| |
| 106 | +| **p99 latency** (100 clients) | 0.44 ms | 0.13 ms | |
| 107 | +| **Throughput** (10-byte msgs) | 208k msg/s | 147k msg/s | |
| 108 | +| **Throughput** (64 KB msgs) | 39k msg/s | 27k msg/s | |
| 109 | +| **Sustained load** (10 min) | 8,981 msg/s | 9,012 msg/s | |
| 110 | +| **Memory** (100 connections) | 2.6 MB | 2.5 MB | |
| 111 | +| **Connection churn** (100k cycles) | 1,496 conn/s | 1,548 conn/s | |
| 112 | +| **Memory leaks** | 0 MB | 0 MB | |
| 113 | + |
| 114 | +All benchmarks run on loopback, `ReleaseSafe` mode, Zig 0.15.2. Methodology and raw results: [`benchmarks/`](benchmarks/README.md). |
98 | 115 |
|
99 | 116 | --- |
100 | 117 |
|
101 | | -**Note**: This project is under active development. The API and architecture may change significantly. |
| 118 | +### Current Limitations |
| 119 | + |
| 120 | +QoS 0 only (at most once delivery), no persistent sessions, no retained messages, single-node deployment. These are scope decisions for the initial release — multi-node and QoS 1/2 are on the roadmap. |
| 121 | + |
| 122 | +### Contributing |
| 123 | + |
| 124 | +Contributions are welcome. If you're interested in MQTT internals, Protobuf wire format, or systems programming in Zig, there's plenty to dig into. See [FEATURES.md](FEATURES.md) for the full feature set and [FAQ.md](FAQ.md) for deployment and configuration guides. |
0 commit comments