Show HN: PicoMQ – Durable Streams over HTTP, on object storage

This sounds like a kafka-like streaming system, but backed onto s3-objects?

Doesn't this mean that write performance is going to be bad?

Regular S3 has write latency 100-150ms, which might be fine depending on your workload anyway.

And it's also fair to question write performance, since it's backed by object storage. The optimization is primarily from the shared WAL across streams, server-side batching, and client-side in-memory pipelining, especially with HTTP/2, without as much connection pool overhead.

In practice, you can go to the extent of achieving up to 100 MiB/s throughput per stream. Considering how granular streams can be, you'd rarely need as much. The latency for a durability ACK is, however, the price to pay, which is going to be ~250 ms, or lower with S3 Express, which I'd say covers most real-time use-cases. The design itself is easy enough to extend to a disk-staged WAL for single-digit durability ACK latency.

I'll be setting up a GCP deployment example similar to AWS soon. I'll be sure to try Rapid Bucket as well, thanks for sharing!

I would also check out Tigris [1], which has an S3-compatible API. Their main claim to fame is that buckets are low-latency, multi-region and replicated by default, so supposedly you get region-local latency no matter where you are reading or writing from. I have not done any rigorous performance comparisons, though. What's amazing, if it does perform well, is that egress is free, and the pricing is otherwise the same as GCS/S3.

[1] https://www.tigrisdata.com/

There are plenty of usecases for lower scale or higher latency (my examples are somewhat unique), and owning the opinionated middle instead of claiming to cover everything is a really useful thing, but acknowledging that the system is opinionated such that it covers a specific set of things well is generally a better argument than 'this basically does everything that people need'.

Where Kafka starts to fall short is routing. If you want to access the data of one user from user-events-topic, that’s expensive to do. Most other streaming technologies are built around the same design, such as Kinesis.

There are other implementations that support the Kafka wire protocol and are cheaper in exchange for latency, e.g., AutoMQ and WarpStream.

That said, I’ll release Disk/EBS-staged WAL soon enough: https://github.com/PicoMQ/picomq/issues/13 as an add-on to cover low-latency needs.

I had heard people talk about the operation pain involved in keeping Kafka alive (which is a thing for sure), but what I was surprised by was how many things behaved in a slightly unobvious manner that wasn't loudly-documented (e.g. if you're using transactions for RWP loops the default rebalance protocol is unsound and transaction markers take up an index in the log so you no longer have contiguous indices in your message stream etc).

But because the Pico's semantics are close to Kafka, but not tightly coupled, it's quite feasible to support the Kafka protocol, * with some restrictions *, such as a topic can only ever have one partition, no support for transactions (since they would only really apply when producers are publishing to multiple streams), and more along those lines. As a result, you'd get Kafka with no topic tax and a lot less operational complexity.

So - in theory - something like a massive chat client, discord like, can be implemented via this solution? And what would be the pricing of such a solution. Cheap-serverless-discord

Exactly, streams can essentially be rooms, and since the ordering is preserved, a Discord-like application is a strong use case. I’m even considering building one using PicoMQ as an example showcase.

The pricing is going to be dirt cheap, and the best part is how easy it is to scale up vertically or add nodes. For some raw numbers, assuming 1M messages/day, 200B per message, ~6GB/month, all-inclusive, it would be $30 to $150 a month, and storage would be the cheapest part.

Is this back of envelope pricing include the traffic/bandwidth of the readers?

That could easily be 10-100x of number of messages.

This solution together with a cheap/free caching layer (especially for non members/writers) could be amazing.

BTW, a classic example would be an hn mirror. ;)

But as we speak, I am in the process of running open-benchmarks on AWS with cost attribution. I'll be posting them in the docs with transparency soon enough.

Read and write through cache already exists today, which would work in favour of both cost, latency, and consumer reads fanout.

I like the HN Mirror over a Discord-like app for simplicity. No doubt that's where my weekend is going!

Initially, I started to compare PicoMQ to https://github.com/s2-streamstore/s2. If you're familiar with them, how would you compare PicoMQ to S2?

SlateDB is great, but it's not purpose-built for streaming workloads. With PicoMQ, the goal is to really do one thing well. It would have been far easier to just extend SlateDB, but instead, the S3Stream engine is built from the ground up with AutoMQ's core primitives.

Not to mention, Pico supports multi-node/cluster deployments and a bunch of other features (no feature gating).

Have you looked into the semantics of their like StreamDB stuff ? Pretty interesting

Also there is this that was built on top of it. Sounds somewhat similar goal as yours but I could be off. Need to explore more https://ursula.tonbo.io/

I did, in fact, initially write PicoMQ with OpenRaft, similar to Ursala, but I really wanted the operational complexity to be minimal and the nodes to be stateless (at least for most use cases), Pico uses SQL database as a metadata command log, inspired by RisingWave.

But Ursala would, without a doubt, have better durability ACK latency, as it wouldn’t have to wait for an ACK from S3. That said, I do plan on extending disk/EBS-staged WAL to hit similar low-latency durability ACK numbers. But again, most use cases don’t need single-digit-millisecond latency for ACKs.

Unfortunate timing and self hosting isn't complex or anything but still as you said, durable streams as a project is most likely going to be abandoned.

If you'd like to consider using PicoMQ, I have examples to deploy on Fly.io and AWS. ursula.tonbo.io is also a good option to try.

but nice product overall - just like s2 lite.

S2-lite is pretty cool, but it's single-node only and built on SlateDB. So, PicoMQ is closer S2 Cloud.

Since you mentioned nodes are stateless and you have examples for Fly.io, how does the system handle rapid node scaling (scaling to zero and back up) in response to bursty traffic? Does the WAL recovery add significant cold-start latency for consumers?