NSQ is a full-featured messaging platform out-of-the-box, whereas zeromq and nanomsq are lower level libraries that you could use to build (the same) functionality.
The Features page says it is "horizontally scalable (no brokers, seamlessly add more nodes to the cluster)", but the Design page says you need to run a 'nsdq' daemon - which is a message broker - i.e. you cannot embed the daemon into your application?
That said, for a larger deployment you'd still probably want to run the lookupd as well, at which point, at least for a typical architecture, running nsqd standalone is probably fine. Embedding is darn handy for testing, though.
That sentence is missing an important word. No centralized brokers, meaning the typical/recommended deployment topology is an nsqd node on all hosts producing messages.
I appreciate all the work that Tyler has put into his series of blog posts on messaging systems (and NSQ bug reports!) but I think this particular article is one that misses the mark [1].
While raw performance is important, I think comparing the guarantees and related operational and development semantics are far more interesting and useful.
These systems vary a great deal in the interfaces and functionality they expose to users building on top of them and operators dealing with them when they're deployed in production (and inevitably break!). This is ultimately what matters most.
P.S. single node (localhost) performance is essentially useless. Most of these tools are designed to be deployed as the backbone of large distributed systems (NSQ in particular), so scalability and performance in that context is a better indication of real-world performance.
NSQ is as much about what it doesn't do as it is what it does. To a certain extent this mirrors, and was inspired by, the language's philosophy (Go) [1].
Also, NSQ was designed to replace an existing home-grown system deployed at scale. This dictated a lot of the initial requirements (and in certain cases excluded off-the-shelf tools).
When we left the experimental phase we realized we had built something that was useful to others, and it turns out that despite not having the features you've identified it can be incredibly effective in lots of use cases that don't need stronger guarantees.
[1] If I'm being honest, NSQ was a vehicle for adoption of Go at bitly as well as the project we used to learn the language. This was a huge risk at the time (almost 3 years ago) but one that has certainly paid off.
Yeah, don't get me wrong, it looks good for what it is, and certainly some of the best products come from skunkworks and hobby.
Is there anywhere I can read about the specific problem at bitly you built it to facilitate? I mean, I've seen a lot of these systems where there are integration and timing concerns, so some kind of message bus is useful. But if they do get to a point where things needs to be distributed, sending messages without order and delivery guarantees just seems to be passing the buck to downstream systems to not mess up.
NSQ treats the message data as an opaque blob so the format wouldn't directly affect it (except on some lower level related to overall message size I suppose). It would impact your producers (encoding) and consumers (decoding), obviously.
For the record... NSQ is a generic messaging platform and is data format agnostic, there is nothing specific about metrics collection (it was just my contrived example).
Agreed. Message queues play an important role for us (bitly) in being a layer of fault tolerance, buffering, and a means to perform various operational tasks.
They're so important to us that we decided to build something that worked exactly like we wanted.
NSQ is a realtime distributed message processing system where we've taken the approach of focussing on making it ops friendly and easy to get started.
IMO, solutions that make it easy to develop on and administrate are most important... because things break. NSQ is straightforward to deploy (limit dependencies), simple to configure (runtime discovery), and client libraries provide a lot of functionality important for handling failures (like backoff, deferred messages, etc.) for a variety of use cases.
We've written an in-depth introductory blog post [2] about NSQ that has more details.
Interesting, thanks for sharing the links. I think building your own message queue is sometimes the only way to get things working exactly as you might want. For people looking for a highly configurable framework for building a tailored MQ, be sure to check out http://www.zeromq.org/ as a basis.
