I recently found a great project for persistent message queues. In a previous project, we were working with a known number of processes communicating over a static list of channels. They started out running on different machines communicating over the network but ended up all running on the same 16-core box.
In that case, all network based persistent queues were too big of a solution. On the other hand, simpler IPC mechanisms like POSIX message queues don’t have persistence across reboots.
Another way to implement a persistent queue is using a database like Berkeley DB or SQLite. But then you’re carrying around an SQL engine or or other indexing mechanism intended for arbitrary record access.
Something home-rolled around shared files seems appropriate here, but not that fun or interesting to implement. That’s how I found JLog.
JLog is exactly how you’d expect a solution in this problem space to be written. File locks around mmap’ed files for quick, safe access. Files are capped at 4MB and deleted by consumers when finished. At 256 bytes per message, that would be 16K messages per file. 1000 files would represent 16 million messages, which is a huge number within a completely manageable file set.
