[zeromq-dev] Custom HWM behavior
bernie at fliptrack.com
Fri Aug 19 18:21:09 CEST 2011
Gehan Gonsalkorale asked a similar question back in May this year, and the
consensus for him was to not use push/pull, but pub/sub instead --
apparently with the primary motivation for the switch is that pub/sub allows
data to be dropped.
While my objectives are similar they are not the same.
I want PHP processess to be able to send events (debug, log, information /
statistics) to a mongo database. I have found that the PHP bindings to
mongo are not super great, and have seen some blocking behavior (even when
configured to not block) / slowness when it comes to either connecting or
writing to mongo.
One solution is to decouple PHP from hitting mongo directly, and insert ZMQ
in the middle. Given the possible huge volumes we're talking about. I
really want unidirectional load balanced sockets that can connect to
multiple event-logging servers.
Under no circumstance should the PHP processess be blocked or delayed due to
event logging. I want to "fire and forget". However, our event logging is
important and possibly high volume enough that I want the option of having
this function scalable and decentralized.
If I was to use pub/sub, and was to add several subscribers, I would have
the same events logged multiple times. That's not at all what I want. I
don't want to MULTIPLY the number of events I add into mongo by a factor of
number of subscribers.
The pub/sub model isn't the logical choice here. My feeling is that from a
pattern perspective, the correct choice is load balanced push/pull, and if
the only reason I switch to pub/sub is because I want to enable data to be
dropped instead of block -- then I would argue I'm using the wrong
networking pattern because of a limitation of the ZMQ design / API. Not
because pub/sub is the logical choice here.
To be clear, the option I would want is to be able to set a socket option
(or something like that) that indicates:
- If we hit HWM condition, drop data
- If we have data at process exit, drop data.
Alternatively, if I can query the queue length of messages that are unsent
from the API, instead of blindly adding the event to be sent over the
socket, I could drop it before it ever hits ZMQ. Does this exist? Of
course, I'd still have to have the ability to "drain the data" to be sent
out at process exit -- I don't want the process to hang / wait indefinitely.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the zeromq-dev