[zeromq-dev] Custom HWM behavior

Andrew Hume andrew at research.att.com
Fri Aug 19 18:55:03 CEST 2011

i would observe that one of teh hallmarks of 0MQ is teh decoupling
of usage pattern from underlying connectivity. bernie's case is where
0MQ has coupled HWM behaviour with teh pattern.

	surely it would be possible to make the HWM behaviour consistent
and independent of pattern?


On Aug 19, 2011, at 9:21 AM, Bernt Habermeier wrote:

> Gehan Gonsalkorale asked a similar question back in May this year, and the consensus for him was to not use push/pull, but pub/sub instead -- apparently with the primary motivation for the switch is that pub/sub allows data to be dropped.
> While my objectives are similar they are not the same.  
> I want PHP processess to be able to send events (debug, log, information / statistics) to a mongo database.  I have found that the PHP bindings to mongo are not super great, and have seen some blocking behavior (even when configured to not block) / slowness when it comes to either connecting or writing to mongo.  
> One solution is to decouple PHP from hitting mongo directly, and insert ZMQ in the middle.  Given the possible huge volumes we're talking about.  I really want unidirectional load balanced sockets that can connect to multiple event-logging servers.
> Under no circumstance should the PHP processess be blocked or delayed due to event logging.  I want to "fire and forget".  However, our event logging is important and possibly high volume enough that I want the option of having this function scalable and decentralized.
> If I was to use pub/sub, and was to add several subscribers, I would have the same events logged multiple times.  That's not at all what I want.  I don't want to MULTIPLY the number of events I add into mongo by a factor of number of subscribers.  
> The pub/sub model isn't the logical choice here.  My feeling is that from a pattern perspective, the correct choice is load balanced push/pull, and if the only reason I switch to pub/sub is because I want to enable data to be dropped instead of block -- then I would argue I'm using the wrong networking pattern because of a limitation of the ZMQ design / API.  Not because pub/sub is the logical choice here.
> To be clear, the option I would want is to be able to set a socket option (or something like that) that indicates:
> If we hit HWM condition, drop data
> If we have data at process exit, drop data.  
> Alternatively, if I can query the queue length of messages that are unsent from the API, instead of blindly adding the event to be sent over the socket, I could drop it before it ever hits ZMQ.  Does this exist?  Of course, I'd still have to have the ability to "drain the data" to be sent out at process exit -- I don't want the process to hang / wait indefinitely.  
> Cheers,
> Bernie
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Andrew Hume  (best -> Telework) +1 623-551-2845
andrew at research.att.com  (Work) +1 973-236-2014
AT&T Labs - Research; member of USENIX and LOPSA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110819/a837df61/attachment.htm>

More information about the zeromq-dev mailing list