[zeromq-dev] Java/Windows issues

[Robin Weisberg]

I've had quite a bit of experience w/ other middleware products, I'd be
happy to have a discussion about it if you are interested. Pieter has
contacted me and seems like I'll be talking to him Tue @ 10, maybe we can
piggyback on that. Its a difficult topic to cover in email because its so
large, but here are some thoughts based on not to much thinking but some
experience as a user.


   - Sequencing could be handled at the level of the Exchange i.e. each
   exchange has different seq numbers/internal msg queues. Queues track these
   seq numbers and can request recovery from the underlying exchange if a
   message is missed.
   - Multiple exchanges could use the same underlying connection (e.g. tcp
   connection or multicast address). The exchangeID/queueID are looked at to
   ensure messages are not received by the application when appropriate.
   - Some middleware products allow you to use wildcards as a convenient way
   to bind multiple queues to multiple exchanges in one shot. Underneath each
   exchange has its own seq numbers/internal msg queues and ordering is only
   guaranteed at the exchange level.
   - Exchanges/Queues should be able to be created w/o using a server
   (zsock?). Basically the server allows you to look up the appropriate
   connection for a given queue, if you don't want to store it in a config
   file, but use of the zmq_server should be optional.
   - When a connection disconnects all queues/exchanges get a message much
   like the gap message although w/ some more details. Same story If there are
   seq gaps (e.g. messages for recovery started/recovery failed/recovery
   success)
   - Some more advanced stuff:
      -  Exchanges could be configured to have a MessageStore the default
      could just keep X messages and/or X seconds of messages in memory.
      - More advances stores could write messages to a file (where its
      configurable whether the OS synch calls are used or not) or db etc.
      - MessageStores should be able to be wrapped in and AsynchMessageStore
      that writes the message to Store in background thread.
      - Also you can configure wheter to persist all messages or only
      messages that have not been explicitly acknowledged by all queues. This
      requires exchanges to become aware of which queues are bound to
them and for
      the queues to acknowledge receipt (these can be batched i.e. every (X
      messages and/or X seconds the queue says I'm at seq number Y). This is
      different from the model of the exchange always keep x seconds worth of
      messages for recovery regardless of who is listening.
      - At some point you will want a message broker/router that can
      configured for routing over wans using compression/encryption and mapping
      multicast to tcp and back. Should be as simple as reading a file
that says
      bind queue X at exchange Y and republish it on exchange Z (which is
      encrypted/compressed). Another broker is configured to bind queue A to
      exchange Z and republish on exchange B (could be noncompressed
multicast at
      a diff site).
      - Embed a web server in the library which you can turn on so you can
      check out stats and debug problems

I know you guys are focused on the high speed stuff which I think is great.
There really isn't another open source solution that supports
multicast/multiple platforms/multiple language bindings. I also think its
important to optimize for the non persistent cases. Of course as a user I
prefer to have one api/product for all the stuff I do so even if the
persistent stuff isn't the best of breed, I'm more likely to use the product
because its a one stop shop.

On Fri, Aug 7, 2009 at 4:50 AM, Martin Sustrik <sustrik at fastmq.com> wrote:

> Robin,
>
>  Good stuff. I'm going to keep pushing my luck here :-) any chance someone
>> could look at the issues I posted here?
>> http://lists.zeromq.org/pipermail/zeromq-dev/2009-July/001064.html
>>
>
> We had a discussion about that one. Although it seems to be a simple
> enhancement of current functionality, it's in fact a different messaging
> model. Multiple publishers change the rules of the game: There's no single
> well-defined ordering of the messages, reporting gaps in the message feed
> makes little sense, etc. We may move in that direction, however, at this
> point it is essential to understand the details of the model, especially
> from the point of view of reliability, failover, restartability etc.
>
> Thoughts?
> Martin
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20090807/1103fee6/attachment.htm>