[zeromq-dev] When unreliability is desired

Sat Dec 7 00:37:17 CET 2013

This is the best place for discussions. The encoder classes are for
TCP only. If you do try to port the UDP transport, my advice is to do
it in many little steps, so we can all get involved. Start with an
empty class if needed. The tipc classes in libzmq may be a help.

On Sat, Dec 7, 2013 at 12:15 AM, Lindley French <lindleyf at gmail.com> wrote:
> Is there a better place for discussion of this effort? I do have a lot of
> questions.
>
> The latest one is about encoder_t etc. This appears to have changed
> significantly recently, and I'm not entirely sure how to adapt the
> Crossroads code to match. Is there documentation on its latest design
> somewhere?
>
>
> On Thu, Dec 5, 2013 at 4:13 PM, Charles Remes <lists at chuckremes.com> wrote:
>>
>> Feel free to "rubber duck" with the list as much as you want. :)
>>
>> cr
>>
>> On Dec 5, 2013, at 3:08 PM, Lindley French <lindleyf at gmail.com> wrote:
>>
>> Disregard the question about timers, it appears this was a change made to
>> Crossroads IO after the fork. I found the relevant commit:
>>
>> https://github.com/crossroads-io/libxs/commit/2df873a435ff139cf9d1b10b666d75e6dc6da442#diff-45c9686e09ed422318a2da658745dedd
>> I will revert the code to the 0MQ way of doing things for now.
>>
>>
>> On Thu, Dec 5, 2013 at 3:48 PM, Lindley French <lindleyf at gmail.com> wrote:
>>>
>>> I'm trying to make sure I understand how the timer_ids work, since these
>>> were not in the Crossroads IO code. It appears several classes define
>>> timer_id constants as anonymous enums. Furthermore, the values of these
>>> constants don't appear to follow any particular pattern, *except* that
>>> constants with the same name (same purpose?) are assigned the same value.
>>> Therefore, I expect I ought to adopt tx_timer_id and rx_timer_id from
>>> pgm_sender without changing their values for udp_sender etc. Am I
>>> understanding this correctly?
>>>
>>>
>>> On Thu, Dec 5, 2013 at 3:06 PM, Lindley French <lindleyf at gmail.com>
>>> wrote:
>>>>
>>>> It appears that at some point, activate_in was renamed to restart_input,
>>>> and likewise with output. I'm assuming there are no functional changes other
>>>> than the names in the UDP case. It also appears that zap_msg_available() can
>>>> be ignored for now. Let me know if I'm wrong about these.
>>>>
>>>>
>>>> On Thu, Dec 5, 2013 at 2:01 PM, Lindley French <lindleyf at gmail.com>
>>>> wrote:
>>>>>
>>>>> Okay, I'm working on porting the UDP stuff to the latest ZMQ now. Most
>>>>> of the changes necessary are purely a matter of namespace, but there are a
>>>>> few others that seem to reflect design changes in ZMQ since XS was forked.
>>>>> These are the ones I'm not immediately sure how to solve:
>>>>>
>>>>> 1) The XS UDP code references encoder_t and decoder_t, but ZMQ has
>>>>> v1_encoder_t and v2_encoder_t etc instead.
>>>>> 2) i_engine appears to have three pure virtual methods that are not
>>>>> defined in the XS code: restart_input, restart_output, and
>>>>> zap_msg_available().
>>>>>
>>>>> Any suggestions how to resolve these?
>>>>>
>>>>>
>>>>> On Thu, Dec 5, 2013 at 10:52 AM, Lindley French <lindleyf at gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Thanks, I found it. I don't see any reason given as to why UDP support
>>>>>> was reverted, though. Are there issues with the code, philosophical
>>>>>> objections, etc?
>>>>>>
>>>>>>
>>>>>> On Wed, Dec 4, 2013 at 5:16 PM, Ivan Pechorin
>>>>>> <ivan.pechorin at gmail.com> wrote:
>>>>>>>
>>>>>>> UDP support was reverted in libxs just before 1.2.0 release: check
>>>>>>> the commits history on github around June 13th, 2012.
>>>>>>>
>>>>>>> 05.12.2013 6:48 пользователь "Lindley French" <lindleyf at gmail.com>
>>>>>>> написал:
>>>>>>>
>>>>>>>> Can you clarify where in the Crossroads IO library the UDP transport
>>>>>>>> code lives? I've downloaded the 1.2.0 tarball here:
>>>>>>>> http://download.crossroads.io/libxs-1.2.0.tar.gz
>>>>>>>> but so far, I don't see a UDP transport in that code. I also checked
>>>>>>>> the github version:
>>>>>>>> https://github.com/crossroads-io/libxs
>>>>>>>> but I don't see it there either.
>>>>>>>>
>>>>>>>> On Wed, Nov 27, 2013 at 11:26 AM, Charles Remes
>>>>>>>> <lists at chuckremes.com> wrote:
>>>>>>>>>
>>>>>>>>> That defunct library (crossroads io) has the code that you want.
>>>>>>>>> That lib was a fork of zeromq, so moving the UDP transport from that library
>>>>>>>>> to zeromq should be easy (for varying degrees of easy). Once it makes it
>>>>>>>>> into zeromq, it will be supported.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Nov 27, 2013, at 9:50 AM, Lindley French <lindleyf at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> I've never used ZeroMQ before so writing up a new transport would
>>>>>>>>> be just a bit ambitious right now. (I did write something in Java last year
>>>>>>>>> that, in retrospect, was solving basically the same problem as ZeroMQ so I
>>>>>>>>> have some familiarity with the problem space.)
>>>>>>>>>
>>>>>>>>> I'm also leery of adopting a defunct library for a new project.
>>>>>>>>>
>>>>>>>>> I'll keep the udp transport option in mind.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Nov 26, 2013 at 1:18 PM, Pieter Hintjens <ph at imatix.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Lindley,
>>>>>>>>>>
>>>>>>>>>> The right solution would be to make a UDP transport for ZeroMQ.
>>>>>>>>>> It's
>>>>>>>>>> not a trivial project but could start with, for instance, just
>>>>>>>>>> pub/sub
>>>>>>>>>> (like PGM).
>>>>>>>>>>
>>>>>>>>>> It might be worth looking at Crossroads.io for that, which is
>>>>>>>>>> abandoned but had afair a UDP transport, and shared the same
>>>>>>>>>> original
>>>>>>>>>> codebase with ZeroMQ.
>>>>>>>>>>
>>>>>>>>>> -Pieter
>>>>>>>>>>
>>>>>>>>>> On Tue, Nov 26, 2013 at 6:15 PM, Lindley French
>>>>>>>>>> <lindleyf at gmail.com> wrote:
>>>>>>>>>> > I have a networking application that I'd like to use ZeroMQ in.
>>>>>>>>>> > However, my
>>>>>>>>>> > use-case demands minimum latency even at the expense of lost
>>>>>>>>>> > messages. I'm
>>>>>>>>>> > weary of using ZeroMQ's TCP transport because if packets are
>>>>>>>>>> > dropped, TCP
>>>>>>>>>> > will block further messages until it has retransmitted the last
>>>>>>>>>> > one, and I
>>>>>>>>>> > don't want that behavior.
>>>>>>>>>> >
>>>>>>>>>> > I don't mind FEC codes or other strategies to improve
>>>>>>>>>> > reliability by sending
>>>>>>>>>> > more data up-front, but I do not need complete reliability and I
>>>>>>>>>> > want to
>>>>>>>>>> > avoid retransmission of messages, or at the least avoid blocking
>>>>>>>>>> > later
>>>>>>>>>> > messages if earlier ones need to be retransmitted.
>>>>>>>>>> >
>>>>>>>>>> > Is there an existing ZeroMQ transport that will provide the
>>>>>>>>>> > behavior I want?
>>>>>>>>>> > I was thinking maybe epgm would do the trick, even though I
>>>>>>>>>> > don't really
>>>>>>>>>> > need multicast. Ideally, I'd want a transport that uses pure UDP
>>>>>>>>>> > for
>>>>>>>>>> > messages, perhaps with some TCP "behind the scenes" for
>>>>>>>>>> > out-of-band
>>>>>>>>>> > handshaking.
>>>>>>>>>> >
>>>>>>>>>> > I may end up just using UDP myself for the time-critical
>>>>>>>>>> > messages, and
>>>>>>>>>> > ZeroMQ for less critical data, but I'd prefer to avoid
>>>>>>>>>> > multiple-API creep.
>>>>>>>>>> >
>>>>>>>>>> > _______________________________________________
>>>>>>>>>> > zeromq-dev mailing list
>>>>>>>>>> > zeromq-dev at lists.zeromq.org
>>>>>>>>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>>>>>> >
>>>>>>>>>> _______________________________________________
>>>>>>>>>> zeromq-dev mailing list
>>>>>>>>>> zeromq-dev at lists.zeromq.org
>>>>>>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> zeromq-dev mailing list
>>>>>>>>> zeromq-dev at lists.zeromq.org
>>>>>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> zeromq-dev mailing list
>>>>>>>>> zeromq-dev at lists.zeromq.org
>>>>>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> zeromq-dev mailing list
>>>>>>>> zeromq-dev at lists.zeromq.org
>>>>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> zeromq-dev mailing list
>>>>>>> zeromq-dev at lists.zeromq.org
>>>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>