[zeromq-dev] Does ZeroMQ pay a lot in terms of marshalling compared to akka and clojure.core.async?

Bennie Kloosteman bklooste at gmail.com
Fri Oct 11 05:56:38 CEST 2013


Yes compared to native langauges all GC languages incur extra IO costs   eg
 When writing both languages they need to read/copy the data to the kernel
, but the GC language needs to tell the GC to pin each buffer so it doesnt
reloccate it ,this can be significant for lots of small packets.

GC pauses can be a huge issue i once had a store and forward message system
which in stress testing ended up having 10 second GC pauses on a 800Meg
heap which caused the system to stop reading incomming packets which then
gave timeout errors . Many get around this by storing the message outside
the GC ( in native or in serialized queues of byte[] ) and then getting the
message when you want.

GC languages also have a card table cost for wrting to a message.

Note pure Java solutions like Java NIO may be better than JeroMQ , there
was a discussion on this earlier . I dont know JeroMQs architecture but
pure solutions tend to consider the GC issues rather than trivially
wrapping the API , they reuse byte[] buffers to copy from and to the kernel
, and may even employ  unsafe / JNI code  to self manage some buffers so
you dont need to  pin the GC .. .

Again except for the GC pauses these issues are only for micro benches and
dwarfed by  the cost of any real work / IO.

My suggestion, build your real work and use a flexible comms API , use the
easiest comms possible ( which would prob be a a Java lib so you dont worry
about deploying zeromq ) then test the whole end to end process if the
comms is too slow  then test zeromq and NIO ( which should be easy since
you put it behind an interface) but not before.

Ben


On Fri, Oct 11, 2013 at 9:10 AM, crocket <crockabiscuit at gmail.com> wrote:

> Does it mean JeroMQ incurs marshalling when communicating to the kernel?
>
>
> On Fri, Oct 11, 2013 at 12:21 AM, Bennie Kloosteman <bklooste at gmail.com>wrote:
>
>> Benchmark it with some real work.
>>
>> Marshalling to native is expensive when doing micro benches with no work
>> but not compared to going over the wire or messages that do real work ,
>> Serialization cost will normally be higher than marshalling to native . If
>> your doing inter process via shared memory / pipes on the same machine and
>> dont do anything with the message it may well be more expensive.  On the
>> other hand  for large scale solutions where your messages  stay in a queue
>> for a while you can leave the message in native and not fill the GC and
>> suffer GC pauses.
>>
>> Note the JVM also incurs marshaling when communicating to the kernel  say
>> for tcp/ip which zeroMq does not .
>>
>> Ben
>>
>>
>>  On Thu, Oct 10, 2013 at 9:22 PM, crocket <crockabiscuit at gmail.com>wrote:
>>
>>>  Someone pointed out that ZeroMQ was great at communicaton between two
>>> or more languages but that for communications in one language, internal
>>> messaging solutions like akka and clojure.core.async would be a lot faster.
>>>
>>> He said marshalling was expensive.
>>>
>>> I looked into core.async, and I found it uses queues for asynchronicity.
>>>
>>> How would one compare core.async and ZeroMQ?
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20131011/991c876d/attachment.htm>


More information about the zeromq-dev mailing list