[zeromq-dev] Does ZeroMQ pay a lot in terms of marshalling compared to akka and clojure.core.async?

Yu Dongmin miniway at gmail.com
Fri Oct 11 06:58:14 CEST 2013


Bennie,

Just FYI, JeroMQ is written with Java NIO as much as possible and has no native code.

Thanks
Min

On Oct 11, 2013, at 12:56 PM, Bennie Kloosteman <bklooste at gmail.com> wrote:

> Yes compared to native langauges all GC languages incur extra IO costs   eg  When writing both languages they need to read/copy the data to the kernel , but the GC language needs to tell the GC to pin each buffer so it doesnt reloccate it ,this can be significant for lots of small packets. 
> 
> GC pauses can be a huge issue i once had a store and forward message system which in stress testing ended up having 10 second GC pauses on a 800Meg heap which caused the system to stop reading incomming packets which then gave timeout errors . Many get around this by storing the message outside the GC ( in native or in serialized queues of byte[] ) and then getting the message when you want. 
> 
> GC languages also have a card table cost for wrting to a message.
> 
> Note pure Java solutions like Java NIO may be better than JeroMQ , there was a discussion on this earlier . I dont know JeroMQs architecture but pure solutions tend to consider the GC issues rather than trivially wrapping the API , they reuse byte[] buffers to copy from and to the kernel , and may even employ  unsafe / JNI code  to self manage some buffers so you dont need to  pin the GC .. . 
> 
> Again except for the GC pauses these issues are only for micro benches and dwarfed by  the cost of any real work / IO. 
> 
> My suggestion, build your real work and use a flexible comms API , use the easiest comms possible ( which would prob be a a Java lib so you dont worry about deploying zeromq ) then test the whole end to end process if the comms is too slow  then test zeromq and NIO ( which should be easy since you put it behind an interface) but not before. 
> 
> Ben
> 
> 
> On Fri, Oct 11, 2013 at 9:10 AM, crocket <crockabiscuit at gmail.com> wrote:
> Does it mean JeroMQ incurs marshalling when communicating to the kernel?
> 
> 
> On Fri, Oct 11, 2013 at 12:21 AM, Bennie Kloosteman <bklooste at gmail.com> wrote:
> Benchmark it with some real work. 
> 
> Marshalling to native is expensive when doing micro benches with no work but not compared to going over the wire or messages that do real work , Serialization cost will normally be higher than marshalling to native . If your doing inter process via shared memory / pipes on the same machine and dont do anything with the message it may well be more expensive.  On the other hand  for large scale solutions where your messages  stay in a queue for a while you can leave the message in native and not fill the GC and suffer GC pauses.  
> 
> Note the JVM also incurs marshaling when communicating to the kernel  say for tcp/ip which zeroMq does not .  
> 
> Ben
> 
> 
> On Thu, Oct 10, 2013 at 9:22 PM, crocket <crockabiscuit at gmail.com> wrote:
> Someone pointed out that ZeroMQ was great at communicaton between two or more languages but that for communications in one language, internal messaging solutions like akka and clojure.core.async would be a lot faster.
> 
> He said marshalling was expensive.
> 
> I looked into core.async, and I found it uses queues for asynchronicity.
> 
> How would one compare core.async and ZeroMQ?
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> 
> 
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> 
> 
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> 
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20131011/5bb30c00/attachment.htm>


More information about the zeromq-dev mailing list