[zeromq-dev] Zero Copy in Java jzmq
Trevor Bernard
trevor.bernard at gmail.com
Sun Feb 17 04:21:11 CET 2013
There is a zero copy API for JZMQ with this commit:
https://github.com/zeromq/jzmq/commit/336a8a1d3269483ba67bb903c8dc0b1482079a33
Cheers
On Sat, Feb 16, 2013 at 5:47 PM, Trevor Bernard
<trevor.bernard at gmail.com> wrote:
> @gonzalo Did you do anything special on the client side to read the
> direct bytebuffer? My recvZeroCopy isn't working ATM.
>
> On Sat, Feb 16, 2013 at 4:28 PM, Trevor Bernard
> <trevor.bernard at gmail.com> wrote:
>> Also you'll need to use ByteBuffer.allocateDirect(...) in order for it
>> to work. I explicitly didn't check GetDirectBufferAddress for perf
>> reasons.
>>
>> For the JZMQ api, I used sendZeroCopy and recvZeroCopy for a lack of a
>> better/fancier name. ;)
>>
>> -Trev
>>
>> On Sat, Feb 16, 2013 at 4:26 PM, Trevor Bernard
>> <trevor.bernard at gmail.com> wrote:
>>> I added support for zero copy send and recv.
>>>
>>> https://github.com/zeromq/jzmq/issues/170
>>>
>>> I haven't had time to test this but it compiles and the unit tests pass.
>>>
>>> -Trev
>>>
>>> On Fri, Feb 15, 2013 at 5:26 PM, The Carrolls <thecarrolls at jiminger.com> wrote:
>>>> Awesome! I'll be watching for it then.
>>>>
>>>> On 02/15/13 13:25, Trevor Bernard wrote:
>>>>> This looks simple enough. I should be able to port this into a jzmq
>>>>> branch fairly easily.
>>>>>
>>>>> I'll likely do it over the weekend.
>>>>>
>>>>> -Trev
>>>>>
>>>>> On Fri, Feb 15, 2013 at 1:55 PM, gonzalo diethelm <gdiethelm at dcv.cl> wrote:
>>>>>> I remembered that I did implement this for a tentative Java binding for Crossroads IO. The relevant methods where send() and recv(). This all worked and performed MUCH better that current binding. Some details follow; if more is required, let me know.
>>>>>>
>>>>>> Usage:
>>>>>>
>>>>>> int size = 128;
>>>>>> ByteBuffer bb = ByteBuffer.allocateDirect(size);
>>>>>> ...
>>>>>> int rc = xs.xs_recv(sock, bb, 0, size, 0);
>>>>>> ...
>>>>>> int rc = xs.xs_recv(sock, bb, 0, size, 0);
>>>>>>
>>>>>> ---------------------
>>>>>> In Java:
>>>>>>
>>>>>> public class XsLibrary {
>>>>>> ...
>>>>>> public native int xs_send(long socket,
>>>>>> ByteBuffer buffer,
>>>>>> int offset,
>>>>>> int length,
>>>>>> int flags);
>>>>>> public native int xs_recv(long socket,
>>>>>> ByteBuffer buffer,
>>>>>> int offset,
>>>>>> int length,
>>>>>> int flags);
>>>>>> }
>>>>>>
>>>>>> ---------------------
>>>>>> In C (the JNI part):
>>>>>>
>>>>>> JNIEXPORT jint JNICALL Java_io_crossroads_jni_XsLibrary_xs_1send(JNIEnv* env,
>>>>>> jobject obj,
>>>>>> jlong socket,
>>>>>> jobject buffer,
>>>>>> jint offset,
>>>>>> jint length,
>>>>>> jint flags)
>>>>>> {
>>>>>> void* sock = 0;
>>>>>> jbyte* buf = 0;
>>>>>> int ret = 0;
>>>>>>
>>>>>> sock = (void*) socket;
>>>>>> XS_ASSERT(sock);
>>>>>> buf = (jbyte*) (*env)->GetDirectBufferAddress(env, buffer);
>>>>>> XS_ASSERT(buf);
>>>>>> ret = xs_send(sock, buf, length, flags);
>>>>>> return ret;
>>>>>> }
>>>>>>
>>>>>> JNIEXPORT jint JNICALL Java_io_crossroads_jni_XsLibrary_xs_1recv(JNIEnv* env,
>>>>>> jobject obj,
>>>>>> jlong socket,
>>>>>> jobject buffer,
>>>>>> jint offset,
>>>>>> jint length,
>>>>>> jint flags)
>>>>>> {
>>>>>> void* sock = 0;
>>>>>> jbyte* buf = 0;
>>>>>> int ret = 0;
>>>>>>
>>>>>> sock = (void*) socket;
>>>>>> XS_ASSERT(sock);
>>>>>> buf = (jbyte*) (*env)->GetDirectBufferAddress(env, buffer);
>>>>>> XS_ASSERT(buf);
>>>>>> ret = xs_recv(sock, buf, length, flags);
>>>>>> return ret;
>>>>>> }
>>>>>>
>>>>>> ---------------------
>>>>>>
>>>>>> --
>>>>>> Gonzalo Diethelm
>>>>>> DCV Chile
>>>>>>
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: zeromq-dev-bounces at lists.zeromq.org [mailto:zeromq-dev-
>>>>>>> bounces at lists.zeromq.org] On Behalf Of Trevor Bernard
>>>>>>> Sent: Friday, February 15, 2013 2:02 PM
>>>>>>> To: thecarrolls at jiminger.com; ZeroMQ development list
>>>>>>> Subject: Re: [zeromq-dev] Zero Copy in Java jzmq
>>>>>>>
>>>>>>>> I'm thinking of extending jzmq to optionally allow the zero-copy
>>>>>>>> technique using the concepts outlined by Martin Thompson here:
>>>>>>> That's awesome
>>>>>>>
>>>>>>>> I'm not 100% sure I'll do it, I need some more experimentation first,
>>>>>>>> but is there any interest or words of advice if someone's tried this
>>>>>>>> already?
>>>>>>> I'd love for send/recv to have a ByteBuffer API.
>>>>>>>
>>>>>>> Some resources and general suggestions:
>>>>>>>
>>>>>>> Have a peak at zmq_msg_t in zmq.h.
>>>>>>>
>>>>>>> Defines ZMTP/2.0 Spec:
>>>>>>> * http://rfc.zeromq.org/spec:15
>>>>>>>
>>>>>>> Also depending on the size of the msg, it may be allocated on the stack or
>>>>>>> heap.
>>>>>>> * http://api.zeromq.org/3-2:zmq-msg-init-size
>>>>>>>
>>>>>>> At some point you'll have to access a malloc'ed array from Java/JNI.
>>>>>>> *
>>>>>>> http://docs.oracle.com/javase/6/docs/technotes/guides/jni/spec/functions.
>>>>>>> html#nio_support
>>>>>>>
>>>>>>> Something like this would do the trick. This wraps a native pointer in a
>>>>>>> ByteBufer
>>>>>>>
>>>>>>> void *data = ...
>>>>>>> jobject bb = (*env)->NewDirectByteBuffer(env, (void*) data,
>>>>>>> sizeof(zmq_msg_t));
>>>>>>>
>>>>>>> Here is another option: Use a library like javolution to do your mapping:
>>>>>>>
>>>>>>> http://javolution.org/target/site/apidocs/javolution/io/Struct.html
>>>>>>>
>>>>>>> class Message extends Struct {
>>>>>>> Unsigned8 _ = new Unsigned8(32);
>>>>>>> Message() {
>>>>>>> setByteBuffer(Message.nativeBuffer(), 0);
>>>>>>> }
>>>>>>> private static native ByteBuffer nativeBuffer(); }
>>>>>>>
>>>>>>> Something along those lines.
>>>>>>>
>>>>>>> Hope this helps.
>>>>>>>
>>>>>>> -Trev
>>>>>>> _______________________________________________
>>>>>>> zeromq-dev mailing list
>>>>>>> zeromq-dev at lists.zeromq.org
>>>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>>
>>>>>> -----------------------------------------
>>>>>> Declaración de confidencialidad: Este Mensaje esta destinado para
>>>>>> el uso de la o las personas o entidades a quien ha sido dirigido y
>>>>>> puede contener información reservada y confidencial que no puede
>>>>>> ser divulgada, difundida, ni aprovechada en forma alguna. El uso no
>>>>>> autorizado de la información contenida en este correo podrá ser
>>>>>> sancionado de conformidad con la ley chilena.
>>>>>> Si usted ha recibido este correo electrónico por error, le pedimos
>>>>>> eliminarlo junto con los archivos adjuntos y avisar inmediatamente
>>>>>> al remitente, respondiendo este mensaje.
>>>>>>
>>>>>> "Before printing this e-mail think if is really necesary".
>>>>>> Disclosure: This Message is to be used by the individual,
>>>>>> individuals or entities that it is addressed to and may include
>>>>>> private and confidential information that may not be disclosed,
>>>>>> made public nor used in any way at all. Unauthorized use of the
>>>>>> information in this electronic mail message may be subject to the
>>>>>> penalties set forth by Chilean law.
>>>>>> If you have received this electronic mail message in error, we ask
>>>>>> you to destroy the message and its attached file(s) and to
>>>>>> immediately notify the sender by answering this message.
>>>>
>>>> _______________________________________________
>>>> zeromq-dev mailing list
>>>> zeromq-dev at lists.zeromq.org
>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
More information about the zeromq-dev
mailing list