[zeromq-dev] Problems with pyczmq.zstr (was Re: pyczmq and ctx.set_linger())

[Michel Pelletier]

Ugh, gmail screwed up my paste:

https://gist.github.com/michelp/8482432

-Michel


On Fri, Jan 17, 2014 at 1:51 PM, Michel Pelletier <
pelletier.michel at gmail.com> wrote:

> It wasn't the recv that was truncating the string, it was the send.
>
> For string that contain null chars, you'll have to use zframe send/recv.
>  Here's your example working with those functions:
>
> File Edit Options Buffers Tools Python Help
>
> from pyczmq import zmq, zctx, zsocket, zstr, zframe
>
> ctx = zctx.new()
> rep = zsocket.new(ctx, zmq.REP)
> zsocket.bind(rep, "tcp://127.0.0.1:5253")
>
> req = zsocket.new(ctx, zmq.REQ)
> zsocket.connect(req, "tcp://127.0.0.1:5253")
>
> out_data = "blah blah hey what is \0 doing here?"
> out_frame = zframe.new(out_data)
> zframe.send(out_frame, req, 0)
>
> in_frame = zframe.recv(rep)
> in_data = bytearray(zframe.data(in_frame))
> assert in_data == out_data, "expected %r, got %r" % (out_data, in_data)
>
>
>
>
>
>
>
>
>
>
>
>
> On Fri, Jan 17, 2014 at 1:20 PM, Michel Pelletier <
> pelletier.michel at gmail.com> wrote:
>
>> Ah you've got a good catch there, browsing the docs cffi does support
>> variadic function calls
>> https://cffi.readthedocs.org/en/release-0.8/#variadic-function-calls
>>
>> I'll study this after work tonight, I'm sure we can address the issue.
>>
>> As for the null char in the middle of a received string, I'll have to
>> think about that one.  I'm open to suggestions.
>>
>> -Michel
>>
>>
>> On Fri, Jan 17, 2014 at 12:55 PM, Greg Ward <greg at gerg.ca> wrote:
>>
>>> On 17 January 2014, Michel Pelletier said:
>>> > zstr however is how you turn messages into strings.  It is not an
>>> > implementation of a string type.  It is critical to the functionality
>>> of
>>> > pyczmq (assuming you want string output from the library).
>>>
>>> I was going to start another thread about this... but what the hell,
>>> as long as it came up, here are the two problems I've spotted with
>>> pyczmq.zstr.send().
>>>
>>> 1. zstr_send() is printf()-style, pyczmq.zstr.send() is not
>>> -----------------------------------------------------------
>>>
>>> The C API for zstr_send() is
>>>
>>>   zstr_send(void *zocket, const char *format, ...)
>>>
>>> which means calling it like
>>>
>>>   char *data = [...get a string from somewhere...]
>>>   zstr_send(mysocket, data);
>>>
>>> is wrong, because 'data' might happen to contain a "%s" or "%d" by
>>> chance. Presumably the innards of zstr_send() will look for further
>>> args on the stack, find gibberish, and send gibberish over the wire.
>>> Or crash. The right way to do this in C is
>>>
>>>   zstr_send(mysocket, "%s", data);
>>>
>>> But the corresponding Python function is
>>>
>>>   def send(sock, string):
>>>       return C.zstr_send(sock, string)
>>>
>>> so if 'string' happens to contain "%s" or "%d", bad stuff happens. But
>>> since pyczmq.zstr.send() doesn't expose a format string, there's no
>>> way to do the right thing.
>>>
>>> Here's an example:
>>>
>>>   $ cat zstr-bug1.py
>>>   from pyczmq import zmq, zctx, zsocket, zstr
>>>
>>>   ctx = zctx.new()
>>>   rep = zsocket.new(ctx, zmq.REP)
>>>   zsocket.bind(rep, "tcp://127.0.0.1:5253")
>>>
>>>   req = zsocket.new(ctx, zmq.REQ)
>>>   zsocket.connect(req, "tcp://127.0.0.1:5253")
>>>
>>>   out_data = "blah blah hey what is %s doing here?"
>>>   zstr.send(req, out_data)
>>>
>>>   in_data = zstr.recv(rep)
>>>   assert in_data == out_data, "expected %r, got %r" % (out_data, in_data)
>>>
>>>   $ python zstr-bug1.py
>>>   zsh: segmentation fault (core dumped)  python zstr-bug1.py
>>>
>>> Here are the innermost stack frames from that segfault:
>>>
>>> """
>>> #0  0x00007f3c06a4bf90 in _IO_vfprintf_internal (s=s at entry
>>> =0x7fffd8435470,
>>>     format=<optimized out>,
>>>     format at entry=0x7f3c07124fa4 "blah blah hey what is %s doing here?",
>>>     ap=ap at entry=0x7fffd84355e8) at vfprintf.c:1655
>>> #1  0x00007f3c06b0f6c0 in ___vsnprintf_chk (
>>>     s=s at entry=0x1120750 "blah blah hey what is ", maxlen=<optimized
>>> out>,
>>>     maxlen at entry=256, flags=flags at entry=1, slen=slen at entry=256,
>>>     format=format at entry=0x7f3c07124fa4 "blah blah hey what is %s doing
>>> here?",
>>>     args=args at entry=0x7fffd84355e8) at vsnprintf_chk.c:63
>>> #2  0x00007f3c04af0d1b in vsnprintf (__ap=0x7fffd84355e8,
>>>     __fmt=0x7f3c07124fa4 "blah blah hey what is %s doing here?", __n=256,
>>>     __s=0x1120750 "blah blah hey what is ")
>>>     at /usr/include/x86_64-linux-gnu/bits/stdio2.h:77
>>> #3  zsys_vprintf (
>>>     format=0x7f3c07124fa4 "blah blah hey what is %s doing here?",
>>>     argptr=argptr at entry=0x7fffd8435638) at zsys.c:386
>>> #4  0x00007f3c04af0499 in zstr_send (zocket=0x11c2d30, format=<optimized
>>> out>)
>>>     at zstr.c:112
>>> #5  0x00007f3c05521adc in ffi_call_unix64 ()
>>>    from /usr/lib/x86_64-linux-gnu/libffi.so.6
>>> """
>>>
>>> 2. zstr_send() is not binary safe
>>> ---------------------------------
>>>
>>> The fact that CZMQ's zstr_send() is not binary safe is just fine: it
>>> says right in the docs that it's a convenience for sending C strings
>>> around. So the fact that pyczmq.zstr.send() is not binary safe is not
>>> surprising (although it is annoying -- I'm used to Python strings, not
>>> C strings). But since you say it's "critical to the functionality of
>>> pyczmq", maybe I should worry about it. Anyways, here's an example:
>>>
>>>   $ cat zstr-bug2.py
>>>   from pyczmq import zmq, zctx, zsocket, zstr
>>>
>>>   ctx = zctx.new()
>>>   rep = zsocket.new(ctx, zmq.REP)
>>>   zsocket.bind(rep, "tcp://127.0.0.1:5253")
>>>
>>>   req = zsocket.new(ctx, zmq.REQ)
>>>   zsocket.connect(req, "tcp://127.0.0.1:5253")
>>>
>>>   out_data = "blah blah hey what is \0 doing here?"
>>>   zstr.send(req, out_data)
>>>
>>>   in_data = zstr.recv(rep)
>>>   assert in_data == out_data, "expected %r, got %r" % (out_data, in_data)
>>>
>>>   $ python zstr-bug2.py
>>>   Traceback (most recent call last):
>>>     File "zstr-bug2.py", line 14, in <module>
>>>       assert in_data == out_data, "expected %r, got %r" % (out_data,
>>> in_data)
>>>   AssertionError: expected 'blah blah hey what is \x00 doing here?', got
>>> 'blah blah hey what is '
>>>
>>> So: until I get to the point of needing secure authentication, pyczmq
>>> just gets in my way and makes life harder. Darn.
>>>
>>>        Greg
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140117/3f049ec3/attachment.htm>