[zeromq-dev] Problems with pyczmq.zstr (was Re: pyczmq and ctx.set_linger())

Pieter Hintjens ph at imatix.com
Sat Jan 18 08:34:08 CET 2014


Yes, I thought it was cool to make a few of the string functions, like
zstr_send, accept printf formats. I've also been bitten more than once
by that.

Perhaps we can consider this API style to be broken, and dangerous,
and remove the printf formatting from zstr_send, putting it into
zstr_sendf instead?

There are a bunch of other cases like this. It'll break the CZMQ API
but I think it's justified if we're fixing a design fault.

-Pieter


On Fri, Jan 17, 2014 at 10:53 PM, Michel Pelletier
<pelletier.michel at gmail.com> wrote:
> Ugh, gmail screwed up my paste:
>
> https://gist.github.com/michelp/8482432
>
> -Michel
>
>
> On Fri, Jan 17, 2014 at 1:51 PM, Michel Pelletier
> <pelletier.michel at gmail.com> wrote:
>>
>> It wasn't the recv that was truncating the string, it was the send.
>>
>> For string that contain null chars, you'll have to use zframe send/recv.
>> Here's your example working with those functions:
>>
>> File Edit Options Buffers Tools Python Help
>> from pyczmq import zmq, zctx, zsocket, zstr, zframe
>>
>> ctx = zctx.new()
>> rep = zsocket.new(ctx, zmq.REP)
>> zsocket.bind(rep, "tcp://127.0.0.1:5253")
>>
>> req = zsocket.new(ctx, zmq.REQ)
>> zsocket.connect(req, "tcp://127.0.0.1:5253")
>>
>> out_data = "blah blah hey what is \0 doing here?"
>> out_frame = zframe.new(out_data)
>> zframe.send(out_frame, req, 0)
>>
>> in_frame = zframe.recv(rep)
>> in_data = bytearray(zframe.data(in_frame))
>> assert in_data == out_data, "expected %r, got %r" % (out_data, in_data)
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Jan 17, 2014 at 1:20 PM, Michel Pelletier
>> <pelletier.michel at gmail.com> wrote:
>>>
>>> Ah you've got a good catch there, browsing the docs cffi does support
>>> variadic function calls
>>> https://cffi.readthedocs.org/en/release-0.8/#variadic-function-calls
>>>
>>> I'll study this after work tonight, I'm sure we can address the issue.
>>>
>>> As for the null char in the middle of a received string, I'll have to
>>> think about that one.  I'm open to suggestions.
>>>
>>> -Michel
>>>
>>>
>>> On Fri, Jan 17, 2014 at 12:55 PM, Greg Ward <greg at gerg.ca> wrote:
>>>>
>>>> On 17 January 2014, Michel Pelletier said:
>>>> > zstr however is how you turn messages into strings.  It is not an
>>>> > implementation of a string type.  It is critical to the functionality
>>>> > of
>>>> > pyczmq (assuming you want string output from the library).
>>>>
>>>> I was going to start another thread about this... but what the hell,
>>>> as long as it came up, here are the two problems I've spotted with
>>>> pyczmq.zstr.send().
>>>>
>>>> 1. zstr_send() is printf()-style, pyczmq.zstr.send() is not
>>>> -----------------------------------------------------------
>>>>
>>>> The C API for zstr_send() is
>>>>
>>>>   zstr_send(void *zocket, const char *format, ...)
>>>>
>>>> which means calling it like
>>>>
>>>>   char *data = [...get a string from somewhere...]
>>>>   zstr_send(mysocket, data);
>>>>
>>>> is wrong, because 'data' might happen to contain a "%s" or "%d" by
>>>> chance. Presumably the innards of zstr_send() will look for further
>>>> args on the stack, find gibberish, and send gibberish over the wire.
>>>> Or crash. The right way to do this in C is
>>>>
>>>>   zstr_send(mysocket, "%s", data);
>>>>
>>>> But the corresponding Python function is
>>>>
>>>>   def send(sock, string):
>>>>       return C.zstr_send(sock, string)
>>>>
>>>> so if 'string' happens to contain "%s" or "%d", bad stuff happens. But
>>>> since pyczmq.zstr.send() doesn't expose a format string, there's no
>>>> way to do the right thing.
>>>>
>>>> Here's an example:
>>>>
>>>>   $ cat zstr-bug1.py
>>>>   from pyczmq import zmq, zctx, zsocket, zstr
>>>>
>>>>   ctx = zctx.new()
>>>>   rep = zsocket.new(ctx, zmq.REP)
>>>>   zsocket.bind(rep, "tcp://127.0.0.1:5253")
>>>>
>>>>   req = zsocket.new(ctx, zmq.REQ)
>>>>   zsocket.connect(req, "tcp://127.0.0.1:5253")
>>>>
>>>>   out_data = "blah blah hey what is %s doing here?"
>>>>   zstr.send(req, out_data)
>>>>
>>>>   in_data = zstr.recv(rep)
>>>>   assert in_data == out_data, "expected %r, got %r" % (out_data,
>>>> in_data)
>>>>
>>>>   $ python zstr-bug1.py
>>>>   zsh: segmentation fault (core dumped)  python zstr-bug1.py
>>>>
>>>> Here are the innermost stack frames from that segfault:
>>>>
>>>> """
>>>> #0  0x00007f3c06a4bf90 in _IO_vfprintf_internal
>>>> (s=s at entry=0x7fffd8435470,
>>>>     format=<optimized out>,
>>>>     format at entry=0x7f3c07124fa4 "blah blah hey what is %s doing here?",
>>>>     ap=ap at entry=0x7fffd84355e8) at vfprintf.c:1655
>>>> #1  0x00007f3c06b0f6c0 in ___vsnprintf_chk (
>>>>     s=s at entry=0x1120750 "blah blah hey what is ", maxlen=<optimized
>>>> out>,
>>>>     maxlen at entry=256, flags=flags at entry=1, slen=slen at entry=256,
>>>>     format=format at entry=0x7f3c07124fa4 "blah blah hey what is %s doing
>>>> here?",
>>>>     args=args at entry=0x7fffd84355e8) at vsnprintf_chk.c:63
>>>> #2  0x00007f3c04af0d1b in vsnprintf (__ap=0x7fffd84355e8,
>>>>     __fmt=0x7f3c07124fa4 "blah blah hey what is %s doing here?",
>>>> __n=256,
>>>>     __s=0x1120750 "blah blah hey what is ")
>>>>     at /usr/include/x86_64-linux-gnu/bits/stdio2.h:77
>>>> #3  zsys_vprintf (
>>>>     format=0x7f3c07124fa4 "blah blah hey what is %s doing here?",
>>>>     argptr=argptr at entry=0x7fffd8435638) at zsys.c:386
>>>> #4  0x00007f3c04af0499 in zstr_send (zocket=0x11c2d30, format=<optimized
>>>> out>)
>>>>     at zstr.c:112
>>>> #5  0x00007f3c05521adc in ffi_call_unix64 ()
>>>>    from /usr/lib/x86_64-linux-gnu/libffi.so.6
>>>> """
>>>>
>>>> 2. zstr_send() is not binary safe
>>>> ---------------------------------
>>>>
>>>> The fact that CZMQ's zstr_send() is not binary safe is just fine: it
>>>> says right in the docs that it's a convenience for sending C strings
>>>> around. So the fact that pyczmq.zstr.send() is not binary safe is not
>>>> surprising (although it is annoying -- I'm used to Python strings, not
>>>> C strings). But since you say it's "critical to the functionality of
>>>> pyczmq", maybe I should worry about it. Anyways, here's an example:
>>>>
>>>>   $ cat zstr-bug2.py
>>>>   from pyczmq import zmq, zctx, zsocket, zstr
>>>>
>>>>   ctx = zctx.new()
>>>>   rep = zsocket.new(ctx, zmq.REP)
>>>>   zsocket.bind(rep, "tcp://127.0.0.1:5253")
>>>>
>>>>   req = zsocket.new(ctx, zmq.REQ)
>>>>   zsocket.connect(req, "tcp://127.0.0.1:5253")
>>>>
>>>>   out_data = "blah blah hey what is \0 doing here?"
>>>>   zstr.send(req, out_data)
>>>>
>>>>   in_data = zstr.recv(rep)
>>>>   assert in_data == out_data, "expected %r, got %r" % (out_data,
>>>> in_data)
>>>>
>>>>   $ python zstr-bug2.py
>>>>   Traceback (most recent call last):
>>>>     File "zstr-bug2.py", line 14, in <module>
>>>>       assert in_data == out_data, "expected %r, got %r" % (out_data,
>>>> in_data)
>>>>   AssertionError: expected 'blah blah hey what is \x00 doing here?', got
>>>> 'blah blah hey what is '
>>>>
>>>> So: until I get to the point of needing secure authentication, pyczmq
>>>> just gets in my way and makes life harder. Darn.
>>>>
>>>>        Greg
>>>> _______________________________________________
>>>> zeromq-dev mailing list
>>>> zeromq-dev at lists.zeromq.org
>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>



More information about the zeromq-dev mailing list