[zeromq-dev] pyczmq import overhead

[MinRK]

On Thu, Jan 16, 2014 at 12:12 PM, Michel Pelletier <
pelletier.michel at gmail.com> wrote:

> Hmm, I don't have any strong feelings either way.  All python imports
> seems slow to me on almost all useful libraries, I use a lot of numpy,
> scipy, pandas, and scikit-learn for my job and I'm used to multi-second
> startup times.  Even ipython takes a second or two on my new macbook.  For
> long running jobs and apps it doesn't seem to be an issue.
>
> If you're running very short scripts that need to fire and exit quickly,
> then I can see that being a problem, but 0.5 isn't a huge win to me.  But
> like I said, I don't have strong feelings, and I think you could count
> pyczmq's users on one hand at this point.  Anyone else have an opinion?
>
> Also, this might be some slowness related to using dlopen instead of
> verify in cffi.  verify is *much* slower the first time due to compilation,
> but then is much faster for subsequent imports.  Maybe investigating that
> approach first would be fruitful.
>

If you use cffi.verify, you can also do the compilation at build time,
as recommended
by cffi<http://cffi.readthedocs.org/en/latest/index.html?highlight=get_extension#distributing-modules-using-cffi>.
 PyZMQ does this<https://github.com/zeromq/pyzmq/blob/f58bb80b3b7583fb45824d0a1037128f7f29d44a/setup.py#L1003>
for
its CFFI backend on PyPy, and it seems to work reasonably well.

-MinRK


>
> -Michel
>
>
> On Thu, Jan 16, 2014 at 9:53 AM, Greg Ward <greg at gerg.ca> wrote:
>
>> Speaking of pyczmq, it is *very* slow to import:
>>
>>   $ time python -c 'import pyczmq'
>>   python -c 'import pyczmq'  1.11s user 0.05s system 100% cpu 1.152 total
>>
>> Turns out this is because pyczmq/__init__.py imports every submodule,
>> as well as some identifiers in a couple of submodules. I'm pretty sure
>> this is considered an anti-pattern. (I dimly recall seeing Guido van
>> Rossum disparage the practice on python-dev many years ago.)
>>
>> I've come to agree with him, if only on performance grounds. E.g.
>> here's the best case:
>>
>>   $ echo -n > pyczmq/__init__.py
>>   $ time python -c 'import pyczmq'
>>   python -c 'import pyczmq'  0.03s user 0.01s system 99% cpu 0.039 total
>>
>> A more realistic test:
>>
>>   $ time python -c 'from pyczmq import zmq, zctx, zsocket'
>>   python -c 'from pyczmq import zmq, zctx, zsocket'  0.46s user 0.05s
>> system 100% cpu 0.508 total
>>
>> Or, for all the submodules needed to implement Pieter's "ironhouse"
>> example in Python:
>>
>>   $ time python -c 'from pyczmq import zmq, zctx, zsocket, zcert, zauth,
>> zstr'
>>   python -c 'from pyczmq import zmq, zctx, zsocket, zcert, zauth, zstr'
>> 0.61s user 0.03s system 100% cpu 0.640 total
>>
>> That's not great, but 0.6 sec is still better than 1.1 sec.
>>
>> Is pyczmq young enough that we can consider breaking backwards
>> compatibility like this? Anyone currently doing
>>
>>   import pyczmq
>>
>>   ctx = pyczmq.zctx.new()
>>   sock = pyczmq.zsocket.new(...)
>>
>> would have to change their imports to:
>>
>>   import pyczmq.zctx
>>   import pyczmq.zsocket
>>
>> If that's OK, I'll work up a patch and send it.
>>
>>        Greg
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140117/149931ca/attachment.htm>