[zeromq-dev] pyczmq import overhead

Michel Pelletier pelletier.michel at gmail.com
Thu Jan 16 21:12:41 CET 2014


Hmm, I don't have any strong feelings either way.  All python imports seems
slow to me on almost all useful libraries, I use a lot of numpy, scipy,
pandas, and scikit-learn for my job and I'm used to multi-second startup
times.  Even ipython takes a second or two on my new macbook.  For long
running jobs and apps it doesn't seem to be an issue.

If you're running very short scripts that need to fire and exit quickly,
then I can see that being a problem, but 0.5 isn't a huge win to me.  But
like I said, I don't have strong feelings, and I think you could count
pyczmq's users on one hand at this point.  Anyone else have an opinion?

Also, this might be some slowness related to using dlopen instead of verify
in cffi.  verify is *much* slower the first time due to compilation, but
then is much faster for subsequent imports.  Maybe investigating that
approach first would be fruitful.

-Michel


On Thu, Jan 16, 2014 at 9:53 AM, Greg Ward <greg at gerg.ca> wrote:

> Speaking of pyczmq, it is *very* slow to import:
>
>   $ time python -c 'import pyczmq'
>   python -c 'import pyczmq'  1.11s user 0.05s system 100% cpu 1.152 total
>
> Turns out this is because pyczmq/__init__.py imports every submodule,
> as well as some identifiers in a couple of submodules. I'm pretty sure
> this is considered an anti-pattern. (I dimly recall seeing Guido van
> Rossum disparage the practice on python-dev many years ago.)
>
> I've come to agree with him, if only on performance grounds. E.g.
> here's the best case:
>
>   $ echo -n > pyczmq/__init__.py
>   $ time python -c 'import pyczmq'
>   python -c 'import pyczmq'  0.03s user 0.01s system 99% cpu 0.039 total
>
> A more realistic test:
>
>   $ time python -c 'from pyczmq import zmq, zctx, zsocket'
>   python -c 'from pyczmq import zmq, zctx, zsocket'  0.46s user 0.05s
> system 100% cpu 0.508 total
>
> Or, for all the submodules needed to implement Pieter's "ironhouse"
> example in Python:
>
>   $ time python -c 'from pyczmq import zmq, zctx, zsocket, zcert, zauth,
> zstr'
>   python -c 'from pyczmq import zmq, zctx, zsocket, zcert, zauth, zstr'
> 0.61s user 0.03s system 100% cpu 0.640 total
>
> That's not great, but 0.6 sec is still better than 1.1 sec.
>
> Is pyczmq young enough that we can consider breaking backwards
> compatibility like this? Anyone currently doing
>
>   import pyczmq
>
>   ctx = pyczmq.zctx.new()
>   sock = pyczmq.zsocket.new(...)
>
> would have to change their imports to:
>
>   import pyczmq.zctx
>   import pyczmq.zsocket
>
> If that's OK, I'll work up a patch and send it.
>
>        Greg
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140116/2a2b7e43/attachment.htm>


More information about the zeromq-dev mailing list