Yet another Python map()

Wed, 16 Apr 2008 08:11:00 CDT

In another article, I described a replacement for Python's built-in map() function that could take advantage of multi-processing systems. That one was based on the standard Unix fork(). Since then, I've written another based on Parallel Python that is much simpler and lets other, better-tested code do all the hard work. It could also be easily extended to run on a cluster instead of just the local system, but I haven't been inclined to tinker with that too much yet.

Note one possibly important difference from the builtin map() function: this version returns a generator that yields values as they are calculated. That way, you can launch the parallel processes then go on with other work while you give the workers a chance to finish their jobs.

Kudos to Connelly for nudging me to finally publish this.

Download: ppmap.py

Multi-processing map() for Python

Fri, 06 Jul 2007 15:35:13 CDT

I think that Python should use multi-processing and/or multi-threading to take advantage of as many opportunities for parallel execution as possible. To this end, I've written a drop-in replacement for map() that runs across as many processes as requested. It should be otherwise identical in every way the built-in version (and if it's not please let me know!).

I also wrote a version based on Parallel Python that is a lot simpler but not quite identical to the original. In particular, it returns a generator instead of a list of values so that program execution doesn't block until the results are fetched.

Drop me a line if you find this interesting or useful or just plain dumb.

Download: forkmap.py

Honeypot.net

Yet another Python map()

Multi-processing map() for Python