Yet Another Python Map

Wed, Apr 16, 2008 One-minute read

In another article, I described a replacement for Python’s built-in map() function that could take advantage of multi-processing systems. That one was based on the standard Unix fork(). Since then, I’ve written another based on Parallel Python that is much simpler and lets other, better-tested code do all the hard work. It could also be easily extended to run on a cluster instead of just the local system, but I haven’t been inclined to tinker with that too much yet.

Note one possibly important difference from the builtin map() function: this version returns a generator that yields values as they are calculated. That way, you can launch the parallel processes then go on with other work while you give the workers a chance to finish their jobs.

Kudos to Connelly for nudging me to finally publish this.