Loading…
This event has ended. View the official site or create your own event → Check it out
This event has ended. Create your own
The attendees list includes all authors (even thought they may not be attending), speakers, artists, etc. 

View the full conference website here:
IEEE Cluster 2013 Conference
View analytic
Wednesday, September 25 • 5:00pm - 5:25pm
Scaling of Many-Task Computing Approaches in Python on Cluster Supercomputers

Sign up or log in to save this to your schedule and see who's attending!

We compare three packages for performing many-task computing in Python: pi4py, IPython Parallel, and Celery. Each of these implementations are based on a different message passing standard, which implies that in an indirect way, we are comparing the Message Passing Interface (MPI)---the current standard for distributed computing---with two emerging technologies, the Advanced Message Queuing Protocol (AMQP) and ZeroMQ. We describe these packages in detail and compare their features as applied to task-based parallel computing on a cluster, including a scaling study using over 12,000 cores and several thousand tasks. Our results suggest that these new, fault-tolerant technologies have a clear place in cluster computing, and that no single technique is the obvious choice. To the best of our knowledge, this is the first time that IPython Parallel and Celery have been run at this scale.


Wednesday September 25, 2013 5:00pm - 5:25pm
15th Floor - Circle City 15 (Hilton) 120 W. Market St, Indianapolis, IN

Attendees (2)