mystic+pathos+dill: let's serialize all of python, and do large-scale high-performance parallel optimization

Date Time: Mike McKerns

We have built a robust framework (mystic) that lowers the barrier to solving complex problems in predictive science. Mystic is built to rigorously solve high-dimensional non-convex optimization problems with highly nonlinear complex constraints. Mystic is capable of solving global optimization problems with thousands of parameters and thousands of constraints, and makes it almost trivial to leverage high-performance parallel computing. Mystic's unique ability to apply highly complex and statistical constraints can be used to find optimal probability distributions, calculate risk, uncertainty, sensitivity, and probability of failure in real-world inverse problems. Mystic leverages a computational job management framework (pathos) that offers a simple, efficient, and consistent user experience across a variety of heterogeneous environments from multi-core workstations to networks of large-scale computer clusters. Both mystic and pathos leverage a serialization package (dill) that can pickle almost the entire set of types in the python standard distribution.

Come join us to help port your favorite optimizer to mystic, to add new constraints solvers and stop conditions, or to extend the new set of parallel ensemble solvers. If you are interested in easy-to-use parallel computing, then help us extend pathos's map-reduce and pipe capabilities to the cloud and beyond. Why not pitch in and either help complete the set of things that dill can pickle, or help us enable multiprocessing and other python core packages to use dill. We are also looking to troubleshoot pickling errors that dill encounters in ipython on types that dill can serialize in standard python.