ApacheCon NA 2015

Batch processing is common in industries for offline data compute. Typically it is bulk-oriented, non-interactive & long running—and is data & computation-intensive.
Batch platform requires business to maintain & operate huge backend infrastructure. While there are hadoop infrastructures, not all batch jobs can be run on hadoop. Compute requirement for batch jobs also varies widely. It is most desirable to have an infrastructure, which is shared, scale on demand, easy to maintain.
We propose an architecture for “Elastic compute for batch platform” using Mesos, Docker, swift object storage & chronos scheduler. We made sure user perception of state-full elastic workload is protected while giving flexibility to the backend infrastructure to isolate compute, storage, environment and monitoring.