ApacheCon NA 2015

High performance workloads have expanded. Today’s HPC users are demanding application frameworks to analyze vast amounts of data created by complex simulations. As the most widely deployed file system for HPC, Lustre software can play a critical role for these data-intensive applications. HPC Adapter for Lustre(HAL) provides adaptability to run Hadoop File System operations on HPC environment without any changes to applications. HPC Adapter for Mapreduce/Yarn(HAM) allows users to run their MapReduce/Yarn applications—without changes—directly on shared, fast, Lustre-powered storage. This optimizes the performance of MapReduce/Yarn tasks, while delivering faster, more scalable, easier-to-manage storage. This session explains the architecture and design level technical details to run Mapreduce and Yarn applications on HPC native schedulers like Slurm, MOAB, etc.