The unprecedented ability to collect massive datasets from large scientific instruments and enterprise data warehouses offers grand challenges for data-intensive computing. Concomitantly, computing infrastructure is itself undergoing swift changes both in architectures and resource access models. With even smart phones being equipped with multiple, heterogeneous CPU cores, distributed computing is the norm rather than the exception.
This confluence of “Big Data” applications with emerging computing infrastructure can lead to transformative scientific advances. However, translating this opportunity to scientific discovery requires advances in the software cyber-infrastructure plane. This spans clean-slate programming abstractions for developers to compose distributed applications, to novel algorithms that can utilize the potential of such abstractions to scale their techniques, to execution frameworks that allow transparent, resilient and efficient access to such advanced computing facilities. Such an integrated view of cyber-infrastructure needs to be as much tuned to the characteristics of the data that they operate upon (e.g. volume, velocity, variety) as to the computational infrastructure that they execute upon (e.g. elasticity, reliability, cost).
The Distributed Research on Emerging Applications and Machines Lab (DREAM:Lab) focuses on holistic distributed systems research that enables the effective and efficient use of emerging distributed data and computing systems, using scalable software architectures, innovative programming and data abstractions, and algorithms for optimal distributed execution, to support data intensive scientific and engineering applications, which can lead to transformative advances to society.
Housed at the Indian Institute of Science‘s Department of Computational and Data Sciences (CDS), a unique inter-disciplinary department in India offering programs on computational and data sciences, the DREAM:Lab explores the vertical data sciences’ stack, from Big Data applications to emerging distributed infrastructure.
- Distributed machines we consider include public and private Clouds, infrastructure and platform as a service (IaaS and PaaS), and virtualized and commodity clusters. With many core architectures, heterogeneous accelerators and low-power CPUs becoming first-class commodity hardware, these offer novel challenges in this distributed environment.
- Scalable software systems range from private Cloud fabrics that manage data-center resources efficiently and scalably to support hundreds-thousands of commodity cores, petabytes of data, and hundreds of users, all the way up to Big Data programming frameworks, execution environments, and scientific gateway portals that support end user scientists. Such software cyber-infrastructure translates theory into practice, and forms the critical enabler in empirically validating the computer science hypothesis or theoretial analysis.
- Programming and data abstractions offer models of computation and data representation that is intuitive to map external applications to, while at the same time providing a structure that allows them to be executed at scale using the distributed resources. Scientific workflows and MapReduce are examples of successful abstractions for Grid and Cloud computing, and many such domain-specific (graph analytics), resource-specific (Clouds) and generalizable (continuous dataflows) abstractions are possible.
- Algorithms span two levels; one to provide optimal execution strategies that imbue resilience, scalability and efficiency when running applications on distributed system; and two, to transform classic shared memory algorithms and applications into distributed algorithms that can leverage the programming abstractions. Advances in the algorithmic space can offer dramatic improvements in performance and scalability.
- Big Data Science Applications: As the scientific and engineering domains contend with an influx of massive data, they offer a valuable context to apply the advances made in distributed systems research as well as a rich space for discovering novel problems that are as yet unaddressed. Cyber Physical Systems and Genomics are initial areas of domain focus.
The research activities of the DREAM:Lab will advance fundamental knowledge on effectively scaling data-driven scientific applications on contemporary and emerging distributed computing infrastructure. Further, the applied nature of this research will translate novel research outcomes into sustainable software prototypes that will help accelerate scientific discovery in critical application domains of national importance. Taking an integrated view across the research stack, from the system to the application, is important. It is avoids conducting research in a vacuum, under idealized conditions detached from reality. This is particularly important for systems research due to the fast changing nature of computing technology and advances in hardware architectures. At the same time, this must not degenerate to building software, systems or applications as an end in themselves, in the absence of tangible research outcomes. Such practical grounding will also illustrate to students the value of inter-disciplinary research while also helping train the research scientists and work-force of the future on advance technologies.