E-infrastructure for climate and atmospheric science research

Abstract

Recent government in e-Infrastructure will transform aspects of environmental science by supporting both fundamental science and innovative uses of environmental data by the commmercial sector. The STFC Centre for Environmental Data Archival (CEDA) is heavily involved in two major projects: JASMIN - a NERC funded facility which will support both data archival and scientific data analysis, and CEMS - the Facility for Climate and Environmental Monitoring from Space - aimed at fostering knowledge exchange and commercial exploitation of environmental data. JASMIN and CEMS will share some hardware. In this presentation, we concentrate on JASMIN, which will consist of multi-Petabyte fast reliable storage and co-located data analysis compute at the STFC Rutherford Appleton Laboratory, with satellite installations at Reading, Leeds and Bristol Universities. JASMIN is a response to the growing use of direct numerical simulation in the environmental sciences resulting in much higher demand for high performance computing. This growth in HPC is accompanied by a transition in its nature, with data intensive HPC becoming an ever increasing part of the mix. (For example, at the time of writing CEDA is currently evaluating the requirements in terms of storage and co-located analysis compute for three grants each of which is expected to produce in excess of 0.5 PB of data over the next three years - this on top of known data acquisition already measured in PB. Clearly every grant round could bring similar requirements.) Such data intensive HPC is being carried out on on many different supercomputers, so it is no longer satisfactory to assume that putting storage alongside the HPC will solve the analysis problem (since such a solution, alone, could result in an NxN data transfer problem for data comparison between results on N supercomputers). Inevitably one needs to reduce the data transfer problem down to as close to Nx1 as possible - hence JASMIN - a facility configured for data storage AND analysis. For analysis, JASMIN will deploy a "private cloud" to allow the community to develop their own analysis environment using their favourite operating system configuration. JASMIN will also be used, along with a large tape facilities provided by STFC, to provide persistent storage for the archival and curation functions which CEDA also provides. These storage and computing advances will be supported by high-bandwidth network connectivity between key collaborating institutions (particularly supercomputing sites), both within the UK and in the Europe, and new light paths have been established alongside the JASMIN activity. JASMIN: Joint Analysis System Meeting e-Infrastructure Needs