Within the framework of the European Copernicus programme, a series of Sentinel satellites have been launched that provide sensing capabilities across the whole measurement spectrum for serving a broad range of applications such as water resources monitoring, rapid mapping of flooding, and agricultural yield forecasting. Due to their advanced sensing concepts and outstanding spatio-temporal sampling characteristics, the Sentinel satellites collect more data than any Earth observation programme before. Every single day they acquire several Terabyte of data, which means that over the years every single Sentinel satellite collects several Petabytes of environmental data. To exploit the full wealth of information contained in this unique data set, it must be possible to process the data over and over again with ever improving scientific algorithms. This is only possible in dedicated data centres that offer the high performance processing capabilities needed to process multi-year global data sets at a fine spatial resolution. Recognizing this need, TU Wien founded the EODC Earth Observation Data Centre together with other Austrian partners in May 2014 as a public-private partnership. The mission of EODC is to work together with its partners from science, the public and the private sectors in order to foster the use of Earth observation data for monitoring of global environmental processes. To archive this the EODC partners have been working together to establish a federated IT infrastructure capable of storing and processing Petabytes of satellite data. The central site of this infrastructure is the Science Centre Arsenal of TU Wien, where a cloud platform and a storage system have been set up and connected to the Vienna Scientific Cluster (VSC). This infrastructure has been continually upgraded since 2014, offering access to a Petabyte-scale satellite data archive and capabilities for processing of high-resolution satellite images on a global scale.
The principal service of the EODC is to provide access to a Petabyte-scale data repository via a cloud platform (OpenStack) suited for scientific analysis and a supercomputer (VSC-3) for large-scale processing activities. The data repository encompasses worldwide Sentinel-1, Sentinel-2, Sentinel-3 Level 1 data, analysis-ready data cubes (e.g. Sentinel-1 and Sentinel-2 data cube over Austria), and various mirrored Earth science data sets (in situ, re-analysis, etc.). Furthermore, the EODC GmbH supports its principal users and EODC cooperation partners to scale-up and operationalise their scientific algorithms, embed their know-how into workflows going from the raw satellite and ground observations up to model predictions, and interfaces to the users.
Methods & Expertise for Research Infrastructure
The EODC infrastructure allows for the possibility to remotely access one’s workspace from anywhere and at the same time directly access all available data without the need to download them to a processing system. Users may connect from their own PC to
(1) a cloud platform, which provides users with virtual machines (VM) for developing and testing the methods, as well as visualizing the results on small spatio-temporal extents.
(2) storage for accessing EO data in the public part of the archive and for storing results in private folders. The storage is realised as a high-performance GPFS clustered file system with a net disk capacity of several Petabytes and a robotic tape library.
(3) the VSC supercomputer, through dedicated login nodes, and, subsequently, the computing nodes.
Wagner, W. (2015) Big Data infrastructures for processing Sentinel data, Photogrammetric Week 2015, Dieter Fritsch (Ed.), Wichmann Verlag/VDE Verlag, Berlin Offenbach, 93-104. (download from https://owncloud.tuwien.ac.at/index.php/s/DmgCkx624fGtLK8 with password "phowo")
Bucur, A., W. Wagner, S. Elefante, V. Naeimi, C. Briese (2018) Development of an Earth observation cloud platform in support to water resources monitoring, in: "Earth Observation Open Science and Innovation", ISSI Scientific Report Series, Vol 15, P. Mathieu, C. Aubrecht (ed.), Springer, 2018, 275-283. https://link.springer.com/chapter/10.1007/978-3-319-65633-5_14