Search
Abstract
The dynamic development of digital technologies, especially those dedicated to devices generating large data streams, such as all kinds of measurement equipment (temperature and humidity sensors, cameras, radio-telescopes and satellites – Internet of Things) enables more in-depth analysis of the surrounding reality, including better understanding of various natural phenomenon, starting from atomic level reactions, through macroscopic processes (e.g. meteorology) to observation of the Earth and the outer space. On the other hand such a large quantitative improvement requires a great number of processing and storage resources, resulting in the recent rapid development of Big Data technologies. Since 2015, the European Space Agency (ESA) has been providing a great amount of data gathered by exploratory equipment: a collection of Sentinel satellites – which perform Earth observation using various measurement techniques. For example Sentinel-2 provides a stream of digital photos, including images of the Baltic Sea and the whole territory of Poland. This data is used in an experimental installation of a Big Data processing system based on the open source software at the Academic Computer Center in Gdansk. The center has one of the most powerful supercomputers in Poland – the Tryton computing cluster, consisting of 1600 nodes interconnected by a fast Infiniband network (56 Gbps) and over 6 PB of storage. Some of these nodes are used as a computational cloud supervised by an OpenStack platform, where the Sentinel-2 data is processed. A subsystem of the automatic, perpetual data download to object storage (based on Swift) is deployed, the required software libraries for the image processing are configured and the Apache Spark cluster has been set up. The above system enables gathering and analysis of the recorded satellite images and the associated metadata, benefiting from the parallel computation mechanisms. This paper describes the above solution including its technical aspects.
Citations
-
0
CrossRef
-
0
Web of Science
-
0
Scopus
Cite as
Full text
- Publication version
- Accepted or Published Version
- License
-
open in new tab
Keywords
Details
- Category:
- Articles
- Type:
- artykuły w czasopismach
- Published in:
-
TASK Quarterly
no. 21,
pages 365 - 377,
ISSN: 1428-6394 - Language:
- English
- Publication year:
- 2017
- Bibliographic description:
- Proficz J., Drypczewski K.: Processing of Satellite Data in the Cloud// TASK Quarterly -Vol. 21,iss. 4 (2017), s.365-377
- DOI:
- Digital Object Identifier (open in new tab) 10.17466/tq2017/21.4/y
- Bibliography: test
-
show (27) hide
- Demchenko Y, De Laat C and Membrey P 2014 CTS, IEEE 104 open in new tab
- Apache Spark -Lightning-fast cluster computing [Online] available at: https://spark. apache.org/ [Accessed: 25-July-2017] open in new tab
- Apache Hadoop -website [Online] available at: https://hadoop.apache.org/ [Acces- sed: 23-June-2017]
- Thensorflow Homepage [Online] available at: https://www.tensorflow.org/ [Accessed: 23-June-2017] open in new tab
- Torch -a scientific computing platform for LuaJIT [Online] available at: http://torch .ch/ [Accessed: 23-June-2017] open in new tab
- Theano -a library to define, optimize, and evaluate mathematical expressions [Online] available at: http://deeplearning.net/software/theano/ [Accessed: 26-June-2017] open in new tab
- Caffe -Deep learning framework [Online] available at: http://caffe. berkeleyvision.org/ [Accessed: 26-June-2017] open in new tab
- Copernicus -observing the Earth, ESA programe [Online] available at: http://www.esa. int/Our Activities/Observing the Earth/Copernicus [Accessed: 26-June-2017]
- Sentinel-2 Homepage [Online] available at: http://www.esa.int/Our Activities/ Observing the Earth/Copernicus/Sentinel-2 [Accessed: 23-June-2017]
- Vega -Arianspace [Online] available at: http://www.arianespace.com/vehicle/vega/ [Accessed: 06-July-2017] open in new tab
- Sentinel-2 MSI, Data formats [Online] available at: https://sentinel.esa.int/web/ sentinel/user-guides/sentinel-2-msi/data-formats [Accessed: 23-July-2017] open in new tab
- Tryton Supercomputer, Academic Computer Centre TASK [Online] available at: https: //task.gda.pl/kdm/sprzet/tryton/ open in new tab
- OpenStack, Open Source Cloud Computing Software [Online] available at: https://www. openstack.org/ [Accessed: 27-June-2017]
- Ceph Homepage [Online] available at: http://ceph.com/ [Accessed: 27-June-2017]
- Swift -OpenStack [Online] available at: https://wiki.openstack.org/wiki/Swift [Accessed: 27-June-2017]
- Apache Hadoop YARN [Online] available at: https://hadoop.apache.org/docs/r2.7.3 /hadoop-yarn/hadoop-yarn-site/YARN.html [Accessed: 27-June-2017]
- Apache Mesos [Online] available at:http://mesos.apache.org/ [Accessed: 28-June-2017]
- Kubernetes -Production-Grade Container Orchestration [Online] available at: https:// kubernetes.io/ [Accessed: 28-June-2017] open in new tab
- HDFS Architecture Guide [Online] available at: https://hadoop.apache.org/docs/r1. 2.1/hdfs design.html [Accessed: 28-June-2017]
- Apache HBase Home [Online] available at: https://hbase.apache.org/ [Accessed: 28-June-2017]
- Apache Cassandra Homepage [Online] available at: http://cassandra.apache.org/ [Accessed: 28-June-2017]
- JPEG 2000 Homepage [Online] available at: https://jpeg.org/jpeg2000/ [Accessed: 28-June-2017] open in new tab
- The ImageIO-Ext Project Home [Online] available at: https://github.com/ geosolutions-it/imageio-ext/ [Accessed: 29-June-2017] open in new tab
- OpenJPEG, an open-source JPEG 2000 codec [Online] available at: http://www. openjpeg.org/ [Accessed: 29-June-2017]
- Odersky M 2014 Scala by Example, Programing Methods Laboratory [Online] available at: http://www.scala-lang.org/docu/files/ScalaByExample.pdf, EPFL [Accessed: 26-July-2017]
- Sentinel-2 User Handbook [Online] available at: https://sentinel.esa.int/documents /247904/685211/Sentinel-2 User Handbook, ESA [Accessed: 03-July-2017] open in new tab
- H2O Homepage [Online] available at: https://www.h2o.ai/ [Accessed: 03-July-2017] open in new tab
- Verified by:
- Gdańsk University of Technology
seen 158 times
Recommended for you
Big Data and the Internet of Things in Edge Computing for Smart City
- J. Balicki,
- H. Balicka,
- P. Dryja
- + 1 authors