Processing of Satellite Data in the Cloud - Publikacja - MOST Wiedzy

Wyszukiwarka

Processing of Satellite Data in the Cloud

Abstrakt

The dynamic development of digital technologies, especially those dedicated to devices generating large data streams, such as all kinds of measurement equipment (temperature and humidity sensors, cameras, radio-telescopes and satellites – Internet of Things) enables more in-depth analysis of the surrounding reality, including better understanding of various natural phenomenon, starting from atomic level reactions, through macroscopic processes (e.g. meteorology) to observation of the Earth and the outer space. On the other hand such a large quantitative improvement requires a great number of processing and storage resources, resulting in the recent rapid development of Big Data technologies. Since 2015, the European Space Agency (ESA) has been providing a great amount of data gathered by exploratory equipment: a collection of Sentinel satellites – which perform Earth observation using various measurement techniques. For example Sentinel-2 provides a stream of digital photos, including images of the Baltic Sea and the whole territory of Poland. This data is used in an experimental installation of a Big Data processing system based on the open source software at the Academic Computer Center in Gdansk. The center has one of the most powerful supercomputers in Poland – the Tryton computing cluster, consisting of 1600 nodes interconnected by a fast Infiniband network (56 Gbps) and over 6 PB of storage. Some of these nodes are used as a computational cloud supervised by an OpenStack platform, where the Sentinel-2 data is processed. A subsystem of the automatic, perpetual data download to object storage (based on Swift) is deployed, the required software libraries for the image processing are configured and the Apache Spark cluster has been set up. The above system enables gathering and analysis of the recorded satellite images and the associated metadata, benefiting from the parallel computation mechanisms. This paper describes the above solution including its technical aspects.

Cytowania

  • 0

    CrossRef

  • 0

    Web of Science

  • 0

    Scopus

Cytuj jako

Pełna treść

pobierz publikację
pobrano 67 razy
Wersja publikacji
Accepted albo Published Version
Licencja
Creative Commons: CC-BY otwiera się w nowej karcie

Słowa kluczowe

Informacje szczegółowe

Kategoria:
Publikacja w czasopiśmie
Typ:
artykuły w czasopismach
Opublikowano w:
TASK Quarterly nr 21, strony 365 - 377,
ISSN: 1428-6394
Język:
angielski
Rok wydania:
2017
Opis bibliograficzny:
Proficz J., Drypczewski K.: Processing of Satellite Data in the Cloud// TASK Quarterly -Vol. 21,iss. 4 (2017), s.365-377
DOI:
Cyfrowy identyfikator dokumentu elektronicznego (otwiera się w nowej karcie) 10.17466/tq2017/21.4/y
Bibliografia: test
  1. Demchenko Y, De Laat C and Membrey P 2014 CTS, IEEE 104 otwiera się w nowej karcie
  2. Apache Spark -Lightning-fast cluster computing [Online] available at: https://spark. apache.org/ [Accessed: 25-July-2017] otwiera się w nowej karcie
  3. Apache Hadoop -website [Online] available at: https://hadoop.apache.org/ [Acces- sed: 23-June-2017]
  4. Thensorflow Homepage [Online] available at: https://www.tensorflow.org/ [Accessed: 23-June-2017] otwiera się w nowej karcie
  5. Torch -a scientific computing platform for LuaJIT [Online] available at: http://torch .ch/ [Accessed: 23-June-2017] otwiera się w nowej karcie
  6. Theano -a library to define, optimize, and evaluate mathematical expressions [Online] available at: http://deeplearning.net/software/theano/ [Accessed: 26-June-2017] otwiera się w nowej karcie
  7. Caffe -Deep learning framework [Online] available at: http://caffe. berkeleyvision.org/ [Accessed: 26-June-2017] otwiera się w nowej karcie
  8. Copernicus -observing the Earth, ESA programe [Online] available at: http://www.esa. int/Our Activities/Observing the Earth/Copernicus [Accessed: 26-June-2017]
  9. Sentinel-2 Homepage [Online] available at: http://www.esa.int/Our Activities/ Observing the Earth/Copernicus/Sentinel-2 [Accessed: 23-June-2017]
  10. Vega -Arianspace [Online] available at: http://www.arianespace.com/vehicle/vega/ [Accessed: 06-July-2017] otwiera się w nowej karcie
  11. Sentinel-2 MSI, Data formats [Online] available at: https://sentinel.esa.int/web/ sentinel/user-guides/sentinel-2-msi/data-formats [Accessed: 23-July-2017] otwiera się w nowej karcie
  12. Tryton Supercomputer, Academic Computer Centre TASK [Online] available at: https: //task.gda.pl/kdm/sprzet/tryton/ otwiera się w nowej karcie
  13. OpenStack, Open Source Cloud Computing Software [Online] available at: https://www. openstack.org/ [Accessed: 27-June-2017]
  14. Ceph Homepage [Online] available at: http://ceph.com/ [Accessed: 27-June-2017]
  15. Swift -OpenStack [Online] available at: https://wiki.openstack.org/wiki/Swift [Accessed: 27-June-2017]
  16. Apache Hadoop YARN [Online] available at: https://hadoop.apache.org/docs/r2.7.3 /hadoop-yarn/hadoop-yarn-site/YARN.html [Accessed: 27-June-2017]
  17. Apache Mesos [Online] available at:http://mesos.apache.org/ [Accessed: 28-June-2017]
  18. Kubernetes -Production-Grade Container Orchestration [Online] available at: https:// kubernetes.io/ [Accessed: 28-June-2017] otwiera się w nowej karcie
  19. HDFS Architecture Guide [Online] available at: https://hadoop.apache.org/docs/r1. 2.1/hdfs design.html [Accessed: 28-June-2017]
  20. Apache HBase Home [Online] available at: https://hbase.apache.org/ [Accessed: 28-June-2017]
  21. Apache Cassandra Homepage [Online] available at: http://cassandra.apache.org/ [Accessed: 28-June-2017]
  22. JPEG 2000 Homepage [Online] available at: https://jpeg.org/jpeg2000/ [Accessed: 28-June-2017] otwiera się w nowej karcie
  23. The ImageIO-Ext Project Home [Online] available at: https://github.com/ geosolutions-it/imageio-ext/ [Accessed: 29-June-2017] otwiera się w nowej karcie
  24. OpenJPEG, an open-source JPEG 2000 codec [Online] available at: http://www. openjpeg.org/ [Accessed: 29-June-2017]
  25. Odersky M 2014 Scala by Example, Programing Methods Laboratory [Online] available at: http://www.scala-lang.org/docu/files/ScalaByExample.pdf, EPFL [Accessed: 26-July-2017]
  26. Sentinel-2 User Handbook [Online] available at: https://sentinel.esa.int/documents /247904/685211/Sentinel-2 User Handbook, ESA [Accessed: 03-July-2017] otwiera się w nowej karcie
  27. H2O Homepage [Online] available at: https://www.h2o.ai/ [Accessed: 03-July-2017] otwiera się w nowej karcie
Weryfikacja:
Politechnika Gdańska

wyświetlono 155 razy

Publikacje, które mogą cię zainteresować

Meta Tagi