NEANIAS Gitlab

Skip to content

GitLab

Commit 7ac63fee authored by Carlos H. Brandt's avatar Carlos H. Brandt
Browse files

Update HRSC readme

parent 390dec08
Hide whitespace changes
Inline Side-by-side
Showing with 141 additions and 56 deletions
datasets/MEX/HRSC/README.md
View file @ 7ac63fee
......@@ -3,76 +3,58 @@
[Neukem et al, 2004]: Neukem_etal-HRSC.pdf
HSRC data are available at different processing levels.
As explained in [Scholten et al, 2005] (hereafter S05) and overviewed in [Neukem et al, 2004]:
As explained in [Scholten et al, 2005][] (hereafter S05) and overviewed in [Neukem et al, 2004][]:
* Level-1: conversion of original data stream (transmitted to ground) to de-compressed data;
* Level-2: radiometric correction of the image data;
* Level-3: map projection of all data;
* Level-4: DTMs and orthoimages.
We are interested in Level-4 data: DTM/DEM. In particular, data taken from
instrument's Nadir observations; Figure [S05 Fig-1] pictures the different lines fo
We are interested in Level-4 data: DTM/DEM, and data taken from
instrument's Nadir observations; Figure [S05 Fig-1][] pictures the different lines fo
sight according to the different CCDs (Nadir represented as `ND`):
![HRSC/SRC technical parameters][S05 Fig-1]
[S05 Fig-1]: Scholten_etal_2005-Fig1_Technical_Params_HRSC_SRC.png
An script to download a set of observation (_e.g._, `0010`) data is available in the Appendix.
### Projection
Projection of level-4 data is **Sinusoidal planetocentric** in a sphere with radius **3396.0 Km**, **positive east**.
## Data archive
HRSC dataset is stored in PDS format (see [../../PDS.md]).
HRSC dataset is stored in PDS format (see [../../PDS.md](../../PDS.md)).
Interfaces to access the archive are provided at:
* https://pds-geosciences.wustl.edu/missions/mars_express/hrsc.htm ;
* https://ode.rsl.wustl.edu/mars/indexDatasets.aspx .
In particular, at https://pds-geosciences.wustl.edu/mex/mex-m-hrsc-5-refdr-dtm-v1/mexhrs_2001
we will find HRSC level-4 data of our interest.
As a PDS archive, we will find there a well defined, standard files/directories
structure.
If we are going straight for the data, our interest is on directories
`browse` and `data`; `browse` provides a _view_ of each data file in a commonly
used format (`PNG`), and `data` provides the actual scientific data.
Metadata, documentation and other support material are found in the
other directories.
> The section below, [Archive Summary] and [Exploring the archive] go into the
structural and content of this archive. For detail I would like to defer the
reader to those sections.
The data (in `data` and `browse`) are split after the mission _orbit-ID_,
inside each _orbit_ directory we will find the respective images.
The number of images is not fixed, but we can generally expect on image
per sensor and per product -- DTM/DEM.
The following list of files was taken from section [Explorin the archive]
and represent the list of files found in orbit `0010`:
```
h0010_0009_bl4.img
h0010_0009_da4.img
h0010_0009_dt4.img
h0010_0009_gr4.img
h0010_0009_ir4.img
h0010_0009_nd4.img
h0010_0009_re4.img
```
. Our interest is over files with indixes `nd`,`dt`,`da` for we want
Nadir, DTM, DEM view/content.
In particular, at https://pds-geosciences.wustl.edu/mex/mex-m-hrsc-5-refdr-dtm-v1/mexhrs_2001
we will find the HRSC/DTM of our interest.
We notice there are no Label (metadata) files accompanying the images.
That's because the (label) metadata is positioned at the header of image files.
In the header we will find virtually all the information we need to know
about each image.
To extract the metadata from the data files is simple and can be done from a
typical shell command, since PDS Labels are _text_ data and obey a formal
syntax (ending always in `END`):
```
$ awk '{print $0; if($0 ~ /^END/){if($1 !~ /END_/){exit}}}' <datafile.img>
```
The `browse/` directory provides a _view_ of each data file in a commonly
used format (`PNG`), while `data/` provides the actual scientific data.
From [hrsc_eaicd.pdf](hrsc_eaicd.pdf):
> The Data Directory contains sub-directories for every orbit which is part of the data set, the diretory names are the four digits orbit number. The delivery contains level-4 data for the nadir sensor and the four color sensors. It also contains Digital Terrain Models (DTMs) with two different reference bodies [21,22]:
>
> - MOLA sphere (dt in the filename)
> - MOLA Areoid (da in the filename)
The _level-4 data_ we are talking about are PDS IMAGE objects. An IMAGE object is a two-dimensional array of values, all of the same type, each of which is referred to as a *sample*.
As IMAGE object is usually accompanied by a LABEL object which is a text file containing the IMAGE's metadata (or HEADER).
In MEX/HRSC, though, _header_ and _image_ are bound in the same (.IMG) file. (See the Appendix for a simple example to read the HEADER from IMAGE.)
For an example of accessing the `data`, see the section [Data access](#data-access), for an walkthrough `browse`, go to section [Browsing the data](#browsing-the-data).
#### Archive summary
Another important directory in a PDS archive is `index/`, in the file `index.tab` one will find the list of data records (files) in the archive. This `index.tab` provide simply the list of files -- path to the data -- and respective information about their release (version).
### Archive summary
Let's quickly summarize some information ([hrsc_eaicd.pdf) regarding the HRSC
PDS archive regarding name and structure conventions in place.
......@@ -142,9 +124,7 @@ The `BROWSE` directory mirrors/mimic the `DATA` structure.
> - improvement of orbit and attittude data [19,20]
> - generation of Digital terrain Models and Orthorectification of image data [19,20]
#### Exploring the archive
### Browsing the data
Our interest now is to access the Level-4, DTM data; Which is under
`MEX-M-HRSC-5-REFDR-DTM-V1.0/MEXHRS_2001`.
......@@ -238,12 +218,12 @@ The corresponding image (10,9) shaded relief, nadir and RGB orthophotos:
![H0010_0009 DTM-ND-RGB](assets/h0010_0009.png)
#### Data access
### Data access
Now that we browsed the images of orbit 0010, and we want `ND4`,`DT4`,`DA4`
files, we go to `DATA` directory, at the same address -- `0010/` -- to check
for the corresponding IMG data:
```
h0010_0009_bl4.img
h0010_0009_da4.img
......@@ -264,11 +244,9 @@ with an `awk` call:
$ awk '{print $0; if($0 ~ /^END/){if($1 !~ /END_/){exit}}}' h0010_0009_nd4.img
```
, which output the IMG's Label content.
Section [Metadata] presents the content of such Label/header.
Section [Metadata](#metadata) presents the content of such Label/header.
##### Metadata
#### Metadata
In PDS Label files/content we find the concise description of the data of
reference (see [/datasets/PDS.md](/datasets/PDS.md)).
......@@ -427,3 +405,110 @@ misaligned data. It is a discussion between Matt and Trent on handling data in
ArcMap + GDAL:
* https://isis.astrogeology.usgs.gov/IsisSupport/index.php?topic=2726.0
# Appendix
## Download script
```
#!/bin/bash
#set -ue
DRY_RUN=${DRY_RUN:-""}
ORBIT=${ORBIT:-0100}
function usage() {
echo ""
echo "Usage:"
echo ""
echo " $ ${BASH_SOURCE} docs|meta|data"
echo ""
echo " docs : documents (txt) and light metadata"
echo " meta : essentially the INDEX (contains GEO_MARS.TAB)"
echo " data : data sample -- browse/extra files as well."
echo ""
echo "Variables can be used to change defaults/states:"
echo " ORBIT : sample data orbit #ID, default is '$ORBIT'"
echo " DRY_RUN : do not download data if defined (e.g., 'y'),"
echo " echoes what 'would be' run"
echo ""
echo "Examples:"
echo ""
echo " Download data for default Orbit, $ORBIT:"
echo ""
echo " $ ./download_sample.sh data"
echo ""
echo " With DRY_RUN set, just print what would be run for docs/meta:"
echo ""
echo " $ DRY_RUN=y ./download_sample.sh docs"
echo " $ DRY_RUN=y ./download_sample.sh meta"
echo ""
echo " Print (wget) commands for downloading Data for Orbit I957:"
echo ""
echo " $ DRY_RUN=y ORBIT=0103 ./download_sample.sh data"
echo ""
echo " Download from NASA instead of ESA (default):"
echo ""
echo " $ SRV=NASA ORBIT=0103 ./download_sample.sh data"
echo ""
}
SRV=${SRV:-ESA}
if [ "$SRV" = 'ESA' ]; then
SERVER='ftp://psa.esac.esa.int'
PDSATH='pub/mirror/MARS-EXPRESS/HRSC/MEX-M-HRSC-5-REFDR-DTM-V1.0'
else
[ "$SRV" = 'NASA' ] || { echo '$SRV should be either ESA or NASA'; exit; }
SERVER='ftp://pds-geosciences.wustl.edu'
PDSATH='mex/mex-m-hrsc-5-refdr-dtm-v1/mexhrs_2001'
fi
docs=(AAREADME.TXT CATALOG/CATINFO.TXT DOCUMENT EXTRAS/EXTRINFO.TXT INDEX/INDXINFO.TXT VOLDESC.CAT)
meta=(INDEX)
data=(BROWSE/${ORBIT} DATA/${ORBIT} EXTRAS/H${ORBIT}*.PNG)
function do_wget() {
# Input argument is a list (space-sep) of files/directories to download
#
content=$@
for c in ${content[@]}; do
data="${SERVER}/${PDSATH}/${c}"
[ "$SRV" = 'NASA' ] && data=$(echo "$data" | tr '[:upper:]' '[:lower:]')
if [ ! -n "$DRY_RUN" ]; then
# download
wget -r -nH --cut-dirs=4 --directory-prefix=${SRV} ${data}
else
# dry-run
echo "wget -r -nH --cut-dirs=4 --directory-prefix=${SRV} ${data}"
fi
done
}
opt=${1:-'nopt'}
case $opt in
docs)
echo "Downloading 'docs' .."
do_wget ${docs[@]}
;;
meta)
echo "Downloading 'meta' .."
do_wget ${meta[@]}
;;
data)
echo "Downloading 'data' .."
do_wget ${data[@]}
;;
*)
usage
exit
;;
esac
```
[hrsc_eaicd.pdf]:
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment