
======================================================================
HOWTO install and use SatPy for EUMETCast under Debian GNU/Linux amd64
======================================================================
Version 1.0    2018/10/??                            by Ernst Lobsiger
Version 2.0    2019/04/12                            by Ernst Lobsiger
Version 3.0    2021/01/08                            by Ernst Lobsiger



1) INTRODUCTION
---------------

SatPy is still in development. It is the highest integration of
other PyTROLL packages to read, process and store satellite data.

http://www.pytroll.org

PyTROLL/SatPy is now used by a couple of European met offices 24/7.
As a user of EUMETCast you can profit from this Open Source project.

There is an important note (at the time of writing SatPy is at 0.25!):
**********************************************************************
* "SatPy's interfaces are not guaranteed stable and may change until *
* version 1.0 when backwards compatibility will be a main focus."    *
**********************************************************************

When I was new to SatPy (and Python!) in 2018 a big problem was the
sparse documentation. So already an installation of a working PyTROLL
environment meant a steep learning curve. Much of my try and error
has been done with both Debian 9 (Stretch) and Devuan 2 (ASCII) still
using Python 2.7 and later (2.7 unmaintained since January 1st 2020)
with Debian 10 (Buster) and Devuan 3 (Beowulf) using Python 3.7 + 3.8.

Under GNU/Linux you basically have three installation methods. You can
install Python and SatPy packages as user root in /usr with installer
pip. You can use virtualenv and pip that lets you do most of the SatPy
install as an unprivileged user in your home directory or you can take
Anaconda (or better Miniconda) to do everything as an unprivileged user.

While I have used all three methods in the past, the SatPy documentation
recommends virtualenv under Ubuntu also making this the prefered install
for Debian or Devuan. After my rather technical Version 2.0 of the HOWTO
for GNU/Linux I issued a Version 1.0 of a much more user friendly HOWTO
for users under Windows 10. As I tried to synchronize as much as possible
my SatPy development work under GNU/Linux with what I finally offered for
Windows 10 the only possibility was to base everything on Miniconda3.

This HOWTO 3.0 for GNU/Linux now uses basically the same data structures
and as I used in the HOWTO 1.0 for Windows 10. In an attempt to further
synchronize my work for both Operating Systems I just issued a Version
3.0 of the Windows HOWTO that is the practical equivalent of this paper.

Installation and usage of my packet in both Operating Systems is now
straight forward. The only few differences left are explained below:

a)
Under GNU/Linux it is assumed that you have ImageMagick (IM) installed.
Under Windows I provided a version of "convert.exe" that does the job.

b)
Under GNU/Linux I provide xRITDecompress as compiled under Debian 10
(Buster) amd64. In the rare case that this should not work on an other
GNU/Linux amd64 distribution, I also provide an improved source code
of this decompressor that has been released as open source by EUMETSAT.
Under Windows 10 I provide xRITDecompress.exe as compiled under CygWin.

c)
Under GNU/Linux I use the common GNU system uncompression utilities.
Under Windows I use a portable version of 7za.exe for similar tasks.

d)
There are some minor differences due to the shells Bash and CMD.exe.


Now let's start the install of Miniconda3 and my structures and scripts.
As usual I take the CLI perspective. On a graphic system use a terminal.
We use a generic names "user" and "host". This either stands for an image
processsor "user" in an EUMETCast receiver "host" or a human unprivileged
user that happens to have access to satellite data received via EUMETCast.



2) Installation of Miniconda3 with PyTROLL/Satpy under Debian 10 GNU/Linux
--------------------------------------------------------------------------
We only take small Miniconda3 as a full blown Anaconda3 is just bloatware.


2.1.) Download the Miniconda3 installer (situation 2021/01/11)
--------------------------------------------------------------

With a graphic system (this can also be Windows) go to

https://docs.conda.io/en/latest/miniconda.html

and under "Linux installers" spot and download

Python 3.8       Miniconda3 Linux 64-bit       89.9 MiB

copy and paste the SHA256 hash (to check later your file is intact)

1314b90489f154602fd794accfc90446111514a5a72fe1f71ab83e07de9504a7

If you are on a CLI only system use wget and the direct link

user@host:~$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

and you can then compare the SHA256 on a screen of a graphic system.


2.2) Check the SHA256 hash
--------------------------

This test file integrity test is optional but highly recommended!

user@host:~$ sha256sum Miniconda3-latest-Linux-x86_64.sh      ...must output exactly the same hash:

1314b90489f154602fd794accfc90446111514a5a72fe1f71ab83e07de9504a7  Miniconda3-latest-Linux-x86_64.sh

If not something went wrong, Miniconda3-latest-Linux-x86_64.sh might be tainted or damaged!


2.3) Installation of Miniconda3 and PyTROLL/SatPy
-------------------------------------------------

This method can completely be done by an unprivileged user in his home
directory. Installation is very easy, every help is found on anaconda.com.

Installation by an unprivileged user is started at the Command Line Interface (CLI) by

user@host:~$ sh Miniconda3-latest-Linux-x86_64.sh


- Accept the license (you have no choice)
- Install Miniconda3 in your home directory (default)
- Make the installer initialize Miniconda3 (type "yes")

You are back to the prompt and almost done. Initialize with "yes" added the lines below

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/eumetcast/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/home/eumetcast/miniconda3/etc/profile.d/conda.sh" ]; then
        . "/home/eumetcast/miniconda3/etc/profile.d/conda.sh"
    else
        export PATH="/home/eumetcast/miniconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

to your local .bashrc file. This is executed whenever you log in and will make you fall
directly to the Miniconda3 "base" environment. To make sure these changes are in effect
without logging out and in again this time just issue "user@host:~$ source .bashrc" and
your prompt will change to "(base) user@host:~$" saying you are in "base" environment.

Your installer and "Swiss Army Knife" is now the command "conda".
There is no manual page but conda has a plenthora of help built in.

REMEMBER: Whenever you login later you will fall to the (base) environment. If
you don't like that you can do a "conda config --set auto_activate_base false"

Let's update basic stuff and conda to the very latest versions first:

At the prompt "(base) user@host:~$" (Basic Environment) enter
conda update -n base -c defaults conda

The SatPy documentation says that you have to add conda-forge as repository.
At the prompt "(base) user@host:~$" (in Basic Environment) enter
conda config --add channels conda-forge

Now we create a special environment for PyTROLL with the command
conda create --name pytroll
and accept the default location at /home/user/miniconda3/envs/pytroll

NOTE: If you do not activate environment pytroll before installing
packages then these packages will all go to the (base) environment.
THEREFORE ACTIVATE pytroll BEFORE INSTALLING PYTROLL/SATPY STUFF:

conda activate pytroll

This will change your path and your prompt will have (pytroll) in front.
Activation in one user terminal does not activate in another terminal !!

Anaconda packages are differently packed from pip packages. To make all
my PyTROLL/SatPy image generation scripts run I just had to (be pacient):

(pytroll) user@host:~$ conda install satpy
(pytroll) user@host:~$ conda install pycoast
(pytroll) user@host:~$ conda install pyhdf

The packages installed above are now found under a rather lengthy path:
/home/user/miniconda3/envs/pytroll/lib/python3.8/site-packages

A command "conda list satpy" should at least indicate version 0.25.1 You will
not have to update for about half a year (*Never touch a running system!*).

To get back to the normal prompt  "user@host:~$" issue "conda deactivate".

BACKGROUND INFO:

Activating and deactivating essentially changes your PATH variable. This
is where Windows searches for executable commands you type at the prompt.

You can check that while in different environments in the following way:

(pytroll) user@host:~$ echo $PATH
(base) user@host:~$ echo $PATH
user@host:~$ echo $PATH



3) Untar the data structures used for this HOWTO
------------------------------------------------

If you are reading this you should also find the files EMCtools.tar and EMCdata.tar.
EMCtools contains the scripts and software, EMCdata is where all your data will go.

In most cases you just untar those in your home directory. If you are the admin of
your system and have root access you might want to give "user" access to some place
for EMCdata (e.g. if the system and /home is an SSD and you have a big HDD as well).

In any case as a user you can only untar the two files where you have write access.
Make sure your path never contains blanks (spaces) as sometimes seen under Windows.
Let's assume the most simple case where you untar both files in your "/home/user".
Please note: Files have been tared by user "eumetcast" that may not exist on your
host and even if you generate that, it will not have the same numerical user ID.

If you don't know how to use untar just do a "user@host:~$ man untar" and do RTFM.

After untar you should have a directory /home/user/EMCtools with subdirectories and
a /home/user/EMCdata with subdirectories. It's high time to explore what is where.



4) Get the shorelines, rivers and borders
-----------------------------------------

Go to

http://www.soest.hawaii.edu/wessel/gshhg/

gshhs and wdb files are part of GSHHG. Current version is GSHHG-2.3.7.
We need the native shape files that come in different resolutions and
can all be downloaded on the bottom of the page above. Look for the link:

CAUTION: Do not take the binary files as used by Hugo in EUMETCastView!

"
For ESRI shapefiles of shoreline polygons, rivers, and borders you can choose to get via ftp or http:

    FTP: GSHHG coastlines, political borders and rivers in shapefile format (zip archive)
    HTTP: GSHHG coastlines, political borders and rivers in shapefile format (zip archive)
"

You should get a file named gshhg-shp-2.3.7.zip

Extract archive gshhg-shp-2.3.7.zip into directory /home/user/EMCdata/gshhg-shp

On a CLI only system you can do a (the *.zip is 149157845 bytes June 15th 2017)

user@host:~$ wget http://www.soest.hawaii.edu/wessel/gshhg/gshhg-shp-2.3.7.zip

user@host:~$ unzip gshhg/gshhg-shp-2.3.7.zip -d /home/user/EMCdata/gshhg-shp

Get the file names and organisation from the output of the commands

user@host:~$ ls -la EMCdata/gshhg-shp
total 48
drwxr-xr-x 4 user user  4096 May 11  2020 .
drwxr-xr-x 8 user user  4096 Jan  9 14:01 ..
-rw------- 1 user user  7816 Jan 22  2014 COPYING.LESSERv3
drwxr-xr-x 7 user user  4096 Oct  5  2018 GSHHS_shp
-rw------- 1 user user  1375 Jan 22  2014 LICENSE.TXT
-rw------- 1 user user 15350 Jun 15  2017 README.TXT
-rw------- 1 user user  2590 Apr  1  2016 SHAPEFILES.TXT
drwxr-xr-x 7 user user  4096 Oct  5  2018 WDBII_shp

user@host:~$ ls -la EMCdata/gshhg-shp/GSHHS_shp
total 44
drwxr-xr-x 7 eumetcast eumetcast  4096 Oct  5  2018 .
drwxr-xr-x 4 eumetcast eumetcast  4096 May 11  2020 ..
drwxr-xr-x 2 eumetcast eumetcast  4096 Oct  5  2018 c
drwxr-xr-x 2 eumetcast eumetcast  4096 Oct  5  2018 f
drwxr-xr-x 2 eumetcast eumetcast  4096 Oct  5  2018 h
drwxr-xr-x 2 eumetcast eumetcast  4096 Oct  5  2018 i
drwxr-xr-x 2 eumetcast eumetcast  4096 Oct  5  2018 l
-rw------- 1 eumetcast eumetcast 15350 Jun 15  2017 README.TXT

user@host:~$ ls -la EMCdata/gshhg-shp/WDBII_shp
total 44
drwxr-xr-x 7 eumetcast eumetcast  4096 Oct  5  2018 .
drwxr-xr-x 4 eumetcast eumetcast  4096 May 11  2020 ..
drwxr-xr-x 2 eumetcast eumetcast  4096 Oct  5  2018 c
drwxr-xr-x 2 eumetcast eumetcast  4096 Oct  5  2018 f
drwxr-xr-x 2 eumetcast eumetcast  4096 Oct  5  2018 h
drwxr-xr-x 2 eumetcast eumetcast  4096 Oct  5  2018 i
drwxr-xr-x 2 eumetcast eumetcast  4096 Oct  5  2018 l
-rw------- 1 eumetcast eumetcast 15350 Jun 15  2017 README.TXT

The subdirectory names stand for line resolution of coastlines/boundaries.

f    full resolution: Original (full) data resolution.
h    high resolution: About 80 % reduction in size and quality.
i    intermediate resolution: Another ~80 % reduction.
l    low resolution: Another ~80 % reduction.
c    crude resolution: Another ~80 % reduction.

Resolution can be chosen later in PyTROLL/SatPy scripts that include coastlines.
GSHHG shapes are further organized in "Levels". You will have to remember that
if you want to picture overlay the shorelines of "Lake Geneva" or "Loch Ness".



5) Environment variables
------------------------

When started PyTROLL/SatPy is looking for two important environment variables:

a) The Path and name of the wavelet decompression tool xRITDecompress
This variable is called XRIT_DECOMPRESS_PATH and must only be set if you
rely on a recent feature of SatPy to automatically decompress xRIT files.
It should be XRIT_DECOMPRESS_PATH="/home/user/EMCtools/exefiles/xRITDecompress"
This was not available when I started my scripts. I use my own decompression
script code and therefore you can forget about XRIT_DECOMPRESS_PATH for now.

b) SatPy is also looking for an environment variable named PPP_CONFIG_DIR.
In this directory you manage your own configuration files locally. This is
really recommended if you begin to experiment with your own map projections.
Those are defined in file areas.yaml where I added westminster, isleofman, cyprus,
... This variable *must* be set to PPP_CONFIG_DIR="/home/user/EMCtools/pppconfig"

The original default areas are found under the lengthy path (don't change those!):
/home/user/miniconda3/envs/pytroll/lib/python3.8/site-packages/satpy/etc/areas.yaml

Environment variables can be added/set for your user account during login in your
.bashrc file. To accomplish this add somthing like the lines below (replace "user").

# This is the *full* path to your xRIT decompression binary
export XRIT_DECOMPRESS_PATH="/home/user/EMCtools/exefiles/xRITDecompress"
# In this directory you manage some configuration files locally
export PPP_CONFIG_DIR="/home/user/EMCtools/pppconfig"

Of course variables can also be set at the prompt for the terminal you are in:
user@host:~$ set PPP_CONFIG_DIR="/home/user/EMCtools/pppconfig"
As long as the terminal stays open, this environment variable is known.

Finally they can be set easily in shell files before the Python scripts are invoked:
set PPP_CONFIG_DIR="/home/user/EMCtools/pppconfig"



6) Working with my EUMETCast PyTROLL/SatPy scripts
--------------------------------------------------

There are three scripts directories. GEOscripts contains all scripts for geostationary
satellites. LEOscripts contains all scripts for polar orbiting satellites. USRscripts
is where you will work. I propose to use GEOscripts and LEOscripts just as archives.

Let's e.g. start with Meteosat-11 also known as MSG4. Copy file MSG4.py from GEOscripts
to USRscripts. In directory USRscripts open MSG4.py and in the your prefered Editor. On a
CLI system you probably use the GNU Midnight Commander (mc) that has syntax highlighting.

Before you can start any of the Python scripts you *HAVE TO* edit the variable "segdir"
inside. That's where you have the data segments/files received for the satellites at hand.
My TC receivers provide satellite files in a /srv/rec_x/EUMETCastChannel/YEAR/MONTH/DAY
structure. If you use Hugo's marvellous program EUMETCastView and also run EUMETCastWatcher
24/7 you should find a similar data structure. If you have a more bulk data structure with
segments from recent few days in a /PATH/EUMETCastChannel directory this should work as well.
Again your segdir should not contain any blanks (spaces) or evil might be round the corner.
Obviously you must have at least *READ ACCESS* to where the raw segment files are located.
Different from real satellite data managers my scripts do not know when the files arrive.
If you ask early in the morning for an image today 12:00 the files will just not be found.
Please note that you do *NOT* have to set the scripts executable (e.g. with chmod +x *.py).
But maybe you have to set the owner of the files to your username (e.g chown user:user *.py).


If you have installed EMCtools and EMCdata in your home directory you have to change the
variable segdir for your MSG4 files and replace my user "eumetcast" with your user name.

toodrp = '?'    is the path where you installed EMCtools (recommended /home/YourUserName).
datdrp = '?'    is the path where you installed EMCdata  (recommended /home/YourUserName).

For what follows below I assume you made the install as proposed on in your home directory:

open a terminal with a prompt "(base) user@host:~$"

ACTIVATE pytroll by entering "conda activate pytroll"
your prompt is now "(pytroll) user@host:~$"


Change to where you have your script with "cd EMCtools/USRscripts". Command completion
comes in handy and your prompt now looks like "(pytroll) user@host:~/EMCtools/USRscripts$".

All scripts are started with "python YourScript.py parameter"

Parameter is normally a date or datetime string as explained in the script comment header.

For Meteosat-11 it's something like "python MSG4.py 202101061200"

There will be a lot of warning output in the CMD window. Normally you can just ignore it.

If you got your first image you will find it in EMCdata. Now play around with this MSG4.py
changing parameters like composite and area until you have the result you are looking for.
If this happens to be composite 'realistic_colors' and area 'westminster' I propose you
rename your tuned copy in USRscripts from MSG4.py to MSG4-realistic_colors-westminster.py.
Of course you can invent "any_other_name.py" because the script doesn't care about its name.

You noticed westminster wasn't known as area? You probably cheated and let out point 5)!

Before you start working with LEOscripts you must check the age of the "my_TLE_file.txt"
in directory ../EMCtools/pppconfig. If it's more than one month old you should update it.
There is a file "Update_TLE_file.cmd" in directory ../EMCtools/cmdfiles that does the job.

CAUTION: Do only use caching if you know what you do. This is an advanced feature that
is useful if you have final designs of images that you produce daily with a scheduler.



7) Debugging when things go wrong
---------------------------------

There are 3 levels of debugging information. First of all there is the Python
interpreter itself that is very noisy talking about this and that as soon as
you start your script. If things don't work Python might already tell you why.
So in case of problems please read and try to understand what Phyton tells you.

There is a second level of debugging that can be activated in SatPy code. You
have to use an Editor again. All scripts have three lines not far from the top:

# Why to hell is it not working?
# from satpy.utils import debug_on
# debug_on()

# is a character that starts a comment just for users and ignored by Phyton.

If you uncomment (delete the # and make the code flush left) the above to

# Why to hell is it not working?
from satpy.utils import debug_on
debug_on()

Phyton has two more lines of code that switch on the internal SatPy debugging.

One special feature of Python is the meaning of indents. These are used to mark
begin and end of code blocks: *THOU SHOUDEST NOT MESS WITH INDENTS IN PYTHON*!
If the two lines are not flush left as shown below you added another problem.

# Why to hell is it not working?
 from satpy.utils import debug_on
 debug_on()

There is one more thing you always *must* remember: Python like GNU/Linux and
unlike Windows is case sensitive. DEBUG.txt and debug.txt are different files.
And YES: The Python interpreter understands debug_on() but never debug_ON().

If you did this right you get additional info what SatPy actually does and where
and why it stops. Again try to read and understand, it's pure English after all.

The third level of debugging is the script that tells you why it stopped or
that should at least tell you what list of segment files it has been given.

If you cannot read the debug information that flashes over the terminal or scroll
back in your Miniconda3 window you must redirect it into a text file as described
below. Then take a cup of coffee and read the file twice before you ask for help.

That's what we do (in UNIX since 50 years, copied by MS-DOS, also in Windows 10):

python script.py param  > DEBUG.txt         The script output goes to DEBUG.txt
python script.py param  1> DEBUG.txt        This is exactly the same as above !

python script.py param  2> DEBUG.txt        The other output goes to DEBUG.txt
python script.py param  > 2>&1 DEBUG.txt    The Full Monty will go to DEBUG.txt

There is well known  "/dev/null" that works as a black hole sucking all in

python script.py param  2> /dev/null        Just the script will have a say
python script.py param  > /dev/null 2>&1    All text output will just vanish

For more info about above lines google for "GNU/Linux redirection stdout stderr".

And here is my point: If you cannot find the problem after having read all
info and double checked your segdir path and "The Full Monty" DEBUG.txt then
send this latter file to your local PyTROLL/SatPy guru or post it on MSG-1.



8) Batch processing using "bash"
--------------------------------

If you work under GNU/Linux you should certainly know a minimum of shell programming.
As you also should know there are a bunch of different shells. I mostly use bash.
Oftem there is a link "sh" that points to a shell. Under Debian 10 you can make a

user@host:~$ which sh

and the system will tel you where "sh" is. The answer is "/usr/bin/sh" and if you go
there you see that "sh" is a link that actually points to dash (a slimed down bash).

Shell scripts are mainly used with a scheduler. You want the system to automatically
produce certain images every day unattended. One problem you have with Miniconda3 is
that a shell file should activate your (pytroll) environment before calling the python
script. Fortunately Miniconda3/bin provides conda that can be used for activation.

Check as example the shell script .../EMCtools/cmdfiles/MSG4-overview-westminster.sh.

I let it as a user exercise to find out when the corresponding data is due to arrive.
It has to do with LTAN/LTDN, the lat/lon position of the POI and the EARS timeliness.

The mostly used scheduler under GNU/Linux is cron with configuration file /etc/crontab.

# This will start a PyTROLL job running in miniconda3 as user eumetcast
30 6-18/1 * * * eumetcast /home/eumetcast/EMCtools/cmdfiles/MSG4-overview-westminster.sh



9) Where do we go from here
---------------------------

You certainly already changed the 'TBS-6909X' to your receiver name. Did you?

What readers are available? Have a look (but don't screw up) here:
/home/user/miniconda3/envs/pytroll/lib/python3.8/site-packages/satpy/readers

What composites need what instrument channels? Find answers here:
/home/user/miniconda3/envs/pytroll/lib/python3.8/site-packages/satpy/etc/composites

Learn more Python. You will find every possible help on the internet.

If you have not already RTFM go to this link and do it:
https://satpy.readthedocs.io/en/latest/

Read and maybe join, ask and contribute here:
https://groups.google.com/forum/#!forum/pytroll

Learn more about conda. There is no automatic update of your PyTROLL/SatPy packages.

Read the scrips of other PyTROLL/SatPy enthusiasts and share your own code.
Windows users should help other Windows users (I want to go back to GNU/Linux).

If you begin to work with the code you should use an editor with syntax highlighting
for Python. If you do not already use it have a look at GNU Midnight Commander (mc).



10) Uninstallation of the whole stuff
-------------------------------------

Uninstallation of Miniconda3 is childs play easily done with mc. You just delete the
directory /home/user/miniconda3 and /home/user/.conda and also the files .condarc and
python_history. Then you tidy up your .bashrc to what it was before and you are done.


***************************************************************************************
GNU/Linux PyTROLL/Satpy HOWTO V 3.0. I will not consider a next release until at least
50 Windows 10 or GNU/Linux EUMETCast users report success with PyTROLL/SatPy. And like
David Taylor I would love to read about PyTROLL/SatPy in the GEO Quarterly journal. But
this article must be written in clean English by a Windows 10 EUMETCast user in the UK.

Belp, Oase 2021/01/17                                                    Ernst Lobsiger
***************************************************************************************
