---------------------------------------------------------------------------------------
All I learnt about SatPy caching when producing images                   Ernst Lobsiger
---------------------------------------------------------------------------------------

Caching is a feature you should only use if you produce many products with a scheduler.
An application is e.g. making 5min RSS frames for 'dust' and 'ash' of a volcano eruption.


A) Projection caching (GEO satellites only, original data has constant geographic layout)

When you resample (reproject) the original data as seen from the GEO satellite to a user
defined area, then depending on your resampler options, SatPy will produce area dependant
lookup tables for the projection of the data. More than one lookup table will be used if
your satellite data has different resolutions (e.g. composites using seviri 'HRV'). When
GEOcache is switched on, SatPy/Pyresample makes a hash of your 'area' (yaml or area_def?)
and looks for lookup table (lut) directories (or sets them up when missing) of the form:

nn_lut-1da10292445ed4dc671dea0785447aa1cc2913f2.zarr      When using resampler='nearest'
bil_lut-3ca4c51d6fc1438c2354bf8911777233fc0146da.zarr     When using resampler='bilinear'

No lut files are produced when using resampler='gradient_search' (install shapely first).
The fastest resampler is 'nearest' (default). The slowest is 'bilinear' but once cached
it can be as fast as resampler='gradient_search', method='bilinear' with similar results.


B) Overlay caching (GEO and LEO satellites, but cannot be used with LEO dynamic bb areas)

If overlays are used and OVRcache is ON Pycoast makes a hash of the 'area' (yaml or area_def?)
and the whole overlays dictionary (coasts, borders, rivers, grid, points and more ...). It
then produces a mostly transparent overlay file with a user choosable prefix, an underline,
a hash and a *.png suffix. I use 'area' as prefix, so my overlay cache images have names like:

westminster_261129563374870086_35748414afb6d1c538ca288e77f46b5a5231b69d86bcdf29ac2f21a14bb55f10.png

Next time Pycoast will calculate the hash and compare with names of available overlay *.png
images. If it finds a *.png image with the current hash in the name it uses this as overlay.
If the hash has changed (because you e.g. changed the colour of coastlines) an new overlay
*.png image will be created. Note that adding a single point, which does not even make it on
the final image, because it was defined outside the area_extent, will ask for a new *.png.


C) Tests made on a 10 years old PC with Intel XEON and 16GB RAM

Times found in (real) seconds when making 5 different MSG4 composites area 'westminster'.
All hrit files have been xRITDecompressed beforehand. Many Pycoast overlay features only
available in the current main branch (2022/07/01) have been used to produce coastlines,
borders, rivers, semitransparent filled lakes using shapefiles, grid, city circles using
cities, EARS network using star5 points and a station marker using triangle text points.

Three different resamplers have been testet, numbers indicate (real) seconds using time:

.., resampler='nearest', radius_of_influence=20000, reduce_data=False)
1 No caching (cache emptied)    42
2 Switching both ON             80  (setting up GEOcache luts takes time)
3 GEOcache ON only              34
4 GEOcache ON + OVRcache ON     16

.., resampler='bilinear', reduce_data=True)
1 No caching (cache emptied)   138
2 Switching both ON            152  (setting up GEOcache luts takes time)
3 GEOcache ON only              50
4 GEOcache ON + OVRcache ON     29

.., resampler='gradient_search', method='bilinear', reduce_data=False)
1 No caching (cache emptied)    51
2 Switching both ON             51  (setting GEOcache ON has no effect)
3 GEOcache ON only              50
4 GEOcache ON + OVRcache On     29

Image production has been performed with "global_scene.load(composites, generate=False)".
Times include the generation and addition of an annotation at left using ImageMagick (IM).
All tests have been done under GNU/Linux. Tests on a similar Windows PC 2-3 times slower.


Belp, 2022/07/01                                                          Ernst Lobsiger

