{
"cells": [
{
"cell_type": "markdown",
"id": "0",
"metadata": {},
"source": [
"[](https://demo.leafmap.org/lab/index.html?path=notebooks/87_actinia.ipynb)\n",
"[](https://colab.research.google.com/github/opengeos/leafmap/blob/master/docs/notebooks/87_actinia.ipynb)\n",
"[](https://mybinder.org/v2/gh/opengeos/leafmap/HEAD)\n",
"\n",
"**Cloud-based geoprocessing with Actinia**"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1",
"metadata": {},
"outputs": [],
"source": [
"# %pip install -U leafmap"
]
},
{
"cell_type": "markdown",
"id": "2",
"metadata": {},
"source": [
"The cloud based geoprocessing platform [actinia](https://github.com/actinia-org) is able to ingest and analyse large volumes of geodata in the cloud.\n",
"\n",
"For the following actinia example we use the [actinia-python-client](https://actinia-org.github.io/actinia-python-client/) ([source code](https://github.com/actinia-org/actinia-python-client)) to establish the connection to an actinia instance. \n",
"First install the actinia-python-client (for latest version, see [actinia-python-client releases](https://github.com/actinia-org/actinia-python-client/releases))."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import leafmap"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4",
"metadata": {},
"outputs": [],
"source": [
"%pip install actinia_python_client"
]
},
{
"cell_type": "markdown",
"id": "5",
"metadata": {},
"source": [
"The results of [actinia](https://actinia.mundialis.de/) ephemeral processing are available via object storage as GeoTIFF/COG or GeoPackage files."
]
},
{
"cell_type": "markdown",
"id": "6",
"metadata": {},
"source": [
"Add a helper function for \"pretty printing\" of actinia results:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7",
"metadata": {},
"outputs": [],
"source": [
"from json import dumps as json_dumps\n",
"\n",
"\n",
"def print_dict(input_dict, text=None):\n",
" if text:\n",
" print(text)\n",
" if \"region\" in input_dict:\n",
" input_dict[\"region\"] = input_dict[\"region\"].__dict__\n",
" print(json_dumps(input_dict, sort_keys=True, indent=4))\n",
"\n",
"\n",
"def print_dict_keys(input_dict, text=None):\n",
" if text:\n",
" print(text)\n",
" print(\", \".join(input_dict.keys()))"
]
},
{
"cell_type": "markdown",
"id": "8",
"metadata": {},
"source": [
"Connect to the default actinia server which is defined in the actinia-python-client, currently https://actinia.mundialis.de."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9",
"metadata": {},
"outputs": [],
"source": [
"# connect to the actinia server\n",
"from actinia import Actinia\n",
"\n",
"# connect to default actinia server (https://actinia.mundialis.de)\n",
"actinia_mundialis = Actinia()\n",
"\n",
"# retrieve metadata about actinia server and related software versions\n",
"version = actinia_mundialis.get_version()\n",
"print_dict(version, \"Version is:\")"
]
},
{
"cell_type": "markdown",
"id": "10",
"metadata": {},
"source": [
"Set the authentication settings of the actinia demo user to gain access to the actinia server functionality."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "11",
"metadata": {},
"outputs": [],
"source": [
"actinia_user = \"demouser\"\n",
"actinia_password = \"gu3st!pa55w0rd\"\n",
"\n",
"# we use the default actinia server\n",
"actinia_mundialis.set_authentication(actinia_user, actinia_password)\n",
"print(\"Connected to actinia server.\")"
]
},
{
"cell_type": "markdown",
"id": "12",
"metadata": {},
"source": [
"Obtain the list of locations and retrieve the metadata of a selected location."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "13",
"metadata": {},
"outputs": [],
"source": [
"# obtain the list of projects (called \"locations\") which are accessible to current user\n",
"locations = actinia_mundialis.get_locations()\n",
"print_dict_keys(locations, \"Locations: \")"
]
},
{
"cell_type": "markdown",
"id": "14",
"metadata": {},
"source": [
"Retrieve the metadata of a selected location (this shows the respective projection information, spatial extent, resolution, etc.) to get an idea how the output looks like."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "15",
"metadata": {},
"outputs": [],
"source": [
"print_dict(actinia_mundialis.locations[\"nc_spm_08\"].get_info(), \"Location info:\")"
]
},
{
"cell_type": "markdown",
"id": "16",
"metadata": {},
"source": [
"At this point the connection to the selected actinia server is properly established."
]
},
{
"cell_type": "markdown",
"id": "17",
"metadata": {},
"source": [
"**Reading the online data resource into the actinia server**"
]
},
{
"cell_type": "markdown",
"id": "18",
"metadata": {},
"source": [
"Next we demonstrate the data processing of a raster map available online in actinia, here a sample DEM GeoTIFF file."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "19",
"metadata": {},
"outputs": [],
"source": [
"# define raster elevation map name\n",
"raster_layer_name = \"srtm90\"\n",
"\n",
"# cache file locally\n",
"out_dir = os.getcwd()\n",
"dem_file = os.path.join(out_dir, f\"{raster_layer_name}.tif\")\n",
"\n",
"# dem_url = (\n",
"# \"https://drive.google.com/file/d/1vRkAWQYsLWCi6vcTMk8vLxoXMFbdMFn8/view?usp=sharing\"\n",
"# )\n",
"dem_url = f\"https://github.com/giswqs/data/raw/main/raster/{raster_layer_name}.tif\"\n",
"\n",
"# leafmap.download_file(dem_url, dem_file, unzip=False, overwrite=True)"
]
},
{
"cell_type": "markdown",
"id": "20",
"metadata": {},
"source": [
"Prepare actinia location and mapset, i.e. generate a subproject for data processing."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "21",
"metadata": {},
"outputs": [],
"source": [
"# request list of all locations\n",
"locations = actinia_mundialis.get_locations()\n",
"print([loc for loc in locations])"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "22",
"metadata": {},
"outputs": [],
"source": [
"# remove leftover location from previous run\n",
"# actinia_mundialis.locations[\"latlong_wgs84\"].delete()\n",
"#\n",
"# remove leftover mapset from previous run\n",
"locations[\"latlong_wgs84\"].delete_mapset(\"elevation\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "23",
"metadata": {},
"outputs": [],
"source": [
"# Create a new location for the data processing in actinia\n",
"new_location = actinia_mundialis.create_location(\"latlong_wgs84\", 4326)\n",
"print(new_location.name)\n",
"print(new_location.region)\n",
"print([loc for loc in actinia_mundialis.locations])\n",
"\n",
"# request list of mapsets in selected location\n",
"mapsets = actinia_mundialis.locations[\"latlong_wgs84\"].get_mapsets()\n",
"print_dict_keys(mapsets, \"Mapsets in latlong_wgs84:\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "24",
"metadata": {},
"outputs": [],
"source": [
"# Create a new mapset for the data processing in actinia\n",
"mapset_name = \"elevation\"\n",
"locations[\"latlong_wgs84\"].create_mapset(mapset_name)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "25",
"metadata": {},
"outputs": [],
"source": [
"## Optional: Upload the sample DEM data set to actinia (indeed not needed since we use `vsicurl/` below\n",
"## to directly retrieve the online dataset).\n",
"# locations[\"latlong_wgs84\"].mapsets[mapset_name].upload_raster(raster_layer_name, dem_file)\n",
"# print_dict_keys(locations[\"latlong_wgs84\"].mapsets[mapset_name].raster_layers, \"Raster maps in new mapset:\")"
]
},
{
"cell_type": "markdown",
"id": "26",
"metadata": {},
"source": [
"#### Ephemeral Processing with actinia\n",
"\n",
"**Ephemeral processing** is used to keep computed results, including user-generated data and temporary data, only for a limited period of time (e.g. 24 hours by default in the actinia demo server). This reduces cloud storage costs.\n",
"\n",
"In contrast, **persistent processing** refers to the persistent retention of data without a scheduled deletion time, even in the event of a power outage, resulting in corresponding storage costs. In the geo/EO context, persistent storage is used to provide, for example, basic cartography, i.e. elevation models, road networks, building footprints, etc."
]
},
{
"cell_type": "markdown",
"id": "27",
"metadata": {},
"source": [
"**Hillshading example**\n",
"\n",
"Here an example for an ephemeral processing job: We download and import the remotely available GeoTIFF file. Then we use [r.relief](https://grass.osgeo.org/grass-stable/manuals/r.relief.html) to generate a hillshading map and pre-define the resolution to 10 m. The computational region is set to the input elevation map. The resulting `hillshade.tif` raster map is then provided as a resource for download and visualization."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "28",
"metadata": {},
"outputs": [],
"source": [
"pc = {\n",
" \"list\": [\n",
" {\n",
" \"id\": \"importer_0\",\n",
" \"comment\": \"Import of remote data source (here: COG)\",\n",
" \"module\": \"r.import\",\n",
" \"inputs\": [\n",
" {\"param\": \"input\", \"value\": f\"/vsicurl/{dem_url}\"},\n",
" {\"param\": \"memory\", \"value\": \"2000\"},\n",
" {\"param\": \"extent\", \"value\": \"input\"},\n",
" ],\n",
" \"outputs\": [{\"param\": \"output\", \"value\": f\"{raster_layer_name}\"}],\n",
" },\n",
" {\n",
" \"id\": \"r.info_1\",\n",
" \"comment\": \"Print metadata of imported raster map\",\n",
" \"module\": \"r.info\",\n",
" \"inputs\": [{\"param\": \"map\", \"value\": f\"{raster_layer_name}\"}],\n",
" },\n",
" {\n",
" \"id\": \"computational_region_2\",\n",
" \"comment\": \"Set computational region to imported map, and print settings\",\n",
" \"module\": \"g.region\",\n",
" \"inputs\": [{\"param\": \"raster\", \"value\": f\"{raster_layer_name}\"}],\n",
" \"stdout\": {\"id\": \"region\", \"format\": \"kv\", \"delimiter\": \"=\"},\n",
" \"flags\": \"g\",\n",
" },\n",
" {\n",
" \"id\": \"create_hillshading_3\",\n",
" \"comment\": \"Compute hillshading map\",\n",
" \"module\": \"r.relief\",\n",
" \"inputs\": [{\"param\": \"input\", \"value\": f\"{raster_layer_name}\"}],\n",
" \"outputs\": [{\"param\": \"output\", \"value\": \"hillshade\"}],\n",
" },\n",
" {\n",
" \"id\": \"exporter_4\",\n",
" \"comment\": \"Export hillshading map to COG file\",\n",
" \"module\": \"exporter\",\n",
" \"outputs\": [\n",
" {\n",
" \"export\": {\"type\": \"raster\", \"format\": \"COG\"},\n",
" \"param\": \"map\",\n",
" \"value\": \"hillshade\",\n",
" }\n",
" ],\n",
" },\n",
" ],\n",
" \"version\": \"1\",\n",
"}\n",
"\n",
"\n",
"print(pc)\n",
"job = actinia_mundialis.locations[\"latlong_wgs84\"].create_processing_export_job(\n",
" pc, \"hillshading\"\n",
")\n",
"job.poll_until_finished()\n",
"\n",
"print(job.status)\n",
"print(job.message)\n",
"exported_raster = job.urls[\"resources\"][0]\n",
"print(exported_raster)"
]
},
{
"cell_type": "markdown",
"id": "29",
"metadata": {},
"source": [
"It will take a moment, then the communication by actinia is shown: \"Status of hillshading job is accepted: Resource accepted\" continued by further communication messages."
]
},
{
"cell_type": "markdown",
"id": "30",
"metadata": {},
"source": [
"In case an error occurs, check the process log (use [x] with x being the step number in the process chain). Examples:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "31",
"metadata": {},
"outputs": [],
"source": [
"# check step 0 (r.import)\n",
"print_dict(job.process_log[0])"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "32",
"metadata": {},
"outputs": [],
"source": [
"# check step 2 (g.region)\n",
"print_dict(job.process_log[2])"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "33",
"metadata": {},
"outputs": [],
"source": [
"# check step 3 (r.relief)\n",
"print_dict(job.process_log[3])"
]
},
{
"cell_type": "markdown",
"id": "34",
"metadata": {},
"source": [
"Inject `user:password@server` into `exported_raster` URL (i.e., the actinia resource)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "35",
"metadata": {},
"outputs": [],
"source": [
"url = exported_raster.replace(\"//\", f\"//{actinia_user}:{actinia_password}@\")\n",
"print(url)"
]
},
{
"cell_type": "markdown",
"id": "36",
"metadata": {},
"source": [
"Visualize the `hillshade` map in leafmap (colorbar inspired by [this notebook](https://leafmap.org/notebooks/07_colorbar/)):"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "37",
"metadata": {},
"outputs": [],
"source": [
"m = leafmap.Map()\n",
"m.add_basemap(\"OpenTopoMap\")\n",
"m.add_colormap(\n",
" cmap=\"terrain\",\n",
" label=\"Elevation\",\n",
" width=3,\n",
" height=0.3,\n",
" orientation=\"horizontal\",\n",
" vmin=0,\n",
" vmax=4000,\n",
")\n",
"m.add_cog_layer(\n",
" url,\n",
" name=\"SRTM90 hillshaded map\",\n",
" attribution='<a href=\"https://e4ftl01.cr.usgs.gov/MEASURES/\">https://e4ftl01.cr.usgs.gov/MEASURES/</a>',\n",
")\n",
"# show map\n",
"m"
]
},
{
"cell_type": "markdown",
"id": "38",
"metadata": {},
"source": [
"Find further leafmap (styling) tools in the upper-right toolbox of leafmap."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}