s3-download ingest notebook BIOMASS-2371

11fc59a7 · algodeveloper1@free.fr · 569f164a · 11fc59a7
Commit 11fc59a7 authored 3 years ago by algodeveloper1@free.fr
--- a/edav/user-data-ingest-from-module.ipynb
+++ b/edav/user-data-ingest-from-module.ipynb
@@ -2,7 +2,7 @@
 "cells": [
  {
   "cell_type": "markdown",
-   "id": "4de51eb6",
+   "id": "c20c96a7",
   "metadata": {},
   "source": [
    "<h1>Sample user data ingestion notebook</h1>"
@@ -10,7 +10,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "8b45f9ee",
+   "id": "f5188e6b",
   "metadata": {},
   "source": [
    "As a first step add the ingestion module location to path. In the future the ingestion module could be automatically put in the python module path."
@@ -18,8 +18,8 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 11,
+   "execution_count": null,
-   "id": "5a994d73",
+   "id": "e09fede3",
   "metadata": {},
   "outputs": [],
   "source": [
@@ -30,7 +30,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "11cb449a",
+   "id": "2608ef04",
   "metadata": {},
   "source": [
    "Then import the product ingestion utils from the edav_ingest module. The module uses the configuration information set in `/projects/.maap/edav.ini`.\n",
@@ -46,7 +46,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "3987a8cd",
+   "id": "cc099be6",
   "metadata": {},
   "source": [
    "<h3>Local file ingestion</h3>\n",
@@ -56,25 +56,10 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": null,
-   "id": "a089ae96",
+   "id": "eddc45b7",
   "metadata": {},
-   "outputs": [
+   "outputs": [],
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "b97de56bbeea4429b6e7caf1a2739756",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "VBox(children=(HBox(children=(Text(value='georef_UK_classif', description='Dataset ID*:', layout=Layout(width=…"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    }
-   ],
   "source": [
    "from edav_form import generateIngestionFormForLocalProduct\n",
    "\n",
@@ -83,7 +68,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "bb1cbec4",
+   "id": "bbef0ed1",
   "metadata": {},
   "source": [
    "<h3>Remote file ingestion</h3>\n",
@@ -93,25 +78,10 @@
  },
  {
   "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": null,
-   "id": "13f4e740",
+   "id": "38621199",
   "metadata": {},
-   "outputs": [
+   "outputs": [],
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "5f1d32eb92124389907392e2f02baf15",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "VBox(children=(HBox(children=(Text(value='georef_Ecosse_classif_test', description='Dataset ID*:', layout=Layo…"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    }
-   ],
   "source": [
    "from edav_form import generateIngestionFormForRemoteProduct\n",
    "\n",
@@ -121,7 +91,7 @@
  {
   "cell_type": "code",
   "execution_count": null,
-   "id": "87872c47",
+   "id": "6b397258",
   "metadata": {},
   "outputs": [],
   "source": []

-%% Cell type:markdown id:4de51eb6 tags:
+%% Cell type:markdown id:c20c96a7 tags:
 <h1>Sample user data ingestion notebook</h1>
-%% Cell type:markdown id:8b45f9ee tags:
+%% Cell type:markdown id:f5188e6b tags:
 As a first step add the ingestion module location to path. In the future the ingestion module could be automatically put in the python module path.
-%% Cell type:code id:5a994d73 tags:
+%% Cell type:code id:e09fede3 tags:
 ``` python
 import os
 import sys
 sys.path.insert(0, os.path.abspath('./user-data-ingest-scripts'))
 ```
-%% Cell type:markdown id:11cb449a tags:
+%% Cell type:markdown id:2608ef04 tags:
 Then import the product ingestion utils from the edav_ingest module. The module uses the configuration information set in `/projects/.maap/edav.ini`.
 It depends on rasterio and ipywidgets libraries that are currently not available in the conda environment. To prepare the environment the following commands were run before importing the script:
 `conda install --freeze-installed rasterio`
 `conda install --freeze-installed ipywidgets`
 `jupyter labextension install @jupyter-widgets/jupyterlab-manager`
-%% Cell type:markdown id:3987a8cd tags:
+%% Cell type:markdown id:cc099be6 tags:
 <h3>Local file ingestion</h3>
 First we run the local ingestion script passing the location of a local product file. This will generate a form. Complete the missing information and submit it to complete the ingestion. The function uses the `maap-s3.py` script internally to upload the product to S3 (in `[user_data_remote_s3_path]/[user_data_upload_location]/[USER_EMAIL]/` as specified in `/projects/.maap/edav.ini` e.g. `maap-scientific-data/shared/edav/edav_esa-maap.org`). Be sure it is available and that the information in `/projects/.maap/auth.ini` are correct
-%% Cell type:code id:a089ae96 tags:
+%% Cell type:code id:eddc45b7 tags:
 ``` python
 from edav_form import generateIngestionFormForLocalProduct
 generateIngestionFormForLocalProduct('/projects/data/georef_UK_classif.tif')
 ```
-%% Output
+%% Cell type:markdown id:bbef0ed1 tags:
-%% Cell type:markdown id:bb1cbec4 tags:
 <h3>Remote file ingestion</h3>
 To ingest a product already on S3 user data location, run the remote ingestion script passing the relative location of the product (relative to the `user_data_local_s3_mount` set in `/projects/.maap/edav.ini`). The assumption is that the S3 location is locally mounted in the jupyter env (e.g. in /project/s3-drive/user-data, configurable in `/projects/.maap/edav.ini`) in order to extract the required information from the product metadata
-%% Cell type:code id:13f4e740 tags:
+%% Cell type:code id:38621199 tags:
 ``` python
 from edav_form import generateIngestionFormForRemoteProduct
 generateIngestionFormForRemoteProduct('GPU_TEST_TIF/georef_Ecosse_classif_test.tiff')
 ```
-%% Output
+%% Cell type:code id:6b397258 tags:
-%% Cell type:code id:87872c47 tags:
 ``` python
 ```