maap-s3-laptop
This python script is used to interact with the esa maap s3.
The script is by default in the stack theia and Jupyter and can be used as a shell script. From you laptop you shoud prefix the script with the command 'python'.
Prerequisite
To use the script some env variables must be defined. Execute the command below from an eclipse che workspace to see the values you have to setup.
echo $MAAP_ENV_TYPE
echo $CLIENT_ID
echo $AUTH_ACCESS_TOKEN_URLCheck if you have request library, or install it if missing.
How to set up env variables on windows:
PowerShell:
$Env:MAAP_ENV_TYPE = "<Value>"
$Env:CLIENT_ID = "<Value>"
$Env:AUTH_ACCESS_TOKEN_URL = "<Value>"CMD:
set MAAP_ENV_TYPE=<Value>
set CLIENT_ID=<Value>
set AUTH_ACCESS_TOKEN_URL=<Value>Linux based OS
Bash
export MAAP_ENV_TYPE=<Value>
export CLIENT_ID=<Value>
export AUTH_ACCESS_TOKEN_URL=<Value>Info
Maap S3 script is the script used to interact with the bucket S3 to share data with other maap users or to save data when workspace is full. Please don't forget the env variable token and maap_env_type The script provide following features:
To see all options run maap-s3.py help
**upload used to push the data to S3**
param 1: sourceFile
ex: /projects/test/toto.txt
param 2: s3 destination path
ex: maap-scientific-data/shared/test/tkossoko/obs.jar
exemple: maap-s3.py upload /projects/test/maap.tiff maap-scientific-data/shared/test/tkossoko/maap.tiff
!WARNING! Upload is only allowed on a few buckets (see BUCKETS3_TEMPORARY_SIGNATURE_PUT_INCLUDE_PATHS configuration parameter in backend for the list of allowed buckets)
resume use to continue a failed multipart previous upload. When your file is more than 5Giga, the python script use multipart upload. During the process, if the upload fails, please run resume without parameter to continue the upload.
**download to download a data from S3**
param 1: s3 path
ex: maap-scientific-data/shared/test/demo/maap.tiff
param 2: Workspace destination path
ex: /projects/test/maap.tiff
exemple: `maap-s3.py download maap-scientific-data/shared/test/tkossoko/maap.tiff /projects/test/maap.tiff`
**list to list a folder from S3**
param 1: s3 bucket folder
ex: maap-scientific-data/shared/test/demo
exemple: `maap-s3.py list maap-scientific-data/shared/test/demo`
exemple: `maap-s3.py list bmap-catalogue-data/Campaign_data`
**delete to delete a file from S3**
param 1: s3 file path
ex: maap-scientific-data/shared/test/demo/maap.tiff
exemple: `maap-s3 delete maap-scientific-data/shared/test/demo/hello-bjorn.txt`
**refresh to reset credentials and passwords when init fails because you type wrong password or email.**
exemple: `maap-s3.py refresh`Private data feature
To manage your personal user data you can upload, download, list and delete your files from your user data folder using -p or --private flag.
Note: -p/--private can be positioned anywhere after calling your script
example : All three following commands are correct
1- maap-s3.py upload -p /projects/test/maap.tiff test/tkossoko/maap.tiff
2- maap-s3.py upload /projects/test/maap.tiff -p test/tkossoko/maap.tiff
2- maap-s3.py upload /projects/test/maap.tiff test/tkossoko/maap.tiff --private
The -p/--private flag serves to indicate that the action should be taken in the user's private bucket. thus the bucket name is known for the algorithm and no need to include it in your destination.
when using "test/myfile.txt" as your destination, the -p/--private flag systematically instruct the algorithm to work on this file in "private-bucket/user-space/test/myfile.txt". Nonetheless the structure of commands stays the same as demonstrated in the previous paragraph.
Here are some example:
*** upload ***
The following command will upload your local file "/projects/test/maap.tiff" to the user's s3 folder "$private-bucket/$user-name/test/tkossoko/maap.tiff"
exemple: `maap-s3.py upload -p /projects/test/maap.tiff test/tkossoko/maap.tiff`
*** list ***
When including -p or --private option you can list files and folders in your private storage.
example1 : `maap-s3.py list -p /`
example2 : `maap-s3.py list -p test/tkossoko/`
*** download ***
Use -p or --private to download your file from you personal storage.
example : `maap-s3.py download -p test/tkossoko/maap.tiff /projects/test/maap.tiff`
*** delete ***
Use -p to delete a file from your personal space.
example : `maap-s3.py delete -p test/tkossoko/maap.tiff`Info for the stacks
Theia The shebang of the script must be replaced by #!/usr/local/bin/python or use the prefix 'python'
Jupyter The shebang of the script must be replaced by #!/opt/conda/bin/python