HDX-9931 - deploy ckan locally using devcontainers

.devcontainer/README.md

@@ -1,39 +1,73 @@
 # HDX CKAN VSCode development
 
-## First Time Setup
+## Installation
 1. Install and open VSCode
 1. Add the "Dev Containers" extension
 1. Clone this git repository
 1. Open this repo in VSCode
-1. Click the popup saying "Reopen in container"
-1. Wait for a few minutes while everything is set up for the first time
+1. Remember to create an `.env.secrets` file in the `.devcontainer` folder from `.env.secrets.template`. Values for secrets can be obtained from OCHA Infrastructure. These secrets also need to be added to `/etc/ckan/prod.ini` in the ckan container.
+1. In VS Code click the popup saying "Reopen in container"
+1. Wait for an hour while the containers are built and Python packages are installed.
+1. The `proxy` network should be produced automatically, if not then run `docker network create proxy` from a shell prompt in the host machine.
+1. A value needs to be added to `/etc/ckan/prod.ini` for `beaker.session.secret` inside the ckan container. It does not matter what this value is.
+1. Create a `hdx-test-core.ini` file (?) and ensure it has the correct values for the following entries:
+```
+solr_url = http://solr:8983/solr/ckan
+sqlalchemy.url = postgresql://ckan:[secret]@db:5432/ckan_test
+ckan.datastore.write_url = postgresql://ckan:[secret]@db:5432/datastore_test
+ckan.datastore.read_url = postgresql://datastore:[secret]@db:5432/datastore_test
+```
 
-## Daily development
-1. Open this repo in VSCode
-1. Click the popup saying "Reopen in container"
-1. Wait for a bit until the container starts
-1. Open a VSCode terminal
+1. The `awscli` commandline tool needs to be installed by running the `awscli-install-run-once.sh` script.
 
-Note: All commands shown below are typed in the ckan container; if you do use VSCode Dev Containers, that would be the terminal inside your VSCode
+## All-in-one magic one time setup
 
-## All-in-one magic setup
-1. Type in: `hdxckantool magic`
-1. Type in: `y`, then your jenkins username and password (note that the password characters will not be echoed out in the terminal)
+The "magic" setup prepares the ckan databases and loads them with the latest (minimal) backups, with the `--test` flag databases are prepared but not loaded with data.
+
+1. Type in: `hdxckantool --verbose magic`
+1. Type in: `y`, then your jenkins username and password (note that the password characters will not be echoed out in the terminal) - these can be obtained from OCHA Infrastructure;
 1. This will take quite a while; go get a few coffees and walks. In the mean time, the following will be done for you (not necessarily in this order):
     1. pull the latest database backups and files backups
     1. restore the files (not filestore)
-    1. syncronise the s3 filestore with the dev filestore
+    1. syncronise the s3 filestore with the dev filestore - this is slow (hours) because it transfers all of the files in HDX to the developer's S3 bucket 
     1. [re]create the databases and restores from the last backups
     1. [re]create the solr collection and reindexes it
+1.Type in: `hdxckantool --verbose magic --test` - this does similar but does not restore databases or rebuild the solr index
 Enjoy!
 
+
+
+## Running tests
+Once "magic" setup has been carried out, a local instance of the HDX website can be launched with: 
+
+```ckan -c /etc/ckan/prod.ini run --disable-reloader```
+
+VS Code automatically forwards the 5000 port from the ckan container, so that the website can be reached in the host at `localhost:5000`.
+
+Tests are run with commandlines like:
+
+`pytest --ckan-ini=hdx-test-core.ini ./ckanext-hdx_package/ckanext/hdx_package/tests`
+
+This invocation runs 119 test which take about 20 minutes.
+
+## Daily development
+1. Note that Desktop Docker starts some of the ckan containers, but not all (solr and zookeeper are not started), on launch so stop them all and allow VS Code to launch them all;
+1. Open this repo in VSCode
+1. Click the popup saying "Reopen in container"
+1. Wait for a bit until the container starts
+1. Open a VSCode terminal
+
+Note: All commands shown below are typed in the ckan container; if you do use VSCode Dev Containers, that would be the terminal inside your VSCode
+
+
+
 ## No magic for me, I want hardcore stuff
 
 ### Database Setup
 1. The user and database creation is done automatically, but you do need to restore.
 1. Set the proper database password in `.pgpass` so you wont have to type it everytime:
     1. Type in: `hdxckantool db pgpass`
-1. Make sure the public schems belongs to the database owner:
+1. Make sure the public schema belongs to the database owner:
     1. Type in: `hdxckantool db schema`
 1. Pull the latest backups:
     1. In a new terminal inside your VSCode type: `hdxckantool db pull -a`
@@ -56,4 +90,6 @@ Enjoy!
     `hdxckantool solr reindex --clear --fast`
 
 ## Sync your S3 filestore
+This moves something like 250GB of files from the production filestore to your developer bucket, taking around 6 hours for an initially empty bucket.
+
 1. Type in: `hdxckantool filestore sync -c`

.devcontainer/awscli-install-run-once.sh

@@ -0,0 +1,5 @@
+set -x
+apt install unzip
+curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
+unzip awscliv2.zip
+./aws/install

.gitattributes

@@ -1,2 +1,6 @@
 #*.po -diff
 *.mo -diff
+
+* text=auto eol=lf
+*.sh eol=lf
+

.gitignore

@@ -62,3 +62,5 @@ travis.env
 
 # cypress
 cypress/videos
+
+src/

.vscode/settings.json

@@ -0,0 +1,7 @@
+{
+    "python.testing.pytestArgs": [
+        "ckan/tests"
+    ],
+    "python.testing.unittestEnabled": false,
+    "python.testing.pytestEnabled": true
+}

ckan/public/base/i18n/am.js

@@ -0,0 +1,156 @@
+{
+  "": {
+    "domain": "ckan",
+    "lang": "am",
+    "plural-forms": "nplurals=2; plural=(n > 1);"
+  },
+  "Add Filter": [
+    null,
+    "ማጣሪያ ጨምር"
+  ],
+  "Are you sure you want to perform this action?": [
+    null,
+    " ይህንን እርምጃ ማከናወን መፈለግዎን እርግጠኛ ነዎት?"
+  ],
+  "Cancel": [
+    null,
+    "ሰርዝ"
+  ],
+  "Confirm": [
+    null,
+    "አረጋግጥ"
+  ],
+  "Could not load view": [
+    null,
+    "እይታን መጫን አልቻለም"
+  ],
+  "DataProxy returned an error": [
+    null,
+    "የመረጃ ተኪው(DataProxy) ስህተት መልሷል"
+  ],
+  "DataStore returned an error": [
+    null,
+    "የመረጃ ማከማቻው(DataStore) ስህተት መልሷል"
+  ],
+  "Edit": [
+    null,
+    "አድስ"
+  ],
+  "Failed to load data API information": [
+    null,
+    "የኤፒአይ መረጃ መጫን አልተሳካም"
+  ],
+  "File": [
+    null,
+    "ፋይል"
+  ],
+  "Filters": [
+    null,
+    "ማጣሪያዎች"
+  ],
+  "Follow": [
+    null,
+    "ተከተል"
+  ],
+  "Graph": [
+    null,
+    "ግራፍ"
+  ],
+  "Grid": [
+    null,
+    "ፍርግርግ"
+  ],
+  "Hide": [
+    null,
+    "ደብቅ"
+  ],
+  "Image": [
+    null,
+    "ምስል"
+  ],
+  "Input is too short, must be at least one character": [
+    "Input is too short, must be at least %(num)d characters",
+    "ግብዓቱ በጣም አጭር ነው ፤ ቢያንስ አንድ ቁምፊ(ትዕምርት) መሆን አለበት",
+    "ግብዓቱ በጣም አጭር ነው ፤ ቢያንስ %(num)d ቁምፊዎች(ትዕምርቶች) መሆን አለበት"
+  ],
+  "Link": [
+    null,
+    "አገናኝ(ሊንክ)"
+  ],
+  "Link to a URL on the internet (you can also link to an API)": [
+    null,
+    "በኢንተርኔት ላይ ወደ ዩአርኤል አገናኝ (ወደ ኤ.ፒ.አይ መገናኘት ይችላሉ)"
+  ],
+  "Loading...": [
+    null,
+    "በመጫን ላይ"
+  ],
+  "Map": [
+    null,
+    "ካርታ"
+  ],
+  "No matches found": [
+    null,
+    "ምንም ተዛማጆች አልተገኙም"
+  ],
+  "Please Confirm Action": [
+    null,
+    "እባክዎን እርምጃውን(ድርጊቱን) ያረጋግጡ"
+  ],
+  "Remove": [
+    null,
+    "አስወግድ"
+  ],
+  "Reset": [
+    null,
+    "ዳግም አጽድቅ"
+  ],
+  "Save order": [
+    null,
+    "ቅደም-ተከተሉን አስቀምጥ"
+  ],
+  "Saving...": [
+    null,
+    "እያስቀመጠ..."
+  ],
+  "Select a field": [
+    null,
+    "መስክ ምረጥ"
+  ],
+  "Show more": [
+    null,
+    "ተጨማሪ አሳይ"
+  ],
+  "Start typing…": [
+    null,
+    "መተየብ(መጻፍ) ይጀምሩ..."
+  ],
+  "There are unsaved modifications to this form": [
+    null,
+    "በዚህ ቅጽ ላይ ያልተቀመጡ(ያልተያዙ) ለውጦች አሉ"
+  ],
+  "There is no API data to load for this resource": [
+    null,
+    "ለዚህ የመረጃ-ምንጭ ምንም የሚጫን የኤፒአይ መረጃ የለም"
+  ],
+  "URL": [
+    null,
+    "ዩአርኤል"
+  ],
+  "Unfollow": [
+    null,
+    "አትከተል"
+  ],
+  "Upload": [
+    null,
+    "አስቀምጥ"
+  ],
+  "Upload a file on your computer": [
+    null,
+    "በኮምፒዩተርዎ ውስጥ ያለ ፋይል ያስቀምጡ"
+  ],
+  "error loading view": [
+    null,
+    "እይታን በመጫን ላይ የተፈጠረ ስህተት"
+  ]
+}