HDX-9931 - deploy ckan locally using devcontainers

.devcontainer/README.md

@@ -1,39 +1,73 @@
 # HDX CKAN VSCode development
 
-## First Time Setup
+## Installation
 1. Install and open VSCode
 1. Add the "Dev Containers" extension
 1. Clone this git repository
 1. Open this repo in VSCode
-1. Click the popup saying "Reopen in container"
-1. Wait for a few minutes while everything is set up for the first time
+1. Remember to create an `.env.secrets` file in the `.devcontainer` folder from `.env.secrets.template`. Values for secrets can be obtained from OCHA Infrastructure. These secrets also need to be added to `/etc/ckan/prod.ini` in the ckan container.
+1. In VS Code click the popup saying "Reopen in container"
+1. Wait for an hour while the containers are built and Python packages are installed.
+1. The `proxy` network should be produced automatically, if not then run `docker network create proxy` from a shell prompt in the host machine.
+1. A value needs to be added to `/etc/ckan/prod.ini` for `beaker.session.secret` inside the ckan container. It does not matter what this value is.
+1. Create a `hdx-test-core.ini` file (?) and ensure it has the correct values for the following entries:
+```
+solr_url = http://solr:8983/solr/ckan
+sqlalchemy.url = postgresql://ckan:[secret]@db:5432/ckan_test
+ckan.datastore.write_url = postgresql://ckan:[secret]@db:5432/datastore_test
+ckan.datastore.read_url = postgresql://datastore:[secret]@db:5432/datastore_test
+```
 
-## Daily development
-1. Open this repo in VSCode
-1. Click the popup saying "Reopen in container"
-1. Wait for a bit until the container starts
-1. Open a VSCode terminal
+1. The `awscli` commandline tool needs to be installed by running the `awscli-install-run-once.sh` script.
 
-Note: All commands shown below are typed in the ckan container; if you do use VSCode Dev Containers, that would be the terminal inside your VSCode
+## All-in-one magic one time setup
 
-## All-in-one magic setup
-1. Type in: `hdxckantool magic`
-1. Type in: `y`, then your jenkins username and password (note that the password characters will not be echoed out in the terminal)
+The "magic" setup prepares the ckan databases and loads them with the latest (minimal) backups, with the `--test` flag databases are prepared but not loaded with data.
+
+1. Type in: `hdxckantool --verbose magic`
+1. Type in: `y`, then your jenkins username and password (note that the password characters will not be echoed out in the terminal) - these can be obtained from OCHA Infrastructure;
 1. This will take quite a while; go get a few coffees and walks. In the mean time, the following will be done for you (not necessarily in this order):
     1. pull the latest database backups and files backups
     1. restore the files (not filestore)
-    1. syncronise the s3 filestore with the dev filestore
+    1. syncronise the s3 filestore with the dev filestore - this is slow (hours) because it transfers all of the files in HDX to the developer's S3 bucket 
     1. [re]create the databases and restores from the last backups
     1. [re]create the solr collection and reindexes it
+1.Type in: `hdxckantool --verbose magic --test` - this does similar but does not restore databases or rebuild the solr index
 Enjoy!
 
+
+
+## Running tests
+Once "magic" setup has been carried out, a local instance of the HDX website can be launched with: 
+
+```ckan -c /etc/ckan/prod.ini run --disable-reloader```
+
+VS Code automatically forwards the 5000 port from the ckan container, so that the website can be reached in the host at `localhost:5000`.
+
+Tests are run with commandlines like:
+
+`pytest --ckan-ini=hdx-test-core.ini ./ckanext-hdx_package/ckanext/hdx_package/tests`
+
+This invocation runs 119 test which take about 20 minutes.
+
+## Daily development
+1. Note that Desktop Docker starts some of the ckan containers, but not all (solr and zookeeper are not started), on launch so stop them all and allow VS Code to launch them all;
+1. Open this repo in VSCode
+1. Click the popup saying "Reopen in container"
+1. Wait for a bit until the container starts
+1. Open a VSCode terminal
+
+Note: All commands shown below are typed in the ckan container; if you do use VSCode Dev Containers, that would be the terminal inside your VSCode
+
+
+
 ## No magic for me, I want hardcore stuff
 
 ### Database Setup
 1. The user and database creation is done automatically, but you do need to restore.
 1. Set the proper database password in `.pgpass` so you wont have to type it everytime:
     1. Type in: `hdxckantool db pgpass`
-1. Make sure the public schems belongs to the database owner:
+1. Make sure the public schema belongs to the database owner:
     1. Type in: `hdxckantool db schema`
 1. Pull the latest backups:
     1. In a new terminal inside your VSCode type: `hdxckantool db pull -a`
@@ -56,4 +90,6 @@ Enjoy!
     `hdxckantool solr reindex --clear --fast`
 
 ## Sync your S3 filestore
+This moves something like 250GB of files from the production filestore to your developer bucket, taking around 6 hours for an initially empty bucket.
+
 1. Type in: `hdxckantool filestore sync -c`

.devcontainer/awscli-install-run-once.sh

@@ -0,0 +1,5 @@
+set -x
+apt install unzip
+curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
+unzip awscliv2.zip
+./aws/install

.gitattributes

@@ -1,2 +1,6 @@
 #*.po -diff
 *.mo -diff
+
+* text=auto eol=lf
+*.sh eol=lf
+

.gitignore

@@ -62,3 +62,5 @@ travis.env
 
 # cypress
 cypress/videos
+
+src/

.vscode/settings.json

@@ -0,0 +1,9 @@
+{
+    "python.testing.pytestArgs": [
+        "--ckan-ini=hdx-test-core.ini",
+        "ckan/tests",
+        "ckanext-hdx_package/ckanext/hdx_package/tests"
+    ],
+    "python.testing.unittestEnabled": false,
+    "python.testing.pytestEnabled": true
+}

docker-compose.devcontainer.yml

@@ -69,7 +69,7 @@ services:
     #   context: ./
     #   dockerfile: Dockerfile
 
-    mem_limit: 16g
+    mem_limit: 4g
     restart: always
     # volumes:
     # - "./:/srv/ckan"
@@ -99,7 +99,7 @@ services:
       TERM: xterm
     hostname: hdx-local-dbckan
     image: postgis/postgis:11-3.2-alpine
-    mem_limit: 16g
+    mem_limit: 4g
     ports:
       - "0.0.0.0:25432:5432"
     restart: always
@@ -115,7 +115,7 @@ services:
       TERM: xterm
     hostname: hdx-local-redis
     image: public.ecr.aws/unocha/redis:5
-    mem_limit: 16g
+    mem_limit: 4g
     restart: always
     volumes:
     - "redis-data:/var/lib/redis"