Cloud-Harness v2.3.1

Hotfix release -- fixes typer dependency issue that prevents harness-deployment to run

What's Changed

• Release/2.3.0 to dev by @filippomc in #739
• Feature/ch 125 remove z2jh by @vgdmetacell in #743
• Update README.md by @afonsobspinto in #751
• CH-131 fix: Only include base images build on codefresh configuration… by @afonsobspinto in #753
• Hotfix release 2.3.1 by @filippomc in #755

New Contributors

• @vgdmetacell made their first contribution in #743

Full Changelog: v2.3.0...v2.3.1

Cloud-Harness v2.3.0

This new release introduces a few small features and bug fixes

• Ingress setting to forward headers (now true by default)
• Open Sentry init parameters in Python
• Update PR template to match the new workflow based on Jira
• Fix stage pipeline

See also https://metacell.atlassian.net/projects/CH/versions/10156/tab/release-report-all-issues

What's Changed

• Release/2.2.0 by @filippomc in #713
• Update cloud-harness-wsl2-setup.md by @afonsobspinto in #715
• Update Python dependencies by @filippomc in #714
• Add dev specific doc for CloudHarness CLI tools by @aranega in #711
• Add tag / build reference api by @filippomc in #718
• Allow to clone an external repo before the build by @filippomc in #720
• Bump fastapi from 0.77.1 to 0.109.1 in /infrastructure/common-images/cloudharness-django/libraries/fastapi by @dependabot in #723
• Bump fastapi from 0.77.1 to 0.109.1 in /infrastructure/common-images/cloudharness-fastapi/libraries/fastapi by @dependabot in #722
• Bump cryptography from 41.0.7 to 42.0.0 in /libraries/cloudharness-common by @dependabot in #724
• Bump axios from 0.21.4 to 1.6.0 in /applications/samples/frontend by @dependabot in #729
• Bump fonttools from 4.38.0 to 4.43.0 in /libraries/models by @dependabot in #730
• CH-123 add parameters to Sentry init by @filippomc in #736
• CH-111 chore: update pull request template by @filippomc in #726
• CH-124 add forwarded headers configuration by @filippomc in #738
• Git clone functionality fixes by @filippomc in #721
• CH-122 fix stage template by @filippomc in #733
• Release/2.3.0 by @filippomc in #740

Full Changelog: v2.2.0...v2.3.0

Cloud-Harness v2.2.0

This new release on CloudHarness introduces a few small features, fixes some bugs and improved documentation.

New features

• Configure proxy timeouts and payload size #701

What's Changed

• Doc/tutorial update by @D-GopalKrishna in #687
• Release/2.1.0 back to develop by @filippomc in #669
• Configure proxy timeouts and payload size by @filippomc in #702
• Fix and improve api tests by @filippomc in #704
• Feature/707 by @filippomc in #708
• Feature/ch 87 by @aranega in #710
• Release/2.2.0 by @filippomc in #712

New Contributors

• @D-GopalKrishna made their first contribution in #687

Full Changelog: v2.1.0...v2.2.0

Cloud-Harness v2.1.0

Summary

This new release on CloudHarness introduces a few new features and fixes some bugs.

Main new features

• User quotas mechanism #637 #638 #690
• New events ui

See also https://github.com/MetaCell/cloud-harness/milestone/7?closed=1 to see the scope of the release

What's Changed

• Feature/498 by @aranega in #581
• chore(deps): bump oauthlib from 3.2.0 to 3.2.1 in /infrastructure/common-images/cloudharness-flask by @dependabot in #580
• #625 Enhance events ui by @filippomc in #627
• #630 feat: Add workflow name to on exit event by @afonsobspinto in #631
• #632 feat: add switch to retrieve group attributes by @zsinnema in #633
• #628 justMyCode added by @filippomc in #634
• #635 feat: add user defined exception for use for general pod spawn errors by @zsinnema in #636
• Feature/638 by @zsinnema in #642
• #637 feat: implement harness quota by @zsinnema in #641
• Feature/652 by @zsinnema in #653
• Release/1.0.2 by @zsinnema in #646
• Release/2.1.0 by @filippomc in #656
• Feature/598 by @zsinnema in #609
• #657 chore: add support for auto pulling CloudHarness tasks onto JupyterHub nodes with the JH prepuller by @zsinnema in #658
• Feature/639 by @filippomc in #661
• #666 change issuer certificate name by @filippomc in #667
• Feature/592 Avoid doing the same repeated build in the dev/test pipelines by @filippomc in #680
• #688 add django user service function by @filippomc in #689
• Release/2.1.0 by @filippomc in #698

Full Changelog: v2.0.0...v2.1.0

Cloud-Harness v1.0.3

Summary

Add support to jupyterhub profiles

Upgrade instructions

Nothing to report.

Addressed issues:

• #645

What's Changed

• feat: support setting the JH image through profiles using the ws image attribute
• Release/0.4.0 by @filippomc in #366
• Release/1.0.0 by @filippomc in #434
• Release/1.0.1 by @zsinnema in #521

Full Changelog: v1.0.0...v1.0.3

Cloud-Harness v2.0.0

What's Changed

• Added testing framework
• Added Django + FastAPI application templates
• Accounts event listener
• Test coverage improved
• Update to Python 3.9
• New Typing/model layer
• Bug fixes

Detailed log:

• Feature/474 by @zsinnema in #475
• Release back to develop by @filippomc in #467
• Python 3.9 upgrade by @ddelpiano in #402
• Feature/483 by @filippomc in #486
• #418 Force restart of gatekeeper on conf change by @filippomc in #484
• #451: model and types by @filippomc in #454
• #444 implement multiple environments functionality by @filippomc in #487
• Bump flask-cors from 3.0.8 to 3.0.9 in /infrastructure/common-images/cloudharness-flask by @dependabot in #482
• Feature/493 by @zsinnema in #494
• Fix samples application by @filippomc in #488
• #499 Fix codefresh templates overriding by @filippomc in #500
• 248/test sequencer by @SimaoBolota-MetaCell in #497
• #513 chore: Add openapi generator requirements overrides by @afonsobspinto in #515
• #513 chore: Optimize imports by @afonsobspinto in #522
• Release/1.0.1 by @filippomc in #523
• #517 Implement workflows filter by @filippomc in #518
• #510 remove double volume type spec by @filippomc in #511
• #508 update k8s spec for cron jobs by @filippomc in #509
• Workflows and volumes enhancements by @filippomc in #507
• #525 fixes None type error when setting an attribute on an user without attributes by @zsinnema in #526
• Update README.md by @afonsobspinto in #531
• Feature/529: FastApi + Django application templates by @zsinnema in #530
• #533 feature: keycloak / accounts add admin interface kafka publisher by @zsinnema in #534
• Feature/532 Django app configuration to enable keycloak gatekeeper integration by @zsinnema in #535
• Update README.md by @afonsobspinto in #539
• Feature/538 by @zsinnema in #540
• #544 chore: moved Django and FastAPI images from base to common by @zsinnema in #545
• CloudHarness Django & FastAPI images fixes by @zsinnema in #547
• #548 dynamic generation of codefresh prepare cmd by @filippomc in #549
• Feature/248 Test framework and more by @filippomc in #503
• #503 merge fix by @filippomc in #550
• #548 fix .overrides directory being included in cf by @filippomc in #551
• #553 +Default allow option on gk by @filippomc in #554
• #562 fix: create a Member record for new KC users by @zsinnema in #565
• #563 fix: crashing openapi backend generator for route based params by @zsinnema in #566
• #561 fix: add CSRF_TRUSTED_ORIGINS to fix CSRF admin issue by @zsinnema in #564
• Feature/516 by @afonsobspinto in #568
• Add tuto about a simple clock/date webapp creation by @aranega in #557
• #571 fix: added get_dependencies() call to generator methods to check & get the dependencies for openapi by @zsinnema in #572
• #573 default gk http-only-cookie false by @filippomc in #574
• #575 fix codefresh template by @filippomc in #576
• #577 fix codefresh issue with _: replace _ with __ by @filippomc in #578
• Feature/587 by @afonsobspinto in #588
• Feature/584 by @afonsobspinto in #585
• #589 fix: Fix main.jinja2 template by @afonsobspinto in #590
• Feature/599 by @filippomc in #606
• Feature/373 by @filippomc in #607
• Release/2.0.0 by @filippomc in #603

New Contributors

• @SimaoBolota-MetaCell made their first contribution in #497

Full Changelog: v1.0.1...v2.0.0

CloudHarness v1.0.1

What's Changed

• Fix to properly support jupyterhub communication to the accounts microservice

Full Changelog: v1.0.0...v1.0.1

CloudHarness v1.0.0

Version 1.0.0 includes general stability improvements and upgrades over the previous version.

Main changes

• Upgrade Kubernetes specifications to v1 #342
• Argo, JupyterHub (#342), Keycloak (#362), upgrade
• Gatekeeper upgraded #130
• Fix Keycloak user for backend api calls #361 #395
• Build dependencies resolution improved (images that are not declared as build dependencies are not included) #346
• Node affinity selectors per for JupyterHub applications #327
• Efficiency improvements #422
• Bug fixes #265 #216 #423 #222 #347 #380 #394

See also https://github.com/MetaCell/cloud-harness/milestone/5

Upgrading from previous version

No major api update is implemented from 0.4.0.

The upgrade can break the deployments for the following reasons:

• Kubernetes definitions upgrade to v1. Kubernetes versions < 1.19 are not supported anymore.
• Keycloak version upgraded to 15.1.1. Deployments with old Keycloak versions may break in a hot upgrade. See also https://www.keycloak.org/docs/latest/upgrading/
• Applications using the user accounts client api is now required to add accounts as a hard dependency
• JupyterHub helm chart upgraded. Some overridden configurations may need changes
• Argo helm chart changed. Some overridden configurations may need changes
• Default databases version changes from latest to a specific version. May need to set the desired version globally or specifically per application
• Directory utilities renamed to tools. Codefresh pipelines may need to change the install command. Change the deployment-configuration/codefresh-ENV.yaml to replace utilities with tools and regenerate the deployment with harness-deployment ... -e ENV

What's Changed

• #251 chore: add CH_CURRENT_USER_ID as an env var to set the current user during development by @zsinnema in #252
• #257 add simple command to tasks and specific task by @filippomc in #258
• #261 chore: add flask 404 handler by @zsinnema in #262
• #271 chore: refactored extract-download task by @zsinnema in #272
• New image names #264 #267 by @filippomc in #270
• Fix build when no registry is set by @filippomc in #273
• #274 fix: added -L to curl to follow redirects by @zsinnema in #275
• #279 add proxy services for main app by @filippomc in #280
• #282 chore: added startupProbe to deployment template by @zsinnema in #283
• #240 chore: implemented secrets through mounting instead of env vars and make sure secret values aren't in the allvalues.yaml by @zsinnema in #284
• Feature/287: Fix image dependencies using Skaffold by @filippomc in #289
• chore: add performance trace sample rate to sentry init by @zsinnema in #292
• Feature/293 cache current user by @zsinnema in #294
• #293 chore: use cachetools for caching by @zsinnema in #295
• #292 hotfix: added missing admin client to user_realm_roles by @zsinnema in #296
• #297 chore: add create topic to produce when kafkatimeout error occurs & refrased some error messages by @zsinnema in #298
• Feature/skaffold fixes by @filippomc in #308
• #305 #306 add test coverage by @filippomc in #309
• #304 chore: enable docker buildkit in skaffold yaml by @zsinnema in #307
• #310 expandable deployment pipelines implemented by @filippomc in #311
• #313 chore: enable kafka topic auto creation by @zsinnema in #314
• #315 chore: removed one stacktrace dump from async consume by @zsinnema in #316
• #317 chore: remove subpart from the configmap mount by @zsinnema in #318
• #300 #301 chore: added notifications server and send_event decorator by @zsinnema in #302
• #322 Generic backend api error handling improved by @filippomc in #323
• #322 Parameter name corrected by @filippomc in #324
• Hotfix/325 by @zsinnema in #326
• Feature/328: Jupyterhub enable cpu/mem resources per application by @zsinnema in #329
• #330 chore: removed all refs to mnp from jupyterhub keycloak authentication and made it generic by @zsinnema in #331
• #333 support for multi-level domains and aliases by @filippomc in #334
• #335 #336 Volume affinity and no cache parameter by @filippomc in #337
• fix: update collection related imports by @sanjayankur31 in #341
• Feature/338: enable buildkit for codefresh by @zsinnema in #340
• #356 ssh:default support for docker buildkit builds to support cloning of private git repos by @zsinnema in #357
• Release/0.4.0 by @filippomc in #367
• Fix databases versions by @filippomc in #351
• Feature/361: stop using the admin account for api calls to KC by @zsinnema in #364
• Feature/342: Update deprecated Kubernetes chart templates by @filippomc in #345
• #347 add test coverage for values.yaml precedence by @filippomc in #377
• #263 chore fix -e parameter description by @filippomc in #376
• #380 chore: fixed issue with wrong replace of accounts subdomain in CH_ACCOUNTS_AUTH_DOMAIN by @zsinnema in #381
• #361 chore: use CH secrets for api acocunt secret, use a random string for nullable secrets default value by @zsinnema in #389
• #387 chore: add auto secrets to codefresh.yaml only if the secret has a value defined in the values.yaml by @zsinnema in #393
• #390 chore: reset the k8s resource values by @zsinnema in #391
• #395 chore: set accounts admin api username to admin_api by @zsinnema in #396
• #394 fix common images dependencies by @filippomc in #399
• #397 chore: accounts secrets not created by @zsinnema in #398
• Feature/216-384 by @filippomc in #385
• Add documentation by @filippomc in #409
• Feature/346 Improve build dependencies: remove unused base images by @filippomc in #400
• Feature/327: implement jupyter node selectors by @zsinnema in #332
• #404 Pull request template by @filippomc in #405
• #365 Add validation checks for dependencies by @filippomc in #378
• #415 rename utilities->tools by @filippomc in #416
• Feature/344: gatekeeper improvements by @filippomc in #379
• #401 implement ch middleware for Flask and Django to set/get the ch application state. Refactored ch kc authentication to disconnect it from Flask by @zsinnema in #417
• #425 #422 Clean, Improve and document operations api by @filippomc in #426
• #423 change the argo executor by @filippomc in #424
• #384 fix: mount secret also on accounts pod by @filippomc in #428
• #423 change argo executor to pns by @filippomc in #429
• Fix/384: load accounts secret only at need by @filippomc in #430

New Contributors

• @sanjayankur31 made their first contribution in #341

Full Changelog: v0.4.0...v1.0.0-RC

CloudHarness v1.0.0 Release Candidate

Version 1.0.0 includes general stability improvements and upgrades over the previous version.

Main changes

• Upgrade Kubernetes specifications to v1 #342
• Argo, JupyterHub (#342), Keycloak (#362), upgrade
• Gatekeeper upgraded #130
• Fix Keycloak user for backend api calls #361 #395
• Build dependencies resolution improved (images that are not declared as build dependencies are not included) #346
• Node affinity selectors per for JupyterHub applications #327
• Efficiency improvements #422
• Bug fixes #265 #216 #423 #222 #347 #380 #394

See also https://github.com/MetaCell/cloud-harness/milestone/5

Upgrading from previous version

No major api update is implemented from 0.4.0.

The upgrade can break the deployments for the following reasons:

• Kubernetes definitions upgrade to v1. Kubernetes versions < 1.19 are not supported anymore.
• Keycloak version upgraded to 15.1.1. Deployments with old Keycloak versions may break in a hot upgrade. See also https://www.keycloak.org/docs/latest/upgrading/
• Applications using the user accounts client api is now required to add accounts as a hard dependency
• JupyterHub helm chart upgraded. Some overridden configurations may need changes
• Argo helm chart changed. Some overridden configurations may need changes
• Default databases version changes from latest to a specific version. May need to set the desired version globally or specifically per application
• Directory utilities renamed to tools. Codefresh pipelines may need to change the install command. Change the deployment-configuration/codefresh-ENV.yaml to replace utilities with tools and regenerate the deployment with harness-deployment ... -e ENV

What's Changed

• #251 chore: add CH_CURRENT_USER_ID as an env var to set the current user during development by @zsinnema in #252
• #257 add simple command to tasks and specific task by @filippomc in #258
• #261 chore: add flask 404 handler by @zsinnema in #262
• #271 chore: refactored extract-download task by @zsinnema in #272
• New image names #264 #267 by @filippomc in #270
• Fix build when no registry is set by @filippomc in #273
• #274 fix: added -L to curl to follow redirects by @zsinnema in #275
• #279 add proxy services for main app by @filippomc in #280
• #282 chore: added startupProbe to deployment template by @zsinnema in #283
• #240 chore: implemented secrets through mounting instead of env vars and make sure secret values aren't in the allvalues.yaml by @zsinnema in #284
• Feature/287: Fix image dependencies using Skaffold by @filippomc in #289
• chore: add performance trace sample rate to sentry init by @zsinnema in #292
• Feature/293 cache current user by @zsinnema in #294
• #293 chore: use cachetools for caching by @zsinnema in #295
• #292 hotfix: added missing admin client to user_realm_roles by @zsinnema in #296
• #297 chore: add create topic to produce when kafkatimeout error occurs & refrased some error messages by @zsinnema in #298
• Feature/skaffold fixes by @filippomc in #308
• #305 #306 add test coverage by @filippomc in #309
• #304 chore: enable docker buildkit in skaffold yaml by @zsinnema in #307
• #310 expandable deployment pipelines implemented by @filippomc in #311
• #313 chore: enable kafka topic auto creation by @zsinnema in #314
• #315 chore: removed one stacktrace dump from async consume by @zsinnema in #316
• #317 chore: remove subpart from the configmap mount by @zsinnema in #318
• #300 #301 chore: added notifications server and send_event decorator by @zsinnema in #302
• #322 Generic backend api error handling improved by @filippomc in #323
• #322 Parameter name corrected by @filippomc in #324
• Hotfix/325 by @zsinnema in #326
• Feature/328: Jupyterhub enable cpu/mem resources per application by @zsinnema in #329
• #330 chore: removed all refs to mnp from jupyterhub keycloak authentication and made it generic by @zsinnema in #331
• #333 support for multi-level domains and aliases by @filippomc in #334
• #335 #336 Volume affinity and no cache parameter by @filippomc in #337
• fix: update collection related imports by @sanjayankur31 in #341
• Feature/338: enable buildkit for codefresh by @zsinnema in #340
• #356 ssh:default support for docker buildkit builds to support cloning of private git repos by @zsinnema in #357
• Release/0.4.0 by @filippomc in #367
• Fix databases versions by @filippomc in #351
• Feature/361: stop using the admin account for api calls to KC by @zsinnema in #364
• Feature/342: Update deprecated Kubernetes chart templates by @filippomc in #345
• #347 add test coverage for values.yaml precedence by @filippomc in #377
• #263 chore fix -e parameter description by @filippomc in #376
• #380 chore: fixed issue with wrong replace of accounts subdomain in CH_ACCOUNTS_AUTH_DOMAIN by @zsinnema in #381
• #361 chore: use CH secrets for api acocunt secret, use a random string for nullable secrets default value by @zsinnema in #389
• #387 chore: add auto secrets to codefresh.yaml only if the secret has a value defined in the values.yaml by @zsinnema in #393
• #390 chore: reset the k8s resource values by @zsinnema in #391
• #395 chore: set accounts admin api username to admin_api by @zsinnema in #396
• #394 fix common images dependencies by @filippomc in #399
• #397 chore: accounts secrets not created by @zsinnema in #398
• Feature/216-384 by @filippomc in #385
• Add documentation by @filippomc in #409
• Feature/346 Improve build dependencies: remove unused base images by @filippomc in #400
• Feature/327: implement jupyter node selectors by @zsinnema in #332
• #404 Pull request template by @filippomc in #405
• #365 Add validation checks for dependencies by @filippomc in #378
• #415 rename utilities->tools by @filippomc in #416
• Feature/344: gatekeeper improvements by @filippomc in #379
• #401 implement ch middleware for Flask and Django to set/get the ch application state. Refactored ch kc authentication to disconnect it from Flask by @zsinnema in #417
• #425 #422 Clean, Improve and document operations api by @filippomc in #426
• #423 change the argo executor by @filippomc in #424
• #384 fix: mount secret also on accounts pod by @filippomc in #428
• #423 change argo executor to pns by @filippomc in #429
• Fix/384: load accounts secret only at need by @filippomc in #430

New Contributors

• @sanjayankur31 made their first contribution in #341

Full Changelog: v0.4.0...v1.0.0-RC

Cloud-Harness v0.4.0

• Runtime Python library improvement and bugfixes #325 #315
  • Implement event decorator for sending objects into a kafka queue for CDC purpose #301
  • Enable kafka auto create topics #313
  • Improve default exception handling in Flask backend apis #322
  • Optimize auth current user retrieval (caching) #293
  • Add a 404 handler for webapps #261
  • Use CH_CURRENT_USER_ID environment variable for development env where no "request user" is present #251
  • Add simple command workflow task #257
• Helm chart auto deployment improvements
  • Auto refresh configmaps in pods #317
  • Support for multi-level domains and alias subdomains #333
  • Create support (proxy) for calling services on the current service url #149
  • Add startupProbe to auto deployment template and add failureTreshold #282
  • Handle secrets #240
• Jupyterhub application improvements
  • enable cpu/mem resources per application #328
  • Add user volume affinity for jupyterhub #335
• Improve extract download task #271 #274
• Create a notification microservice for sending different types of notifications (email, slack, ....) #300
• Build and deployment improvements
  • Add argument to selectively disable cache on image builds#336
  • Enable docker buildkit in skaffold #304
  • Fix task images not included in skaffold build without specifying parameter -i in harness-deployment #306
  • Fix build dependencies not working with Skaffold #287
  • Fix workflows images retrieval to work with Skaffold build #267
  • Add context to image tags in pipelines. #264
  • Use buildkit on codefresh builds #338
• Add Sentry performance monitoring option #291
• Improve deployment script
  • Improve multiple deployment pipeline script generation #310
• Database backups #269

See also https://github.com/MetaCell/cloud-harness/milestone/4?closed=1