Releases: alan-turing-institute/data-safe-haven
v5.0.0
Release v5.0.0
Upgrading
This is a major release and it not compatible with any previous versions.
To use this version you must start a new TRE deployment.
Changes
- Complete rewrite of code in Python using IAC and configuration management tools Pulumi and Ansible
What's Changed
- Release v4.0.1 candidate by @jemrobinson in #1324
- Proof-of-concept migration to Pulumi for deployment by @jemrobinson in #1316
- Release v4.0.2 candidate by @jemrobinson in #1353
- Release v4.0.3 candidate by @jemrobinson in #1365
- Add instructions for installing documentation build dependencies by @JimMadge in #1370
- Update docs with how to resize VMs by @edwardchalstrey1 in #1367
- Update Badges by @JimMadge in #1371
- Update Powershell module requirements by @craddm in #1368
- Allow -UseDeviceAuthentication switch in
Deploy_SHM.ps1
by @craddm in #1378 - Prevent removal of backup data during dry run by @JimMadge in #1383
- Pulumi: Fix user list retrieval by @craddm in #1386
- Policy for software package requests by @jemrobinson in #1387
- Add firewall to Pulumi by @jemrobinson in #1375
- Add
arrow
CRAN package to Tier 3 allowlist by @craddm in #1391 - ⬆️ Update caching in allowlists workflow by @jemrobinson in #1395
- Update user management guide to explain adding users to security group and changing a phone number by @edwardchalstrey1 in #1389
- Add Python type-hinting throughout Pulumi codebase by @jemrobinson in #1390
- Add instructions for GPU VM resizing by @edwardchalstrey1 in #1399
- Simplify Pulumi secret handling by @jemrobinson in #1400
- Add separate docs section GPU VMs and specify NVIDIA required by @edwardchalstrey1 in #1406
- Add Linux update server proxy by @jemrobinson in #1404
- Remove reference to unused System Administrators Security Group by @edwardchalstrey1 in #1407
- Add automated updates to Pulumi by @jemrobinson in #1412
- Refactor SRD creation by @jemrobinson in #1416
- Add SHM bastion by @jemrobinson in #1417
- Fix allowlist generation by @jemrobinson in #1422
- Update SRD image by @jemrobinson in #1421
- Fix incorrect logic around automated PR creation by @jemrobinson in #1426
- Update PyPI and CRAN allow lists by @github-actions in #1425
- Add new servicebus endpoints for self-service password reset by @edwardchalstrey1 in #1423
- Update PyPI and CRAN allow lists by @github-actions in #1428
- Update PyPI and CRAN allow lists by @github-actions in #1429
- Remove egress steps not carried out by System Manager by @edwardchalstrey1 in #1434
- Update SRE user troubleshooting by @edwardchalstrey1 in #1435
- Update SRD package versions by @github-actions in #1433
- Update PyPI and CRAN allow lists by @github-actions in #1437
- Update SRD package versions by @github-actions in #1440
- Add RPostgreSQL to t3 extra cran allowlist by @edwardchalstrey1 in #1441
- Revert "Add RPostgreSQL to t3 extra cran allowlist" by @JimMadge in #1442
- Better package name matching for Nexus by @craddm in #1447
- Update PyPI and CRAN allow lists by @github-actions in #1454
- Update PyPI and CRAN allow lists by @github-actions in #1456
- Update SRD package versions by @github-actions in #1460
- Update VM resizing note to suggest stopping the VM before increasing the quota by @edwardchalstrey1 in #1408
- Add data preparation guidance (including data integrity) by @JimMadge in #1459
- Migrate docs to readthedocs.io by @JimMadge in #1453
- Create users with no password expiry on AD by @craddm in #1461
- Modify location of requirements.txt in Dockerfile by @craddm in #1464
- Merge documentation changes into release branch by @JimMadge in #1468
- cherrypick devcontainer fix to release branch by @JimMadge in #1469
- Update servicebus endpoints used for self-service password reset by @jemrobinson in #1466
- Correct path to Scriberia cartoon in README.md by @JimMadge in #1475
- Replace deprecated Set-AzDiagnosticSetting by @jemrobinson in #1470
- Update PyPI and CRAN allow lists by @github-actions in #1477
- Correct link on citation badge by @JimMadge in #1474
- Add CODEOWNERS for docs by @jemrobinson in #1478
- Update documentation dependencies by @JimMadge in #1476
- Enable pdf and html downloads on readthedocs by @JimMadge in #1462
- Update SRD package versions by @github-actions in #1482
- Updating SSL certificate doc + gitignore change + undo duplication of docs building by @edwardchalstrey1 in #1432
- Mount data and user directories in SRD by @jemrobinson in #1480
- Change servicebus firewall rule by @craddm in #1485
- Folder typo for SHM deployment by @edwardchalstrey1 in #1488
- Update SRD package versions by @github-actions in #1489
- Force az login before reading Pulumi encryption key by @jemrobinson in #1490
- Clarify PR template by @jemrobinson in #1491
- Offline linkcheck by @JimMadge in #1486
- Pulumi: Add Git and Markdown servers by @jemrobinson in #1492
- Fixing the build warnings for documentation by @craddm in #1483
- Add Nexus repositories by @jemrobinson in #1499
- Pin container images by @JimMadge in #1501
- Automate user synchronisation by @jemrobinson in #1500
- Switch CLI interface to Typer by @jemrobinson in #1502
- Refactor config files by @jemrobinson in #1510
- Add portal.azure.com to lychee ignore list by @JimMadge in #1520
- Bump certifi from 2023.5.7 to 2023.7.22 in /docs by @dependabot in https://git...
Release v5.0.0rc2
Release v5.0.0rc2
This release is not ready for production usage.
Known Issues
- ClamAV not configured
- Unstable container service IP addresses
- Lacking Nvidia utils
What's Changed
- Use pip-compile for package resolution by @jemrobinson in #1514
- Add pip-tools to NON_IMPORTABLE_PACKAGES by @edwardchalstrey1 in #1537
- Add May 2023 DSG to versioning by @jemrobinson in #1545
- Release v4.1.0 cloud init changes by @edwardchalstrey1 in #1548
- Update SRD package versions by @github-actions in #1578
- Update PyPI and CRAN allow lists by @github-actions in #1579
- Fix deployment issues with MSSQL and PyPi mirrors by @craddm in #1582
- Update PyPI and CRAN allow lists by @github-actions in #1588
- Update SRD package versions by @github-actions in #1587
- Updates for Release v4.1.0 by @craddm in #1590
- Release v4.1.0 by @craddm in #1586
- Remove CoCalc by @craddm in #1554
- Merge 'latest' into 'develop' by @craddm in #1593
- Add script to automate account deletion by @edwardchalstrey1 in #1508
- Add @craddm to CODEOWNERS by @jemrobinson in #1594
- Update PyPI and CRAN allow lists by @github-actions in #1595
- Remove pulumi testing files from develop branch by @craddm in #1597
- Update PyPI and CRAN allow lists by @github-actions in #1601
- Update SRD package versions by @github-actions in #1616
- Update SRD package versions by @github-actions in #1622
- Bump urllib3 from 2.0.2 to 2.0.6 in /docs by @dependabot in #1625
- Improve Pulumi error messages by @craddm in #1624
- Update PyPI and CRAN allow lists by @github-actions in #1627
- Update PyPI and CRAN allow lists by @github-actions in #1631
- Update SRD package versions by @github-actions in #1630
- Improve Python documentation by @jemrobinson in #1635
- Use Pulumi random provider by @jemrobinson in #1629
- Pulumi: Fix selectors not updating by @JimMadge in #1621
- Bump urllib3 from 2.0.6 to 2.0.7 in /docs by @dependabot in #1647
- Remove hyphens from SHM and SRE names by @craddm in #1650
- Update PyPI and CRAN allow lists by @github-actions in #1646
- Update SRD package versions by @github-actions in #1652
- Pulumi: Improve login flow by @JimMadge in #1617
- Update PyPI and CRAN allow lists by @github-actions in #1654
- Add all contributors table and instructions for how to update by @edwardchalstrey1 in #1649
- Update PyPI and CRAN allow lists by @github-actions in #1656
- Update PyPI and CRAN allow lists by @github-actions in #1668
- Update SRD package versions by @github-actions in #1669
- Update devcontainer configuration by @craddm in #1662
- Update outdated parameters that cause breaking change warnings by @craddm in #1663
- Change default lun from lun1 to lun0 by @craddm in #1667
- Add context command by @JimMadge in #1655
- Pulumi: Update dependencies, enable pinning by @JimMadge in #1660
- Remove unneeded opening bracket in SRE network configuration script by @craddm in #1670
- Update PyPI and CRAN allow lists by @github-actions in #1671
- Use memory for the /tmp directory by @craddm in #1672
- Factor out storage creation from SHM scripts by @craddm in #1673
- Add missing import for logging module by @JimMadge in #1681
- Update PyPI and CRAN allow lists by @github-actions in #1682
- Update help text for Powershell command
shmId
andsreId
arguments by @craddm in #1683 - Update contributors by @JimMadge in #1684
- Document removal of persistent SRE storage accounts by @craddm in #1685
- docs: update @helendduncan as a contributor by @JimMadge in #1686
- Update PyPI and CRAN allow lists by @github-actions in #1688
- Update SRD package versions by @github-actions in #1692
- Update PyPI and CRAN allow lists by @github-actions in #1693
- Update PyPI and CRAN allow lists by @github-actions in #1694
- Update DBeaver drivers using Github workflow by @craddm in #1696
- Update SRD package versions by @github-actions in #1698
- Bump jinja2 from 3.1.2 to 3.1.3 in /docs by @dependabot in #1700
- Update SRD package versions by @github-actions in #1701
- Update PyPI and CRAN allow lists by @github-actions in #1702
- Update PyPI and CRAN allow lists by @github-actions in #1703
- Handle no selected context by @JimMadge in #1691
- Add basic config commands by @JimMadge in #1674
- Fixing DBeaver driver issues on T2+ SREs by @craddm in #1704
- Use Pydantic for validation and serialisation by @JimMadge in #1661
- Improve handling of spaces in file paths by @craddm in #1705
- Update PyPI and CRAN allow lists by @github-actions in #1706
- Create pulumi container by @jemrobinson in #1711
- Fix private link scope by @jemrobinson in #1713
- Improve handling of SRE names by @JimMadge in #1699
- Apply changes from updated black version by @jemrobinson in #1718
- Bump black version by @JimMadge in #1719
- Fix some issues with context handling at deployment time by @jemrobinson in #1716
- Update SRD package versions by @github-actions in #1723
- Correct file path for clamonacc service by @craddm in #1725
- Add additional multiple data provider guidance to docs by @craddm in #1707
- Update SRD package versions by @github-actions in #1727
- Fix Pos...
Release 4.2.2 (2024-07-15)
⚠️ Update Requires Manual Intervention ⚠️
If you are using a 4.2.x
SHM and want to upgrade to 4.2.2
, please follow the steps below:
For the SHM:
- Add a
docker
section to your SHM config with a username and personal access token (following the SHM deployment instructions) - Re-run
Setup_SHM_Networking.ps1 -shmId {shm}
fromdeployment/safe_haven_management/setup
For any SRE that you deployed using an earlier 4.2.x
version:
- Delete the
GUACAMOLE-SRE-{sreId}
VM and associated resources from the
RG_SHM_{shmId}_SRE_{sreId}_REMOTE_DESKTOP
resource group - Re-run the deployment script
Deploy_SRE.ps1 -shmId {shm} -sreId {sre} -VmSizes {as before}
fromdeployment/secure_research_environment/setup
Known issues
- As for 4.2.0, 4.2.1
Bug Fixes
- Workaround for an issue where Let's Encrypt refused to provide certificates for uppercase FQDNs #1938
- Fix for change in Azure supported public IP address SKU for VPNs, which prevented deployment of the virtual network gateway for accessing domain controllers #1947
- Require supply of Docker Hub credentials to work round change in Docker download rate limits #1994
- Update approved IP address list for Ubuntu apt repositories
- Update to backup policy rules for Blob storage #1988
Full Changelog: v4.2.1...v4.2.2
Release v4.2.1 (2024-05-31)
⚠️ Update Requires Manual Intervention ⚠️
If you are using a 4.2.0
SHM and want to upgrade to 4.2.1
, please follow the steps below:
- Delete the
GUACAMOLE-SRE-{sreId}
VM and associated resources from theRG_SHM_{shmId}_SRE_{sreId}_REMOTE_DESKTOP
resource group - Re-run the deployment script
Deploy_SRE.ps1 -shmId {shm} -sreId {sre} -VmSizes {as before}
fromdeployment/secure_research_environment/setup
Known issues
- As for 4.2.0
Bug Fixes
- Update Guacamole to 1.5.5 to avoid this known bug
Full Changelog: v4.2.0...v4.2.1
Release 4.2.0 (2024-03-28)
⚠️ Update Requires Manual Intervention ⚠️
If you are using a 4.1.0
SHM and want to upgrade to 4.2.0
, please follow the steps below:
- Run
Setup_SHM_Firewall.ps1 -shmId {shmid}
- Run
Setup_SHM_Networking.ps1 -shmId {shmid}
- Delete
LINUX-UPDATES-SHM-{shmid}
VM and associated resources from theRG_SHM_{shmid}_MONITORING
resource group - Delete
RG_SHM_{shmid}_PACKAGE_REPOSITORIES
resource group and all resources - Run
Setup_SHM_Update_Servers.ps1 -shmId {shmid}
(Note that this needs to happen before any further resources are deployed, since any further Linux resources will need access to the Linux update proxy). - Run
Setup_SHM_Package_Repositories -shmId {shmid}
- Run
Setup_SHM_Monitoring.ps1 -shmId {shmid}
Known issues
- Jupyter notebook launched from GUI menu could not launch Python kernel, so it has been removed from the menu 0657647
New Features
- Remove Microsoft Remote Desktop support: #1535
- Remove CoCalc: #1554
- Install dev dependencies in container: #1747
- Add script to renew NFS share Stored Access Policies: #1739
- Add script to automate account deletion: #1508
- Factored out storage creation from SHM scripts #1673
- SRD image updated, with latest Python versions available f3e890a
Bug Fixes
- Update DBeaver drivers using Github workflow: #1696
- Fixing DBeaver driver issues on T2+ SREs: #1704
- Improve handling of spaces in file paths: #1705
- Correct file path for Clam OnAccess scanning service: #1725
- Fix PostgreSQL permissions and data schema, and relevant docs: #1708
- Update outdated parameters that cause breaking change warnings: #1663
- Change default lun from lun1 to lun0: #1667
- Increase apt proxy server disk to 64 Gb: #1726
- Remove
omsagent
from VM build image: #1732 - Remove hyphens from SHM and SRE names in #1650
- Update devcontainer configuration in #1662
- Use memory for the /tmp directory in #1672
- Remove unneeded opening bracket in SRE network configuration script #1670
- Add missing import for logging module #1681
- Fix
cloud-init
log parser using old name for event 58a85bc - Detect and remove
omsagent
installed on SRD image before generalization e168b05
Security Fixes
- Update software on Guacamole and Nginx to latest versions: #1741
- Update Nexus proxy server for T2/T3 package access: in #1744
- Update CodiMD server version: #1743
- Improve hardcoded domains and IP addresses: #1745
- Prevent Nginx version information from appearing in http headers
Documentation updates
- Add guidance on resizing NFS shares: #1749
- Update documents to reflect change to Microsoft Entra ID: #1665
- Update deprecation warning for MS RDS: #1542
- Add explanation of how to change allowed inbound IP addresses: #1484
- Add all contributors table and instructions for how to update: #1649
- Update contributors: #1684
- Document removal of persistent SRE storage accounts: #1685
- docs: update contributors: #1686
- Add additional multiple data provider guidance to docs: #1707
- Add links to guides for terminal, Xfce, and Guacamole: #1737
- Update help text for Powershell command
shmId
andsreId
arguments #1683
Full Changelog: v4.1.0...v4.2.0
Release v5.0.0-rc.1 (2023-09-27)
First version of migration to Python using Pulumi. Penetration tested in September 2023.
Known Issues
This release is not ready for production usage.
Release 4.1.0 (2023-09-06)
⚠️ Update Requires Manual Intervention ⚠️
If you are using a 4.X.Y
SHM and want to upgrade to 4.1.0
, please follow the steps below:
- Run
./deployment/safe_haven_management/setup/Setup_SHM_Networking.ps1 -shmId <your SHM ID>
- Restart the virtual machine at
RG_SHM_<SHM name>_MONITORING/LINUX-UPDATES-SHM-<SHM name>
in the Azure portal
Known Issues
Only phone call authentication works for MS RDS. This provides no on-screen MFA Prompt.
New Features
- Allow device authentication in SHM deployment #1378
- Add
arrow
CRAN package to Tier 3 core list #1391 - Update Python in SRD images #1421
Bug Fixes
- Update Powershell module requirements: #1368
- Update supported Powershell version to
7.3.6
- Prevent removal of backup data during dry run: #1383
- Better package name matching for Nexus: #1447
- Update SRD image: #1421
- Add new servicebus endpoints for self-service password reset: #1423 and #1466
- Modify location of requirements.txt in Dockerfile: #1469
- Fixes of the SRD build related to python packages: #1514 and #1537
- Fix allowlist generation: #1422
- Update badges: #1371
- Update caching in allowlists workflow: #1395
- Fix incorrect logic around automated PR creation: #1426
- Update Ubuntu apt server addresses #1548
- Add docker.io to allowed-FQDNs #1548
- Change cloud-init files to automatically select appropriate disk partition #1548
- Fix MS-SQL database deployment #1580
- Fix PyPi Tier 3 mirror failures #1581
Security Fixes
- Fix non-allowed CRAN packages beginning with allowed name being installable: #1447
- Update to firewall rules: #1519
Documentation Updates
- Add instructions for installing documentation build dependencies: #1370
- Add instructions to resize VMs: #1367
- Update user management guide to explain adding users to security group and changing a phone number: #1389
- Add instructions for GPU VM resizing: #1399
- Add note on NVIDIA GPU support: #1406
- Remove reference to unused System Administrators Security Group: #1407
- Remove egress steps not carried out by System Manager: #1434
- Update SRE user troubleshooting: #1435
- Move from GitHub pages to ReadTheDocs #1468
- Add Policy for software package requests: #1387
- Add deprecation warning for MSRDS #1542
- Add warning that MSRDS does not work with the Microsoft Authentication app. #1589
- Add step for adding SSL certificate in step-by-step instructions for Guacamole #1590
Full Changelog: v4.0.3...release-v4.1.0
Release 4.0.3 (2023-01-27)
Bug fixes
- Update maximum allowed Powershell version
- Fix disk mounting issue when upgrading SRDs
Documentation updates
- Minor fixes
Release 4.0.2 (2023-01-05)
Bug fixes
- Add missing Powershell module imports
- Fix
-Upgrade
option when adding new SRD - Fix
tensorflow
installation in SRD base image - Register
Microsoft.DataProtection
on subscriptions that an SRE will be deployed into - Support cross-subscription role assignments for backup
- Switch to correct subscription before deploying update automation
- Update Powershell version requirements to avoid upstream bug
- Update SRD package versions
- Use process-scope when retrieving Graph authorization tokens with Connect-MgGraph
Security fixes
- Remove unnecessary information from deployment logging
Documentation updates
- Add link to teardown docs to deployment page
- Add a VSCode
.devcontainer
for use in deployment - Clarify that IP addresses are required in SRE config file
- Consolidate MFA setup description
- Update documentation build triggers to also run on
latest
Release 4.0.1 (2022-10-24)
Bug fixes
- Add additional modules to requirements checker
- Add check for non-existing AzureAD security group
- Switch CI tests from Travis to GitHub Actions
Documentation updates
- Updated issue templates
- Fix documentation building