Dictionary 1.18 merge to master

[hknahal]

• Related to Proposed solution for making a clinical exception on the "prescribed_cumulative_drug_dose" and "actual_cumulative_drug_dose"  #409 and Update validation scripts to take into account exception values (Unknown and Not applicable) #407
• Update validation scripts to take into account exception values (Unknown and Not applicable)
• solution for making a clinical exception on the "prescribed_cumulative_drug_dose" and "actual_cumulative_drug_dose"

Test in QA/Staging

[hknahal]

Testing in QA:

Tested chemotherapy_drug_dose_units exception using HNAHAL-CA

Test case 1: Missing dose when chemotherapy_drug_dose_units has been submitted with an enum value from the dictionary (ie. mg/m2).
If chemotherapy_drug_dose_units is an enum value from the dictionary (ie.chemotherapy_drug_dose_units = mg/m2) and prescribed_cumulative_drug_dose and actual_cumulative_drug_dose fields are left empty, the validation should fail because it's expected the dose be submitted if the dose units have been submitted.

Sample submission files:
sample_registration .csv
donor.csv
treatment .csv
chemotherapy_test_case1 .csv

Result: Validation fails as expected

Test case 2: Dose field(s) are submitted when there is a clinical exception for chemotherapy_drug_dose_units = Not applicable. If the chemotherapy_drug_dose_units field has a clinical exception, and both/either the prescribed_cumulative_drug_dose and actual_cumulative_drug_dose fields are submitted, the validation should fail.

Sample submission files:
chemotherapy_drug_dose_units_exception.csv
donor .csv
treatment_chemo .csv
chemotherapy_test_case2 .csv

Result: Validation fails as expected

Test case 3: Validation should pass if prescribed_cumulative_drug_dose and actual_cumulative_drug_dose are empty when chemotherapy_drug_dose_units has a clinical exception value of Not applicable.

Sample submission files:
donor .csv
treatment_chemo .csv
chemotherapy_test_case3 .csv

Result: Validation passes as expected (row 3)

[hknahal]

Testing in QA:

Tested validation script changes related to lymph_nodes_examined_method field in QA using HNAHAL-CA:

Test case 1: Validation should fail when lymph_nodes_examined_status has a clinical exception of Unknown and lymph_nodes_examined_method is submitted (is status of whether lymph nodes were examined or not is unknown, then the method must be unknown as well)

Sample submission files:
lymph_nodes_exception.csv
donor .csv
primary_diagnosis_test_case1 .csv

Submission in primary_diagnosis.tsv file:

Error message in clinical submission system:

Result: Validation failed as expected. Both lymph_nodes_examined_status and lymph_nodes_examined_method must have the same clinical exception of Unknown.

Test case 2: Validation should fail if number_lymph_nodes_examined and/or number_lymph_nodes_positive is submitted when lymph_nodes_examined_status has a clinical exception value of Unknown

Sample submission files:
donor .csv
primary_diagnosis_test_case2.csv

Submission in primary_diagnosis.tsv file:

Error message in clinical submission system:

Result: Validation fails as expected. Rows 1 and 2 of the submission fail as expected, while Rows 3 and 4 pass validation as expected.

Test case 3: Validation should pass if number_lymph_nodes_examined and number_lymph_nodes_positive are submitted when lymph_nodes_examined_status = Yes and lymph_nodes_examined_method is an enum value from the dictionary. (NOTE: This is normal validation that existed before the validation script updates, so just making sure validation is still working as expected when a valid case is submitted)

Sample submission files:
donor .csv
primary_diagnosis_test_case3.csv

Result: Validation passes as expected (row 2)

[hknahal]

Testing in QA:

Tested validation script changes for tumour_grade field in QA using HNAHAL-CA

Test case 1: If tumour_grading_system has a clinical exception value of Not applicable, but tumour_grade is submitted with an enum value from the dictionary, the validation should fail (if tumour grading system is not applicable, then tumour_grade must be not applicable as well).

Sample submission files:
tumour_grade_exception.csv
donor .csv
specimen .csv

Result: Validation fails as expected. Both the tumour_grading_system and tumour_grade fields must have the same clinical exception value of Not applicable.

Updated error message in clinical submission system indicates to the data submitter that the tumour_grade field must be submitted as Not applicable as well when tumour_grading_system has a clinical exception value of Not applicable:

Test case 2: If both tumour_grading_system and tumour_grade have a clinical exception value of Not applicable, the validation should pass.

Sample submission files:
donor .csv
specimen.csv

Result: Validation passes as expected.
Clinical exceptions file with both tumour_grading_system and tumour_grade are Not applicable:
HNAHAL-CA_program_exception_chemotherapy_drug_dose_units.csv

[hknahal]

Deployed Dictionary version 1.18 to Staging

Tested the same test cases that were done in QA (documented in above comments) in Staging as well. Used the same sample submission files (included in comments above).
All test cases passed/failed validation in Staging as expected (same results as QA testing)

[hknahal]

Dictionary version 1.18 deployed to prod August 25, 2023.

Tested same test cases (documented above) in prod. All test cases passed/failed validation in prod as expected (same results as Staging).