GFDL 20231113 candidate PR failure on wcoss2

[jiandewang]

GFDL's 20231213 candidate PR (mom-ocean#1616) failed to retain UFSWM BL on wocss2 for all non-debug S2S jobs. With many testing and help from @DeniseWorthen I was able to narrow down to two sepcific commits (among ~150 commits) which caused the answer changes. GFDL is actively studing the modified codes for these two commits.

[jiandewang]

based on last week's discussion with @junwang-noaa, I merged the PR code to dev/emc and re-generated BL. Re-run it matched this new BL so the code in this PR is reproductciable by self.
FYI @DeniseWorthen and @JessicaMeixner-NOAA as I always assign you two as code reviewers.
In the meantime let's wait to see if GFDL can figure it out or not. The above will be the last option we will adapt.

[jiandewang]

with Marshall and GFDL's detailed work, the updated code is able to retain UFS wcoss2 results for 1x1, half and quarter degree settings. C48-5x5 setting will requires a new BL. hafs-mom6 setting will also requires a new BL as there is a bug fixing in this PR.
From now on I will monite and test dev/gfdl code in UFS at about every 10 days ferquency to have early detecting of FMA enabled related issue. Thanks for @DeniseWorthen and @jkbk2004 help me tested on platforms that I don't have access to. I will have PR to dev/emc soon.

[jiandewang]

close