Add FATES tree recruitment scheme tests

[glemieux]

Description of changes

This simple pull request adds tests for the two recruitment namelist options for the FATES tree recruitment scheme. There are two different modes for turning on the scheme, which now have a test. No FATES parameter file modifications are necessary to activate the test.

Resolves #2760.

Note that there is an known FATES mass balance issue, NGEET/fates#1246, that might be triggered by the FatesColdTRS testmod. This will be added to the expected fail list if it comes up during testing.

Specific notes

Contributors other than yourself, if any:

CTSM Issues Fixed (include github issue #): #2760

Are answers expected to change (and if so in what way)? No

Any User Interface Changes (namelist or namelist defaults changes)? No

Does this create a need to change or add documentation? Did you do so? No

Testing performed, if any:
(List what testing you did to show your changes worked as expected)
(This can be manual testing or running of the different test suites)
(Documentation on system testing is here: https://github.com/ESCOMP/ctsm/wiki/System-Testing-Guide)
(aux_clm on derecho for intel/gnu and izumi for intel/gnu/nag/nvhpc is the standard for tags on master)

NOTE: Be sure to check your coding style against the standard
(https://github.com/ESCOMP/ctsm/wiki/CTSM-coding-guidelines) and review
the list of common problems to watch out for
(https://github.com/ESCOMP/CTSM/wiki/List-of-common-problems).

[glemieux]

Running the TRS test mods on derecho currently.

[glemieux]

The FatesColdTRSNoSeedDynamics runs successfully, but the FatesColdTRS test mod is failing with an issue different from NGEET/fates#1246. I'll create a new issue on the fates side and update the expected failure list.

Alternatively, we could attempt to fix the issue and turn this PR into an API update associated with whichever FATES tag fixes the TRS issue.

[glemieux]

TO DO:

• Review history output to add to test
• Add expected failure

[glemieux]

Running aux_clm regression tests against ctsm5.4.021.

[glemieux]

Testing on izumi against ctsm5.4.021 is B4B on all expected tests.

Testing on derecho against the same baseline is showing almost all expected tests are B4B with the following exceptions:

pr3757-aux_clm-ctsm54021_int: 208 tests
    PEND ERP_P256x2_Ld30.f45_f45_mg37.I2000Clm60FatesRs.derecho_intel.clm-mimicsFatesCold COMPARE_base_rest
    FAIL RXCROPMATURITYSKIPGEN_Ld1097.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-cropMonthOutput RUN time=70
    FAIL SUBSETDATAPOINT_Ld5_D_Mmpi-serial.CLM_USRDAT.I2000Clm60BgcCropCrujra.derecho_intel.clm-default RUN time=24
    FAIL SUBSETDATAREGION_Ld5_D_Mmpi-serial.CLM_USRDAT.I2000Clm60BgcCropCrujra.derecho_intel.clm-default RUN time=92

@ekluzek @slevis-lmwg did you see these fail during the testing for the last b4b-dev tag? Apologies in advance if I missed a conversation in the chat or elsewhere about these.

Results (izumi): /home/glemieux/scratch/ctsm-tests/tests_0222-215030iz
Results (derecho):

[slevis-lmwg]

Testing on derecho against the same baseline is showing almost all expected tests are B4B with the following exceptions:

pr3757-aux_clm-ctsm54021_int: 208 tests
    PEND ERP_P256x2_Ld30.f45_f45_mg37.I2000Clm60FatesRs.derecho_intel.clm-mimicsFatesCold COMPARE_base_rest
    FAIL RXCROPMATURITYSKIPGEN_Ld1097.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-cropMonthOutput RUN time=70
    FAIL SUBSETDATAPOINT_Ld5_D_Mmpi-serial.CLM_USRDAT.I2000Clm60BgcCropCrujra.derecho_intel.clm-default RUN time=24
    FAIL SUBSETDATAREGION_Ld5_D_Mmpi-serial.CLM_USRDAT.I2000Clm60BgcCropCrujra.derecho_intel.clm-default RUN time=92

@ekluzek @slevis-lmwg did you see these fail during the testing for the last b4b-dev tag? Apologies in advance if I missed a conversation in the chat or elsewhere about these.

I confirmed that the RX and SUBSETDATA tests passed in ctsm5.4.021 by looking at Teststatus here:
/glade/campaign/cgd/tss/ctsm_baselines/ctsm5.4.021

[glemieux]

Looking at the cesm.log error message it appears to be an instance of #3590 as I'm seeing the same thing as mentioned in #2930 (comment). I'll try rerunning these without defining a custom test-id.

[glemieux]

Looking at the cesm.log error message it appears to be an instance of #3590 as I'm seeing the same thing as mentioned in #2930 (comment). I'll try rerunning these without defining a custom test-id.

Looks like this is the cause of the issue I was seeing. Re-running these tests individually with the default test-id resulted in all three previous run failures passing b4b.