DAOS-18710 test: Update provider tests to use dc_x for UCX.

[jgmoore-or]

Steps for the author:

• Commit message follows the guidelines.
• Appropriate Features or Test-tag pragmas were used.
• Appropriate Functional Test Stages were run.
• At least two positive code reviews including at least one code owner from each category referenced in the PR.
• Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

• Gatekeeper requested (daos-gatekeeper added as a reviewer).

[github-actions]

Ticket title is 'Change to using ucx+dc_x as the provider string for daily test run with UCX and for the weekly provider tests.'
Status is 'Open'
Labels: 'request_for_2.8'
https://daosio.atlassian.net/browse/DAOS-18710

[daosbuild3]

Test stage Functional on Leap 15.6 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17746/1/display/redirect

[phender]

Note: for master we'll also want to update https://github.com/daos-stack/daos/blob/provider-testing/Jenkinsfile#L100 in a separate PR.

[daosbuild3]

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17746/2/execution/node/322/log

[daosbuild3]

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17746/2/execution/node/274/log

[daosbuild3]

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17746/3/execution/node/494/log

[daosbuild3]

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17746/3/execution/node/266/log

[daosbuild3]

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17746/4/execution/node/520/log

[daosbuild3]

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17746/5/execution/node/386/log

[daosbuild3]

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17746/5/execution/node/409/log

[johannlombardi]

@jgmoore-or please review the failures and let us know whether there are either existing issues or new ones introduced by the switch from UD to DC.

[phender]

Analysis of https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17746/5/#showFailuresLink failures:

• Known issue https://daosio.atlassian.net/browse/DAOS-15518
  • [HW Medium] 1-./control/dmg_network_scan.py:DmgNetworkScanTest.test_dmg_network_scan_basic
  • [HW Medium MD on SSD] 1-./control/dmg_network_scan.py:DmgNetworkScanTest.test_dmg_network_scan_basic
• Known issue https://daosio.atlassian.net/browse/DAOS-16553
  • [HW Medium] 1-./dfuse/pil4dfs_fio.py:Pil4dfsFio.test_pil4dfs_vs_dfs
  • [HW Medium] 3-./dfuse/pil4dfs_fio.py:Pil4dfsFio.test_pil4dfs_vs_dfs
  • [HW Medium] 4-./dfuse/pil4dfs_fio.py:Pil4dfsFio.test_pil4dfs_vs_dfs
  • [HW Medium] 5-./dfuse/pil4dfs_fio.py:Pil4dfsFio.test_pil4dfs_vs_dfs
  • [HW Medium MD on SSD] 1-./dfuse/pil4dfs_fio.py:Pil4dfsFio.test_pil4dfs_vs_dfs
  • [HW Medium MD on SSD] 3-./dfuse/pil4dfs_fio.py:Pil4dfsFio.test_pil4dfs_vs_dfs
  • [HW Medium MD on SSD] 4-./dfuse/pil4dfs_fio.py:Pil4dfsFio.test_pil4dfs_vs_dfs
  • [HW Medium MD on SSD] 5-./dfuse/pil4dfs_fio.py:Pil4dfsFio.test_pil4dfs_vs_dfs
• NEW issue
  • [HW Medium MD on SSD] 3-./pool/create.py:PoolCreateTests.test_create_no_space_loop
    • This last test case times out on loop 81 of 100 while attempting to create pools that are expected to fail due to DER_NOSPACE.
    • I do not see any open or recently closed tickets on this issue.
• Known issue https://daosio.atlassian.net/browse/DAOS-18276
  • [HW Medium MD on SSD] 05-./pool/create_all_hw.py:PoolCreateAllHwTests.test_recycle_pools_hw
  • [HW Medium MD on SSD] 07-./pool/create_all_hw.py:PoolCreateAllHwTests.test_recycle_pools_hw
  • [HW Medium MD on SSD] 08-./pool/create_all_hw.py:PoolCreateAllHwTests.test_recycle_pools_hw
• Known issue https://daosio.atlassian.net/browse/DAOS-17616
  • [HW Medium MD on SSD] 1-./server/daos_server_restart.py:DaosServerTest.test_daos_server_reformat

[jgmoore-or]

The daily testing does not run the HW Medium MD on SSD test for UCX, so the new failure noted by @phender is not something new introduced by the switch to dc_x from ud_x. This test is also not run in the weekly provider tests, which will be switched to dc_x in a separate PR.

I agree that we need a ticket for this failure, but it is not pertinent to this ticket (which just changes from ud_x to dc_x for the Functional Hardware Medium UCX Provider test). Note that none of the tests incurring failures for this PR are run as part of the Functional Hardware Medium UCX Provider test.

The dmg network scan failure is fixed by: #17636 (currently under review). The pil4dfs_fio tests failures are due to a limitation in UCX that requires a change to the UCX code to resolve. We have such a fixed (developed by Lei Huang) and plan to push this to UCX.

The other MD-on-SSD failures listed by Phil do not appear to be specific to UCX.

[phender]

The daily testing does not run the HW Medium MD on SSD test for UCX, so the new failure noted by @phender is not something new introduced by the switch to dc_x from ud_x. This test is also not run in the weekly provider tests, which will be switched to dc_x in a separate PR.

It should be noted that build #5 ran all the Functional HW stages with ucx+dc_x due to having both the TestProvider and TestProviderUCX parameter to ucx+dc_x. The code change in this PR should only be setting the defafult TestProviderUCX value to ucx+dc_x. Normally TestProvider would be left empty (default value) resulting in verbs being used for the non-Provider Functional HW stages.