Add Jenkinsfile for the ND pipeline

[astawast-cisco]

ND Nightly Pipeline – Combined Draft and Work Done So Far

Summary

This draft combines the existing ND Pipeline1 flow with reliability, security and reporting improvements from my nightly-pipeline work.

Pipeline1 remains the baseline for integration-module coverage, smoke playbooks, repository checkout, dependency installation, linting, fabric reset and Webex presentation.

The proposed additions and enhancements include workspace-local runtime handling, safer Consul downloads, an ND precheck, protected inventory generation, stronger target-level failure handling, cleanup verification, stricter result parsing, artifact collection and direct Webex API delivery.

This is currently a draft for comparison and discussion. The pipeline framework is implemented, but complete end-to-end module execution is pending the testbed-variable fixes documented below.

Baseline retained from Pipeline1

The following capabilities were retained from the existing Pipeline1 flow:

• Docker-based execution using the ansible_nd_setup image.
• ND 4.2 test configuration.
• Existing integration-module coverage.
• Existing smoke-playbook coverage.
• One ansible-playbook invocation per integration target.
• Embedded run_integration_module.yaml runner.
• Best-effort pre-cleanup and post-cleanup inside the integration runner.
• Checkout of the cisco.nd develop branch.
• Python 3.12 virtual-environment setup.
• Basic Consul integration for downloading pipeline assets.
• Dependency installation.
• Dynamic inventory generation.
• Target-output collection.
• Ansible-lint execution and summary generation.
• Fabric reset using reset_fabric.yaml.
• Jenkins artifact archiving.
• Webex nightly-notification presentation.
• Protection against concurrent builds.
• Ten-hour overall pipeline timeout.

These features were used as the baseline and were extended where required.

Pipeline and Jenkins setup enhancements

The following Jenkins-related changes were incorporated:

• Added the nightly schedule.

• Added skipDefaultCheckout because repository checkout is handled explicitly by the pipeline.

• Added Jenkins timestamps so execution duration and the point of failure are easier to trace.

• Added a 30-day retention policy for builds and artifacts.

• Retained the ten-hour overall timeout.

• Retained protection against concurrent builds because the tests operate on a shared testbed.

• Added configurable parameters for:

  • RUN_SMOKE
  • LINT_GATES_BUILD
  • RESET_FABRIC_AFTER_RUN

The DEBUG parameter is currently reserved and does not yet modify pipeline behaviour.

Repository and environment setup

Python 3.12 was already used by Pipeline1. The environment handling was extended as follows:

• Added explicit Python 3.12 validation before reusing an existing virtual environment.

• Added automatic virtual-environment recreation when the existing environment uses a different Python version.

• Moved the virtual environment into the Jenkins workspace:

  $WORKSPACE/.jenkins-runtime

• Moved the collection checkout into:

  $WORKSPACE/nd

• Retained checkout of the cisco.nd develop branch.

• Added explicit comparison of the local and remote commit.

• Reused the repository when it is already current.

• Re-cloned the repository when the remote branch has changed.

• Added validation to ensure an existing nd path is a valid Git repository.

This reduces dependency on manually maintained files under the Jenkins home directory.

Consul and required pipeline assets

Pipeline1 already downloaded configuration and playbooks from Consul. The download process was strengthened with:

• Separate Consul prefixes for Pipeline1 assets and newer safety assets.
• Validation of Consul prefixes and filenames.
• curl --fail for HTTP error handling.
• Connection and total-request timeouts.
• Downloading into a temporary file.
• Moving the temporary file into place only after a successful download.
• Automatic cleanup of incomplete temporary downloads.
• Required-file validation before dependency installation.

The required-file check covers:

• requirements.txt
• requirements.yaml
• ansible.cfg
• nd_precheck.yaml
• nd_cleanup_verify.yaml
• reset_fabric.yaml
• run_integration_module.yaml

Smoke playbooks are downloaded only when RUN_SMOKE is enabled.

Dependency installation

The pipeline:

• Installs Ansible Core using the supported range:

  ansible-core>=2.19,<2.20

• Installs Python packages from requirements.txt.

• Installs Ansible collections from requirements.yaml.

• Validates Python 3.12.

• Reports the installed Python version.

• Reports the installed Ansible version.

• Reports the installed ansible-lint version when available.

A specific ansible-lint version is not currently enforced.

Inventory and credential handling

Pipeline1 already generated an inventory. The combined version changes this to protected JSON inventory generation.

The inventory mechanism:

• Uses Jenkins-provided ND credentials.
• Passes credentials through environment variables.
• Avoids inserting passwords directly into Groovy-interpolated strings.
• Includes HTTPAPI and connection settings.
• Includes the configured fabric and testbed variables.
• Creates the inventory in a Jenkins temporary directory.
• Applies mode 0600.
• Removes the inventory in post/always, including when the build fails.

The protected inventory mechanism is implemented, but explicit testbed-value validation and final switch-variable alignment are still pending.

ND precheck

An ND precheck was added before integration and smoke execution.

Its purpose is to detect:

• Controller connectivity problems.
• Authentication failures.
• HTTPAPI configuration problems.
• Testbed or environment issues.

The precheck has a separate ten-minute timeout.

If the precheck fails, integration and smoke targets are not started.

The precheck is implemented but has not yet been validated end to end because the current inventory problem occurs before it can run.

Integration target execution

The currently enabled integration targets are:

• nd_manage_policy
• nd_manage_policy_group
• nd_manage_networks
• nd_manage_vrfs
• nd_vpc_pair

Pipeline1’s one-invocation-per-target design was retained and extended with:

• Independent target output files.
• Independent target duration.
• A 120-minute timeout for every integration target.
• Target-specific result processing.
• Target-specific cleanup verification.
• Target-specific artifact collection.
• Target-name validation before passing a name to shell commands.
• Accumulation of failures across targets.

After target execution begins, a normal failure in one target is recorded and the remaining targets can continue. At the end, failed targets are aggregated and used to update the overall build result.

Failures that occur before the target loop, such as inventory-generation or dependency failures, still prevent all modules from starting.

The five targets are configured, but they have not yet completed an end-to-end run in the combined pipeline.

Cleanup handling

The integration runner retains Pipeline1’s cleanup behaviour:

• Detect whether the target contains cleanup.yaml.
• Perform best-effort pre-cleanup when available.
• Run best-effort post-cleanup through an Ansible always block.

The combined pipeline additionally runs:

nd_cleanup_verify.yaml

after every integration target.

This separates two concerns:

• Cleanup attempts to remove test objects.
• Cleanup verification checks whether the expected objects were actually removed.

Cleanup problems are tracked separately using the CLEANUP_FAILED status instead of being included as normal module-task failures.

Cleanup verification has a separate 15-minute timeout.

Smoke tests

The smoke-playbook list was retained from Pipeline1 and made configurable through RUN_SMOKE.

The smoke coverage includes:

• External fabric
• eBGP VXLAN
• iBGP VXLAN
• AI eBGP VXLAN
• AI iBGP VXLAN
• Prefix list

Each smoke playbook has a 45-minute timeout.

Smoke validation is pending. The current focus is to get the integration flow working first.

Ansible exit-code handling

Ansible output is sent through tee so it can be displayed in Jenkins and saved in the target output file.

Without additional handling, a shell pipeline can return the status from tee instead of the status from ansible-playbook.

The pipeline captures the real Ansible exit code using:

PIPESTATUS[0]

This prevents a successful tee operation from hiding an Ansible failure.

Failure handling

Target results can use the following statuses:

• PASSED
• FAILED
• UNREACHABLE
• ERROR
• CLEANUP_FAILED

The failure-handling flow includes:

• Recording normal Ansible task failures.
• Recording unreachable-host results.
• Recording execution errors.
• Recording cleanup-verification failures separately.
• Continuing with remaining targets after normal target-level failures.
• Aggregating target failures at the end.
• Creating a synthetic failure when a target starts but does not produce a PLAY RECAP.
• Keeping compact failure reasons for Webex readability.

Synthetic target failures apply only after runTarget() starts. Early setup or inventory failures are not currently included in the module counters.

Result parsing

The pipeline parses each target’s Ansible PLAY RECAP for:

• ok
• failed
• skipped
• unreachable

It also records:

• Execution duration
• Ansible exit code
• Timeout information
• Missing PLAY RECAP
• Execution or infrastructure errors

The cleanup verifier produces its own PLAY RECAP. That section is excluded from the integration target’s totals so cleanup tasks do not incorrectly increase the target’s passed count.

The reported Passed value is based on Ansible’s ok= count. It therefore represents successful Ansible tasks and not necessarily the number of individual test cases.

Artifact collection

Pipeline1 already contained artifact collection. The combined flow keeps target-specific evidence by:

• Creating a separate output file for every target.
• Saving complete Ansible output.
• Saving target duration.
• Saving the exit code.
• Saving cleanup-verification output.
• Archiving the output through Jenkins.
• Enabling artifact fingerprinting.

This allows a Webex target result to be traced back to its raw Jenkins and Ansible output.

Artifact creation has not yet been validated end to end because Build 16 did not reach target execution.

Ansible-lint

Pipeline1’s ansible-lint execution was retained and integrated with the new reporting flow.

The combined pipeline:

• Runs ansible-lint --profile=production.
• Saves the complete lint output.
• Archives the full lint output.
• Produces a condensed Webex summary.
• Uses LINT_GATES_BUILD to control whether lint failures fail the build.

There is a known issue where an old lint_summary.txt can remain in the workspace when lint does not run. This can result in a current NOT_RUN status being displayed with an older summary.

Lint-summary cleanup or conditional display is still pending.

Fabric reset

Pipeline1’s fabric-reset capability was retained and made configurable through:

RESET_FABRIC_AFTER_RUN

The reset stage uses reset_fabric.yaml and the generated inventory.

The current Groovy draft defaults this parameter to true. During validation, it must be manually disabled until the reset playbook and its fabric-specific values have been verified.

Before merge, either:

• The reset playbook must be validated against VXLAN_Fabric, or
• The parameter default should be changed to disabled.

Because this stage modifies the fabric, it should not be enabled for the nightly run until it has been tested separately.

Webex notification

Pipeline1 already produced Webex notifications. The combined pipeline changes the delivery mechanism to call the Webex API directly instead of downloading and executing a separate notification script.

The implementation:

• Uses a Jenkins credential for the Webex bearer token.
• Builds a valid JSON request payload.
• Masks the authorization header.
• Handles Webex failures separately.
• Prevents a notification failure from hiding the original pipeline failure.
• Keeps failure reasons compact.

Once target execution succeeds, the notification is designed to include:

• Overall build result
• Individual target results
• Fabric name
• ND version
• Target type
• Passed-task count
• Failed-task count
• Skipped-task count
• Unreachable count
• Duration
• Target status
• Cleanup/suite error count
• Lint information
• Failure details
• Jenkins console link

The Webex notification was successfully delivered for Build 16. This confirms that direct API delivery works. Target-level reporting is still pending because Build 16 did not execute any integration target.

Validation completed so far

The following has been completed or validated:

• Combined pipeline structure created.
• Pipeline1 coverage retained.
• Workspace-local runtime added.
• Python 3.12 validation added.
• Safe Consul download handling added.
• Build 15 permission fix implemented.
• Embedded runner generation implemented.
• Protected inventory mechanism implemented.
• ND precheck flow implemented.
• Target-execution framework implemented.
• Result parser implemented.
• Cleanup-verification framework implemented.
• Lint reporting framework implemented.
• Direct Webex API integration implemented.
• Webex notification successfully delivered for Build 16.

Proposed changes for consideration

The main enhancements proposed for incorporation are:

• Workspace-local runtime handling.
• Explicit checkout with skipDefaultCheckout.
• Jenkins timestamps and retention.
• Safe Consul downloads.
• Separate Consul prefixes.
• Required-file validation.
• Protected temporary inventory.
• Inventory cleanup in post/always.
• ND precheck.
• Target-name validation.
• Per-target timeouts.
• Correct exit-code handling using PIPESTATUS.
• Continue-on-failure and aggregate failure handling.
• Explicit target statuses.
• Separate test and cleanup errors.
• Cleanup verification.
• Strict PLAY RECAP parsing.
• Exclusion of cleanup results from module totals.
• Synthetic failures for missing PLAY RECAP.
• Compact failure reporting.
• Target artifacts and fingerprinting.
• Configurable smoke execution.
• Configurable lint gating.
• Configurable fabric reset.
• Direct Webex API delivery.
• Webex token masking.
• Detailed target-level Webex reporting.
• Webex API error handling.

Next steps

1. Get protected inventory generation working.
2. Validate the ND precheck.
3. Run an integration-only build with reset and smoke disabled.
4. Validate one integration target end to end.
5. Validate PLAY RECAP parsing and cleanup verification.
6. Verify artifact creation and fingerprinting.
7. Verify the corresponding target-level Webex result.
8. Run the remaining enabled integration targets.
9. Validate smoke playbooks separately.
10. Fix stale lint-summary handling.
11. Validate fabric reset separately.
12. Enable fabric reset only after successful testbed validation.

[astawast-cisco]

The draft PR has been narrowed to the Jenkinsfile only. The duplicated root-level test assets and unrelated .gitignore changes have been removed; the existing integration tests under tests/integration/targets remain the test source.