[Nexthop] Move send LLDP packets to retry loop

[vvasavada-nexthop]

Pre-submission checklist

• I've ran the linters locally and fixed lint errors related to the files I modified in this PR. You can install the linters by running pip install -r requirements-dev.txt && pre-commit install
• pre-commit run

Summary

TLDR

TX path is still settling when early LLDP sends happen, initial sends fail/partially fail, and neighbor count recovers gradually but times out at 29/32.

Detailed debugging

Switch/agent is coming up

17:12:05.491080 MultiSwitchThriftHandler.cpp:28] failing thrift prior to switch configuration: co_syncHwStats
17:12:05.885758 MultiSwitchThriftHandler.cpp:28] failing thrift prior to switch configuration: co_notifyRxPacket
17:12:06.351933 MultiSwitchThriftHandler.cpp:28] failing thrift prior to switch configuration: co_notifyLinkChangeEvent
17:12:06.521503 SwSwitch.cpp:31] SwitchRunState changed from INITIALIZED to CONFIGURED

LLDP manager tries to send but actually the packet cannot be sent out of port 1. This happens for any ports.

17:12:06.522541 SwSwitch.cpp:3443] Error sending packet via thrift stream to switch 0
17:12:06.522550 SwSwitch.cpp:3344] failed to send packet out port 1
17:12:06.522556 LldpManager.cpp:527] sent LLDP on port 1 with CPU MAC 2e:03:69:9f:71:97 port id eth1/1/1 and vlan 2000

Test verify starts in retry loops. We slowly start getting rx packets.

17:12:08.522739 AgentHwTest.h:70] verify()
17:12:08.523160 AgentPacketSendTests.cpp:671] LLDP neighbors found: 0
17:12:08.525357 MultiSwitchThriftHandler.cpp:297] Got rx packet from switch 0 for port 1
17:12:08.525389 LldpManager.cpp:113] got LLDP packet: local_port=1 chassis=2e:03:69:9f:71:97 port=eth1/1/1 name=wdg154

Neighbor count is gradually increasing even after ~30s as tx/rx path converges

17:12:09.523222 AgentPacketSendTests.cpp:671] LLDP neighbors found: 22
17:12:12.523424 AgentPacketSendTests.cpp:671] LLDP neighbors found: 25
17:12:17.523759 AgentPacketSendTests.cpp:671] LLDP neighbors found: 27
17:12:32.524864 AgentPacketSendTests.cpp:671] LLDP neighbors found: 28
17:12:37.525254 AgentPacketSendTests.cpp:671] LLDP neighbors found: 29

The test timeout is at 30s which may not be enough. And so we assert and exit before all LLDP neighbors are found.
Note that LLDP manager has its own retry waves at every 5s interval. However, it is possible that tx path is fully ready before test timeouts after the last retry attempt but test timesout in the next 5s interval before next LLDP wave. Hence, moving sendLldpOnAllPorts to retry loop gives more robustness as we'd retry every 1s by explicitly sending packets.

Test Plan

Was able to reproduce failure in 2-3 attempts without the fix.
The test AgentPacketSendLldpTest.LldpLoopbackTest passes 10/10 times with the fix.