DHCP failed, N-S traffic is intermittent - lets debug why

Recently one of my peers from Sales team reached out and said that he encountered a weird problem during a NSXT customer POC. Problem statement as follows.
1. DHCP didnt work
2. N-S traffic fails intermittently. rebooting the edge remediate the issue short-term.

Product versions:
NSXT 2.4
ESXi 6.7U2

In his internal lab he deployed a similar topology[Pic1] where he could not recreate the issue.

Pic1

Started debugging the dhcp issue, my initial observations below

i)  DHCPREQUEST never reaches the edge (dhcp service port on T1). But the geneve tunnels between edge and hypervisor is up, DFW allows the dhcp traffic as well.

ii) Tried assigning a static ip to the dhcptest(client) vm - 192.168.2.3 and pinged the dhcp server ip -192.168.2.2 , didn’t work(desitnation not reachable messages were seen). 

iii)  Did a traceflow between vm ip and the dhcp server ip(Pic2), this went through fine!!.  Traffic reached DHCP server , reply came back to client as well.

Point number iii) increased my curiosity because when traceflow can work why ping/dhcp didnt go through.

Pic2

My suspicion - traceflow could have worked because it knew the destination mac i.e dhcp server port mac , but ping involves arp resolution. For clarity,  I have added a static arp entry on the test vm for the dhcp server ip, and the ping worked.  This led me to believe we may have issues in ARP resolution.

Further deepened my investigation.

Assigned a static ip on the dhcp test VM and Initiated a ping request towards dhcp server, captured it on the UplinkSnd point of the SRC VTEP vmnic.

[root@hypervisor:~] pktcap-uw --uplink vmnic2 --capture UplinkSndKernel --vni 65544  -o test.pcap

12:38:30.052479 00:50:56:60:d6:ea > 00:50:56:6a:4a:05, ethertype IPv4 (0x0800), length 118: 172.16.16.6.56840 > 172.16.16.4.6081: Geneve, Flags [C], vni 0x10008, proto TEB (0x6558), options [8 bytes]: 00:50:56:80:bf:03 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.2.2 tell 192.168.2.3, length 46

12:38:30.052480 00:50:56:60:d6:ea > 00:50:56:80:12:6c, ethertype IPv4 (0x0800), length 118: 172.16.16.6.56840 > 172.16.16.7.6081: Geneve, Flags [C], vni 0x10008, proto TEB (0x6558), options [8 bytes]: 00:50:56:80:bf:03 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.2.2 tell 192.168.2.3, length 46

Above you can see the arp broadcast on the uplink of SRC TEP.

However this traffic was never seen on the remote VTEP’s Uplink- (note: edge is in vm form factor, edge vtep portgroup is connected to vmnic2 on the base host and the capture was done there) . 

[root@edgevm:~] pktcap-uw --uplink vmnic2 --capture UplinkRcvKernel --vni 65544

No packets.

 This is the same case with DHCP broadcast traffic as well.

Since the packets are not even received by UplinkRcvKernel, I started doubting on nic driver of the server that hosts edge vm.  Noticed Edge cluster hosts had QLogic network adapters while compute clusters had intel.  To confirm my understanding , I moved the edge vm to a server that had intel adapters , dhcp traffic  worked just fine!!.   N-S traffic issue was also resolved, both the issues are related . Hence, the problem with QLogic adapter was confirmed.

On QLogic website found a new driver with below release notes, 

 Problem: GENEVE/VxLAN rx pkts drop with inner broadcast/multicast MAC.

"GENEVE/VxLAN rx pkts were dropped because VNI were not programmed. FW added support for any_vni flag. Set any_vni when VxLAN/Geneve  filtering is enabled."

Basically if the inner packet of VxLAN/GENEVE traffic contains broadcast/multicast mac then driver drops such traffic - this is the root cause.

The default driver shipped with 6.7 U2 had this issue. Upgrading to QLC_bootbank_qedentv_3.11.7.0-1OEM.670.0.0.8169922.vib fixed this once and for all!!