1.0 Executive Summary

This document addresses a critical communication failure within a Cisco SD-WAN Hub-and-Spoke lab environment. The core issue is the inability of spoke sites to communicate with each other, despite both spokes having full connectivity to the central hub site. The initial action of applying a standard hub-and-spoke centralized control policy via vManage did not resolve the issue. This document provides a comprehensive, systematic troubleshooting methodology to diagnose and rectify the underlying cause, focusing on control plane verification, data plane forwarding, and policy configuration best practices.

2.0 Problem Analysis

The reported issue is constrained to a specific traffic path within the SD-WAN fabric. A summary of the network state is as follows:

• Topology: A standard Hub-and-Spoke topology has been deployed.
• Working Communication Paths:
  • Spoke-to-Hub communication is fully functional.
  • Hub-to-Spoke communication is fully functional.
• Failing Communication Path:
  • Spoke-to-Spoke communication fails. The intended path for this traffic is from the source spoke, through the hub, and then to the destination spoke.

This symptom pattern strongly suggests that while basic site-to-hub control and data plane connections are established, the mechanism for advertising routes between spoke sites via the hub is not functioning as intended. The failure likely resides in the application of the control policy, the Overlay Management Protocol (OMP) route advertisement process, or data plane forwarding at the hub router.

3.0 Review of Initial Action

The initial step taken was the application of a pre-defined hub-and-spoke control policy from the vSmart controller. This is the correct strategic approach for enforcing this topology. However, the failure of this policy to enable communication indicates one of the following possibilities:

• The policy was not correctly constructed or applied to the appropriate site lists and VPNs.
• The policy is being applied correctly, but OMP is not advertising or processing the routes as expected.
• The control plane is functioning, but the data plane on the hub router is not correctly forwarding the hairpin traffic between spokes.

4.0 Comprehensive Troubleshooting and Resolution Plan

To systematically isolate and resolve the root cause, the following steps must be executed in order. This plan moves from the control plane (OMP) to the data plane (forwarding).

4.1 Control Plane Verification (OMP Route Advertisement)

The control policy’s primary function is to manipulate OMP route advertisements. We must verify that the hub is correctly re-advertising spoke routes to other spokes.

1. Verify Policy Application on vSmart:

   • Confirm that the centralized control policy is active and correctly applied. Ensure the policy is applied in the outbound direction to the site list containing the spoke sites.
   • Validate that the policy correctly identifies the hub TLOCs in its set tloc action.

2. Verify Route Reception at the Hub:

   • On the hub router’s CLI, confirm it is receiving OMP routes for the service-side (LAN) prefixes from both spokes.
   • Command: show omp routes vpn <VPN-ID>
   • Look for the prefixes of each spoke’s LAN segment.

3. Verify Route Advertisement from the Hub:

   • This is the most critical step. Check if the hub is advertising the routes learned from Spoke A to Spoke B, and vice-versa. The control policy should modify the TLOC of these advertised routes to be the hub’s own TLOC.
   • Command: show omp routes advertised
   • Filter for a specific spoke’s prefix and check if it is being advertised to the other spoke’s System-IP.

4. Verify Route Reception at the Spokes:

   • On Spoke B’s CLI, check the OMP routes received for Spoke A’s LAN prefix.
   • Command: show omp routes vpn <VPN-ID>
   • Crucially, the TLOC listed for Spoke A’s prefix should be the System-IP and color of the Hub router, not Spoke A. If you see Spoke A’s original TLOC, the control policy is not working correctly. The FROM PEER should be the System-IP of the vSmart controller.

4.2 Data Plane Verification (Packet Forwarding)

If the control plane verification confirms that routes are being advertised correctly, the issue lies within the data plane.

1. Check the Spoke’s Routing Table:

   • On Spoke B, verify that the route for Spoke A’s LAN prefix is installed in the correct VPN routing table with the hub as the next hop.
   • Command: show ip route vpn <VPN-ID> <spoke-A-prefix>
   • The output should show the route is known via the IPsec tunnel pointing towards the hub.

2. Verify Hub Forwarding:

   • Initiate a continuous ping from a host behind Spoke A to a host behind Spoke B.
   • On the hub router, check if traffic is being received from Spoke A and forwarded towards Spoke B. You can use an access-list applied to the hub’s service-side interface or check interface statistics to see packets being routed between tunnels.
   • Command: show platform hardware qfp active feature ipsec-in stats and show platform hardware qfp active feature ipsec-out stats can help verify if encrypted/decrypted packet counts are increasing.

3. Inspect for Data Plane Restrictions:

   • Check for any data policies or Access Control Lists (ACLs) that might be dropping the inter-spoke traffic.
   • Verify the hub router’s zone-based firewall configuration (if applicable). Traffic entering one tunnel and exiting another may be considered inter-zone traffic and could be blocked by default security posture.

5.0 Conclusion

The failure of spoke-to-spoke communication in a properly configured hub-and-spoke SD-WAN fabric is most often attributed to an improperly applied control policy or a misinterpretation of OMP route advertisements. By following the structured verification plan outlined above—validating the OMP control plane from vSmart to the spokes, and subsequently verifying the data plane forwarding path on the hub—the exact point of failure will be identified. This systematic approach ensures a rapid and accurate resolution.