Skip to content

14. Traffic classes and packet walks

In this chapter

You will learn

  • what each of the six traffic classes - L2, L3-BL, L3-DCFW, L3-ENTFW, DSCP and LAB - measures and why it exists
  • how to derive the source and destination of any stream: attachment point, S-VLAN, C-VLAN, IP and MAC address
  • how to follow a frame hop by hop: tags on every link, VXLAN encapsulation, MAC learning, MAC rewrite and TTL at routing hops
  • what a measured latency contains - fabric hops, routing and firewall hops, DCI and TX-to-TX fibre crossings
  • why intra-RZC streams are slower than intra-RZK streams, and how you calibrate this fibre offset
  • how the centralised gateway in Kalk and the location of the active firewall member change the path
  • which configuration each class needs and how each class typically breaks

Prerequisites: Chapter 5, Chapter 6, Chapter 10, Chapter 12, Chapter 13

A latency of 183 µs or a loss of 40 packets means nothing until you know which path the packets took. This chapter is the map behind every number on the dashboards: for each class it follows a frame from the TRex port through the TX switch, the Leaf, the spines, the Border Leafs, the DCI and the firewalls back to TRex, and shows what each device does to it. When a stream gets slower, this is the chapter you reason with: which devices does it cross, and which of them changed?

The six classes at a glance

A traffic class is a family of streams that exercises one kind of forwarding. Each class has its own UDP destination port, so a capture or a firewall log tells you immediately which class a packet belongs to.

Class Question it answers Source and destination First hop seen by TRex UDP dst
L2 Does the fabric deliver frames inside one VLAN, within a DC and across the DCI? two APs, same VLAN the remote TRex client 32001
L3-BL What does the centralised gateway on sw-rzk-BL add? VLAN 1314 at X to VLAN 1315 at Y gateway 10.3.14.1 / 10.3.15.1 32002
L3-DCFW What does routing through the DC firewall CPF-ZV05/06 cost? production VLAN A to VLAN B DC-FW cluster address (VIP) 32003
L3-ENTFW What does the path from VPN-CORE through the Enterprise firewall cost? VLAN 1314 to the TRex firewall zone 10.3.14.1 / zone VIP 10.9.90.1 32004
DSCP Does any device treat marked packets differently? BE, EF, AF41 and CS1 copies of one L2 and one L3-ENTFW pair as the copied class 32005
LAB How much can a path carry, how fast does it converge? pinned lab ports, on demand as the tested class 32009
Five horizontal lanes, one per monitoring class, each showing the devices a stream crosses from TRex back to TRex
The device sequence of each monitoring class, drawn for an RZK source. Orange: TX switches; indigo/purple: fabric; red: firewalls. Every lane starts and ends at the TRex server in RZK. The L3-ENTFW lane shows firewall-zone option B, the plan's default: the firewall returns the frame to sw-rzk-BL, which bridges it to the K-BL attachment point.

Building blocks shared by all classes

The TRex clients and their addresses

Every stream runs between two emulated hosts - EMU clients (Chapter 6), one per combination of attachment point (AP) and VLAN. Their addresses are computed, not configured by hand:

  • IP address = pool start of the VLAN + (AP index - 1)
  • MAC address = 02:54:52:<AP index>:<VLAN high byte>:<VLAN low byte> in hexadecimal. 02 marks a locally administered unicast address; 54:52 happens to be ASCII "TR", which makes TRex clients easy to spot in MAC tables.
AP (index) S-VLAN VLAN IP address MAC address
K-L11 (1) 3101 1314 (0x0522) 10.3.14.101 02:54:52:01:05:22
K-L21 (3) 3103 1315 (0x0523) 10.3.15.103 02:54:52:03:05:23
K-BL (6) 3110 1314 10.3.14.106 02:54:52:06:05:22
K-BL (6), zone client 3110 4003 (0x0FA3) 10.9.90.106 02:54:52:06:0f:a3
C-L11 (7) 3201 1314 10.3.14.107 02:54:52:07:05:22
C-L21 (10) 3204 1314 10.3.14.110 02:54:52:0a:05:22
C-L22 (11) 3205 1315 10.3.15.111 02:54:52:0b:05:23
C-BL (13), zone client 3210 4003 10.9.90.113 02:54:52:0d:0f:a3
FW (14), only with firewall-zone option A 3300 1420 (0x058C) 10.9.90.114 02:54:52:0e:05:8c

The two zone clients in VLAN 4003 belong to firewall-zone option B, the plan's default: the TRex zone behind the Enterprise firewall is the existing transfer VSI 4003, which TRex reaches through the Border Leaf APs K-BL and C-BL like any other VLAN (Chapter 15). The last row exists only if the alternative, option A, is chosen.

Comware shows the same MACs as 0254-5201-0522 (K-L11, VLAN 1314). Every stream also has a fixed UDP source port (40000 + stream index), TTL 64 and a frame size of 128 bytes without FCS (256 bytes for DSCP copies), so it is one stable 5-tuple - see ECMP, LACP and stable 5-tuples. Pools, the zone subnet and the production VLANs are planning values (Chapter 16).

In the LVR platform

In the planned inventory, monitoring port p0 (sw-rzk-TX1 XGE1/1/21) hosts the clients of the RZK APs, including the zone client at K-BL, p1 (XGE1/1/22) those of the RZC APs, including the zone client at C-BL. Both trunks carry every S-VLAN; a stream leaves TRex on the port of its source client and arrives on the port of its destination client.

What every frame does at the edge

The edge behaviour is identical for all classes (Chapter 12, Chapter 13):

  1. TRex to TX switch: two 802.1Q tags - the outer S-tag selects the AP of the source client, the inner C-tag is the production VLAN.
  2. TX switch: the trunk classifies by the outer tag only. The AP port is an access port of that S-VLAN with qinq enable: the S-tag is removed on egress and the port's PVID pushed onto every incoming frame ("A QinQ-enabled port tags all incoming frames (tagged or untagged) with the PVID tag", L2CG PDF p. 220).
  3. Leaf or Border Leaf: one Ethernet service instance per VLAN (encapsulation s-vid 1314 / xconnect vsi VPN-1314). In the default VLAN access mode the VTEP removes the tag when the frame enters the VSI and adds it when it sends to the local site; VXLAN packets carry untagged inner frames (VXCG PDF pp. 13-14, VXCR PDF p. 55).
  4. VXLAN: 8-byte VXLAN header with the VNI (here equal to the VLAN number), 8-byte outer UDP header with destination port 4789, 20-byte outer IP header (VXCG PDF p. 7), plus the outer Ethernet header of the routed underlay link.
Frame formats on each link of an L2 stream: double-tagged 128 bytes at TRex, single-tagged 124 bytes on the AP link, VXLAN-encapsulated 170 bytes in the fabric
The same monitoring frame on each link of its path. Only the outer headers change; the inner IPv4/UDP packet and the TRex payload with the latency signature travel unchanged in the L2 class.

How the fabric knows where a MAC address is

The examined fabric switches (sw-rzk-L11, SW-RZC-L21, sw-rzk-BL, sw-rzc-BL) run vxlan tunnel mac-learning disable: a VTEP learns MACs only on its local ports and receives all remote MACs as BGP EVPN routes. On the Border Leafs, peer ... next-hop-local towards their own spines and peer ... router-mac-local towards the other Border Leafs (bgp 65001, address family l2vpn evpn on sw-rzk-BL) make every RZC MAC appear behind sw-rzk-BL (172.31.0.254) in RZK, and vice versa. In February 2025 the EVPN MAC table of sw-rzk-L11 had no next hops other than 172.31.0.242-245 (the other RZK Leafs) and 172.31.0.254.

The VSI on the Border Leaf therefore has two kinds of tunnels. verified Real output, diag_sw-rzk-BL_20250217-163816.txt, display evpn auto-discovery imet (EVPN CR PDF p. 18), VSI VPN-1314:

VSI name: VPN-1314
RD                    PE_address      Tunnel_address  Tunnel mode VXLAN ID
1:1314                172.31.0.241    172.31.0.241    VXLAN       1314
1:1314                172.31.0.242    172.31.0.242    VXLAN       1314
1:1314                172.31.0.243    172.31.0.243    VXLAN       1314
1:1314                172.31.0.244    172.31.0.244    VXLAN       1314
1:1314                172.31.0.245    172.31.0.245    VXLAN       1314
1:1314                172.31.1.254    172.31.1.254    VXLAN-DCI   1314
1:1314                172.31.2.254    172.31.2.254    VXLAN-DCI   1314

172.31.0.241 is the loopback of sw-rzk-L11, 172.31.1.254 sw-rzc-BL, 172.31.2.254 sw-rzd-BL. The VSI exists on all five RZK Leafs; on SW-RZC-L21 the same command lists the other five RZC Leafs (172.31.1.241-246) and sw-rzc-BL. A frame between the data centres is decapsulated and re-encapsulated on both Border Leafs - three VXLAN tunnels in a row.

What TRex measures

TRex sends and receives every frame on the same server clock, so it measures one-way latency per stream, from the TX core to the RX core (Chapter 5). The value contains every serialisation, queue, lookup and fibre on the way, plus a small constant share of the TRex server itself (NIC queues, polling RX core) that is part of every baseline. Streams are unidirectional; every pair is monitored in both directions with two pg_ids, because several LVR paths are asymmetric.

TRex recognises its own packets by marks it writes itself. It rewrites the IPv4 identification field of every statistics packet: a flow-stat stream gets a reserved base (0xFC00 in the v3.x source; the documentation names the range 0xff00-0xffff) plus an internal hardware ID - not the pg_id - and every latency stream gets 0xFFFF. A latency packet also carries a 16-byte signature at the end of its payload: an internal ID, a sequence number and the TX timestamp. And TRex sets bit 0 of the TOS byte on all these packets: a stream built with TOS 0x00 leaves as 0x01, one built with EF (0xB8) as 0xB9. The DSCP - the upper six bits - is unchanged; only the ECN field reads 01. Expect these values in every capture (Chapter 5).

Class L2 - Layer 2 inside one VLAN

L2 streams answer the basic question: does the EVPN-VXLAN fabric deliver frames between two ports of the same VLAN, and how fast? Source and destination are clients in the same VLAN at two APs; the destination MAC is the remote client's MAC, known from the addressing plan - no ARP, no routing. Intra-DC pairs connect each Leaf AP with the BL AP of its DC, inter-DC pairs an RZK AP with an RZC AP.

Packet walk intra-DC: K-L11 to K-BL in VLAN 1314

Watch for three moments in the animation: the TX switch removing the outer tag towards the Leaf, the Leaf exchanging the 802.1Q tag for a VXLAN header, and the TX switch pushing a different S-tag (3110 instead of 3101) on the way back - that S-tag tells TRex at which AP the frame left the fabric.

# Link or device The frame What happens
1 TRex p0 to sw-rzk-TX1 XGE1/1/21 DA 02:54:52:06:05:22, SA 02:54:52:01:05:22, S 3101, C 1314, IP 10.3.14.101 to .106, UDP to 32001, TTL 64, 128 B The payload ends with the 16-byte latency signature (internal ID, sequence number, TX timestamp)
2 sw-rzk-TX1, VLAN 3101 unchanged DA learned on XGE1/1/1 (from the reverse stream): the frame goes there
3 XGE1/1/1 to sw-rzk-L11 XGE1/3/1 S-tag removed: C 1314, 124 B An ordinary tagged server frame
4 sw-rzk-L11 (VTEP 172.31.0.241) VXLAN, outer IP 172.31.0.241 to 172.31.0.254, UDP 4789, VNI 1314; inner frame untagged; 170 B without FCS Service instance 1314 maps it to VSI VPN-1314; the DA is known from an EVPN route with next hop 172.31.0.254
5 sw-rzk-S1 or S2 outer MACs rewritten, outer TTL - 1 Routes the outer packet; the inner frame is untouched
6 sw-rzk-BL decapsulated, tag 1314 added The DA is on the local AP port XGE1/3/6
7 XGE1/3/6 to sw-rzk-TX1 XGE1/1/6 C 1314, 124 B qinq enable pushes PVID 3110
8 sw-rzk-TX1 to TRex p0 S 3110, C 1314, 128 B, TTL 64 The RX core matches the signature and computes the latency

Nothing in the inner packet changes - same MACs, same TTL, same payload. That makes L2 the cleanest measurement of the fabric itself.

Packet walk inter-DC: K-L11 to C-L11

The first hops are identical (S 3101, C 1314); the destination is now the C-L11 client 02:54:52:07:05:22:

  1. sw-rzk-L11 finds the DA behind 172.31.0.254 and tunnels the frame to sw-rzk-BL.
  2. sw-rzk-BL decapsulates, finds the DA behind the VXLAN-DCI tunnel to 172.31.1.254 and re-encapsulates. The packet leaves over Route-Aggregation1 (2 x 40G, MACsec) - the single DCI crossing of this stream.
  3. sw-rzc-BL re-encapsulates towards sw-rzc-L11, which adds tag 1314 and sends the frame to its AP.
  4. sw-rzc-TX1 pushes S 3201 and forwards the frame over the TX-to-TX fibre (HGE1/2/1 on sw-rzk-TX1) to sw-rzk-TX1, which delivers it to TRex p1 (S 3201 / C 1314).

The reverse stream starts on p1 and crosses the TX-to-TX fibre on the way out. Each direction crosses the DCI once and the TX-to-TX fibre once.

What the latency contains

Stream type TX switch transits Fabric switch transits DCI TX-to-TX fibre
L2 intra-RZK (K-Lxx to K-BL) 2 3 (Leaf, spine, BL) 0 0
L2 intra-RZC (C-Lxx to C-BL) 4 3 0 2
L2 inter-DC 3 6 (Leaf, spine, BL, BL, spine, Leaf) 1 1

Expected orders of magnitude, to be replaced by the measured baseline: tens of microseconds for intra-RZK L2; every DCI or TX-fibre crossing adds about 5 µs per kilometre. The examined 5950s forward store-and-forward (cut-through enable is not configured and is disabled by default, L2CG PDF p. 377), so each hop adds the serialisation time of the frame - about 0.1 µs for 128 bytes on a 10G link.

What it needs and what breaks it

  • Configuration: the VLAN on both AP ports (service instance, port trunk permit vlan), the VSI on the Leaf (VPN-1314 and VPN-1315 exist on all RZK and RZC Leafs according to the IMET tables of February 2025; for any other VLAN check first, because not every VSI exists on every Leaf), the S-VLANs on the TRex trunks and, for RZC APs, on the TX-to-TX trunk. No gateway, no firewall rule.
  • MAC learning: every client also sends (the reverse stream), so its MAC is learned at its AP and distributed by EVPN. An unknown destination MAC is flooded in the VSI - still delivered, but copied to every port of the VLAN, for VLAN 1314/1315 including the DC firewall via BAGG31 on both Border Leafs.
  • Typical failures: AP link or IRF member down (all streams of that AP), missing service instance or trunk permission after a change (one VLAN only), EVPN or DCI problems (inter-DC streams only), duplicate addresses if pools are not reserved.

The TX-to-TX fibre offset

There is one TRex server, and it stands in RZK. Every frame starts and ends in RZK - even a stream between two RZC APs. Watch in the animation how often each stream crosses the TX-to-TX fibre.

Stream between Crosses the TX-to-TX fibre When
RZK AP and RZK AP never -
RZK AP to RZC AP once on the way back to TRex
RZC AP to RZK AP once on the way out to the source AP
RZC AP to RZC AP twice out and back
Grid of source DC versus destination DC showing the number of TX-to-TX fibre and DCI crossings for L2 and L3-BL streams
Fibre (F) and DCI (D) crossings per direction. The gateway in Kalk makes every routed RZC stream cross the DCI; the TRex location in Kalk makes every RZC endpoint cross the TX-to-TX fibre.

Light in fibre needs about 5 µs per kilometre. An intra-RZC stream therefore shows two one-way TX-fibre delays (plus two extra TX-switch transits) more than the same stream type in RZK, although both fabrics are built alike. That is the geometry of the measurement, not a fabric problem.

Calibrating the offset

illustrative values Intra-RZK L2 baseline 30 µs, intra-RZC L2 baseline 182 µs. The difference of 152 µs equals 2 x (one-way TX-fibre delay + one sw-rzc-TX1 transit). With a transit of a few µs, the one-way fibre delay is roughly 74 µs - about 15 km. Every RZK-RZC stream then contains about 74 µs and every RZC-RZC stream about 148 µs of measurement-path fibre.

Two rules follow: compare each stream with its own baseline and report absolute values together with the baseline instead of silently subtracting an offset; and remember that an inter-DC L2 stream crosses the DCI once per measured direction and the TX-to-TX fibre once - with similar fibre routes, roughly half of its latency is measurement-path fibre.

Why not a second TRex server in RZC?

One-way latency needs TX and RX on the same clock. With one server the offset is a constant you calibrate once; two servers would need a time base synchronised to microseconds for cross-DC streams - a much harder problem.

Class L3-BL - routing on the centralised gateway

The VPN-CORE gateways of the test VLANs exist only on sw-rzk-BL; sw-rzc-BL has no server-VLAN gateways. L3-BL streams measure this centralised routing hop and its asymmetry between the data centres. verified Existing configuration, sw-rzk-BL running configuration (17 Feb 2025), Vsi-interface915 is built the same way:

vsi VPN-1314
 description Tier 3 Services DataCenter
 gateway vsi-interface 914
 vxlan 1314
#
interface Vsi-interface914
 description Performance-Test VLAN 1314
 ip binding vpn-instance VPN-CORE
 ip address 10.3.14.1 255.255.255.0
Line What it does Why it matters for TRex
gateway vsi-interface 914 Makes Vsi-interface914 the gateway of VSI VPN-1314 Frames to the gateway MAC are routed here and nowhere else
ip binding vpn-instance VPN-CORE Places the interface in VPN-CORE L3-BL is "L3 within one VRF"
ip address 10.3.14.1 ... Gateway address EMU resolves it by ARP for every VLAN-1314 client

Packet walk: K-L11 (VLAN 1314) to K-L21 (VLAN 1315)

  1. TRex p0 sends from the K-L11 client: DA = the gateway MAC that EMU resolved for 10.3.14.1 (ec9b-8b67-37b8 in February 2025), S 3101 / C 1314, IP 10.3.14.101 to 10.3.15.103, TTL 64, UDP destination 32002.
  2. sw-rzk-L11 knows the gateway MAC from an EVPN route flagged as gateway (BG, next hop 172.31.0.254, its display evpn route mac of February 2025) and tunnels the frame to sw-rzk-BL with VNI 1314.
  3. sw-rzk-BL decapsulates; the DA is its own MAC, so it routes in VPN-CORE: 10.3.15.103 is on the direct network of Vsi915, the ARP entry (answered by EMU) gives 02:54:52:03:05:23. It writes a new Ethernet header (DA = client, SA = its own MAC), decrements the TTL to 63 and encapsulates with VNI 1315 towards sw-rzk-L21.
  4. sw-rzk-L21 adds tag 1315; sw-rzk-TX1 pushes S 3103 and delivers the frame to TRex p0 (S 3103 / C 1315, TTL 63).
The L3-BL frame before and after sw-rzk-BL: destination and source MAC swapped to the client and gateway MACs, TTL 64 to 63, VNI 1314 to 1315
What the routing hop changes. Everything that identifies the stream for TRex - IP addresses, UDP ports, the IPv4 ID marker, the latency signature - stays the same.

In the LVR platform: the service loopback on the Border Leafs

A 5950 4-slot switch (JH404A) acting as EVPN gateway needs a service loopback group of the VSI-gateway type (EVPN CG PDF p. 28); such a group consists of ports that loop packets sent out by the device back into it (L2CG PDF p. 375). Both Border Leafs have service-loopback group 1 type vsi-gateway with HundredGigE1/1/8 and HundredGigE2/1/8. The routing hop of L3-BL and L3-ENTFW includes this internal recirculation; its cost is part of what you measure.

RZC sources: the centralised gateway effect

Take C-L21 (VLAN 1314, 10.3.14.110) to C-L22 (VLAN 1315, 10.3.15.111). Both clients are in Chorweiler, their gateway is in Kalk:

  1. TRex p1 sends with S 3204 / C 1314 over the TX-to-TX fibre (crossing 1) to SW-RZC-L21.
  2. The frame goes via sw-rzc-BL over the DCI (crossing 1) to the gateway on sw-rzk-BL.
  3. sw-rzk-BL routes it into VLAN 1315 (TTL 63) and sends it back over the DCI (crossing 2) to sw-rzc-BL and sw-rzc-L22.
  4. sw-rzc-TX1 pushes S 3205 and sends it over the TX-to-TX fibre (crossing 2) to TRex p1.

An RZC-to-RZC L3-BL stream crosses the DCI and the TX-to-TX fibre twice each; a stream between an RZC and an RZK AP crosses each once; an RZK-to-RZK stream crosses neither. The difference is the price of the centralised gateway for Chorweiler - exactly what this class shows.

Confirm during bring-up: does the gateway answer ARP from behind a Leaf?

confirm on device For a centralised EVPN gateway the HPE EVPN configuration guide requires ARP flood suppression on the VTEPs: with remote ARP learning disabled, the gateway does not answer ARP requests for its MAC that arrive through VXLAN tunnels (HPE FlexFabric 5950 EVPN Configuration Guide R630x, PDF p. 28). No VSI in the examined configurations has arp suppression enable, and vxlan tunnel arp-learning disable is not configured, so remote ARP learning is at its default. In February 2025 the ARP table of sw-rzk-BL held entries learned through the DCI tunnel to sw-rzc-BL - hosts in RZC resolve the RZK gateways through a VXLAN tunnel - but none learned through the tunnels of the RZK Leafs. Confirm in rollout phase 3 that an EMU client behind a Leaf AP (for example K-L11 in VLAN 1314) resolves 10.3.14.1 (dgw.resolve true, Chapter 6). Fallbacks: arp suppression enable in VSI view (HPE FlexFabric 5950 VXLAN Command Reference R630x, PDF pp. 5-6; disabled by default - a production change on the Leaf VSIs) or running the L3 classes from the Border Leaf APs.

Confirm during bring-up: does every RZC Leaf know the gateway MAC?

confirm on device In February 2025 sw-rzk-L11 had the gateway MAC ec9b-8b67-37b8 in VSI VPN-1314 (flags BG) and sw-rzc-BL had it behind its DCI tunnel, but the EVPN MAC table of SW-RZC-L21 (Release 6205P03) had no entry for it. If an RZC Leaf does not know the gateway MAC, routed frames of RZC clients are unknown unicast in the RZC part of the VSI: they still reach the gateway via sw-rzc-BL, but copies are flooded to the other RZC VTEPs and leave through every RZC AP carrying VLAN 1314 - TRex then sees duplicates (RX above TX, dup counters). Check display evpn route mac vsi VPN-1314 (EVPN CR PDF p. 23) on every RZC Leaf first.

What it needs and what breaks it

  • Configuration: VLANs 1314 and 1315 on both APs; the gateways exist. Vsi914/915 relay DHCP (servers 10.10.252.252 and 10.10.254.254), so the DHCP scopes must exclude the TRex pools.
  • Verification on sw-rzk-BL: display arp vpn-instance VPN-CORE (L3 IP Services CR PDF p. 33) lists the TRex clients; display evpn route arp vpn-instance VPN-CORE (EVPN CR PDF p. 20) shows the gateway entries with flags GL.
  • Typical failures: gateway not resolved by EMU (the stream cannot be built - see the ARP caveat above), a lost ARP entry for a destination client, BL IRF events (all L3-BL streams), DCI problems (RZC pairs only), the unknown gateway MAC above.

Class L3-DCFW - routing through the DC firewall

Most server VLANs at LVR have their default gateway on the DC firewall CPF-ZV05/06, attached to both Border Leafs via Bridge-Aggregation31 (--> Channel to DC-FW Vlan-all). L3-DCFW streams go from production VLAN A to production VLAN B, both with their gateway on the DC firewall. The VLANs are not chosen yet; test-oriented candidates on BAGG31 of both BLs are 171, 176 "DC-Testnetz RZ Deutz/RZ-Chorweiler", 724 "Test Server Netz" and 1395/1396 "t3_139x_ngfw-test" decision. Not every VSI exists on every Leaf: VPN-171 and VPN-724 are configured on sw-rzk-L11 but not on SW-RZC-L21 (running configurations, February 2025), so check every Leaf of the chosen pairs before the VLAN is added to an AP.

Packet walk: K-L11 (VLAN A) to K-L41 (VLAN B)

  1. TRex p0 sends from the K-L11 client of VLAN A: DA = the MAC that EMU resolved for the DC-FW address of VLAN A, S 3101 / C A, TTL 64, UDP destination 32003.
  2. sw-rzk-L11 finds the firewall MAC behind sw-rzk-BL. If the active member is attached in RZK, sw-rzk-BL sends the frame out of its BAGG31 (tag A); otherwise it crosses the DCI and sw-rzc-BL delivers it on its BAGG31.
  3. The active firewall member checks its rules, routes into VLAN B (TTL 63, new Ethernet header) and returns the packet on the same bundle with tag B.
  4. That Border Leaf finds the client behind sw-rzk-L41 - locally, or across the DCI if the firewall was in RZC - and the frame reaches TRex p0 with S 3105 / C B.

In the LVR platform: where the active member was in February 2025

The DC-FW address 10.9.0.35 on the VPN-CORE transfer VSI 4004 (next hop of all "zur DC-FW" static routes) resolved to 506b-4b24-7280 - learned locally on sw-rzc-BL and identical to the MAC of member address 10.9.0.37 (display evpn route arp on both BLs). Traffic to the firewall was delivered in Chorweiler. If the production VLANs behave the same way, the RZK pairs of L3-DCFW cross the DCI twice and the RZC pairs not at all - and a failover reverses that.

Needs: both VLANs on the APs that take part, 16 reserved addresses per VLAN, and DC-FW rules between the pools (UDP 32000-32099, ICMP, no NAT) - no new firewall interface (Chapter 15). Expect tens to hundreds of microseconds. Typical failures: a missing rule (100 % loss on L3-DCFW while L2 in the same VLANs stays clean); a failover (a step on every L3-DCFW stream at once - two DCI crossings more for the pairs in one DC, two fewer in the other); and a stale destination MAC. TRex writes the resolved firewall MAC into the stream's packet when the stream is built. In February 2025 the cluster answered with a member MAC (box above); if it does so after a failover, the new active member answers with its MAC, and both directions of every L3-DCFW pair keep sending to the old MAC until they are rebuilt. EMU re-resolves the gateway periodically, and trexctl's gateway-MAC tracker watches the resolved gateway MACs and rebuilds the affected streams with the new one (event gateway_mac_changed, Chapter 18); how quickly that happens after a failover is explained in Chapter 15. A ClusterXL virtual MAC would avoid the issue. Whether the cluster uses one must be confirmed by the firewall team, and the behaviour tested during a planned failover confirm in a maintenance window.

Class L3-ENTFW - through the Enterprise firewall

The Enterprise firewall cluster CPF-ZV01 (RZK) / CPF-ZV02 (RZC) is the hub between VPN-WLAN, VPN-WAN and VPN-CORE (transfer VSIs 4000, 4001, 4002 on Bridge-Aggregation10 of both Border Leafs; VSI 4003 on the same bundle is free). L3-ENTFW streams measure the path every inter-VRF flow takes: VPN-CORE, the routing hop on sw-rzk-BL, the transfer network Vsi40 (10.9.0.16/29) and the firewall, into a dedicated TRex zone - the subnet 10.9.90.0/24 behind the firewall, cluster address (VIP) 10.9.90.1. The firewall zone is the existing, free transfer VSI 4003: the firewall team adds a VLAN 4003 interface on the bond that already connects both cluster members to the Border Leafs, and TRex reaches the zone through the Border Leaf attachment points K-BL and C-BL like any other VLAN. This is firewall-zone option B, which the plan recommends and uses as the default; option A - a new firewall port per member on the TX switches, the customer's original wish - remains the alternative decision (Chapter 15).

The zone client always sits at the BL AP of the source's data centre: the sources K-L11 and K-BL send to 10.9.90.106 at K-BL (p0, S 3110 / C 4003), the sources C-L21 and C-BL to 10.9.90.113 at C-BL (p1, S 3210 / C 4003).

Packet walk: K-L11 to the zone client at K-BL

Watch the animation: the frame gets a new Ethernet header twice - at the gateway on sw-rzk-BL and at the firewall - and loses one TTL each time. Then watch the way back into the zone: the firewall returns the frame over the same bundle, now tagged 4003, and sw-rzk-BL only bridges it to the K-BL attachment point. The last step shows what option A would change.

  1. TRex p0 sends from the K-L11 client: DA = the gateway MAC that EMU resolved for 10.3.14.1, S 3101 / C 1314, IP 10.3.14.101 to 10.9.90.106, TTL 64, UDP destination 32004.
  2. sw-rzk-TX1 removes S 3101; sw-rzk-L11 bridges the frame in VSI VPN-1314 and tunnels it (VNI 1314) to the VTEP that owns the gateway MAC, sw-rzk-BL.
  3. sw-rzk-BL routes in VPN-CORE towards 10.9.0.22 - today through the default route 0.0.0.0/0 (O_ASE1 via 10.9.0.22 on Vsi40), later through an explicit route for 10.9.90.0/24. New Ethernet header (DA = the MAC answering for 10.9.0.22), TTL 63, VSI VPN-4002.
  4. VSI VPN-4002 delivers the frame on BAGG10 (tag 4002) to the member that answers for 10.9.0.22 - locally to CPF-ZV01, or across the DCI to CPF-ZV02.
  5. The firewall matches the rule for the TRex objects (UDP 32000-32099), creates or refreshes the session and routes into the zone interface, a VLAN 4003 sub-interface on the same inside bond: new Ethernet header (DA = 02:54:52:06:0f:a3, the zone client at K-BL), TTL 62.
  6. The frame comes back on BAGG10 with tag 4003. The Border Leaf only bridges it in VSI VPN-4003, which has no gateway: from CPF-ZV01 it reaches sw-rzk-BL directly; from CPF-ZV02 it arrives on sw-rzc-BL and crosses the DCI back to sw-rzk-BL, where the zone client's MAC is local.
  7. sw-rzk-BL sends it out of the K-BL AP port XGE1/3/6 (service instance 4003, tag 4003); sw-rzk-TX1 XGE1/1/6 pushes S 3110, and TRex p0 receives S 3110 / C 4003 with TTL 62.

The reverse stream starts at the zone client: p0 sends S 3110 / C 4003 with DA = the MAC that answers for the zone VIP 10.9.90.1, and sw-rzk-BL bridges the frame in VSI VPN-4003 to the active member. The firewall routes it to 10.3.14.0/24, learned by OSPF from sw-rzk-BL (10.9.0.17) - the only BL with Vsi-interface914, which redistributes its connected networks (import-route direct in ospf 1) - and sw-rzk-BL routes it into VLAN 1314 towards K-L11.

RZC sources (C-L21 and C-BL) send to the zone client at C-BL. Their frames first cross the TX-to-TX fibre and the DCI to the only VLAN-1314 gateway on sw-rzk-BL. Behind the firewall, the zone leg to C-BL crosses the DCI again when CPF-ZV01 is active, or stays in RZC when CPF-ZV02 is active - then the second DCI crossing happens earlier, between sw-rzk-BL and the firewall. The frame returns to TRex p1 over the TX-to-TX fibre.

With option A, the alternative, the zone leg would not touch the fabric: the firewall would send the frame (tag 1420) out of a new zone port to the option-A port of its TX switch - CPF-ZV01 to sw-rzk-TX1 XGE1/1/7, CPF-ZV02 to sw-rzc-TX1 XGE1/1/8 and then over the TX-to-TX fibre; both ports are already configured for S-VLAN 3300 but shut down - and TRex p0 would receive S 3300 / C 1420 for the single option-A zone client 10.9.90.114 (AP FW). That removes the Border Leaf transit from the zone leg but makes the TX switches and the TX-to-TX fibre part of a firewall interface (Chapter 15).

Where the active member is matters

In February 2025 the address 10.9.0.22 resolved to 001c-7f65-ff14, the MAC of member address 10.9.0.21 behind sw-rzc-BL: traffic to the firewall was delivered to CPF-ZV02 in Chorweiler (output in Chapter 15). With the gateway in Kalk and the zone on VSI 4003 (option B), an L3-ENTFW stream crosses per direction:

Source AP in Member answering for the cluster addresses DCI crossings TX-to-TX fibre crossings
RZK CPF-ZV01 (RZK) 0 0
RZK CPF-ZV02 (RZC) 2 (to the firewall, and back on VSI 4003) 0
RZC CPF-ZV01 (RZK) 2 (to the gateway, and on the zone leg to C-BL) 2
RZC CPF-ZV02 (RZC) 2 (to the gateway, and from sw-rzk-BL to the firewall) 2

A failover of the Enterprise firewall therefore moves the RZK L3-ENTFW streams at the same moment by two DCI crossings - up when CPF-ZV02 takes over, down when CPF-ZV01 does - while L2 and L3-BL stay unchanged. The RZC L3-ENTFW streams cross the DCI twice with either member; only the leg of the second crossing changes. L3-DCFW pairs, which lie inside one DC, move in opposite directions for RZK and RZC. Both patterns point to a firewall event, not a fabric problem. With option A the pattern would differ: one DCI and one TX-fibre crossing per direction for an RZK source with CPF-ZV02 or an RZC source with CPF-ZV01, two of each for an RZC source with CPF-ZV02 - so a failover would shift all L3-ENTFW streams in the same direction.

Needs: the zone (with option B a VLAN 4003 interface on the firewall's existing inside bond and VLAN 4003 with service instance 4003 on the K-BL and C-BL attachment points), a route for it in VPN-CORE, rules in both directions, no NAT (Chapter 15). Typical failures: missing rule or route (100 % loss, often in one direction only), anti-spoofing drops, a missing service instance 4003 on one BL AP (the zone streams of that data centre fail while L2 and L3-BL at the same AP stay clean), a failover (latency step and short loss) and a stale firewall MAC after a failover; with option A also any problem of the TX switches or the TX-to-TX fibre, which would become part of a firewall interface. The stale MAC concerns the streams that start at a zone client - the reverse L3-ENTFW streams and the reverse DSCP copies - because their destination is the MAC that answers for the VIP 10.9.90.1. On the transfer network the cluster answered with a member MAC in February 2025 (001c-7f65-ff14 for 10.9.0.22). If the zone VIP behaves the same, these streams keep losing packets after a failover until trexctl's gateway-MAC tracker rebuilds them with the newly resolved MAC - how long that takes is discussed in Chapter 15; a virtual MAC avoids this. The forward streams recover by themselves, because sw-rzk-BL re-resolves 10.9.0.22 by ARP.

Class DSCP - QoS verification

The DSCP group sends one L2 pair and one L3-ENTFW pair in four copies that differ only in the TOS byte: BE (0x00, the reference), EF (0xB8, DSCP 46), AF41 (0x88, DSCP 34) and CS1 (0x20, DSCP 8). The default profile uses K-L11 to C-L21 (L2, VLAN 1314, 100 pps; group dscp-l2-interdc) and K-L11 to the zone client at K-BL (L3-ENTFW, VLAN 1314 to 4003, 50 pps; group dscp-entfw), both directions, 256-byte frames: 4 markings x 2 pairs x 2 directions = 16 latency streams. All other streams of the profile are BE (0x00).

  • Expectation: the fabric configuration has no QoS trust or scheduling policy; the only QoS policy on the Border Leafs, iperf-counting, merely counts UDP port 5001 (accounting packet). Equal treatment is expected in the fabric. A difference between markings - most likely under load, during a lab test - points to a device that re-marks or queues by DSCP, for example a firewall.
  • TX switches are neutral: the S-tag's 802.1p is copied from the C-tag only with qos trust dot1p (L2CG PDF p. 224), which is not configured; the TOS byte is never touched.
  • Compare each marking with its BE copy. The BE reference has the same pair, frame size and rate, so serialisation and frame-size effects are identical; what remains is the effect of the marking. The copies still have different source ports and can be hashed onto different ECMP or LACP members, so a few microseconds of difference are not yet a QoS finding.
  • Re-marking check: capture on the RX port and read the TOS byte, for example capture record start --rx 1 --limit 200 and capture record stop --id <N> -o /tmp/dscp.pcap confirm on the bench. TRex sets bit 0 itself, so an untouched packet reads 0x01 (BE), 0xB9 (EF), 0x89 (AF41) or 0x21 (CS1); only a change of the upper six bits is a re-marking.

Class LAB - on-demand performance tests

LAB tests run on the second instance, trex-lab, on the lab ports p4-p7 (NIC2, sw-rzk-TX1 XGE1/1/9-12), each pinned to one AP (default pins K-L11, K-L41, FWZ and C-L21; FWZ is the inventory alias for the firewall zone - with option B the K-BL attachment point with VLAN 4003): RFC 2544-style throughput searches for the no-drop rate, soak tests, convergence measurements (1000 pps for 1 ms loss resolution) and ASTF tests through the Enterprise firewall (the ASTF variant uses its own configuration with p4 as client side and p6 as server side, Chapter 17).

The packet walk differs only at the TX switch. A lab port is itself a QinQ access port with the S-VLAN of its pinned AP as PVID, so TRex sends single-tagged frames: the lab port pushes the S-tag, the AP port pops it, and in the return direction the frame leaves the lab port single-tagged again. A lab client is identified differently from a monitoring client: a lab port keeps one fixed MAC (src_mac in the TRex configuration, 02:54:52:f0:00:04 for p4 ... :07 for p7) whatever it is pinned to, because TRex answers ARP for a lab port with the port MAC and the port is re-pinned over time; only its IP follows the pinned AP: lab pool start .151 + (AP index - 1), for example 10.3.14.151 for p4 at K-L11 in VLAN 1314. The lab pool .151-.166 needs a second IPAM reservation and DHCP exclusion in every VLAN used for lab tests. The lab port p6 (PVID 3110) sends VLAN 4003 frames into the zone; its lab zone client is 10.9.90.156 / 02:54:52:f0:00:06. With one VLAN tag the X710 counts flow statistics in hardware (127 streams, packet counters only) at line rate.

Production impact of LAB tests

A lab test shares the AP link with that AP's monitoring, the TX-to-TX fibre with all RZC monitoring, the DCI (2 x 40G) with production, and the firewalls are production devices. A lab frame with an unlearned destination MAC is flooded in the S-VLAN - into the trex-mon trunks - and in the production VSI. Static MAC entries for lab clients on the TX switch, learning traffic before each run and rate ramps are the mitigations; read Chapter 24 before any LAB test.

ECMP, LACP and stable 5-tuples

Almost every hop offers parallel paths: the examined Leafs have two routed uplinks, one per spine and IRF member (sw-rzk-L11 and SW-RZC-L21: HundredGigE1/1/3 to Spine-1, HundredGigE2/1/3 to Spine-2); the DCI Route-Aggregation1 has two members (HundredGigE1/2/1, HundredGigE2/2/1); each Enterprise firewall member is attached with a two-port bundle (BAGG10: Ten-GigabitEthernet1/3/5 and 2/3/5); the DC firewall with BAGG31.

Switches keep a flow on one member by hashing header fields. For VXLAN the outer UDP source port is recommended to be a hash of fields from the inner packet (RFC 7348, section 5), so the inner headers decide the spine; the exact load-sharing fields of the 5950 are not covered here. TRex keeps all fields of a stream constant, in particular the source port, so:

  • each stream stays on one path - stable latency, no reordering by load sharing, a meaningful baseline;
  • different streams spread over the members - a fault on one LAG member or uplink hits a subset of the streams of a class, a useful troubleshooting hint;
  • two streams of the same pair can take different members - one reason to compare DSCP copies with care;
  • each stream is exactly one firewall session, kept alive by its own packets (Chapter 15).

Why not randomise ports to cover all paths?

A stream with a changing 5-tuple would mix several paths in one latency value, could be reordered, and would open a new firewall session for every new 5-tuple. Path coverage comes from many streams with fixed, different 5-tuples.

Summary of the classes

Class Example stream TRex sends / receives Routing hops (TTL at RX) DCI / TX-fibre crossings Expected magnitude
L2 intra-RZK K-L11 to K-BL, 1314 S 3101/C 1314 - S 3110/C 1314 0 (64) 0 / 0 tens of µs
L2 intra-RZC C-L21 to C-BL, 1314 S 3204/C 1314 - S 3210/C 1314 0 (64) 0 / 2 intra-RZK + 2 x TX fibre
L2 inter-DC K-L11 to C-L11, 1314 S 3101/C 1314 - S 3201/C 1314 0 (64) 1 / 1 + DCI fibre + TX fibre
L3-BL K-L11 (1314) to K-L21 (1315) S 3101/C 1314 - S 3103/C 1315 1 (63) RZK 0/0, RZC 2/2 intra-DC + gateway hop
L3-DCFW K-L11 (A) to K-L41 (B) S 3101/C A - S 3105/C B 1 (63) 0/0 or 2/0 tens to hundreds of µs
L3-ENTFW K-L11 (1314) to the zone client at K-BL (4003) S 3101/C 1314 - S 3110/C 4003 2 (62) RZK 0/0 or 2/0, RZC 2/2 tens to hundreds of µs
DSCP K-L11 to C-L21, BE/EF/AF41/CS1 as copied class as copied class as copied class as copied class
LAB pinned lab ports C-tag only as tested class as tested class load-dependent
Class Needs Typical break
L2 VLAN on both APs; S-VLANs on the TRex trunks and, for RZC, on the TX-to-TX trunk AP down, missing service instance or trunk permission, DCI/EVPN for inter-DC pairs
L3-BL gateways on sw-rzk-BL (exist) gateway not resolved, BL events, unknown gateway MAC on RZC Leafs
L3-DCFW DC-FW rules, reserved pools rules, failover, stale firewall MAC in the stream (rebuild)
L3-ENTFW zone (option B: VLAN 4003 interface on the firewall, service instance 4003 on K-BL and C-BL), route, ENT-FW rules rules, route, anti-spoofing, missing service instance 4003, failover, stale VIP MAC in the reverse streams (rebuild)
DSCP nothing extra re-marking or queuing by a device
LAB change control, flood protection flooding, shared links

Key takeaways

  • A stream is defined by its source and destination client; their IP and MAC addresses follow from AP index and VLAN (02:54:52:<index>:<VLAN>), so a MAC table tells you which TRex client you are looking at.
  • On the TRex link a frame carries the S-VLAN of the AP where it enters (outbound) or leaves (inbound) the fabric; in the fabric it is VXLAN with an untagged inner frame; on the AP link it carries only the production VLAN tag.
  • L2 frames are never modified; routed frames get a new Ethernet header and lose one TTL per routing hop - 63 after sw-rzk-BL, 62 after sw-rzk-BL and the Enterprise firewall.
  • RZK Leafs reach everything in RZC through sw-rzk-BL; an inter-DC frame passes three VXLAN tunnels and crosses the DCI once per direction.
  • Because TRex stands in RZK, RZK-RZC streams cross the TX-to-TX fibre once and RZC-RZC streams twice; calibrate the offset with intra-RZK versus intra-RZC streams and judge every stream against its own baseline.
  • The centralised gateway makes routed RZC-to-RZC streams cross the DCI twice (RZC-to-RZK streams once); the active firewall member adds or removes DCI crossings (with the zone on VSI 4003, two for the RZK L3-ENTFW streams), so a failover changes the firewall streams at the same moment. Streams whose destination is a firewall MAC must be rebuilt after a failover - the job of trexctl's gateway-MAC tracker - unless the cluster uses a virtual MAC.
  • The firewall zone of the plan's default (option B) is VLAN 4003 on the existing firewall bundle; its zone clients sit at the Border Leaf APs (10.9.90.106 at K-BL, 10.9.90.113 at C-BL), and the frame reaches them through VSI VPN-4003 - the fabric is part of the zone leg.
  • TRex marks its packets itself - IPv4 ID (internal ID, not the pg_id), latency signature and TOS bit 0 (0xB8 leaves as 0xB9); the DSCP class compares EF, AF41 and CS1 with an otherwise identical BE copy.
  • Fixed 5-tuples keep each stream on one ECMP/LACP path and in one firewall session; coverage comes from many streams.

Check your understanding

1. Which tags does TRex send and receive for the L2 inter-DC stream K-L31 to C-L22 in VLAN 1314, and what does the stream cross?

It is sent on p0 with S 3104 (AP K-L31) / C 1314 and received on p1 with S 3205 (AP C-L22) / C 1314. In between the frame passes three VXLAN tunnels (K-L31 to sw-rzk-BL, sw-rzk-BL to sw-rzc-BL, sw-rzc-BL to sw-rzc-L22), crosses the DCI once and the TX-to-TX fibre once, on the way back to TRex.

2. The intra-RZC L2 streams show about 150 µs more latency than the intra-RZK L2 streams. Is the RZC fabric slower?

Not necessarily. Every intra-RZC stream crosses the TX-to-TX fibre twice and passes sw-rzc-TX1 twice, because TRex is in RZK. The difference is 2 x (one-way fibre delay + one TX-switch transit) - about 15 km of fibre here. Use it to calibrate the offset; a fabric problem shows up as a change against the stream's own baseline.

3. A capture on TRex shows an L3-BL frame from K-L11 to K-L21. Which source MAC, TTL and tags do you expect?

Source MAC = the gateway MAC of sw-rzk-BL (ec9b-8b67-37b8 in February 2025), destination MAC 02:54:52:03:05:23, TTL 63, outer tag S 3103 (AP K-L21), inner tag C 1315. IP addresses, UDP ports and the TRex payload are unchanged.

4. At 14:02:17 the four RZK L3-DCFW streams step up by about 150 µs and the four RZC L3-DCFW streams step down by about the same amount; L2 and L3-BL are unchanged. What happened, and how do you check?

The signature of a DC-firewall failover from a member in RZK to a member in RZC: the RZK pairs now cross the DCI twice to reach the firewall, the RZC pairs no longer do. On sw-rzk-BL, display evpn route arp vpn-instance VPN-CORE shows which MAC answers for 10.9.0.35 and whether it is local (DL) or learned from the other Border Leaf (B). Look for a short loss burst at the same moment, ask the firewall team for the cluster log and annotate the event. If the cluster answers with member MACs, also check that the L3-DCFW streams were rebuilt with the new firewall MAC - otherwise their loss continues after the burst. An Enterprise-firewall failover looks different: the RZK L3-ENTFW streams shift by two DCI crossings in the same direction, while the RZC L3-ENTFW streams keep their number of crossings.

5. What would happen if an RZC Leaf does not know the gateway MAC of VLAN 1314?

Routed frames from RZC clients would be unknown unicast in the RZC part of VSI VPN-1314. They would still reach the gateway, because sw-rzc-BL knows the MAC behind its DCI tunnel, but copies would be flooded to the other RZC VTEPs and out of every RZC AP carrying VLAN 1314; TRex would count duplicates. display evpn route mac vsi VPN-1314 on the RZC Leafs shows whether the MAC is known.

6. Why does every L2 pair run in both directions, even if you only care about one direction?

Besides revealing asymmetric paths, the reverse stream makes the destination client send, so its MAC is learned at its AP, advertised by EVPN and learned by the TX switch. Otherwise the MAC could age out and every frame would be flooded in the VSI - for VLAN 1314/1315 also to the DC firewall via BAGG31.

7. The four DSCP copies K-L11 to C-L21 (BE, EF, AF41, CS1) differ by up to 2 µs in average latency on an idle network, and a capture shows TOS 0xB9 for the EF copy. Is that a QoS effect?

Probably not. The copies have different source ports and can be hashed onto different spine uplinks or DCI members - the BE reference as well. A QoS effect shows up as a systematic difference between a marking and its BE copy that grows with load, or as a changed DSCP in a capture. 0xB9 is not a re-marking: TRex sets bit 0 of the TOS byte itself, the DSCP (upper six bits) is still 46.

8. CPF-ZV02 is the active member. How often does the L3-ENTFW stream K-L11 to 10.9.90.106 cross the DCI, and with which tags and TTL does TRex receive it?

Twice, and it never crosses the TX-to-TX fibre. sw-rzk-BL routes the frame into VSI VPN-4002, and the MAC answering for 10.9.0.22 is behind sw-rzc-BL, so the frame crosses the DCI to reach CPF-ZV02. The firewall returns it on its bundle with tag 4003; sw-rzc-BL bridges it in VSI VPN-4003 back over the DCI to sw-rzk-BL, where the zone client's MAC is local. TRex p0 receives it with S 3110 (AP K-BL) / C 4003 and TTL 62. After a failover to CPF-ZV01 both crossings disappear - the latency of the stream steps down.

References

  • TRex stateless guide, https://trex-tgn.cisco.com/trex/doc/trex_stateless.html, "Tutorial: Per stream statistics" and "Tutorial: Per stream latency/jitter/packet errors" (local copy artifacts/tmp/docs/trex_stateless.txt).
  • trex-core source, https://github.com/cisco-system-traffic-generator/trex-core: src/stx/stl/trex_stl_fs.cpp (IPv4 ID = reserve base + hardware ID for flow-stat streams, FLOW_STAT_PAYLOAD_IP_ID 0xFFFF for latency streams, set_tos_to_cpu()), src/bp_sim.h (TOS_GO_TO_CPU 0x1).
  • HPE FlexFabric 5950 VXLAN Configuration Guide R630x, PDF p. 7 (VXLAN packet format, UDP port 4789), PDF pp. 13-14 (VLAN access mode).
  • HPE FlexFabric 5950 VXLAN Command Reference R630x, PDF pp. 5-6 (arp suppression enable, default disabled), PDF p. 13 (display l2vpn mac-address), PDF p. 55 (xconnect vsi, default access mode VLAN).
  • HPE FlexFabric 5950 EVPN Configuration Guide R630x, PDF p. 28 (section "Configuring a centralized EVPN gateway": ARP flood suppression on the VTEPs, service loopback group of the VSI-gateway type on JH404A).
  • HPE FlexFabric 5950 EVPN Command Reference R630x, PDF p. 18 (display evpn auto-discovery), PDF p. 20 (display evpn route arp), PDF p. 23 (display evpn route mac).
  • HPE FlexFabric 5950 Layer 2 - LAN Switching Configuration Guide R630x, PDF p. 220 (QinQ tagging), p. 224 (qos trust dot1p), p. 375 (service loopback groups), p. 377 (cut-through forwarding, disabled by default).
  • HPE FlexFabric 5950 Layer 3 - IP Services Command Reference R630x, PDF p. 33 (display arp vpn-instance).
  • LVR running configurations of 17/18 Feb 2025: sw-rzk-BL (vsi VPN-1314/1315/4003, Vsi-interface914/915/40, service-loopback group 1 type vsi-gateway, bgp 65001, Route-Aggregation1, Bridge-Aggregation10/31, qos policy iperf-counting), sw-rzc-BL, sw-rzk-L11, SW-RZC-L21.
  • LVR diag outputs: diag_sw-rzk-BL_20250217-163816.txt (display evpn auto-discovery imet, display evpn route arp, display arp all - dynamic entries via Tunnel5 to sw-rzc-BL), diag_sw-rzc-BL_20250217-163904.txt (display evpn route arp, display evpn route mac), diag_sw-rzk-L11_20250217-164257.txt and diag_SW-RZC-L21_20250218-075107.txt (display evpn route mac, display evpn auto-discovery imet).
  • Plan document v0.92 (14 October 2026), sections 4.3 (traffic classes; 4.3.4 L3-ENTFW path with option B, 4.3.5 firewall-zone options), 4.6.1-4.6.3 (attachment points, VLANs, deterministic addressing and lab client identity) and 9.4 (profile groups).
  • RFC 7348, Virtual eXtensible Local Area Network (VXLAN), section 5 (outer UDP source port).