TRex Network Monitoring and Performance Test Platform

Document control

Conventions used in this document

Table of contents

1. Executive summary

LVR-InfoKom operates an HPE FlexFabric 5950 EVPN-VXLAN fabric spanning the data centres Kalk (RZK, BGP AS 65001) and Chorweiler (RZC, BGP AS 65002). The fabric has no built-in way to measure end-to-end delay, jitter and packet loss between arbitrary points, across the inter-DC link, across the Border Leaf routing functions or across the Check Point firewalls. This document describes a permanent, low-footprint measurement platform based on the open-source traffic generator Cisco TRex, plus an on-demand performance test capability, with all results stored in InfluxDB and visualised in Grafana.

The core design decisions are:

• One TRex server in RZK (HPE ProLiant, 2 x Intel X710-DA4 = 8 x 10G SFP+ DPDK ports, RHEL/Rocky 9) connected to a dedicated HPE 5950 "TX switch" in each DC. The two TX switches are joined by a dedicated dark-fibre/DWDM link so that measurements never depend on the network under test for the return path.
• Each TX switch has one 10G link to every Leaf pair, to the Border Leaf and to the local Enterprise-firewall cluster member. Each such link is an attachment point (AP). On the TX switches every AP is an isolated QinQ (802.1Q-in-802.1Q) domain: the outer tag (S-VLAN) selects the AP, the inner tag is the production VLAN. The TX switch therefore needs no production VLAN configuration at all; adding a VLAN under test only touches the Leaf port and the TRex profile. The QinQ design is validated against the 5950 R630x Layer 2 configuration guide (section 7.1).
• Two independent TRex instances on the same server: a monitoring instance (software mode, one data-plane core per port pair so that latency stays accurate, TRex EMU for ARP, QinQ frames, up to 128 latency-measuring streams plus up to 1023 loss-counting streams, continuous measurement at low packet rates) and a lab instance (hardware-assisted stateless line-rate tests or stateful ASTF firewall tests) on separate NICs, so that performance tests never interrupt the permanent monitoring.
• Traffic classes covering all requirements: L2 inside one VLAN (intra- and inter-DC), L3 within VPN-CORE routed by the Border Leaf (using the already existing "Performance-Test" VLANs 1314/1315), L3 between production VLANs routed by the DC firewall, L3 from VPN-CORE through the Enterprise firewall into a new dedicated TRex firewall zone (the inter-VRF/firewall path; two implementation options, section 4.3.5), DSCP-marked variants of the above with a best-effort reference stream, and stateful TCP tests through the Enterprise firewall.
• A project-specific Python orchestrator trexctl (specified in section 9) generates streams from a YAML inventory, provisions EMU clients, runs tests, and exports statistics to InfluxDB every two seconds. Grafana dashboards and alert rules are specified in section 10.

The expected monitoring footprint is about 21 kpps and 25 Mbit/s in total (176 streams: 74 latency streams at 50-200 pps and 102 loss-only streams at 100 pps, 128-byte packets), i.e. negligible for the fabric and the firewalls. Sections 7 and 8 contain step-by-step configuration and installation procedures, section 11 the test and operations procedures, and section 12 the phased implementation plan with acceptance criteria and open items.

2. Requirements, scope and decisions

2.1 Customer requirements (as stated)

1. TRex server with several NICs.
2. Send traffic inside multiple production VLANs to measure delay permanently and to run occasional performance tests.
3. Traffic types: Layer 2 (inside one VLAN); Layer 3 within a single VRF (default routing table or a single vpn-instance); Layer 3 between different VRFs (vpn-instances); traffic through the firewall.
4. The TRex server is connected to one of two dedicated switches; the two switches are in different DCs (RZK, RZC), one per DC, each with physical connections to every Leaf, Border Leaf and FW in its DC.
5. Integration with Grafana for visualisation.
6. Reference: the MAN Truck & Bus TRex documentation v1.3 (German) as inspiration, not a 1:1 copy.

2.2 Interpretation against the real fabric

The diagnostic files show that the fabric uses centralised Layer 3: the Leafs are pure Layer 2 VTEPs (VLAN to VSI mapping via Ethernet service instances), and all vpn-instance gateways live on the Border Leafs. Most server VLANs do not have a switch gateway at all - their default gateway is the DC firewall (CPF-ZV05/06). The default routing table (public instance) carries only the underlay (loopbacks, OSPF 172, BGP EVPN), so "L3 in the default routing table" has no end-system meaning in this fabric. The requirement list therefore maps to these measurable paths:

2.3 Decisions taken with the customer (12 October 2026)

2.4 Assumptions

• CPF-ZV01 (RZK) and CPF-ZV02 (RZC) are the two members of one stretched Check Point ClusterXL (sync VLAN 123 "Sync_FW-innen" is a stretched VSI; a single OSPF neighbour 10.9.0.22 is seen from sw-rzk-BL in VPN-CORE; in VSI VPN-4002 the Check Point MAC 001c-7f65-8ec8 is local on sw-rzk-BL and 001c-7f65-ff14 is learned via sw-rzc-BL). The same is assumed for the DC firewall CPF-ZV05/06 (sync VLAN 196). The TRex firewall zone is therefore one Layer 2 domain reachable from both DCs (option A: stretched S-VLAN 3300; option B: stretched VSI 4003). CONFIRM with the firewall team.
• Both clusters answer ARP for their virtual addresses with the MAC of the active member, not with a virtual MAC: on 2025-02-17 sw-rzk-BL resolved 10.9.0.22 (ENT-FW) to 001c-7f65-ff14 and 10.9.0.35 (DC-FW) to 506b-4b24-7280, both learned via the DCI tunnel from RZC, and its log contains ARP_DUPLICATE_IPADDR_DETECT events in which 001c-7f65-8ec8 and 001c-7f65-ff14 claimed the same address 10.9.0.22 at failover times. Consequence: TRex streams whose destination MAC is a firewall MAC must be rebuilt after a failover (section 4.3.3/4.3.4, 9.5) unless the firewall team enables a ClusterXL virtual MAC. DECISION
• The spare 5950s are 4-slot chassis (JH404A) with at least one 24-port SFP+ module. If they are 32-port QSFP28 units (JH321A), 100G ports are split into 4 x 10G breakout ports (using tengige, see 7.1.4).
• The production VLANs to be monitored and their reserved TRex address ranges are supplied by LVR IP address management; this document uses the existing test VLANs 1314/1315 plus placeholders.
• Firewall rule changes for the TRex address objects are approved by the firewall team (section 7.5).
• The 5950 series is end of sale; JH321A reached end of support life on 28.02.2026 and JH404A reaches it on 31.10.2026 (from "LVR-InfoKom - Comware Switch Analysis v1.0.xlsx"). Using spare 5950s as TX switches is acceptable for a test tool, but no HPE software support should be expected for them.

3. Environment analysis (as found)

Sources: diag_*_20250217/18.txt for sw-rzk-S1/S2, sw-rzk-L11, sw-rzk-BL, SW-RZC-S1/S2, SW-RZC-L21, sw-rzc-BL, sw-rzd-BL; "Big Picture-Rechenzentren-RZKC.pdf" (page 1); "LVR-InfoKom - Comware Switch Analysis v1.0.xlsx".

3.1 Devices and roles

3.2 Layer 2 / Layer 3 model relevant for TRex

• Access model. Every access port is a trunk with one Ethernet service instance per VLAN: service-instance N / encapsulation s-vid N / xconnect vsi VPN-N. VSIs are EVPN-VXLAN (vxlan N, route-distinguisher auto, vpn-target 1:N). Example: sw-rzk-L11 Ten-GigabitEthernet1/3/3 (ESH-ZV12 trunk) carries VLANs 147, 176, 252, 1054, 1080-1082, 1280, 1309, 1380, 1383, 1397-1398 this way. VALIDATED
• VLAN 1314 and 1315 (VLAN names t3_1314_webserv_win / t3_1315_winserv; VSI VPN-1314/1315 "Tier 3 Services DataCenter"; Vsi-interface descriptions "Performance-Test VLAN 1314/1315") exist on all switches, have gateways 10.3.14.1/24 and 10.3.15.1/24 on Vsi-interface914/915 of sw-rzk-BL bound to VPN-CORE (DHCP relay configured) and are not mapped to any access port on the Leafs examined. They are mapped as service instances on BAGG31 "Channel to DC-FW Vlan-all" of both Border Leafs, so broadcasts and unknown unicast in these VLANs also reach the DC firewall. In February 2025 the VSIs held no learned MAC addresses and the ARP tables only the gateway itself. They are the intended carriers for the permanent monitoring and the L3-BL class; LVR must confirm that they are reserved for tests and that the DC firewall has no interface in them. CONFIRM The gateway exists on sw-rzk-BL only, so RZC-sourced routed traffic always transits the DCI link to Kalk.
• VPN-CORE routing. Static routes for 10.0.0.0/16 ... 10.245.0.0/16 point to the DC firewall (10.9.0.35 on Vsi44); OSPF 1 runs on the transfer VSI 40 (10.9.0.16/29) with the Enterprise firewall (10.9.0.22), sw-rzc-BL (10.9.0.18) and sw-rzd-BL (10.9.0.19), redistributing direct and static. The routing table already contains a default route 0.0.0.0/0 O_ASE1 via 10.9.0.22, so a new subnet behind the Enterprise firewall is reachable from VPN-CORE without any switch change; an explicit OSPF advertisement by the firewall is still the cleaner choice. Routed traffic on the Border Leafs passes the VSI-gateway service-loopback group (HGE1/1/8, HGE2/1/8).
• STP. PVST is enabled; STP is disabled for most server VLANs (undo stp vlan ... enable, incl. 1082-1399 on sw-rzk-L11 and 1314/1315 on all examined switches). Server ports have no edge-port configuration.
• Centralised gateway and ARP. The HPE EVPN Configuration Guide R630x (PDF p. 28) requires ARP flood suppression on the VTEPs when a centralised EVPN gateway is used, because the gateway does not answer ARP requests that arrive through VXLAN tunnels when remote ARP learning is disabled. LVR has neither arp suppression enable nor vxlan tunnel arp-learning disable (remote ARP learning is therefore at its default). In February 2025 sw-rzk-BL held 188 dynamic ARP entries learned through the DCI tunnel from RZC but none through the RZK Leaf tunnels, and SW-RZC-L21 had no EVPN MAC route for the sw-rzk-BL gateway MAC ec9b-8b67-37b8 (sw-rzk-L11 and sw-rzc-BL have it; the RZC Leafs receive RZK MAC routes only re-originated by sw-rzc-BL, and the gateway-flagged routes are apparently not re-originated). Both points are commissioning checks for the L3 classes (section 11.2); fall-backs are to source routed streams from the Border Leaf APs or to enable ARP suppression in the affected VSIs (a production change). CONFIRM
• VSI coverage differs per Leaf. Not every VSI exists on every Leaf (for example VPN-724 and VPN-171 exist on sw-rzk-L11 but not on SW-RZC-L21). Before a VLAN is added to an attachment point, verify the VSI on that Leaf (display l2vpn vsi name VPN-<id>).
• Previous test tooling. An ACL/QoS policy iperf-counting (UDP destination port 5001 accounting) is applied inbound on BAGG31 and RAGG1 of sw-rzk-BL and on RAGG1 of sw-rzc-BL, and port FGE1/4/26 of sw-rzk-BL is labelled "ALLEGRO-TEST". This confirms earlier ad-hoc performance measurements and the need for a permanent solution.
• DCI optics. The RAGG1 members HGE1/2/1 and HGE2/2/1 of sw-rzk-BL carry 40G_BASE_ER4_QSFP_PLUS transceivers (40 km class). The RZK-RZC fibre is therefore probably longer than the 10 km reach of LR4 optics; the TX-to-TX optics must be chosen after the real distance and attenuation are known (section 5.2).
• Free 10G SFP+ ports (as of 2025-02-17, must be re-verified): sw-rzk-L11 XGE1/3/1 (default config, down); SW-RZC-L21 XGE1/4/4 (default config, down, but an HPE transceiver from 2020 is inserted - possibly reserved); sw-rzk-BL XGE1/3/6; sw-rzc-BL XGE1/3/14. Slot 2 (L11) / slot 4 (BL) modules are 10GBase-T (JH182A) and are not suitable for SFP+ cabling.
• Jumbo frames. 5950 Ethernet interfaces accept jumbo frames up to 9416 bytes by default (Layer 2 Configuration Guide, PDF p. 41), so the 4 additional QinQ bytes need no MTU change on the TX switches.
• Unused VLAN IDs. Across all nine configurations the ranges 2907-3999 and 1617-2903 are unused; 3000-3099 is reserved on the Border Leafs (reserve-vlan-interface 3000 to 3099 global). The S-VLAN plan therefore uses 3101-3399.

3.3 Lessons taken from the reference design (MAN, man_trex_documentation_v1.3)

4. Solution concept

4.1 Components and overall architecture

4.2 Attachment-point model: QinQ on the TX switches

The TX switch must connect a small number of TRex ports to many production devices without ever bridging production VLANs between two attachment points (which would create a loop through the stretched VXLAN VLANs). Comware QinQ solves this elegantly: a QinQ-enabled port adds the port PVID as an outer tag to every incoming frame, tagged or untagged, and removes it on egress (Layer 2 Configuration Guide, "Configuring QinQ", PDF pp. 219-222). By giving every AP-facing port its own PVID (S-VLAN), each AP becomes an isolated two-port bridge domain: {AP port, TRex port(s)}.

Two kinds of TRex-facing ports exist on sw-rzk-TX1:

4.3 Traffic classes and paths

4.3.1 Class L2 - Layer 2 inside one VLAN

Source and destination TRex clients are in the same production VLAN (C-VLAN) at two different APs. The destination MAC is the MAC of the remote TRex client (known from the inventory - no ARP needed; EMU gratuitous ARP keeps the remote MAC learned in the fabric). Path: TX switch - Leaf A - Spine - Leaf B (VXLAN) - TX switch. Inter-DC variant: Leaf A - Spine - sw-rzk-BL - RAGG1 (MACsec) - sw-rzc-BL - Spine - Leaf B - sw-rzc-TX1 - fibre - sw-rzk-TX1. Measures fabric forwarding, VXLAN encapsulation, DCI link and IRF behaviour.

4.3.2 Class L3-BL - Layer 3 within VPN-CORE routed on the Border Leaf

Source in VLAN 1314 (gateway 10.3.14.1 on sw-rzk-BL Vsi-interface914), destination in VLAN 1315 (10.3.15.1, Vsi-interface915), both bound to VPN-CORE. Destination MAC = gateway MAC resolved by EMU ARP. Path: Leaf A - fabric - sw-rzk-BL (routing, VXLAN re-encapsulation) - fabric - Leaf B. Sources in RZC also cross RAGG1 twice (gateway only in Kalk). Measures the centralised routing hop and its asymmetry.

4.3.3 Class L3-DCFW - Layer 3 between production VLANs routed by the DC firewall

Source in production VLAN A and destination in production VLAN B, both with their default gateway on CPF-ZV05/06 (VIP). Path: Leaf - fabric - sw-rzk-BL/sw-rzc-BL - BAGG31 "Vlan-all" - DC-FW (active member) - BAGG31 - fabric - Leaf. Requires a DC-FW rule permitting the TRex UDP port range between the TRex host objects. Measures firewall forwarding latency and its cluster state (active member location changes the path). The destination MAC of these streams is the MAC of the active cluster member (section 2.4): after a failover the stream keeps sending to the old MAC until trexctl detects the new gateway MAC and rebuilds the stream (section 9.5); the gap seen in Grafana therefore includes the rebuild time, which must be considered when failover times are read from these streams.

4.3.4 Class L3-ENTFW - VPN-CORE through the Enterprise firewall into the TRex firewall zone

Source in VLAN 1314 (VPN-CORE); destination TRex client in a new TRex firewall zone (subnet 10.9.90.0/24, VIP 10.9.90.1, proposal) which is an interface of the Enterprise-FW cluster. Two ways to build the zone are described in 4.3.5: option A - a new physical interface per cluster member cabled to the TX switches (AP "FW", S-VLAN 3300 stretched over both TX switches, FW VLAN 1420); option B - the existing, unused transfer VSI 4003 that is already trunked to both cluster members on BAGG10, reached by TRex through the Border Leaf APs like any other VLAN. Path (option B): Leaf - fabric - sw-rzk-BL (VPN-CORE) - Vsi40 transfer (10.9.0.16/29) - ENT-FW active member - VLAN 4003 interface - BAGG10 - sw-rzk-BL or sw-rzc-BL VSI VPN-4003 - fabric - BL AP (K-BL or C-BL) - TX switch - TRex. VPN-CORE already carries a default route to the firewall (10.9.0.22), so the zone is reachable without switch changes; the firewall should nevertheless advertise 10.9.90.0/24 into OSPF 1 explicitly. The zone-side streams (zone client to inside) have the firewall MAC as destination and are subject to the rebuild-after-failover rule of 4.3.3. This is the inter-VRF/firewall path requested; extending it to VPN-WAN/VPN-WLAN later only requires TRex clients in those VRFs.

4.3.5 Firewall zone design options

Both options use the same subnet, the same TRex clients (addressed by the AP-index formula of 4.6.3), the same firewall objects and rules, and the same trexctl profile group (l3-entfw); only the inventory entry of the zone differs. The remainder of this document describes option B as the default and marks the option-A specifics where they occur.

4.3.6 Class DSCP - QoS verification

Selected L2 and L3 pairs are duplicated with four TOS bytes on the same pair, size and rate: BE = 0x00 as the reference, EF = 0xB8, AF41 = 0x88 and CS1 = 0x20. The BE copy is essential: each class is compared with a stream that is identical except for the TOS byte. Because the fabric configuration contains no QoS trust or scheduling policy, equal treatment is expected; differences in latency or loss under load (lab tests) would indicate a (re)marking or queuing effect, e.g. on the firewalls. Two TRex properties must be known when captures are inspected: TRex sets bit 0 of the TOS byte on every flow-statistics and latency packet (TOS_GO_TO_CPU 0x1 in src/bp_sim.h, applied in src/stx/stl/trex_stl_fs.cpp), so the streams leave with 0x01 / 0xB9 / 0x89 / 0x21 - the DSCP (upper six bits) is unchanged, the ECN field reads 01; and the IPv4 identification field is rewritten (4.4). On the TX switches the 802.1p value of the S-tag is copied from the C-tag only when qos trust dot1p is configured (L2 Configuration Guide PDF p. 224); this is irrelevant for the measurement and not configured.

4.3.7 Class LAB - performance tests

• Stateless throughput between two pinned APs (or AP to itself through the FW zone): binary search for the no-drop rate (RFC 2544 style) with fixed frame sizes or IMIX, per direction up to 10 Gbit/s per TRex port, hardware flow statistics.
• Stateful (ASTF) client side pinned to a VPN-CORE VLAN AP, server side pinned to the FW zone AP: HTTP/HTTPS-like TCP profiles, connections per second, concurrent sessions, goodput through the Enterprise firewall.
• Convergence tests during planned maintenance (IRF member reboot, link failure, firewall failover): the monitoring instance already provides 5 ms loss resolution; for 1 ms resolution the affected streams are temporarily raised to 1000 pps.

4.4 Measurement methodology

• Per-stream statistics. TRex offers two kinds of per-stream statistics, both keyed by a user-chosen packet group id (pg_id): flow statistics (STLFlowStats: TX/RX packet and byte counters, i.e. loss) and latency statistics (STLFlowLatencyStats: the same counters plus one-way latency average/minimum/maximum, last-window maximum, RFC 3550 jitter, a logarithmic histogram and sequence-error counters). Internally TRex rewrites the IPv4 identification field: flow-statistics packets carry 0xFC00 + internal id (reserved range 0xffff - 1023 upwards in the v3.x source; the documentation still quotes 0xff00), latency packets carry 0xFFFF plus a 16-byte signature with sequence number and TX timestamp at the end of the payload. The timestamp is written per packet by the TX core without recalculating the UDP checksum; latency streams are therefore built with UDP checksum 0 (legal for IPv4) so that devices that verify UDP checksums do not drop them (CONFIRM firewall behaviour on the bench). Latency is measured one-way because TX and RX happen on the same server clock; accuracy is in the microsecond range when the RX core reads the port queue directly (ONE_QUE mode, section 4.7).
• Resolution. Loss granularity = 1 / pps: 200 pps gives 5 ms, 100 pps 10 ms, 1000 pps 1 ms. Latency streams are processed by the RX core; TRex recommends keeping the total latency-stream rate well below 5 Mpps (stateless guide, "Latency streams"). The monitoring budget (~21 kpps) is far below that.
• Fixed offsets. Every frame enters and leaves the network at the TRex server in RZK. An RZK-to-RZC or RZC-to-RZK stream crosses the TX-to-TX fibre once, an RZC-to-RZC stream twice; an inter-DC L2 stream crosses the DCI once per measured direction. Propagation delay in fibre is roughly 5 us per km (the DCI uses 40 km class optics, so expect tens of microseconds). The TX fibre contribution can be measured directly: the same stream type intra-RZC versus intra-RZK differs by twice the one-way TX-fibre delay (plus the extra TX switch hop). Report absolute values together with the measured baseline rather than subtracting.
• Direction. Streams are unidirectional; each AP pair is monitored in both directions (two pg_ids) so that asymmetric paths (gateway in Kalk only, active firewall member location) become visible.
• Stream limits. Latency streams: 128 per TRex instance, "in addition to the streams which collect per stream statistics" (stateless guide, "Per stream latency"; MAX_FLOW_STATS_PAYLOAD 128 in src/trex_defs.h). Flow-statistics streams: 1023 in --software mode (trex-mon), 127 on the X710 in hardware mode (trex-lab; MAX_FLOW_STATS_X710) where RX byte counters are not available (packet counters only; use --no-hw-flow-stat or software mode when byte counts are required). The permanent profile therefore combines latency streams for the pairs where delay matters with loss-only flow-statistics streams for coverage (4.5).
• Packet sizes. Monitoring: 128 bytes L2 without FCS (room for two tags + IP + UDP + the 16-byte TRex latency signature). DSCP class: 256 bytes. Lab: 64/128/256/512/1024/1518 or IMIX; jumbo up to 9000 bytes is possible (fabric supports 9416 by default).
• Latency statistics semantics (STL API get_pgid_stats): average is low-pass filtered (new = previous/2 + last period/2), total_max/total_min are lifetime values, last_max is the maximum of the last 0.5-second server window only, histogram holds cumulative counts per logarithmic bucket. An exporter that polls every 2 s must therefore derive the interval maximum from histogram deltas (or poll every 0.5 s); last_max alone misses three of four windows (section 9.7).

4.5 Permanent monitoring matrix and packet budget

The profile is split into latency streams (limit 128 per instance) for the pairs where delay and jitter matter, and loss-only streams (flow statistics, limit 1023) that extend the coverage to further VLANs. Both kinds report TX/RX counters and therefore loss and outage duration.

Guard rails enforced by trexctl for the permanent profile: at most 120 latency streams (hard limit 128), at most 600 loss-only streams (hard limit 1023), at most 50 kpps and 50 Mbit/s in total, at most 100 pps per stream through a firewall. About 50 latency streams remain free for ad-hoc latency measurements. The profile is a YAML file (section 9.4), so pairs, kinds and rates are adjusted without code changes. If more than ~120 latency streams are ever needed, a second monitoring instance (own 128 latency streams) on p2/p3 is the way forward; it needs bench validation because the EMU helper process is shared per host (section 11.12).

4.6 Addressing, VLAN and naming plan

4.6.1 Attachment points and S-VLANs

AP indexes 1-49 are reserved for attachment points; lab clients use index + 50 (4.6.3).

4.6.2 VLANs under test and TRex address pools

Candidate production VLANs seen in the configurations that are already test-oriented and therefore low-risk: 176 ("DC-Testnetz RZ Deutz/RZ-Chorweiler", 10.180/24-DC), 171 ("Server Testnetz und Altnetz von Deutz"), 724 ("Test Server Netz"), 1395/1396 ("t3_139x_ngfw-test"). Their subnets and gateways are not visible in the switch configurations (gateway on the DC firewall) and must be supplied by LVR; note that VPN-171 and VPN-724 do not exist on SW-RZC-L21 (3.2). TBD Every pool (monitoring and lab) needs an IPAM reservation and, where DHCP relay exists (Vsi914/915), a DHCP scope exclusion.

4.6.3 Deterministic TRex client addressing

For VLAN V with pool start P and attachment point index i (table 4.6.1): IP = P + (i - 1). MAC = 02:54:52:<ii>:<VV>:<VV> where ii is the AP index and VVVV the VLAN id in hex (locally administered, unique per AP and VLAN). Example: AP K-L11 (1) in VLAN 1314 (0x0522): 10.3.14.101, 02:54:52:01:05:22. Clients attached to the lab instance use index + 50, i.e. the lab pool .151-.166 and MACs 02:54:52:33.. onwards, so that monitoring and lab clients never collide. Examples: lab client at K-L11 in VLAN 1314 = 10.3.14.151 / 02:54:52:33:05:22; at K-BL = 10.3.14.156 / 02:54:52:38:05:22; at C-L21 = 10.3.14.160 / 02:54:52:3c:05:22; zone lab client option B (K-BL, VLAN 4003 = 0x0FA3) = 10.9.90.156 / 02:54:52:38:0f:a3, option A (index 64, VLAN 1420 = 0x058C) = 10.9.90.164 / 02:54:52:40:05:8c.

4.6.4 UDP port and pg_id plan

4.6.5 Names and management addresses

4.7 Two TRex instances on one server

4.8 Security and operational impact

• TRex data ports are DPDK devices without a kernel IP stack; only EMU answers ARP/ICMP for the configured client addresses. The server is managed exclusively via the management NIC (SSH, HTTPS). No routing or forwarding between data ports is possible.
• Control-port protection. The TRex control channels (RPC 4501/4601, publisher 4500/4600, EMU 4510, Scapy 4507) listen on all addresses without authentication; anyone who reaches them can start traffic. In addition the TRex start script trex-cfg executes systemctl stop firewalld.service on Red Hat-family systems at every start, so a firewalld-based host firewall is off whenever TRex runs. Host protection is therefore implemented with the native nftables service (allow-list of management hosts, section 8.3) and management-network ACLs; firewalld is disabled. DECISION
• TRex clients appear in the fabric as ordinary hosts (MAC/IP in EVPN). Their traffic is restricted by firewall rules to the UDP range 32000-32099 and ICMP between TRex objects. Production hosts never receive TRex traffic (destinations are always TRex clients).
• Packet budget (section 4.5) is enforced in trexctl (profile validation refuses to apply a profile above the budget without --force).
• Monitoring streams are UDP with fixed 5-tuples: one firewall session per stream; sessions are kept alive by the continuous traffic (Check Point UDP virtual session timeout, 40 s by default CONFIRM with the firewall team).
• Flooding risk of lab tests. Every S-VLAN also contains the trex-mon trunk ports, and a VLAN under test is a stretched VSI. A line-rate lab stream whose destination MAC is not learned is flooded as unknown unicast: up to 10 Gbit/s into the trex-mon ports (software RX) and into every port of that VLAN in the fabric, including the DC-firewall channel for VLAN 1314/1315. Mitigations (11.9): learn the destination before every run (reverse stream or gratuitous ARP), static MAC entries for the deterministic lab-client MACs on the TX switch lab ports, rate ramps, test VLANs only for line-rate tests. The bundled TRex NDR tool (./ndr) must never be used on this server: it connects to the default RPC port 4501 (= trex-mon) and resets all ports.
• Change management: adding a VLAN under test touches the Leaf AP port (service instance), possibly a firewall rule and the trexctl inventory - all low risk. The TX switches never need production VLAN changes.
• Failure modes: a TX switch or TX-fibre failure only affects measurements. It cannot affect production traffic with firewall-zone option B; with option A it can trigger a ClusterXL failover unless the zone interface is made non-critical (4.3.5). A TRex server failure is detected by Grafana (no data) and by systemd restart. The TX-to-TX link is shared by the monitoring and lab traffic of all RZC APs; lab tests close to its capacity disturb the RZC monitoring streams.

5. Bill of materials

5.1 TRex server

5.2 TX switches and attachment links

6. Physical design and cabling

6.1 Port assignment sw-rzk-TX1 (RZK)

Port names assume a 5950 4-slot chassis with a 24-port SFP+ module in slot 1 and a QSFP28 module in slot 2 CONFIRM. Adjust if the module layout differs.

6.2 Port assignment sw-rzc-TX1 (RZC)

6.3 Cabling and labelling rules

• Label both ends of every AP link with the AP code (e.g. TRX K-L11 / sw-rzk-L11 XGE1/3/1). Record the labels in the trexctl inventory (device_port field) - the inventory is the single source of truth and is also used to generate switch configuration snippets.
• Use one IRF member for the AP link as decided (member 1 ports in the proposals above). If a Leaf IRF member is rebooted during maintenance, the AP goes down and the monitoring shows this as "path down" for that AP - a desired effect, but it must be understood when interpreting results.
• Keep the TRex-to-TX cabling order identical to the TRex port numbering (p0 = lowest PCI address of NIC1). Verify with dpdk_setup_ports.py -s and link LEDs before the first test. Remember that p0-p7 are server-wide labels: inside the trex-lab instance p4 is port 0 and p7 is port 3 (section 4.7).
• The inter-DC TX link carries only S-VLANs; its MTU does not need adjustment (5950 default jumbo frame limit 9416 bytes). If a DWDM transponder with a lower MTU is used, keep the TRex packet size below the transponder limit minus 8 bytes.

7. Switch and firewall configuration

All Comware commands in this section follow the 5950 R630x documentation and the configuration style found in the diagnostic files. Commands marked VALIDATED use syntax already present on the customer's 5950s or documented in the cited guides; values marked CONFIRM depend on the physical port layout of the spare units or on live state.

7.1 TX switch sw-rzk-TX1

7.1.1 Preparation

1. Factory-reset the spare 5950 (restore factory-default, reboot) and upgrade to R6301P03 (same release as the Border Leafs) using the HPE release notes HPE5950CMW710R6301P03RN.pdf. CONFIRM file names on the flash.
2. Confirm the module layout (display device) and derive the interface names. The configuration below assumes a 24-port SFP+ module in slot 1 (Ten-GigabitEthernet1/1/x) and a QSFP28 module in slot 2 (HundredGigE1/2/x).
3. Connect the management port to the switch management network and apply the base configuration via console.

7.1.2 Base configuration

# --- system -------------------------------------------------------------
system-view
 sysname sw-rzk-TX1
 clock timezone COLOGNE add 01:00:00
 clock summer-time MEST 02:00:00 March last Sunday 02:00:00 October last Sunday 01:00:00
 clock protocol ntp
# management VRF, same pattern as the production switches
ip vpn-instance MGMT
 route-distinguisher 10:1
 description MGMT/Monitoring
 vpn-target 10:1 import-extcommunity
 vpn-target 10:1 export-extcommunity
quit
interface M-GigabitEthernet0/0/0
 description mgmt_oobm
 ip binding vpn-instance MGMT
 ip address <MGMT-IP-TX-K> <MASK>
quit
 ip route-static vpn-instance MGMT 0.0.0.0 0 <MGMT-GW> description mgmt-DEFAULT-GATEWAY
 ntp-service enable
 ntp-service source M-GigabitEthernet0/0/0
 ntp-service unicast-server 10.0.31.21 vpn-instance MGMT
 ntp-service unicast-server 10.0.31.23 vpn-instance MGMT
 info-center loghost vpn-instance MGMT 10.0.61.10
# SNMPv3 / SSH / local users: copy the LVR standard from an existing Leaf
# (prefer SNMPv3 as recommended in the switch analysis)
 ssh server enable
 lldp global enable
# --- no spanning tree on the TX switch (see 4.2) -------------------------
 undo stp global enable
# --- S-VLANs: one per attachment point, plus the stretched FW zone -------
 vlan 3101 to 3110
 vlan 3201 to 3210
# (bulk creation stays in system view; naming requires VLAN view)
vlan 3101
 name TRX-AP-K-L11
 quit
vlan 3102
 name TRX-AP-K-L12
 quit
vlan 3103
 name TRX-AP-K-L21
 quit
vlan 3104
 name TRX-AP-K-L31
 quit
vlan 3105
 name TRX-AP-K-L41
 quit
vlan 3110
 name TRX-AP-K-BL
 quit
vlan 3300
 name TRX-AP-FW-ZONE-stretched
 quit

QinQ, STP and VLAN commands: 5950 Layer 2 Configuration Guide R630x, PDF pp. 219-222 (QinQ), p. 306 (undo stp global enable); info-center loghost [ vpn-instance name ] ip: Network Management and Monitoring Command Reference R630x, PDF p. 440 (the production switches log in the public instance: info-center loghost 10.0.61.10); ip binding vpn-instance MGMT on M-GigabitEthernet0/0/0 and the NTP/VPN pattern as on sw-rzk-BL and sw-rzc-BL (sw-rzk-L11 uses ip address dhcp-alloc without a VPN binding on its management port). VALIDATED

7.1.3 Attachment-point ports (QinQ access)

# one isolated S-VLAN per attachment point; QinQ pushes the PVID on ingress and pops it on egress
interface Ten-GigabitEthernet1/1/1
 description TRX AP K-L11 --> sw-rzk-L11 XGE1/3/1
 port link-type access
 port access vlan 3101
 qinq enable
quit
interface Ten-GigabitEthernet1/1/2
 description TRX AP K-L12 --> sw-rzk-L12 <PORT>
 port link-type access
 port access vlan 3102
 qinq enable
quit
interface Ten-GigabitEthernet1/1/3
 description TRX AP K-L21 --> sw-rzk-L21 <PORT>
 port link-type access
 port access vlan 3103
 qinq enable
quit
interface Ten-GigabitEthernet1/1/4
 description TRX AP K-L31 --> sw-rzk-L31 <PORT>
 port link-type access
 port access vlan 3104
 qinq enable
quit
interface Ten-GigabitEthernet1/1/5
 description TRX AP K-L41 --> sw-rzk-L41 <PORT>
 port link-type access
 port access vlan 3105
 qinq enable
quit
interface Ten-GigabitEthernet1/1/6
 description TRX AP K-BL --> sw-rzk-BL XGE1/3/6
 port link-type access
 port access vlan 3110
 qinq enable
quit
interface Ten-GigabitEthernet1/1/7
 description TRX AP FW-ZONE --> CPF-ZV01 <FW-PORT>
 port link-type access
 port access vlan 3300
 qinq enable
quit
# unused ports administratively down
interface range Ten-GigabitEthernet1/1/8 Ten-GigabitEthernet1/1/13 to Ten-GigabitEthernet1/1/20
 shutdown
quit

7.1.4 TRex-facing ports

# multiplexed monitoring ports: plain 802.1Q trunk, frames stay double-tagged towards TRex
interface Ten-GigabitEthernet1/1/21
 description TRX p0 trex-mon (double-tagged, all APs)
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 3101 to 3110 3201 to 3210 3300
quit
interface Ten-GigabitEthernet1/1/22
 description TRX p1 trex-mon (double-tagged, all APs)
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 3101 to 3110 3201 to 3210 3300
quit
interface Ten-GigabitEthernet1/1/23
 description TRX p2 trex-mon reserve
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 3101 to 3110 3201 to 3210 3300
 shutdown
quit
interface Ten-GigabitEthernet1/1/24
 description TRX p3 trex-mon reserve
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 3101 to 3110 3201 to 3210 3300
 shutdown
quit
# pinned lab ports: QinQ access, PVID selects the attachment point; TRex sends single-tagged frames
interface Ten-GigabitEthernet1/1/9
 description TRX p4 trex-lab pinned [K-L11]
 port link-type access
 port access vlan 3101
 qinq enable
quit
interface Ten-GigabitEthernet1/1/10
 description TRX p5 trex-lab pinned [K-BL]
 port link-type access
 port access vlan 3110
 qinq enable
quit
interface Ten-GigabitEthernet1/1/11
 description TRX p6 trex-lab pinned [FW-ZONE]
 port link-type access
 port access vlan 3300
 qinq enable
quit
interface Ten-GigabitEthernet1/1/12
 description TRX p7 trex-lab pinned [C-L21]
 port link-type access
 port access vlan 3204
 qinq enable
quit

7.1.5 Inter-DC trunk

interface HundredGigE1/2/1
 description TRX inter-DC --> sw-rzc-TX1 HGE1/2/1 (dark fibre)
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 3201 to 3210 3300
quit
save force

If the inter-DC path is 10G, use a Ten-GigabitEthernet port with the same VLAN configuration. If 40G is used on a QSFP28 port, no breakout is needed (FortyGigE transceiver in a 100G port is supported on the 8-port QSFP28 module; confirm with the installation guide for the module in use CONFIRM).

7.1.6 Variant: spare units are 5950 32QSFP28 (JH321A)

Split 100G ports into four 10G breakout ports with a QSFP28-to-4xSFP+ breakout cable (Layer 2 Configuration Guide R630x, "Splitting a 100-GE interface and combining 10-GE breakout interfaces", PDF pp. 37-38). The resulting interfaces are named Ten-GigabitEthernet1/0/N:1 ... :4 and support the same QinQ configuration. VALIDATED (syntax) / CONFIRM (cable type per installation guide).

interface HundredGigE1/0/1
 using tengige
quit
interface Ten-GigabitEthernet1/0/1:1
 description TRX AP K-L11 --> sw-rzk-L11 XGE1/3/1
 port link-type access
 port access vlan 3101
 qinq enable
quit

7.2 TX switch sw-rzc-TX1

Identical base configuration with sysname sw-rzc-TX1, its own management address and the RZC S-VLANs:

 vlan 3201 to 3210
vlan 3201
 name TRX-AP-C-L11
 quit
vlan 3202
 name TRX-AP-C-L12
 quit
vlan 3203
 name TRX-AP-C-L13
 quit
vlan 3204
 name TRX-AP-C-L21
 quit
vlan 3205
 name TRX-AP-C-L22
 quit
vlan 3206
 name TRX-AP-C-L23
 quit
vlan 3210
 name TRX-AP-C-BL
 quit
vlan 3300
 name TRX-AP-FW-ZONE-stretched
 quit
interface Ten-GigabitEthernet1/1/1
 description TRX AP C-L11 --> sw-rzc-L11 <PORT>
 port link-type access
 port access vlan 3201
 qinq enable
quit
# ... XGE1/1/2-6 analogously with 3202-3206 ...
interface Ten-GigabitEthernet1/1/4
 description TRX AP C-L21 --> SW-RZC-L21 XGE1/4/4
 port link-type access
 port access vlan 3204
 qinq enable
quit
interface Ten-GigabitEthernet1/1/7
 description TRX AP C-BL --> sw-rzc-BL XGE1/3/14
 port link-type access
 port access vlan 3210
 qinq enable
quit
interface Ten-GigabitEthernet1/1/8
 description TRX AP FW-ZONE --> CPF-ZV02 <FW-PORT>
 port link-type access
 port access vlan 3300
 qinq enable
quit
interface HundredGigE1/2/1
 description TRX inter-DC --> sw-rzk-TX1 HGE1/2/1 (dark fibre)
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 3201 to 3210 3300
quit
interface range Ten-GigabitEthernet1/1/9 to Ten-GigabitEthernet1/1/24
 shutdown
quit
save force

7.3 Leaf attachment ports

The Leaf port is configured exactly like an existing server trunk: 802.1Q trunk, one Ethernet service instance per VLAN under test, cross-connected to the existing VSI. The pattern is taken from sw-rzk-L11 Ten-GigabitEthernet1/3/3 (running configuration 2025-02-17). VALIDATED Only the VLAN list differs per AP and is generated by trexctl inventory render-leaf (section 9.6).

7.3.1 Template

interface <LEAF-PORT>
 port link-mode bridge
 description TRX AP <AP-CODE> --> <TX-SWITCH> <TX-PORT> (TRex attachment point)
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 1314 to 1315 <PROD-VLAN-A> <PROD-VLAN-B>
 stp edged-port
 #
 service-instance 1314
  encapsulation s-vid 1314
  xconnect vsi VPN-1314
 #
 service-instance 1315
  encapsulation s-vid 1315
  xconnect vsi VPN-1315
 #
 service-instance <PROD-VLAN-A>
  encapsulation s-vid <PROD-VLAN-A>
  xconnect vsi VPN-<PROD-VLAN-A>
 #
 service-instance <PROD-VLAN-B>
  encapsulation s-vid <PROD-VLAN-B>
  xconnect vsi VPN-<PROD-VLAN-B>

stp edged-port (L2 Configuration Guide PDF p. 294) makes the port forward immediately in the PVST-enabled VLANs. It is not used on the existing server ports; if the team prefers to stay consistent, omit it and accept a 30 s forwarding delay after link-up. Do not enable BPDU guard on these ports unless the BPDU behaviour of the TX switch has been confirmed (section 4.2).

7.3.2 Example: sw-rzk-L11 (AP K-L11)

system-view
interface Ten-GigabitEthernet1/3/1
 port link-mode bridge
 description TRX AP K-L11 --> sw-rzk-TX1 XGE1/1/1 (TRex attachment point)
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 1314 to 1315
 stp edged-port
 #
 service-instance 1314
  encapsulation s-vid 1314
  xconnect vsi VPN-1314
 #
 service-instance 1315
  encapsulation s-vid 1315
  xconnect vsi VPN-1315
quit
save force

Prerequisite check before configuring: the VSIs exist on the Leaf (display l2vpn vsi name VPN-1314 verbose; not every VSI exists on every Leaf, e.g. VPN-724 and VPN-171 are missing on SW-RZC-L21) and the port is really unused (display interface Ten-GigabitEthernet1/3/1, display current-configuration interface Ten-GigabitEthernet1/3/1). CONFIRM

7.3.3 Example: SW-RZC-L21 (AP C-L21)

interface Ten-GigabitEthernet1/4/4
 port link-mode bridge
 description TRX AP C-L21 --> sw-rzc-TX1 XGE1/1/4 (TRex attachment point)
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 1314 to 1315
 stp edged-port
 #
 service-instance 1314
  encapsulation s-vid 1314
  xconnect vsi VPN-1314
 #
 service-instance 1315
  encapsulation s-vid 1315
  xconnect vsi VPN-1315
quit
save force

SW-RZC-L21 runs R6205P03. The commands are identical on that release (its own running configuration uses the same service-instance/xconnect vsi syntax), but the switch analysis recommends upgrading it to R6301P03 anyway.

7.4 Border Leaf attachment ports and optional routes

7.4.1 sw-rzk-BL (AP K-BL)

interface Ten-GigabitEthernet1/3/6
 port link-mode bridge
 description TRX AP K-BL --> sw-rzk-TX1 XGE1/1/6 (TRex attachment point)
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 1314 to 1315 4003
 stp edged-port
 #
 service-instance 1314
  encapsulation s-vid 1314
  xconnect vsi VPN-1314
 #
 service-instance 1315
  encapsulation s-vid 1315
  xconnect vsi VPN-1315
 #
# firewall zone option B: the existing transfer VSI 4003 ("Transfer ENTFW *frei*") becomes the TRex zone
 service-instance 4003
  encapsulation s-vid 4003
  xconnect vsi VPN-4003
quit

sw-rzk-BL already uses this access style (e.g. BAGG11 "CPF-ZV01 - MAIN - DMZ" with service instances 125, 137, ...). VALIDATED Note that the BL runs hardware-resource vxlan border; adding two more service instances is far below any resource limit.

7.4.2 sw-rzc-BL (AP C-BL)

interface Ten-GigabitEthernet1/3/14
 port link-mode bridge
 description TRX AP C-BL --> sw-rzc-TX1 XGE1/1/7 (TRex attachment point)
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 1314 to 1315 4003
 stp edged-port
 #
 service-instance 1314
  encapsulation s-vid 1314
  xconnect vsi VPN-1314
 #
 service-instance 1315
  encapsulation s-vid 1315
  xconnect vsi VPN-1315
 #
 service-instance 4003
  encapsulation s-vid 4003
  xconnect vsi VPN-4003
quit

VSI VPN-4003 exists on both Border Leafs (vxlan 4003, no gateway interface) and is already a service instance on BAGG10 towards each Enterprise-firewall member. With option A the 4003 entries are omitted.

7.4.3 Optional: static routes for scalable ASTF tests (sw-rzk-BL only)

Only required for the stateful test variant with routed client ranges (section 11.7, option B). The TRex ASTF client port (lab port p4 pinned to a VLAN-1314 AP) gets 10.3.14.151; the client range 10.9.92.0/24 is routed to it. OSPF 1 already redistributes static routes (import-route static), so the Enterprise firewall learns the return path automatically.

 ip route-static vpn-instance VPN-CORE 10.9.92.0 24 10.3.14.151 description TRex-ASTF-clients

Syntax identical to the existing VPN-CORE static routes on both Border Leafs. VALIDATED Remove the route after the test campaign (undo ip route-static vpn-instance VPN-CORE 10.9.92.0 24 10.3.14.151).

7.5 Firewall requirements (Check Point)

Firewall configuration is performed by the LVR firewall team in SmartConsole/Gaia; this section lists the required objects and rules. Object names are proposals.

7.5.1 Enterprise firewall cluster CPF-ZV01/ZV02 - TRex zone

7.5.2 DC firewall cluster CPF-ZV05/ZV06 - inter-VLAN streams

Class L3-DCFW needs rules between the TRex pools of the selected production VLANs: TRX-mon-<VLAN-A> <-> TRX-mon-<VLAN-B>, services TRX-udp and ICMP, Accept, Log; no NAT. No interface changes are needed on the DC firewall.

7.6 Verification commands

7.7 Rollback

• Leaf/BL: undo service-instance N for each instance, then default on the interface (or restore the saved pre-change interface block), save force. The VSIs themselves are untouched.
• TX switches: stand-alone devices; power-off is a complete rollback for the fabric.
• Firewall: disable the TRex rules; remove the zone interface (option B: the VLAN 4003 sub-interface; option A: the new physical interface).
• Always take display current-configuration snapshots before and after a change and store them in the project archive (/data/trexctl/archive/).

8. TRex server installation

Target: Rocky Linux 9 / RHEL 9 (minimal), TRex v3.08, two TRex instances, Python 3.12 virtual environment for trexctl. Commands are shown for Rocky 9; RHEL 9 is identical apart from subscription handling. Everything in this section runs as root unless stated otherwise.

8.1 Firmware and BIOS (iLO / RBSU)

8.2 Operating system installation

• Rocky Linux 9 minimal ISO, language English, time zone Europe/Berlin, hostname trex-rzk-01.
• Partitioning: RAID 1 SSD pair 1: /boot/efi 1 GB, /boot 2 GB, / 120 GB (xfs), swap 16 GB; RAID 1 SSD pair 2: /data (xfs, whole device) for InfluxDB, TRex releases, logs and archives.
• Network: management NIC only (static <MGMT-IP-SERVER>); the X710 ports are left unconfigured (they will be bound to DPDK).
• Create an administrative user in group wheel; root login via SSH disabled.

8.3 Base configuration

# time, NTP (same servers as the switches), tuned profile for low latency
dnf -y update
dnf -y install chrony tuned pciutils numactl tmux git wget tar bzip2 vim-enhanced \
    python3.12 python3.12-pip kernel-devel-$(uname -r) kernel-headers gcc make \
    elfutils-libelf-devel zlib-devel nftables policycoreutils-python-utils
sed -i 's/^pool .*/server 10.0.31.21 iburst\nserver 10.0.31.23 iburst/' /etc/chrony.conf
systemctl enable --now chronyd
tuned-adm profile network-latency

# host firewall: nftables, NOT firewalld. TRex's start script (scripts/trex-cfg) runs
# "systemctl stop firewalld.service" on Red Hat-family systems at every start, and the TRex
# control ports (4500/4501, 4600/4601, 4507, 4510) have no authentication.
systemctl disable --now firewalld 2>/dev/null; dnf -y remove firewalld
cat >/etc/sysconfig/nftables.conf <<'EOF'
flush ruleset
table inet filter {
  set mgmt_hosts {
    type ipv4_addr; flags interval
    elements = { <MGMT-PREFIX>, <JUMPHOST-IP> }
  }
  chain input {
    type filter hook input priority 0; policy drop;
    iif lo accept
    ct state established,related accept
    ct state invalid drop
    ip protocol icmp accept
    # SSH and Grafana from the management hosts only
    ip saddr @mgmt_hosts tcp dport { 22, 443 } accept
    # TRex control channels (RPC, publisher, EMU, Scapy) only from the management hosts
    ip saddr @mgmt_hosts tcp dport { 4500, 4501, 4507, 4510, 4600, 4601 } accept
    # InfluxDB listens on 127.0.0.1 only (10.1); nothing to open here
    counter drop
  }
  chain forward { type filter hook forward priority 0; policy drop; }
  chain output  { type filter hook output  priority 0; policy accept; }
}
EOF
systemctl enable --now nftables
nft list ruleset | head -30

# kernel parameters: IOMMU passthrough + 8 GB of 2 MB hugepages (TRex allocates from them)
grubby --update-kernel=ALL --args="intel_iommu=on iommu=pt default_hugepagesz=2M hugepagesz=2M hugepages=4096"
# CPU isolation: keep the OS, InfluxDB, Grafana and trexctl on cores 0-3 (threads 0-3 and 16-19);
# TRex threads 4-15 and their hyper-thread siblings 20-31 are isolated (layout of 8.5; verify siblings with lscpu -e)
grubby --update-kernel=ALL --args="isolcpus=4-15,20-31 nohz_full=4-15,20-31 rcu_nocbs=4-15,20-31"
reboot

# after reboot: verify
grep -E 'HugePages_(Total|Free)' /proc/meminfo
cat /proc/cmdline
dmesg | grep -i -E 'DMAR|IOMMU' | head
lspci -nn | grep -i ethernet
for d in $(lspci -n | awk '/8086:1572/{print $1}'); do
    echo "$d numa=$(cat /sys/bus/pci/devices/0000:$d/numa_node)"
done

Intel X710 devices show PCI ID 8086:1572 (SFP+). All eight ports should report the same NUMA node (or -1 on a single-socket system).

8.4 TRex installation

mkdir -p /data/trex /opt/trex
cd /data/trex
# the TRex download server uses a certificate chain that some systems do not trust;
# if --no-check-certificate is needed, verify the archive against the checksum on the release page
wget --no-check-certificate https://trex-tgn.cisco.com/trex/release/v3.08.tar.gz
tar xzf v3.08.tar.gz
ln -sfn /data/trex/v3.08 /opt/trex/current
cd /opt/trex/current
./t-rex-64 --help | head -5          # must print the usage text
./dpdk_setup_ports.py -s             # shows NICs, drivers, NUMA, MAC addresses

8.4.1 Binding the X710 ports to DPDK

Run the interactive generator once to let TRex detect the NICs and create a baseline configuration, then split it into the two instance files of section 8.5. Select all eight X710 ports (IDs from the table), keep the default IP suggestions (they are not used in stateless mode), and let the script save to /etc/trex_cfg.yaml.

./dpdk_setup_ports.py -i
# ... choose the 8 X710 interfaces, accept defaults, save to /etc/trex_cfg.yaml
cat /etc/trex_cfg.yaml                # baseline: interfaces list and platform/thread layout
./dpdk_setup_ports.py -s              # X710 ports must now show driver vfio-pci (or igb_uio)

dpdk_setup_ports.py -i only generates the configuration file. The ports are bound to the DPDK driver when TRex starts (t-rex-64 calls trex-cfg, which runs dpdk_setup_ports.py --parent ... with the configuration file) or manually with ./dpdk_setup_ports.py --cfg /etc/trex/trex_mon.yaml. With IOMMU enabled vfio-pci is used; without IOMMU the script compiles and loads igb_uio from ko/src (needs kernel-devel, fails with Secure Boot).

8.5 TRex configuration files for the two instances

Replace the PCI addresses with the values from dpdk_setup_ports.py -s (NIC1 = four consecutive functions .0-.3 of the first X710, NIC2 = the second X710). Thread numbers assume a 16-core / 32-thread single socket with siblings N and N+16 (verify with lscpu -e); threads 0-3 and 16-19 remain for the OS, InfluxDB, Grafana and trexctl. c is the number of data-plane threads per port pair and each dual_if threads list must contain at least c entries.

### /etc/trex/trex_mon.yaml - monitoring instance (NIC1, software mode, EMU)
- version: 2
  interfaces: ['31:00.0', '31:00.1', '31:00.2', '31:00.3']
  port_limit: 4
  prefix: mon
  zmq_pub_port: 4500
  zmq_rpc_port: 4501
  limit_memory: 3072
  port_bandwidth_gb: 10
  c: 1
  services_core: 4
  port_info:
    - ip: 1.1.1.1
      default_gw: 1.1.1.2
    - ip: 1.1.1.2
      default_gw: 1.1.1.1
    - ip: 1.1.2.1
      default_gw: 1.1.2.2
    - ip: 1.1.2.2
      default_gw: 1.1.2.1
  platform:
    master_thread_id: 4
    latency_thread_id: 5
    dual_if:
      - socket: 0
        threads: [6]
      - socket: 0
        threads: [8]

### /etc/trex/trex_lab.yaml - lab instance (NIC2, hardware mode or ASTF)
- version: 2
  interfaces: ['98:00.0', '98:00.1', '98:00.2', '98:00.3']
  port_limit: 4
  prefix: lab
  zmq_pub_port: 4600
  zmq_rpc_port: 4601
  limit_memory: 3072
  port_bandwidth_gb: 10
  c: 4
  port_info:
    - ip: 2.1.1.1
      default_gw: 2.1.1.2
    - ip: 2.1.1.2
      default_gw: 2.1.1.1
    - ip: 2.1.2.1
      default_gw: 2.1.2.2
    - ip: 2.1.2.2
      default_gw: 2.1.2.1
  platform:
    master_thread_id: 10
    latency_thread_id: 11
    dual_if:
      - socket: 0
        threads: [12, 13, 28, 29]
      - socket: 0
        threads: [14, 15, 30, 31]

### /etc/trex/trex_lab_astf.yaml - lab instance in ASTF mode: two ports only.
### ASTF port roles are fixed: even ports = client side, odd ports = server side.
### port 0 = p4 (client, pinned to a VPN-CORE VLAN AP), port 1 = p6 (server, pinned to the FW zone)
- version: 2
  interfaces: ['98:00.0', '98:00.2']
  port_limit: 2
  prefix: lab
  zmq_pub_port: 4600
  zmq_rpc_port: 4601
  limit_memory: 3072
  port_bandwidth_gb: 10
  c: 4
  port_info:
    - ip: 10.3.14.151
      default_gw: 10.3.14.1
      vlan: 1314
    - ip: 10.9.90.156
      default_gw: 10.9.90.1
      vlan: 4003
  platform:
    master_thread_id: 10
    latency_thread_id: 11
    dual_if:
      - socket: 0
        threads: [12, 13, 28, 29]

The port_info entries are placeholders: in stateless mode the real client addresses are configured by EMU (monitoring) or by the trexctl test runner (lab); in ASTF mode the port_info of trex_lab_astf.yaml is real (port address, default gateway and VLAN per port; trexctl test astf rewrites it for the chosen pins, section 11.7; the zone values shown are for option B - option A uses VLAN 1420 and 10.9.90.164). Keys prefix, zmq_pub_port, zmq_rpc_port, limit_memory, services_core, c, port_info (ip, default_gw, vlan) and the platform section are documented in the TRex manual (chapter "Platform configuration"). VALIDATED Because the ASTF configuration uses two of the four ports of NIC2, TRex (i40e) needs --unbind-unused-ports at start, otherwise it exits with an error about bound but unused ports of the same NIC (main_dpdk.cpp option list) CONFIRM on the bench.

8.6 systemd units

mkdir -p /etc/trex /var/log/trex
cat >/etc/systemd/system/trex-mon.service <<'EOF'
[Unit]
Description=TRex monitoring instance (stateless, software mode, EMU)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/trex/current
ExecStart=/opt/trex/current/t-rex-64 -i --software --emu --cfg /etc/trex/trex_mon.yaml --iom 0
Restart=on-failure
RestartSec=15
LimitMEMLOCK=infinity
LimitNOFILE=65536
StandardOutput=append:/var/log/trex/trex-mon.log
StandardError=append:/var/log/trex/trex-mon.log

[Install]
WantedBy=multi-user.target
EOF

cat >/etc/systemd/system/trex-lab.service <<'EOF'
[Unit]
Description=TRex lab instance (stateless, hardware flow statistics)
After=network-online.target
Conflicts=trex-lab-astf.service

[Service]
Type=simple
User=root
WorkingDirectory=/opt/trex/current
ExecStart=/opt/trex/current/t-rex-64 -i --cfg /etc/trex/trex_lab.yaml --iom 0 --no-scapy-server
Restart=no
LimitMEMLOCK=infinity
StandardOutput=append:/var/log/trex/trex-lab.log
StandardError=append:/var/log/trex/trex-lab.log
EOF

cat >/etc/systemd/system/trex-lab-astf.service <<'EOF'
[Unit]
Description=TRex lab instance (advanced stateful, ASTF)
After=network-online.target
Conflicts=trex-lab.service

[Service]
Type=simple
User=root
WorkingDirectory=/opt/trex/current
ExecStart=/opt/trex/current/t-rex-64 -i --astf --cfg /etc/trex/trex_lab_astf.yaml --iom 0 --no-scapy-server --unbind-unused-ports
Restart=no
LimitMEMLOCK=infinity
StandardOutput=append:/var/log/trex/trex-lab.log
StandardError=append:/var/log/trex/trex-lab.log
EOF

systemctl daemon-reload
systemctl enable --now trex-mon.service
sleep 20 && tail -n 30 /var/log/trex/trex-mon.log

The lab units are started on demand (systemctl start trex-lab or trex-lab-astf); Conflicts= guarantees that only one of them owns NIC2. t-rex-64 is a wrapper script that runs trex-cfg (hugepages, DPDK binding, helper daemons) before launching _t-rex-64; the --iom 0 flag suppresses the interactive text UI output in the log. --no-scapy-server on the lab units prevents them from restarting the Scapy helper daemon (a single named daemon per host, started by any interactive STL instance) under trex-mon; the lab units never use --emu, because a second --emu start would restart trex-mon's EMU server as well. After the first start verify in /var/log/trex/trex-mon.log: set dpdk queues mode to ONE_QUE.

8.7 Python environment for trexctl and the console

python3.12 -m venv /opt/trexctl/venv
/opt/trexctl/venv/bin/pip install --upgrade pip
/opt/trexctl/venv/bin/pip install pyyaml scapy influxdb-client rich jsonschema pyzmq simplejson
# make the TRex client libraries importable (STL, EMU, ASTF APIs ship with the TRex release)
echo "/opt/trex/current/automation/trex_control_plane/interactive" \
   > /opt/trexctl/venv/lib/python3.12/site-packages/trex.pth
/opt/trexctl/venv/bin/python -c "from trex.stl.api import STLClient; from trex.emu.api import EMUClient; print('ok')"

TRex v3.08 added Python 3.12 support on the client side and notes that older Python versions may not work with that release; hence the dedicated 3.12 environment. Console access: cd /opt/trex/current && ./trex-console -p 4501 --async_port 4500 --emu (monitoring) or -p 4601 --async_port 4600 (lab).

8.8 Initial self-test (before connecting to the TX switch)

1. Connect NIC1 port 2 and port 3 (p2, p3) with a DAC (loopback). Start trex-mon, open the console and run a short stateless test:

   cd /opt/trex/current && ./trex-console -p 4501 --async_port 4500
   trex> portattr -a --prom on
   trex> start -f stl/udp_1pkt_simple.py -m 1kpps --port 2
   trex> tui            # RX on port 3 must equal TX on port 2; q to quit
   trex> stop -a

2. Repeat with the latency example stl/flow_stats_latency.py and check stats -l and stats --lh (the console shows at most 4-5 pg_ids; use the API for more): latency in the single-digit microsecond range confirms the server-side baseline (DPDK, no switch). Confirm set dpdk queues mode to ONE_QUE in the trex-mon log.
3. Start trex-lab with NIC2 p6-p7 looped (lab ports 2 and 3) and run start -f stl/imix.py -m 100% --port 2 for 60 s: no drops and no "queue full" counters confirm hardware mode line rate. Never run the bundled ./ndr tool on this server: it connects to the default RPC port 4501 (trex-mon) and resets all ports.
4. Capture a few latency packets on the looped ports (capture record start --rx 3 --limit 50, capture record stop --id 1 -o /tmp/lat.pcap) and confirm in Wireshark: IP ID 0xFFFF, TOS bit 0 set (e.g. 0x01), UDP checksum 0 - the expected signature of TRex latency streams (4.4).
5. Record the results in the acceptance protocol (section 12.3).

8.9 Installation troubleshooting

9. Orchestration tooling: trexctl specification

trexctl is a project deliverable: a Python 3.12 command-line tool and a small daemon that turn a declarative description of the environment (inventory) and of the wanted traffic (profiles) into TRex streams and EMU clients, run them, and export statistics to InfluxDB. It replaces the proprietary trex_client.py of the reference design. This section is its functional specification; implementation effort is estimated in section 12.

9.1 Architecture

9.2 Repository and file layout

/opt/trexctl/                         # git checkout (LVR internal repository)
  venv/                               # Python 3.12 virtual environment (8.7)
  trexctl/                            # package: cli.py inventory.py profile.py emu.py streams.py exporter.py tests/
  schemas/inventory.schema.json       # JSON schema for validation
  schemas/profile.schema.json
  dashboards/*.json                   # Grafana dashboards (provisioned)
  alerts/*.yaml                       # Grafana alert rules (provisioned)
/etc/trexctl/
  inventory.yaml                      # environment description (single source of truth)
  profiles/monitoring-default.yaml    # permanent profile
  profiles/dscp-check.yaml
  profiles/lab-*.yaml                 # lab test definitions
  trexctl.yaml                        # tool settings: TRex endpoints, InfluxDB URL/token, sampling interval, budget
  pgid-map.json                       # persisted pg_id allocation (auto-managed)
  secrets.env                         # INFLUX_TOKEN (mode 0600)
/data/trexctl/
  reports/                            # test reports (JSON + Markdown)
  archive/                            # switch config snapshots, pcaps
  state/                              # exporter state
/var/log/trexctl/

9.3 Inventory (inventory.yaml)

version: 1
site:
  name: LVR RZK/RZC
  mac_oui: "02:54:52"            # locally administered prefix for TRex clients
trex:
  mon:  { server: 127.0.0.1, rpc_port: 4501, pub_port: 4500, emu_port: 4510, mode: stl-software-emu }
  lab:  { server: 127.0.0.1, rpc_port: 4601, pub_port: 4600, mode: stl-hw }
ports:                                 # TRex port roles
  mon:
    0: { type: multiplexed, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/21, default_for: [rzk, fw] }
    1: { type: multiplexed, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/22, default_for: [rzc] }
  lab:                                 # keys = instance-local port numbers of trex-lab; label = server-wide name
    0: { label: p4, type: pinned, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/9,  ap: K-L11, astf_role: client }
    1: { label: p5, type: pinned, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/10, ap: K-BL }
    2: { label: p6, type: pinned, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/11, ap: FWZ,   astf_role: server }
    3: { label: p7, type: pinned, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/12, ap: C-L21 }
aps:
  K-L11: { index: 1,  dc: rzk, device: sw-rzk-L11, device_port: Ten-GigabitEthernet1/3/1, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/1, svlan: 3101, vlans: [1314, 1315] }
  K-L12: { index: 2,  dc: rzk, device: sw-rzk-L12, device_port: TBD, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/2, svlan: 3102, vlans: [1314, 1315] }
  K-L21: { index: 3,  dc: rzk, device: sw-rzk-L21, device_port: TBD, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/3, svlan: 3103, vlans: [1314, 1315] }
  K-L31: { index: 4,  dc: rzk, device: sw-rzk-L31, device_port: TBD, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/4, svlan: 3104, vlans: [1314, 1315] }
  K-L41: { index: 5,  dc: rzk, device: sw-rzk-L41, device_port: TBD, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/5, svlan: 3105, vlans: [1314, 1315] }
  K-BL:  { index: 6,  dc: rzk, device: sw-rzk-BL,  device_port: Ten-GigabitEthernet1/3/6, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/6, svlan: 3110, vlans: [1314, 1315, 4003] }
  C-L11: { index: 7,  dc: rzc, device: sw-rzc-L11, device_port: TBD, tx_switch: sw-rzc-TX1, tx_port: XGE1/1/1, svlan: 3201, vlans: [1314, 1315] }
  C-L12: { index: 8,  dc: rzc, device: sw-rzc-L12, device_port: TBD, tx_switch: sw-rzc-TX1, tx_port: XGE1/1/2, svlan: 3202, vlans: [1314, 1315] }
  C-L13: { index: 9,  dc: rzc, device: sw-rzc-L13, device_port: TBD, tx_switch: sw-rzc-TX1, tx_port: XGE1/1/3, svlan: 3203, vlans: [1314, 1315] }
  C-L21: { index: 10, dc: rzc, device: SW-RZC-L21, device_port: Ten-GigabitEthernet1/4/4, tx_switch: sw-rzc-TX1, tx_port: XGE1/1/4, svlan: 3204, vlans: [1314, 1315] }
  C-L22: { index: 11, dc: rzc, device: sw-rzc-L22, device_port: TBD, tx_switch: sw-rzc-TX1, tx_port: XGE1/1/5, svlan: 3205, vlans: [1314, 1315] }
  C-L23: { index: 12, dc: rzc, device: sw-rzc-L23, device_port: TBD, tx_switch: sw-rzc-TX1, tx_port: XGE1/1/6, svlan: 3206, vlans: [1314, 1315] }
  C-BL:  { index: 13, dc: rzc, device: sw-rzc-BL,  device_port: Ten-GigabitEthernet1/3/14, tx_switch: sw-rzc-TX1, tx_port: XGE1/1/7, svlan: 3210, vlans: [1314, 1315, 4003] }
  # firewall zone, option B (default): VSI 4003 reached through the BL APs -> no AP of its own;
  # K-BL and C-BL simply carry VLAN 4003 (added above). FWZ is an alias used by the lab pins.
  FWZ:   { alias_of: K-BL, vlan: 4003 }
  # firewall zone, option A (alternative): direct links, own AP index 14
  # FW:  { index: 14, dc: both, device: CPF-ZV01/ZV02, device_port: TBD, tx_switch: both, tx_port: XGE1/1/7+XGE1/1/8, svlan: 3300, vlans: [1420] }
vlans:
  1314: { vsi: VPN-1314, prefix: 10.3.14.0/24, gateway: 10.3.14.1, gateway_owner: sw-rzk-BL/VPN-CORE, pool_start: 10.3.14.101, lab_pool_start: 10.3.14.151, vrf: VPN-CORE }
  1315: { vsi: VPN-1315, prefix: 10.3.15.0/24, gateway: 10.3.15.1, gateway_owner: sw-rzk-BL/VPN-CORE, pool_start: 10.3.15.101, lab_pool_start: 10.3.15.151, vrf: VPN-CORE }
  4003: { vsi: VPN-4003, prefix: 10.9.90.0/24, gateway: 10.9.90.1, gateway_owner: ENT-FW, pool_start: 10.9.90.101, lab_pool_start: 10.9.90.151, vrf: FW-ZONE, gateway_mac_tracking: true }
  # 1420: { prefix: 10.9.90.0/24, gateway: 10.9.90.1, gateway_owner: ENT-FW, pool_start: 10.9.90.101, lab_pool_start: 10.9.90.151, vrf: FW-ZONE, gateway_mac_tracking: true }   # option A
  <PROD-VLAN-A>: { vsi: VPN-A, prefix: <SUBNET-A>, gateway: <GW-A>, gateway_owner: DC-FW, pool_start: <POOL-A>, vrf: DC-FW, gateway_mac_tracking: true }
budget: { max_latency_streams: 120, max_flowstat_streams: 600, max_pps: 50000, max_bps: 50000000, max_pps_per_fw_stream: 100 }

Rules enforced by the validator: S-VLANs unique; AP indexes unique and 1-49 (lab clients use index+50); every VLAN referenced by an AP exists; the monitoring pool (16 addresses) and the lab pool (16 addresses) do not overlap each other or the gateway and stay inside the prefix; gateway_mac_tracking is set for every VLAN whose gateway is a firewall cluster (the MAC of the active member changes on failover, 2.4). trexctl inventory render-leaf and the acceptance checks also verify that the VSI of every listed VLAN exists on the Leaf (3.2).

9.4 Profiles (profiles/*.yaml)

version: 1
name: monitoring-default
instance: mon
defaults: { kind: latency, pps: 200, size: 128, tos: 0, ttl: 64, bidir: true, dport_base: 32000, sport_base: 40000 }
# kind: latency = STLFlowLatencyStats (max 128 per instance); flowstat = STLFlowStats, loss only (max 1023)
groups:
  - name: l2-intra-rzk
    class: L2                          # L2: same VLAN, dst MAC = remote client MAC
    vlans: [1314]
    pairs: &rzk_pairs [[K-L11, K-BL], [K-L12, K-BL], [K-L21, K-BL], [K-L31, K-BL], [K-L41, K-BL]]
  - name: l2-intra-rzc
    class: L2
    vlans: [1314]
    pairs: &rzc_pairs [[C-L11, C-BL], [C-L12, C-BL], [C-L13, C-BL], [C-L21, C-BL], [C-L22, C-BL], [C-L23, C-BL]]
  - name: l2-inter-dc
    class: L2
    vlans: [1314]
    pairs: &dc_pairs [[K-L11, C-L11], [K-L12, C-L12], [K-L21, C-L21], [K-L31, C-L22], [K-L41, C-L23], [K-BL, C-BL]]
  - name: l2-coverage-1315          # loss-only copies of the three L2 groups on the second carrier VLAN
    class: L2
    kind: flowstat
    pps: 100
    vlans: [1315]
    pairs_from: [l2-intra-rzk, l2-intra-rzc, l2-inter-dc]
  - name: l2-coverage-prod          # loss-only copies on the production VLANs
    class: L2
    kind: flowstat
    pps: 100
    vlans: [<PROD-VLAN-A>, <PROD-VLAN-B>]
    pairs_from: [l2-intra-rzk, l2-intra-rzc, l2-inter-dc]
  - name: l3-bl
    class: L3                          # L3: dst MAC = resolved gateway MAC of the source VLAN
    src_vlan: 1314
    dst_vlan: 1315
    pairs: [[K-L11, K-L21], [K-L31, K-BL], [C-L21, C-L22], [C-L13, C-BL]]
  - name: l3-dcfw
    class: L3
    src_vlan: <PROD-VLAN-A>
    dst_vlan: <PROD-VLAN-B>
    pps: 50
    pairs: [[K-L11, K-L41], [K-L21, K-L31], [C-L12, C-L23], [C-L11, C-L22]]
  - name: l3-entfw
    class: L3
    src_vlan: 1314
    dst_vlan: 4003                     # firewall zone option B (option A: 1420 at AP FW)
    pps: 50
    pairs: [[K-L11, K-BL], [K-BL, K-BL], [C-L21, C-BL], [C-BL, C-BL]]   # zone client sits at the BL AP of the same DC
  - name: dscp-l2-interdc
    class: L2
    vlans: [1314]
    pps: 100
    size: 256
    tos_set: [0x00, 0xB8, 0x88, 0x20]  # one stream per TOS value; 0x00 = best-effort reference
    pairs: [[K-L11, C-L21]]
  - name: dscp-entfw
    class: L3
    src_vlan: 1314
    dst_vlan: 4003
    pps: 50
    size: 256
    tos_set: [0x00, 0xB8, 0x88, 0x20]
    pairs: [[K-L11, K-BL]]
tags: { environment: prod, owner: netzwerk }

Expansion rules: each pair and each VLAN (or VLAN pair) and each TOS value produces one stream; bidir: true adds the reverse stream; pairs_from re-uses the pair lists of other groups. Class L2 requires both APs to carry the VLAN; class L3 requires a gateway for the source VLAN and resolves it per source AP; for class L3 the destination is the client of dst_vlan at the second AP of the pair. kind selects STLFlowLatencyStats (latency, counted against the limit of 128) or STLFlowStats (loss only). The expanded default profile yields 74 latency and 102 loss-only streams (4.5). The class name, kind, DC pair (rzk-rzk, rzk-rzc, rzc-rzc, x-fw), VLANs, APs and TOS become InfluxDB tags.

9.5 Stream generation algorithm

1. Load and validate inventory and profile (JSON schema + semantic rules); compute the stream list; abort if a budget guard rail is exceeded (latency streams > 120, loss-only streams > 600, pps/bps, firewall pps) unless --force (the hard TRex limits 128/1023 are never overridable).
2. Allocate pg_ids: key = class|kind|src_ap|src_vlan|dst_ap|dst_vlan|tos; look up in pgid-map.json, otherwise assign the lowest free id (latency 1-127, loss-only 1000-1999) and persist. Stable ids keep Grafana history continuous.
3. Provision EMU (monitoring instance): for every (TRex port, S-VLAN, C-VLAN) create a namespace with tci=[svlan, cvlan], tpid=[0x8100, 0x8100] (tci[0] is the outer tag); add one client per needed IP with its deterministic MAC, ipv4_dg = VLAN gateway and plugins arp, icmp. EMUClient.load_profile() first calls remove_profile() and deletes all namespaces and clients, so it is used only for the initial load; every later change is incremental (add_ns, add_clients, remove_clients, remove_ns) so that running streams keep their emulated hosts. Set the TRex ports to promiscuous + multicast and to filtered service mode with the mask the console uses for service --emu (NO_TCP_UDP_MASK | DHCP_MASK | MDNS_MASK | TRANSPORT_MASK; NO_TCP_UDP_MASK alone covers ARP/ICMP - confirm on the bench that the statistics UDP streams are counted unchanged). Wait until every client that needs a gateway reports dgw.resolve = true; retry three times, then report unresolved clients and continue with the resolvable streams only.
4. Build packets: Ether(src=client_mac, dst=dst_mac)/Dot1Q(vlan=svlan)/Dot1Q(vlan=cvlan)/IP(src, dst, tos, ttl)/UDP(sport, dport, chksum=0 for latency streams)/Raw(pad), padded to size (at least 16 bytes of payload for the latency signature). For class L2 dst_mac is the remote TRex client MAC; for class L3 it is the resolved gateway MAC. Pinned lab ports omit the outer Dot1Q.
5. Create streams: STLStream(packet=STLPktBuilder(pkt), mode=STLTXCont(pps=pps), flow_stats=STLFlowLatencyStats(pg_id) or STLFlowStats(pg_id), name=key); one TRex profile per group and port (port_id.profile_id syntax, e.g. 0.l2-inter-dc) so that a group can be stopped, removed and re-added without disturbing the others; start(ports=[...profiles...], force=True). Rate changes are implemented as remove + re-add of the group (latency streams ignore the multiplier, so update(mult=) cannot change them; counters restart, the exporter handles the discontinuity).
6. Track gateway MACs: for every VLAN with gateway_mac_tracking the tracker polls the EMU client information (dgw.rmac) every 10 s and additionally forces a re-resolution every 60 s; when the resolved MAC differs from the MAC baked into the running streams, the affected streams (all L3 streams sourced in that VLAN at that AP) are rebuilt and an event gateway_mac_changed is written to trex_event. Until then the streams show loss - the gap measured on firewall classes therefore includes the detection time (worst case ~10 s + rebuild).
7. Record the applied stream table (/data/trexctl/state/applied-<profile>.json) with pg_id, ports, addresses, tags - the exporter reads this file to tag the statistics.

9.6 Command-line interface

9.7 Exporter and InfluxDB data model

Interval: 2 s by default (1 s possible for convergence tests, 5 s for long-term). Each poll calls get_pgid_stats() (flow and latency statistics) and get_stats() (ports, global) through a read-only client: the exporter never acquires the monitoring ports and disconnects with disconnect(stop_traffic=False), because the default disconnect() stops traffic on owned ports. Counters are read from the 'total' entry of the per-port dictionaries (summing all values would double-count). TX/RX packet counters are converted to per-interval deltas; drops per interval = TX delta - RX delta (clamped at 0 and compensated for in-flight packets by using the TRex rx_delay convention); loss is also kept as cumulative counters for exact outage accounting. Latency: average is exported as reported (low-pass filtered by TRex); the interval maximum is derived from the delta of the cumulative histogram (upper edge of the highest bucket that grew in the interval), because last_max only covers the last 0.5-second server window; total_max/total_min are exported as lifetime values; no latency fields are written for an interval in which the stream received no packets (TRex freezes average in that case).

Series cardinality: about 180 streams x up to ~30 fields - trivial for InfluxDB. Writes use the v2 HTTP API with a write token scoped to the trex bucket; batched every interval; on InfluxDB unavailability the exporter buffers up to 10 minutes in memory and logs a warning.

9.8 Test runners

• throughput (lab instance, pinned ports): for each frame size run 60 s trials with non-latency streams (STLFlowStats, hardware counters) plus one low-rate latency stream; binary search between 1% and 100% of line rate until the no-drop rate (loss < 0.001%) is found within 0.5% resolution; record NDR, latency at NDR and at 50% load. Optional bidirectional mode (both pinned ports transmit).
• soak: fixed rate for a long duration; loss and latency time series into trex_test.
• convergence: re-adds the selected monitoring groups at 1000 pps (1 ms resolution; the budget check applies), records cumulative drops per stream before/after the planned event, computes outage duration = drops / pps and produces a table per stream and per AP pair; restores the original rates afterwards. For streams towards a firewall MAC the reported outage includes the gateway-MAC rebuild (9.5).
• astf: writes /etc/trex/trex_lab_astf.yaml for the chosen pins (port 0 = client side, port 1 = server side; even/odd roles are fixed in TRex), starts trex-lab-astf (via systemctl, sudo rule), loads an ASTF Python profile with the chosen client/server ranges, runs for the duration, collects m_active_flows, m_est_flows, tcps_connattempt, tcps_connects, errors, goodput, latency histograms; restores trex-lab (stateless) afterwards.

9.9 Reference implementation sketches

The following excerpts illustrate the TRex API usage the implementation relies on. They follow the TRex STL and EMU API documentation (v3.x).

# EMU profile: one namespace per (port, S-VLAN, C-VLAN), one client per TRex IP
from trex.emu.api import *

def build_emu_profile(clients):
    ns_by_key = {}
    for c in clients:                                   # c: port, svlan, cvlan, mac, ip, gw
        key = (c.port, c.svlan, c.cvlan)
        if key not in ns_by_key:
            tci  = [c.svlan, c.cvlan] if c.svlan else [c.cvlan, 0]
            tpid = [0x8100, 0x8100] if c.svlan else [0x8100, 0]
            ns_key = EMUNamespaceKey(vport=c.port, tci=tci, tpid=tpid)
            ns_by_key[key] = EMUNamespaceObj(ns_key=ns_key,
                                             def_c_plugs={'arp': {'enable': True}, 'icmp': {}})
        ns_by_key[key].add_clients(EMUClientObj(mac=Mac(c.mac).V(),
                                                ipv4=Ipv4(c.ip).V(),
                                                ipv4_dg=Ipv4(c.gw).V()))
    return EMUProfile(ns=list(ns_by_key.values()), def_ns_plugs=None)

emu = EMUClient(server='127.0.0.1', sync_port=4510)
emu.connect()
if first_apply:                                         # load_profile() removes ALL existing namespaces/clients first
    emu.load_profile(profile=build_emu_profile(clients), max_rate=512, tunables=[])
else:                                                   # incremental changes keep the running streams' hosts intact
    emu.add_ns(new_namespaces); emu.add_clients(ns_key, new_clients); emu.remove_clients(ns_key, gone_clients)
# then poll emu.get_info_clients(...) until dgw.resolve is True and read dgw.rmac;
# the gateway-MAC tracker keeps polling dgw.rmac and triggers a stream rebuild when it changes

# stateless streams with latency statistics, double-tagged for multiplexed ports
from trex.stl.api import *
from scapy.layers.l2 import Ether, Dot1Q
from scapy.layers.inet import IP, UDP

def build_stream(s):                                     # s: stream definition (dataclass)
    pkt = Ether(src=s.src_mac, dst=s.dst_mac)
    if s.svlan:
        pkt = pkt / Dot1Q(vlan=s.svlan)
    pkt = pkt / Dot1Q(vlan=s.cvlan) / IP(src=s.src_ip, dst=s.dst_ip, tos=s.tos, ttl=s.ttl)
    if s.kind == 'latency':
        pkt = pkt / UDP(sport=s.sport, dport=s.dport, chksum=0)   # TRex rewrites the payload per packet without fixing the UDP checksum
    else:
        pkt = pkt / UDP(sport=s.sport, dport=s.dport)
    pad = max(16, s.size - len(pkt)) * b'\x00'                 # >= 16 bytes payload for the latency signature
    fs = STLFlowLatencyStats(pg_id=s.pg_id) if s.kind == 'latency' else STLFlowStats(pg_id=s.pg_id)
    return STLStream(name=s.key, packet=STLPktBuilder(pkt=pkt / pad), mode=STLTXCont(pps=s.pps), flow_stats=fs)

c = STLClient(server='127.0.0.1', sync_port=4501, async_port=4500)
c.connect()
c.acquire(ports=[0, 1], force=True)
c.set_port_attr(ports=[0, 1], promiscuous=True, multicast=True)
# filtered service mode with the mask the console uses for "service --emu"
from trex.common.trex_client import NO_TCP_UDP_MASK, DHCP_MASK, MDNS_MASK, TRANSPORT_MASK
EMU_SERVICE_MASK = NO_TCP_UDP_MASK | DHCP_MASK | MDNS_MASK | TRANSPORT_MASK
c.set_service_mode(ports=[0, 1], enabled=False, filtered=True, mask=EMU_SERVICE_MASK)
for (port, group), streams in streams_by_port_group.items():
    profile = f"{port}.{group}"                              # one TRex profile per group and port
    c.add_streams([build_stream(s) for s in streams], ports=[profile])
c.start(ports=[f"{p}.{g}" for (p, g) in streams_by_port_group], force=True)

# exporter poll loop (simplified)
import time
from influxdb_client import InfluxDBClient, Point, WritePrecision

influx = InfluxDBClient(url="http://127.0.0.1:8086", token=TOKEN, org="lvr")
write  = influx.write_api()
c = STLClient(server='127.0.0.1', sync_port=4501, async_port=4500)
c.connect()                                             # read-only: no acquire(); at exit c.disconnect(stop_traffic=False)
prev = {}
while True:
    st  = c.get_pgid_stats()                            # {'flow_stats': {...}, 'latency': {...}}
    now = time.time_ns()
    points = []
    for pg_id, meta in applied.items():                 # meta = tags from the applied stream table
        fs  = st['flow_stats'].get(pg_id, {})
        lat = st['latency'].get(pg_id, {}).get('latency', {})
        tx  = fs.get('tx_pkts', {}).get('total', 0)             # per-port dict also contains the key 'total'
        rx  = fs.get('rx_pkts', {}).get('total', 0)
        dtx = tx - prev.get((pg_id, 'tx'), tx)
        drx = rx - prev.get((pg_id, 'rx'), rx)
        prev[(pg_id, 'tx')] = tx
        prev[(pg_id, 'rx')] = rx
        p = Point("trex_stream").time(now, WritePrecision.NS)
        for k, v in meta.tags.items(): p = p.tag(k, v)
        p = (p.field("tx_pkts", tx).field("rx_pkts", rx)
               .field("drop_pkts", max(0, dtx - drx))
               .field("drop_pkts_total", max(0, tx - rx))
               .field("tx_pps", dtx / INTERVAL).field("rx_pps", drx / INTERVAL))
        if meta.kind == 'latency' and drx > 0:                  # no latency fields for intervals without RX
            hist = lat.get('histogram', {})                      # cumulative counts per logarithmic bucket
            dh = {b: hist[b] - prev.get((pg_id, 'h', b), 0) for b in hist}
            for b in hist: prev[(pg_id, 'h', b)] = hist[b]
            grown = [int(b) for b, v in dh.items() if v > 0]
            interval_max = bucket_upper_edge(max(grown)) if grown else lat.get('last_max', 0)
            p = (p.field("lat_avg_us", lat.get('average', 0.0))
                   .field("lat_max_us", interval_max)              # interval maximum at bucket resolution
                   .field("lat_last_max_us", lat.get('last_max', 0))   # last 0.5 s window only
                   .field("lat_total_max_us", lat.get('total_max', 0))
                   .field("lat_min_us", lat.get('total_min', 0))
                   .field("jitter_us", lat.get('jitter', 0)))
            for b, v in dh.items(): p = p.field(f"hist_{b}", v)
        points.append(p)
    write.write(bucket="trex", record=points)
    time.sleep(INTERVAL)

Latency dictionary keys (average, total_max, total_min, last_max, jitter, histogram, err_cntrs) and the 'total' key of the per-port counter dictionaries are documented in the TRex STL Python API (get_pgid_stats) and in trex_stl_stats.py; the service-mode masks are defined in trex/common/trex_client.py. VALIDATED

9.10 Service integration, permissions and logging

cat >/etc/systemd/system/trexctl-exporter.service <<'EOF'
[Unit]
Description=trexctl statistics exporter (TRex -> InfluxDB)
After=trex-mon.service influxdb.service
Requires=trex-mon.service

[Service]
Type=simple
User=trexctl
Group=trexctl
EnvironmentFile=/etc/trexctl/secrets.env
ExecStart=/opt/trexctl/venv/bin/trexctl exporter run
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF
useradd -r -s /sbin/nologin -d /data/trexctl trexctl
# trexctl needs to start/stop the lab units for ASTF tests:
cat >/etc/sudoers.d/trexctl <<'EOF'
trexctl ALL=(root) NOPASSWD: /usr/bin/systemctl start trex-lab.service, \
    /usr/bin/systemctl stop trex-lab.service, \
    /usr/bin/systemctl start trex-lab-astf.service, \
    /usr/bin/systemctl stop trex-lab-astf.service
EOF
chmod 440 /etc/sudoers.d/trexctl
systemctl daemon-reload && systemctl enable --now trexctl-exporter.service

• Logs: /var/log/trexctl/trexctl.log (rotating), JSON lines for machine processing; important events (apply, stop, unresolved clients, budget violations, InfluxDB write failures) are also written to trex_event.
• A monitoring profile re-apply after a TRex restart is automatic: the exporter detects a server restart (TRex uptime reset) and triggers trexctl apply -p <active profile>.
• Configuration files are kept under version control (git) with a pre-commit hook running trexctl inventory validate and trexctl plan.

9.11 Development and acceptance of trexctl

• Unit tests for inventory/profile expansion (pytest, no TRex needed); integration tests against a TRex instance in loopback (two DAC-connected ports) and against the TRex "stl-sim" simulator for packet building.
• Deliverables: source repository, README, this specification, example inventory/profiles, Grafana dashboards and alert rules, systemd units.
• Acceptance: trexctl apply -p monitoring-default brings up all planned streams (74 latency + 102 loss-only) with resolved gateways within 60 s; trexctl status matches the TRex console (stats -s / stats -l for the first pg_ids) and the API; Grafana shows all streams within two sampling intervals; a simulated gateway-MAC change (bench: swap the gateway MAC) triggers a stream rebuild within 15 s; the exporter survives a trex-mon restart without stopping traffic on reconnect; trexctl test throughput in loopback reports line rate for 1518-byte frames.

10. InfluxDB and Grafana

10.1 InfluxDB OSS 2.x installation

cat >/etc/yum.repos.d/influxdata.repo <<'EOF'
[influxdata]
name = InfluxData Repository - Stable
baseurl = https://repos.influxdata.com/stable/x86_64/main
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdata-archive.key
EOF
dnf -y install influxdb2 influxdb2-cli
# keep the data on /data
mkdir -p /data/influxdb && chown influxdb:influxdb /data/influxdb
cat >/etc/influxdb/config.toml <<'EOF'
bolt-path = "/data/influxdb/influxd.bolt"
engine-path = "/data/influxdb/engine"
http-bind-address = "127.0.0.1:8086"
query-memory-bytes = 2147483648
storage-cache-max-memory-size = 2147483648
EOF
systemctl enable --now influxdb
# initial setup: organisation lvr, bucket trex with 90-day retention;
# the admin token is printed once - store it in the password safe
influx setup --org lvr --bucket trex --retention 90d --username influxadmin \
    --password '<STRONG-PASSWORD>' --force
influx bucket create --org lvr --name trex_longterm --retention 730d
# scoped tokens (bucket IDs from 'influx bucket list')
TREX_ID=$(influx bucket list --org lvr --name trex --hide-headers | awk '{print $1}')
LT_ID=$(influx bucket list --org lvr --name trex_longterm --hide-headers | awk '{print $1}')
influx auth create --org lvr --description "trexctl exporter (write trex)" --write-bucket "$TREX_ID"
influx auth create --org lvr --description "grafana (read)" --read-bucket "$TREX_ID" --read-bucket "$LT_ID"

Pin the package version in the repository configuration (e.g. influxdb2-2.9.x): InfluxDB 3 (Core/Enterprise) is now InfluxData's primary product line and has a different storage engine and API, and InfluxData has announced that its "latest" container tags move to InfluxDB 3 Core (announcement date to be re-checked at installation time CONFIRM). InfluxDB 2.x is still maintained (2.9.1, released 2026-05-12, at the time of writing) and remains the recommended version for this deployment; a later migration to InfluxDB 3 is possible but out of scope.

10.2 Downsampling task

Raw 2-second data is kept for 90 days in trex. A Flux task writes 1-minute aggregates (mean/max/sum per stream) into trex_longterm for two years:

option task = {name: "trex_downsample_1m", every: 10m, offset: 1m}

from(bucket: "trex")
  |> range(start: -20m)
  |> filter(fn: (r) => r._measurement == "trex_stream")
  |> filter(fn: (r) => r._field == "lat_avg_us" or r._field == "jitter_us"
                       or r._field == "rx_pps" or r._field == "tx_pps")
  |> aggregateWindow(every: 1m, fn: mean, createEmpty: false)
  |> to(bucket: "trex_longterm", org: "lvr")

from(bucket: "trex")
  |> range(start: -20m)
  |> filter(fn: (r) => r._measurement == "trex_stream")
  |> filter(fn: (r) => r._field == "lat_max_us" or r._field == "lat_last_max_us")
  |> aggregateWindow(every: 1m, fn: max, createEmpty: false)
  |> to(bucket: "trex_longterm", org: "lvr")

from(bucket: "trex")
  |> range(start: -20m)
  |> filter(fn: (r) => r._measurement == "trex_stream" and r._field == "drop_pkts")
  |> aggregateWindow(every: 1m, fn: sum, createEmpty: false)
  |> to(bucket: "trex_longterm", org: "lvr")

Create it with influx task create --org lvr -f /opt/trexctl/influx/downsample.flux. Disk estimate: 180 streams x 25 fields x 0.5 points/s is about 2 250 points/s raw; with InfluxDB compression this is in the order of 1-2 GB per month, so 90 days of raw data need well under 10 GB.

10.3 Grafana installation

cat >/etc/yum.repos.d/grafana.repo <<'EOF'
[grafana]
name=grafana
baseurl=https://rpm.grafana.com
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://rpm.grafana.com/gpg.key
EOF
dnf -y install grafana            # pin a version if required by LVR policy, e.g. grafana-12.4.x
# TLS: certificate from the LVR PKI (CN trex-rzk-01.<domain>)
install -o root -g grafana -m 640 /path/trex-rzk-01.crt /etc/grafana/tls.crt
install -o root -g grafana -m 640 /path/trex-rzk-01.key /etc/grafana/tls.key
# grafana.ini changes
sed -i -e 's/^;protocol = http/protocol = https/' \
       -e 's/^;http_port = 3000/http_port = 443/' \
       -e 's#^;cert_file =#cert_file = /etc/grafana/tls.crt#' \
       -e 's#^;cert_key =#cert_key = /etc/grafana/tls.key#' \
       -e 's/^;root_url = .*/root_url = https:\/\/trex-rzk-01.<DOMAIN>\//' \
       -e 's/^;min_refresh_interval = 5s/min_refresh_interval = 1s/' /etc/grafana/grafana.ini
# allow the grafana service user to bind port 443 (systemd drop-in survives package upgrades)
mkdir -p /etc/systemd/system/grafana-server.service.d
cat >/etc/systemd/system/grafana-server.service.d/bind443.conf <<'EOF'
[Service]
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
EOF
systemctl daemon-reload
systemctl enable --now grafana-server

Data sources (provisioned)

# /etc/grafana/provisioning/datasources/influx.yaml
apiVersion: 1
datasources:
  - name: InfluxDB-trex
    type: influxdb
    access: proxy
    url: http://127.0.0.1:8086
    isDefault: true
    jsonData: { version: Flux, organization: lvr, defaultBucket: trex, tlsSkipVerify: true }
    secureJsonData: { token: "<GRAFANA-READ-TOKEN>" }
  - name: InfluxDB-trex-longterm
    type: influxdb
    access: proxy
    url: http://127.0.0.1:8086
    jsonData: { version: Flux, organization: lvr, defaultBucket: trex_longterm }
    secureJsonData: { token: "<GRAFANA-READ-TOKEN>" }

Authentication

Local admin plus Active Directory via LDAP ([auth.ldap] enabled = true, config_file = /etc/grafana/ldap.toml with the LVR domain controllers, group mapping: network team = Editor, others = Viewer). Anonymous access disabled. Dashboards and alert rules are provisioned from /opt/trexctl/dashboards and /opt/trexctl/alerts so that the installation is reproducible.

10.4 Dashboards

Example Flux queries

// Latency matrix cell: mean latency per src/dst AP for one class
from(bucket: "trex")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "trex_stream" and r._field == "lat_avg_us")
  |> filter(fn: (r) => r.kind == "latency" and r.class == "${class}" and r.src_vlan == "${vlan}")
  |> group(columns: ["src_ap", "dst_ap"])
  |> mean()
  |> pivot(rowKey: ["src_ap"], columnKey: ["dst_ap"], valueColumn: "_value")

// Drops per stream over time
from(bucket: "trex")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "trex_stream" and r._field == "drop_pkts")
  |> filter(fn: (r) => r.dc_pair =~ /${dc_pair:regex}/)
  |> aggregateWindow(every: v.windowPeriod, fn: sum, createEmpty: false)

// Number of streams currently dropping (last 2 intervals)
from(bucket: "trex")
  |> range(start: -10s)
  |> filter(fn: (r) => r._measurement == "trex_stream" and r._field == "drop_pkts" and r._value > 0)
  |> group(columns: ["pg_id"]) |> count() |> group() |> count()

// Outage table: drops / pps = outage seconds per stream in the selected range
drops = from(bucket: "trex") |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "trex_stream" and r._field == "drop_pkts")
  |> group(columns: ["name", "src_ap", "dst_ap", "class"]) |> sum() |> rename(columns: {_value: "drops"})
pps = from(bucket: "trex") |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "trex_stream" and r._field == "tx_pps")
  |> group(columns: ["name", "src_ap", "dst_ap", "class"]) |> mean() |> rename(columns: {_value: "pps"})
join(tables: {d: drops, p: pps}, on: ["name", "src_ap", "dst_ap", "class"])
  |> map(fn: (r) => ({r with outage_s: float(v: r.drops) / r.pps}))
  |> sort(columns: ["outage_s"], desc: true)

Usage hints carried over from the reference design: keep the auto-refresh at 1-5 s and the time window at 5-30 minutes for live observation; disable auto-refresh when zooming or hovering; prefer "all streams" and filter by tags (class, DC pair, AP) rather than by individual stream ids; use the long-term data source for windows above 24 hours.

10.5 Alert rules (Grafana unified alerting)

Notification channels: e-mail to the network team distribution list; optionally the existing monitoring system (IMC/syslog webhook). Maintenance windows are modelled with Grafana mute timings plus trexctl annotate events.

10.6 Backup and housekeeping

• Nightly influx backup /data/backup/influx-$(date +%F) --org lvr (bucket metadata + data), 14 days kept locally, copied to the LVR backup target.
• Grafana: /var/lib/grafana/grafana.db (users, alert state) in the nightly backup; dashboards and alerts are in git anyway.
• Retention is handled by the bucket policies; monitor free space on /data (node exporter or Grafana server dashboard).
• TLS certificate renewal: yearly, same path, systemctl restart grafana-server.

11. Operations and test procedures

11.1 Roles and daily operation

Daily: check the Overview dashboard (no path-down streams, no dropping streams, latency within baseline, TRex CPU < 50%); review alert history. Weekly: check /data usage, exporter log for InfluxDB write errors, unresolved EMU clients. Monthly: review baseline thresholds and the monitoring matrix.

11.2 Permanent delay monitoring: bring-up and baseline

1. Prerequisites complete: TX switches configured (7.1/7.2), Leaf/BL AP ports configured for VLAN 1314/1315 (7.3/7.4), firewall zone and rules active (7.5), TRex server installed with trex-mon running (8), InfluxDB and Grafana running (10), inventory validated (trexctl inventory validate).
2. Dry run: trexctl plan -p monitoring-default - review the stream table, pg_ids and budget.
3. Apply: trexctl apply -p monitoring-default. Expected output: EMU clients created (13 APs x 2-4 VLANs + 2 zone clients), gateways resolved for all L3 streams, 74 latency + 102 loss-only streams started on ports 0 and 1. Commissioning checks (3.2): the EMU clients behind the Leaf APs (not only behind the BL APs) must resolve 10.3.14.1 / 10.3.15.1 - this proves that the centralised gateway answers ARP arriving through VXLAN; and on an RZC Leaf (display evpn route mac) the sw-rzk-BL gateway MAC ec9b-8b67-37b8 should be present in VSI VPN-1314/1315 once the routed streams run. If either check fails, source the L3 classes from the BL APs until the fabric team has decided on ARP suppression.
4. Verify on TRex: trexctl status shows RX pps = TX pps for every stream after a few seconds; latency values plausible (intra-DC fabric tens of microseconds, inter-DC plus the DCI and TX fibre delays, firewall paths 50-500 us); /var/log/trex/trex-mon.log contains set dpdk queues mode to ONE_QUE.
5. Verify on the network: display l2vpn mac-address vsi VPN-1314 on a Leaf shows the local TRex MAC on the AP port and the remote TRex MACs via VXLAN tunnels; display arp vpn-instance VPN-CORE on sw-rzk-BL shows the TRex IPs; display mac-address vlan 3101 on sw-rzk-TX1 shows MACs on both the AP port and the TRex trunk port.
6. Verify Grafana: Overview shows all streams, no drops; Latency matrix populated.
7. Baseline phase (7 days): let the system run; then derive per-class latency thresholds (e.g. p99 of the 7-day average x 1.5) and set the alert rules (10.5). Document the baseline table in the operations handbook (template below).
8. Enable alert notifications.

11.3 Generic test procedure (applies to 11.4-11.10)

1. Define: purpose, class, APs, VLANs, rate, duration, expected result; create a change/test ticket if production VLANs or firewalls are involved.
2. Pre-checks: trexctl status (monitoring healthy), target AP links up (display interface brief on TX switch), firewall rules in place, no concurrent maintenance.
3. Annotate: trexctl annotate "TEST <id>: <description> start".
4. Execute with trexctl test ... (or raw TRex console for ad-hoc cases).
5. Observe in Grafana (Lab tests / Loss dashboards) and in the switch counters (display counters rate inbound interface on TX switch and AP ports).
6. Post-checks: lab streams stopped (trexctl test stops automatically; verify with the console stats), monitoring unaffected, firewall sessions cleared.
7. Report: trexctl report --test <id> --format md; attach to the ticket; annotate "end".

11.4 Test: Layer 2 inside one VLAN

Goal: loss and delay between two APs in the same VLAN (e.g. VLAN 1314, K-L11 to C-L21, inter-DC). Where: monitoring instance (already running permanently); for a dedicated high-rate run use the lab instance.

1. Permanent view: Grafana "Class detail", class L2, src_ap K-L11, dst_ap C-L21 - read latency/jitter/loss for the time range.
2. Higher resolution for a limited time: trexctl update --group l2-inter-dc --pps 1000 (1 ms resolution; budget check applies; the group's streams are re-added, so their counters restart); revert with --pps 200.
3. Line-rate variant: pin lab ports p4 to K-L11 and p7 to C-L21 (already the default pins), then trexctl test throughput --src K-L11 --dst C-L21 --vlan 1314 --sizes 128,512,1518 --duration 60 --bidir. The runner creates lab EMU-less clients (index+50 addresses), resolves nothing (L2: destination MAC = remote lab client MAC), runs the binary search and reports the NDR per frame size plus latency at NDR. Note that the inter-DC TX fibre carries 10G in each direction in this test; with a 10G inter-DC TX link, bidirectional line rate is the maximum.

Raw TRex alternative (console on the lab instance; p4 and p7 are lab ports 0 and 3, single-tagged):

cd /opt/trex/current && ./trex-console -p 4601 --async_port 4600
trex> portattr -a --prom on
trex> start -f /opt/trexctl/profiles/raw/udp_vlan_latency.py --port 0 -t cvlan=1314,smac=02:54:52:33:05:22,dmac=02:54:52:3c:05:22,sip=10.3.14.151,dip=10.3.14.160,pps=1000
trex> start -f /opt/trexctl/profiles/raw/udp_vlan_latency.py --port 3 -t cvlan=1314,smac=02:54:52:3c:05:22,dmac=02:54:52:33:05:22,sip=10.3.14.160,dip=10.3.14.151,pps=1000,pg=2001
trex> stats -l            # latency per pg_id (console shows the first 5); stats --lh = histogram
trex> tui                 # live view, 'l' for latency; q to quit
trex> stop -a

# /opt/trexctl/profiles/raw/udp_vlan_latency.py - minimal single/double-tagged UDP latency stream with tunables
from trex.stl.api import *

class STLS1(object):
    def get_streams(self, direction=0, svlan=0, cvlan=1314, smac='02:54:52:33:05:22', dmac='02:54:52:3c:05:22',
                    sip='10.3.14.151', dip='10.3.14.160', pps=200, size=128, tos=0, pg=2000, **kwargs):
        pkt = Ether(src=smac, dst=dmac)
        if int(svlan):
            pkt = pkt / Dot1Q(vlan=int(svlan))
        pkt = pkt / Dot1Q(vlan=int(cvlan)) / IP(src=sip, dst=dip, tos=int(tos), ttl=64) \
                  / UDP(sport=40000 + int(pg), dport=32009, chksum=0)   # checksum 0: TRex rewrites the latency payload
        pad = max(16, int(size) - len(pkt)) * 'x'                        # >= 16 bytes for the latency signature
        return [STLStream(packet=STLPktBuilder(pkt=pkt / pad),
                          mode=STLTXCont(pps=int(pps)),
                          flow_stats=STLFlowLatencyStats(pg_id=int(pg)))]

def register():
    return STLS1()

For L2 streams no ARP is needed, but the fabric must have learned the destination MAC before a high-rate run: start the reverse stream (or send a gratuitous ARP from the lab client via service mode arp) at a low rate first and check display l2vpn mac-address vsi VPN-1314 on the destination Leaf. Otherwise the stream is flooded as unknown unicast into every port of the VSI (including the DC-firewall channel for VLAN 1314/1315) and into the trex-mon ports - at line rate this is a production incident, not a test. On the wire the packets carry TOS bit 0 and IP ID 0xFFFF (4.4).

11.5 Test: Layer 3 within VPN-CORE (Border Leaf routing)

Goal: delay added by the centralised gateway, asymmetry RZK vs RZC. Pairs: VLAN 1314 at AP X to VLAN 1315 at AP Y.

1. Permanent: class L3-BL in Grafana (K-L11 to K-L21 and C-L21 to C-L22). Compare RZC-sourced vs RZK-sourced latency: the difference is the double DCI crossing to the gateway in Kalk.
2. Ad-hoc with the monitoring instance: add a temporary group with a new profile file (copy monitoring-default.yaml, keep only an l3-bl group with the wanted pairs) and trexctl apply -p l3bl-adhoc; EMU resolves 10.3.14.1 / 10.3.15.1 for each source AP; verify with trexctl emu show (resolved MAC = sw-rzk-BL Vsi-interface MAC). Remove with trexctl stop -p l3bl-adhoc.
3. Line-rate: pin p4 to K-L11 (VLAN 1314 side) and p5 to K-BL (VLAN 1315 side): trexctl test throughput --src K-L11 --src-vlan 1314 --dst K-BL --dst-vlan 1315 --sizes 512,1518 --duration 60. The runner puts the lab ports in L3 service mode (set_l3_mode with VLAN) to resolve the gateways, then builds streams with the gateway MACs. Expect line rate on the 5950 (hardware routing); any loss points to the single 10G AP link being the bottleneck (both directions share it in bidirectional mode).

Switch-side checks during the test: display counters rate inbound interface Ten-GigabitEthernet1/3/1 on sw-rzk-L11; display arp vpn-instance VPN-CORE on sw-rzk-BL; display interface Vsi-interface914 and Vsi-interface915 for packet counters. VALIDATED

11.6 Test: Layer 3 through the DC firewall (production VLANs)

Goal: inter-VLAN latency and loss through CPF-ZV05/06 between two production VLANs whose gateway is the DC firewall. Prerequisites: TRex pools reserved in both VLANs, Leaf AP ports carry both VLANs, DC-FW rule (7.5.2) active.

1. Permanent: group l3-dcfw (50 pps). Grafana "Class detail", class L3-DCFW. A step change in latency usually indicates a cluster failover (active member moved to the other DC); confirm in the firewall logs and annotate. Because the cluster answers ARP with the active member's MAC, a failover also produces a short loss gap on these streams until the trexctl tracker has rebuilt them with the new MAC (trex_event gateway_mac_changed); this gap is a property of the measurement, not of the firewall.
2. Ad-hoc profile: as in 11.5 with class: L3, src_vlan: <A>, dst_vlan: <B>; keep pps <= 100 per stream through firewalls (budget rule).
3. Load test (coordinate with the firewall team): pin two lab ports to two APs that carry VLAN A and B, trexctl test soak --src K-L11 --src-vlan <A> --dst K-L41 --dst-vlan <B> --rate 1gbps --duration 600. Watch firewall CPU and connection table in SmartView at the same time. Increase the rate stepwise; stop when loss appears or the firewall team requests it.

11.7 Test: VPN-CORE through the Enterprise firewall (firewall zone) and stateful tests

Goal: inter-VRF/firewall path latency (stateless) and firewall performance (stateful).

Stateless (monitoring and lab)

1. Permanent: group l3-entfw (VLAN 1314 at four APs to the FW zone, 50 pps). The zone clients (option B: 10.9.90.106 at K-BL on p0 and 10.9.90.113 at C-BL on p1, VLAN 4003; option A: 10.9.90.114 at AP FW) resolve the VIP 10.9.90.1 via EMU; the inside clients resolve 10.3.14.1 on sw-rzk-BL. Verify the route for 10.9.90.0/24 on both Border Leafs first (display ip routing-table vpn-instance VPN-CORE 10.9.90.0 24; without an explicit advertisement the default route via 10.9.0.22 applies). The zone-side streams are subject to the MAC rebuild after a failover (11.6).
2. Load test: pin p4 (lab port 0) to K-L11 (VLAN 1314) and p6 (lab port 2) to the zone (FWZ: option B = K-BL with VLAN 4003, option A = AP FW with VLAN 1420): trexctl test throughput --src K-L11 --src-vlan 1314 --dst FWZ --dst-vlan 4003 --sizes 512,1518 --duration 60. Stateless UDP through a Check Point gateway creates one session per 5-tuple; the runner uses a small number of 5-tuples (default 16 per direction) so the test measures forwarding, not session setup. Learn the zone client MAC before ramping (11.4).

Stateful (ASTF) - variant 1: port addresses only (no routing changes)

ASTF port roles are fixed in TRex: even ports are client side, odd ports server side. The ASTF unit therefore uses its own two-port configuration /etc/trex/trex_lab_astf.yaml with the interface list [p4, p6]: ASTF port 0 = p4 (client, pinned to a VPN-CORE VLAN AP), ASTF port 1 = p6 (server, pinned to the zone). The pins of p5/p7 are irrelevant during an ASTF run.

1. trexctl test astf --profile lab-fw-http.yaml --mult 200 --duration 300. The runner writes trex_lab_astf.yaml (port 0: 10.3.14.151, gateway 10.3.14.1, VLAN 1314; port 1: zone lab client 10.9.90.156 for option B / .164 for option A, gateway 10.9.90.1, VLAN 4003 / 1420), stops trex-lab, starts trex-lab-astf (with --unbind-unused-ports because only two of the four NIC2 ports are used), loads an ASTF profile whose client range is the single port address and whose server range is the single zone address. With one client IP the connection rate is limited by source-port reuse (~60 000 concurrent flows); sufficient for throughput and latency, not for CPS scalability.

Stateful (ASTF) - variant 2: routed ranges (scalable)

1. Switch team: ip route-static vpn-instance VPN-CORE 10.9.92.0 24 10.3.14.151 description TRex-ASTF-clients on sw-rzk-BL (7.4.3). Firewall team: static route 10.9.93.0/24 via the zone lab client (10.9.90.156 / .164) on the Enterprise firewall and rule TRX-astf-clients to TRX-astf-servers TCP 80/443.
2. Profile lab-fw-http-routed.yaml: client range 10.9.92.1-10.9.92.254, server range 10.9.93.1-10.9.93.254, templates http_simple/http_https (TRex astf/ examples) or a pcap-based L7 profile; --mult scales connections per second.
3. Run as in variant 1; read CPS, established flows, TCP retransmissions and goodput from the ASTF dashboard; correlate with firewall CPU. Remove the static route afterwards.

# ASTF profile skeleton used by trexctl (derived from the TRex astf/http_simple.py example)
from trex.astf.api import *

class Prof1():
    def get_profile(self, tunables, **kwargs):
        ip_gen_c = ASTFIPGenDist(ip_range=["10.9.92.1", "10.9.92.254"], distribution="seq")
        ip_gen_s = ASTFIPGenDist(ip_range=["10.9.93.1", "10.9.93.254"], distribution="seq")
        ip_gen = ASTFIPGen(glob=ASTFIPGenGlobal(ip_offset="1.0.0.0"), dist_client=ip_gen_c, dist_server=ip_gen_s)
        return ASTFProfile(default_ip_gen=ip_gen,
                           cap_list=[ASTFCapInfo(file="../avl/delay_10_http_browsing_0.pcap", cps=10)])

def register():
    return Prof1()

ASTF VLAN handling: the per-port VLAN and default gateway come from port_info in trex_lab_astf.yaml (or from an ASTF topology VIF for finer control); the server side learns VLAN and MAC from the incoming SYN (TRex ASTF guide, "Topology"). Because the lab ports are pinned (single-tagged), no QinQ is involved in ASTF tests. ASTF -m is a plain multiplier of the profile's connections per second, not a rate like in stateless mode.

11.8 Test: DSCP / QoS verification

1. Permanent: groups dscp-l2-interdc and dscp-entfw (BE reference + EF/AF41/CS1); Grafana "DSCP" dashboard compares the four side by side for the same pair. In an unloaded fabric all four must be identical within jitter.
2. Under load: run a lab soak at 8-9 Gbit/s on the same AP pair with TOS 0 while the DSCP monitoring streams keep running; a divergence between EF and BE latency/loss shows whether any device in the path differentiates (fabric: expected no; firewall: depends on configuration).
3. Mark verification: trexctl capture --port 1 --filter "vlan 1314 and udp port 32005" --count 20 (wrapper around the TRex capture API) and inspect the TOS byte at the receiving end to detect remarking. Expected values on the wire are 0x01 / 0xB9 / 0x89 / 0x21, because TRex sets TOS bit 0 on statistics packets (4.3.6); any other value means the path remarked the packet.

11.9 Test: throughput / performance (stateless, lab instance)

1. Pin the two lab ports to the wanted APs (7.1.4), update inventory.yaml ports.lab, run trexctl inventory validate.
2. Decide frame sizes (64, 128, 256, 512, 1024, 1518, 9000 or IMIX) and whether bidirectional.
3. trexctl test throughput --src <AP> --dst <AP> --vlan <V> [--src-vlan/--dst-vlan for L3] --sizes 64,512,1518 --duration 60 [--bidir] [--max-rate 9gbps].
4. Result: table NDR (Mbit/s, pps) per frame size, latency avg/max at NDR and at 50%, drops at line rate; stored under /data/trexctl/reports/ and in trex_test.
5. Interpretation: the 5950 forwards at line rate for all sizes; loss at 64-byte line rate (14.88 Mpps) usually means the TRex RX side (software counters) or the X710 at small frames - check trex_port.ierrors and queue_full; use 128 bytes as the smallest meaningful size unless hardware flow statistics are confirmed to keep up.
6. Flooding safeguards (mandatory for line-rate runs): the runner first sends the reverse stream and a gratuitous ARP at low rate and verifies on the destination device that the lab-client MAC is learned (display l2vpn mac-address vsi VPN-<vlan>); it ramps the rate in steps (10% / 50% / 100%); it aborts when the RX counters of trex-mon ports show unexpected traffic (flooding into the monitoring ports); prefer the test VLANs 1314/1315 for line-rate tests. Optionally add static MAC entries for the deterministic lab-client MACs on the TX switch lab ports (mac-address static <H-H-H> interface Ten-GigabitEthernet1/1/9 vlan 3101, L2 CR R630x PDF p. 16 - verify the exact syntax with ? on the device CONFIRM). Never run the bundled ./ndr tool on this server (it targets port 4501 = trex-mon and resets all ports); NDR searches use trexctl test throughput on port 4601.

11.10 Test: convergence during planned maintenance

1. Before the window: trexctl test convergence --group l2-inter-dc,l3-bl --pps 1000 --duration 3600 re-adds the selected groups at 1000 pps (1 ms resolution; counters restart) and starts recording.
2. trexctl annotate "Maintenance: sw-rzk-L21 IRF member 2 reboot".
3. Perform the maintenance (IRF member reboot, link failover, ISSU, firewall failover ...).
4. After the window the runner restores the rates and prints the outage table: per stream and per AP pair the number of lost packets and the derived outage (drops/pps) plus the time of first and last loss. Streams through the rebooted member's AP show the full downtime (expected); other streams reveal fabric convergence time (EVPN/VXLAN re-route, LACP on servers is not covered because APs use single links). For a firewall failover the L3-DCFW and zone-side L3-ENTFW streams additionally contain the gateway-MAC detection and rebuild time of trexctl (9.5); read the pure failover time from the inside-to-zone direction, whose destination MAC (sw-rzk-BL) does not change.
5. Grafana "Loss and outages" with the annotation marks the event for later review.

11.11 Change procedure: adding a VLAN under test or an attachment point

1. IP address management reserves 16 addresses in the VLAN; firewall team adds the pool object to the TRex rules (if the VLAN's gateway is a firewall or if L3 classes will use it).
2. Edit /etc/trexctl/inventory.yaml: add the VLAN (vsi, prefix, gateway, pool_start, vrf) and append it to the vlans list of the APs that should carry it; trexctl inventory validate.
3. Render and apply the Leaf/BL configuration: trexctl inventory render-leaf --ap K-L11 prints port trunk permit vlan ... and the service-instance/xconnect vsi block; apply on the switch, save force.
4. Extend the profile (new groups or VLAN list), trexctl plan, trexctl apply.
5. Verify RX = TX for the new streams; EMU clients resolved; Grafana shows the new VLAN tag.
6. New AP: additionally patch the fibre, configure the TX switch AP port (QinQ access, new S-VLAN; add the S-VLAN to the trunk permit lists and, for RZC, to the inter-DC trunk), configure the device port, add the AP to the inventory with the next free index.
7. Commit inventory/profile changes to git with the ticket number.

11.12 Maintenance of the platform

• TRex upgrade: download the new release to /data/trex/vX.YY, run ./t-rex-64 --help and the loopback self-test (8.8) with the new version using a test configuration, then switch the /opt/trex/current symlink and restart trex-mon (monitoring gap of about 60 s; the exporter re-applies the active profile). Keep the previous release for rollback. Check the TRex release notes for Python client compatibility and re-check that scripts/trex-cfg still only stops firewalld (not nftables) and that the queue-mode selection has not changed.
• OS patching: kernel updates require the DPDK driver to rebind (automatic with vfio-pci); reboot in a maintenance window; verify hugepages and trex-mon after reboot.
• InfluxDB/Grafana upgrades: minor versions via dnf; backup first (10.6); re-check the provisioned data sources and dashboards.
• TX switch: no routine changes; firmware stays at R6301P03 (device out of support).
• Capacity: if more than ~120 latency streams are needed, a second monitoring instance on p2/p3 provides another 128; this needs bench validation because the EMU server (port 4510) is a single named daemon per host that a second --emu instance would restart (possible approaches: a second EMU server with its own port/prefix, or EMU only on one instance). More loss-only streams (up to 1023 per instance) need no second instance.

11.13 Troubleshooting matrix

12. Implementation plan

12.1 Phases and work packages

12.2 Indicative effort

12.3 Acceptance criteria

1. Server self-test: trex-mon log shows set dpdk queues mode to ONE_QUE; loopback latency < 20 us average, 0 drops at 1 kpps; a captured latency packet shows IP ID 0xFFFF, TOS bit 0 and UDP checksum 0; lab instance 0 drops at 1518-byte line rate for 60 s in loopback; management access survives a TRex restart (nftables, not firewalld).
2. QinQ path: for every AP, a stream from the monitoring instance to a client at that AP shows RX = TX for 10 minutes; display mac-address vlan <svlan> on the TX switch shows exactly the expected MACs.
3. Monitoring profile: 74 latency and 102 loss-only streams started, all L3 gateways resolved (including from Leaf APs), exporter writing without stopping traffic on reconnect, Grafana Overview without path-down streams, 24 h without false alerts after baseline tuning.
4. Firewall classes: L3-DCFW and L3-ENTFW streams RX = TX; a planned firewall cluster failover is visible as a latency step and logged by an annotation; the streams towards the firewall MAC are rebuilt automatically (gateway_mac_changed event) and recover within 15 s of the failover (or immediately if a virtual MAC is used); the inside-to-zone direction shows no loss beyond the failover window.
5. Test runners: throughput report for K-L11 to C-L21 (VLAN 1314) at 512 and 1518 bytes with the flooding safeguards active; convergence report from one maintenance event; ASTF variant-1 report with established flows > 0 and zero TCP errors at the chosen rate.
6. Operations: handbook (this document v1.0) with the baseline table; runbook procedures 11.2-11.12 executed once by the LVR team.

12.4 Risks and mitigations

12.5 Open items

Appendix A - Validated command reference and citations

Appendix B - Source documents

• Big Picture-Rechenzentren-RZKC.pdf (Visio export, 15.05.2024, A3, 1 page) - topology, loopbacks, management addresses, VRF/VNI table.
• diag_sw-rzk-S1/S2, diag_sw-rzk-L11, diag_sw-rzk-BL, diag_SW-RZC-S1/S2, diag_SW-RZC-L21, diag_sw-rzc-BL, diag_sw-rzd-BL (17./18.02.2025) - running configurations, interface tables, routing and EVPN state.
• LVR-InfoKom - Comware Switch Analysis v1.0.xlsx - models, releases, recommendations, EoS/EoSL dates.
• man_trex_documentation_v1.3.docx (MAN Truck & Bus, German) - reference design: TRex switch with dot1q tunnels, trex_client.py concept, InfluxDB/Grafana integration, Grafana usage notes.
• simplified trex topology.png - customer's target picture.
• HPE FlexFabric 5950 Switch Series documentation, Release 630x (Layer 2 - LAN Switching CG/CR, VXLAN CG/CR, EVPN CG, Layer 3 IP Routing CR, Layer 3 IP Services CR), HPE5950CMW710R6301P03RN.pdf.
• Cisco TRex documentation (manual, stateless, ASTF, EMU, STL API), v3.x, October 2026; trex-core source repository (github.com/cisco-system-traffic-generator/trex-core, master = v3.08) for the facts cited in Appendix A.
• HPE UEFI System Utilities User Guide for HPE ProLiant Gen11 servers (workload profile table).
• Review notes of 13 October 2026 (artifacts/trex-plan/REVIEW-NOTES-for-plan-author.md) and the training course derived from the plan (artifacts/trex-training/); all findings were re-verified against the sources above before being incorporated (Appendix D).

Appendix C - Glossary

Appendix D - Changes since draft v0.9 (review round 1)

A technical review of draft v0.9 (13 October 2026) raised 39 items. Each was re-verified against the trex-core source (master = v3.08), the TRex documentation, the diagnostic files and the HPE guides before the plan was changed. Items that could not be verified are marked CONFIRM in the text. Rejected items: none; two items were softened (InfluxDB "latest" tag date, Check Point UDP timeout) because they could not be verified from primary sources.