TRex Network Monitoring and Performance Test Platform

Document control

Conventions used in this document

Table of contents

1. Executive summary

LVR-InfoKom operates an HPE FlexFabric 5950 EVPN-VXLAN fabric spanning the data centres Kalk (RZK, BGP AS 65001) and Chorweiler (RZC, BGP AS 65002). The fabric has no built-in way to measure end-to-end delay, jitter and packet loss between arbitrary points, across the inter-DC link, across the Border Leaf routing functions or across the Check Point firewalls. This document describes a permanent, low-footprint measurement platform based on the open-source traffic generator Cisco TRex, plus an on-demand performance test capability, with all results stored in InfluxDB and visualised in Grafana.

The core design decisions are:

• One TRex server in RZK (HPE ProLiant, 2 x Intel X710-DA4 = 8 x 10G SFP+ DPDK ports, RHEL/Rocky 9) connected to a dedicated HPE 5950 "TX switch" in each DC. The two TX switches are joined by a dedicated dark-fibre/DWDM link so that measurements never depend on the network under test for the return path.
• Each TX switch has one 10G link to every Leaf pair, to the Border Leaf and to the local Enterprise-firewall cluster member. Each such link is an attachment point (AP). On the TX switches every AP is an isolated QinQ (802.1Q-in-802.1Q) domain: the outer tag (S-VLAN) selects the AP, the inner tag is the production VLAN. The TX switch therefore needs no production VLAN configuration at all; adding a VLAN under test only touches the Leaf port and the TRex profile. The QinQ design is validated against the 5950 R630x Layer 2 configuration guide (section 7.1).
• Two independent TRex instances on the same server: a monitoring instance (software mode, TRex EMU for ARP, QinQ frames, up to 1023 measured streams, continuous latency/loss measurement at low packet rates) and a lab instance (hardware-assisted stateless line-rate tests or stateful ASTF firewall tests) on separate NICs, so that performance tests never interrupt the permanent monitoring.
• Traffic classes covering all requirements: L2 inside one VLAN (intra- and inter-DC), L3 within VPN-CORE routed by the Border Leaf (using the already existing "Performance-Test" VLANs 1314/1315), L3 between production VLANs routed by the DC firewall, L3 from VPN-CORE through the Enterprise firewall into a new dedicated TRex firewall zone (the inter-VRF/firewall path), DSCP-marked variants of the above, and stateful TCP tests through the Enterprise firewall.
• A project-specific Python orchestrator trexctl (specified in section 9) generates streams from a YAML inventory, provisions EMU clients, runs tests, and exports statistics to InfluxDB every two seconds. Grafana dashboards and alert rules are specified in section 10.

The expected monitoring footprint is below 50 Mbit/s and 50 kpps in total (roughly 180 streams at 50-200 pps each, 128-byte packets), i.e. negligible for the fabric and the firewalls. Sections 7 and 8 contain step-by-step configuration and installation procedures, section 11 the test and operations procedures, and section 12 the phased implementation plan with acceptance criteria and open items.

2. Requirements, scope and decisions

2.1 Customer requirements (as stated)

1. TRex server with several NICs.
2. Send traffic inside multiple production VLANs to measure delay permanently and to run occasional performance tests.
3. Traffic types: Layer 2 (inside one VLAN); Layer 3 within a single VRF (default routing table or a single vpn-instance); Layer 3 between different VRFs (vpn-instances); traffic through the firewall.
4. The TRex server is connected to one of two dedicated switches; the two switches are in different DCs (RZK, RZC), one per DC, each with physical connections to every Leaf, Border Leaf and FW in its DC.
5. Integration with Grafana for visualisation.
6. Reference: the MAN Truck & Bus TRex documentation v1.3 (German) as inspiration, not a 1:1 copy.

2.2 Interpretation against the real fabric

The diagnostic files show that the fabric uses centralised Layer 3: the Leafs are pure Layer 2 VTEPs (VLAN to VSI mapping via Ethernet service instances), and all vpn-instance gateways live on the Border Leafs. Most server VLANs do not have a switch gateway at all - their default gateway is the DC firewall (CPF-ZV05/06). The default routing table (public instance) carries only the underlay (loopbacks, OSPF 172, BGP EVPN), so "L3 in the default routing table" has no end-system meaning in this fabric. The requirement list therefore maps to these measurable paths:

2.3 Decisions taken with the customer (12 October 2026)

2.4 Assumptions

• CPF-ZV01 (RZK) and CPF-ZV02 (RZC) are the two members of one stretched Check Point ClusterXL (sync VLAN 123 "Sync_FW-innen" is a stretched VSI; a single OSPF neighbour 10.9.0.22 is seen from sw-rzk-BL in VPN-CORE). The same is assumed for the DC firewall CPF-ZV05/06 (sync VLAN 196). The TRex firewall zone is therefore designed as one Layer 2 domain stretched over both TX switches. DECISION If they are independent clusters the zone is simply split into two S-VLANs (section 4.3.5).
• The spare 5950s are 4-slot chassis (JH404A) with at least one 24-port SFP+ module. If they are 32-port QSFP28 units (JH321A), 100G ports are split into 4 x 10G breakout ports (using tengige, see 7.1.4).
• The production VLANs to be monitored and their reserved TRex address ranges are supplied by LVR IP address management; this document uses the existing test VLANs 1314/1315 plus placeholders.
• Firewall rule changes for the TRex address objects are approved by the firewall team (section 7.5).
• The 5950 series is end of sale; JH321A reached end of support life on 28.02.2026 and JH404A reaches it on 31.10.2026 (from "LVR-InfoKom - Comware Switch Analysis v1.0.xlsx"). Using spare 5950s as TX switches is acceptable for a test tool, but no HPE software support should be expected for them.

3. Environment analysis (as found)

Sources: diag_*_20250217/18.txt for sw-rzk-S1/S2, sw-rzk-L11, sw-rzk-BL, SW-RZC-S1/S2, SW-RZC-L21, sw-rzc-BL, sw-rzd-BL; "Big Picture-Rechenzentren-RZKC.pdf" (page 1); "LVR-InfoKom - Comware Switch Analysis v1.0.xlsx".

3.1 Devices and roles

3.2 Layer 2 / Layer 3 model relevant for TRex

• Access model. Every access port is a trunk with one Ethernet service instance per VLAN: service-instance N / encapsulation s-vid N / xconnect vsi VPN-N. VSIs are EVPN-VXLAN (vxlan N, route-distinguisher auto, vpn-target 1:N). Example: sw-rzk-L11 Ten-GigabitEthernet1/3/3 (ESH-ZV12 trunk) carries VLANs 147, 176, 252, 1054, 1080-1082, 1280, 1309, 1380, 1383, 1397-1398 this way. VALIDATED
• VLAN 1314 and 1315 ("Performance-Test VLAN 1314/1315", VSI VPN-1314/1315, "Tier 3 Services DataCenter") already exist on all switches, have gateways 10.3.14.1/24 and 10.3.15.1/24 on Vsi-interface914/915 of sw-rzk-BL bound to VPN-CORE (DHCP relay configured) and are currently not mapped to any access port on the Leafs examined. They are ideal carriers for the permanent monitoring and for the L3-BL class. Note that the gateway exists on sw-rzk-BL only, so RZC-sourced routed traffic always transits the DCI link to Kalk.
• VPN-CORE routing. Static routes for 10.0.0.0/16 ... 10.245.0.0/16 point to the DC firewall (10.9.0.35 on Vsi44); OSPF 1 runs on the transfer VSI 40 (10.9.0.16/29) with the Enterprise firewall (10.9.0.22), sw-rzc-BL (10.9.0.18) and 10.9.0.19, redistributing direct and static. A new subnet advertised by the Enterprise firewall via OSPF is therefore reachable from VPN-CORE without any switch change.
• STP. PVST is enabled; STP is disabled for most server VLANs (undo stp vlan ... enable, incl. 1082-1399 on sw-rzk-L11). Server ports have no edge-port configuration.
• Previous test tooling. An ACL/QoS policy iperf-counting (UDP destination port 5001 accounting) is applied inbound on the DC-FW "Vlan-all" channel and on RAGG1, and port FGE1/4/26 of sw-rzk-BL is labelled "ALLEGRO-TEST". This confirms earlier ad-hoc performance measurements and the need for a permanent solution.
• Free 10G SFP+ ports (as of 2025-02-17, must be re-verified): sw-rzk-L11 XGE1/3/1 (default config, down); SW-RZC-L21 XGE1/4/4; sw-rzk-BL XGE1/3/6; sw-rzc-BL XGE1/3/14. Slot 2 (L11) / slot 4 (BL) modules are 10GBase-T (JH182A) and are not suitable for SFP+ cabling.
• Jumbo frames. 5950 Ethernet interfaces accept jumbo frames up to 9416 bytes by default (Layer 2 Configuration Guide, PDF p. 41), so the 4 additional QinQ bytes need no MTU change on the TX switches.
• Unused VLAN IDs. Across all nine configurations the ranges 2907-3999 and 1617-2903 are unused; 3000-3099 is reserved on the Border Leafs (reserve-vlan-interface 3000 to 3099 global). The S-VLAN plan therefore uses 3101-3399.

3.3 Lessons taken from the reference design (MAN, man_trex_documentation_v1.3)

4. Solution concept

4.1 Components and overall architecture

4.2 Attachment-point model: QinQ on the TX switches

The TX switch must connect a small number of TRex ports to many production devices without ever bridging production VLANs between two attachment points (which would create a loop through the stretched VXLAN VLANs). Comware QinQ solves this elegantly: a QinQ-enabled port adds the port PVID as an outer tag to every incoming frame, tagged or untagged, and removes it on egress (Layer 2 Configuration Guide, "Configuring QinQ", PDF pp. 219-222). By giving every AP-facing port its own PVID (S-VLAN), each AP becomes an isolated two-port bridge domain: {AP port, TRex port(s)}.

Two kinds of TRex-facing ports exist on sw-rzk-TX1:

4.3 Traffic classes and paths

4.3.1 Class L2 - Layer 2 inside one VLAN

Source and destination TRex clients are in the same production VLAN (C-VLAN) at two different APs. The destination MAC is the MAC of the remote TRex client (known from the inventory - no ARP needed; EMU gratuitous ARP keeps the remote MAC learned in the fabric). Path: TX switch - Leaf A - Spine - Leaf B (VXLAN) - TX switch. Inter-DC variant: Leaf A - Spine - sw-rzk-BL - RAGG1 (MACsec) - sw-rzc-BL - Spine - Leaf B - sw-rzc-TX1 - fibre - sw-rzk-TX1. Measures fabric forwarding, VXLAN encapsulation, DCI link and IRF behaviour.

4.3.2 Class L3-BL - Layer 3 within VPN-CORE routed on the Border Leaf

Source in VLAN 1314 (gateway 10.3.14.1 on sw-rzk-BL Vsi-interface914), destination in VLAN 1315 (10.3.15.1, Vsi-interface915), both bound to VPN-CORE. Destination MAC = gateway MAC resolved by EMU ARP. Path: Leaf A - fabric - sw-rzk-BL (routing, VXLAN re-encapsulation) - fabric - Leaf B. Sources in RZC also cross RAGG1 twice (gateway only in Kalk). Measures the centralised routing hop and its asymmetry.

4.3.3 Class L3-DCFW - Layer 3 between production VLANs routed by the DC firewall

Source in production VLAN A and destination in production VLAN B, both with their default gateway on CPF-ZV05/06 (VIP). Path: Leaf - fabric - sw-rzk-BL/sw-rzc-BL - BAGG31 "Vlan-all" - DC-FW (active member) - BAGG31 - fabric - Leaf. Requires a DC-FW rule permitting the TRex UDP port range between the TRex host objects. Measures firewall forwarding latency and its cluster state (active member location changes the path).

4.3.4 Class L3-ENTFW - VPN-CORE through the Enterprise firewall into the TRex firewall zone

Source in VLAN 1314 (VPN-CORE); destination TRex client in the new TRex firewall zone (VLAN 1420, 10.9.90.0/24, proposal) which is a dedicated interface of the Enterprise-FW cluster connected through AP "FW" (S-VLAN 3300, stretched over both TX switches so that both cluster members see the same Layer 2 domain). Path: Leaf - fabric - sw-rzk-BL (VPN-CORE) - Vsi40 transfer (10.9.0.16/29) - ENT-FW active member - zone interface - TX switch - TRex. The firewall advertises 10.9.90.0/24 into OSPF 1 (it is already an OSPF neighbour of both Border Leafs in VPN-CORE); alternatively a static route on both Border Leafs. This is the inter-VRF/firewall path requested; extending it to VPN-WAN/VPN-WLAN later only requires TRex clients in those VRFs.

4.3.5 Firewall zone design options

4.3.6 Class DSCP - QoS verification

Selected L2 and L3 pairs are duplicated with different TOS bytes (EF = 0xB8, AF41 = 0x88, CS1 = 0x20, BE = 0x00). Because the fabric configuration contains no QoS trust or scheduling policy, equal treatment is expected; differences in latency or loss under load (lab tests) would indicate a (re)marking or queuing effect, e.g. on the firewalls. On the TX switches the 802.1p value of the S-tag is copied from the C-tag only when qos trust dot1p is configured (L2 Configuration Guide PDF p. 224); this is irrelevant for the measurement and not configured.

4.3.7 Class LAB - performance tests

• Stateless throughput between two pinned APs (or AP to itself through the FW zone): binary search for the no-drop rate (RFC 2544 style) with fixed frame sizes or IMIX, per direction up to 10 Gbit/s per TRex port, hardware flow statistics.
• Stateful (ASTF) client side pinned to a VPN-CORE VLAN AP, server side pinned to the FW zone AP: HTTP/HTTPS-like TCP profiles, connections per second, concurrent sessions, goodput through the Enterprise firewall.
• Convergence tests during planned maintenance (IRF member reboot, link failure, firewall failover): the monitoring instance already provides 5 ms loss resolution; for 1 ms resolution the affected streams are temporarily raised to 1000 pps.

4.4 Measurement methodology

• Per-stream statistics. Every stream carries a TRex packet group id (pg_id) in the IPv4 identification field (TRex reserves 0xff00-0xffff) and, for latency streams, a timestamp in the payload. TRex counts TX and RX packets per pg_id and computes one-way latency (average, minimum, maximum, last-window maximum, RFC 3550 jitter, logarithmic histogram) because TX and RX happen on the same server clock. Accuracy is in the microsecond range (software mode, DPDK polling). All streams of the monitoring instance are latency streams (STLFlowLatencyStats).
• Resolution. Loss granularity = 1 / pps: 200 pps gives 5 ms, 1000 pps gives 1 ms. Latency streams are processed by the RX core; TRex recommends keeping the total latency-stream rate well below 5 Mpps (stateless guide, "Latency streams"). The monitoring budget (~30 kpps) is three orders of magnitude below that.
• Fixed offsets. Every RZC stream includes the TX-to-TX fibre once (RX direction) and the TX switch hops. The fibre offset is roughly 5 us per km and can be measured directly: run an L2 stream between two RZC APs in the same VLAN and the same stream between two RZK APs; the difference (minus the identical fabric hops) is the fibre contribution. Report absolute values together with the measured baseline rather than subtracting.
• Direction. Streams are unidirectional; each AP pair is monitored in both directions (two pg_ids) so that asymmetric paths (gateway in Kalk only, active firewall member location) become visible.
• Stream limits. trex-mon: 1023 concurrent pg_ids (X710 in software mode, stateless guide Table 14). trex-lab in hardware mode: 127 pg_ids on X710 and no RX byte counters (packet counters only); use --no-hw-flow-stat or software mode when byte counts are required.
• Packet sizes. Monitoring: 128 bytes L2 without FCS (room for two tags + IP + UDP + TRex latency payload). DSCP class: 256 bytes. Lab: 64/128/256/512/1024/1518 or IMIX; jumbo up to 9000 bytes is possible (fabric supports 9416 by default).

4.5 Permanent monitoring matrix and packet budget

Hard budget for the permanent profile: not more than 600 streams, 50 kpps and 50 Mbit/s in total; not more than 100 pps per stream through a firewall. The profile is a YAML file (section 9.4), so pairs and rates are adjusted without code changes.

4.6 Addressing, VLAN and naming plan

4.6.1 Attachment points and S-VLANs

4.6.2 VLANs under test and TRex address pools

Candidate production VLANs seen in the configurations that are already test-oriented and therefore low-risk: 176 ("DC-Testnetz RZ Deutz/RZ-Chorweiler", 10.180/24-DC), 171 ("Server Testnetz und Altnetz von Deutz"), 724 ("Test Server Netz"), 1395/1396 ("t3_139x_ngfw-test"). Their subnets and gateways are not visible in the switch configurations (gateway on the DC firewall) and must be supplied by LVR. TBD

4.6.3 Deterministic TRex client addressing

For VLAN V with pool start P and attachment point index i (table 4.6.1): IP = P + (i - 1). MAC = 02:54:52:<ii>:<VV>:<VV> where ii is the AP index and VVVV the VLAN id in hex (locally administered, unique per AP and VLAN). Example: AP K-L11 (1) in VLAN 1314 (0x0522): 10.3.14.101, 02:54:52:01:05:22. Clients attached to the lab instance use index offset 50 (02:54:52:33.., IP pool second half) so that monitoring and lab clients never collide.

4.6.4 UDP port and pg_id plan

4.6.5 Names and management addresses

4.7 Two TRex instances on one server

4.8 Security and operational impact

• TRex data ports are DPDK devices without a kernel IP stack; only EMU answers ARP/ICMP for the configured client addresses. The server is managed exclusively via the management NIC (SSH, HTTPS). No routing or forwarding between data ports is possible.
• TRex clients appear in the fabric as ordinary hosts (MAC/IP in EVPN). Their traffic is restricted by firewall rules to the UDP range 32000-32099 and ICMP between TRex objects. Production hosts never receive TRex traffic (destinations are always TRex clients).
• Packet budget (section 4.5) is enforced in trexctl (profile validation refuses to apply a profile above the budget without --force).
• Monitoring streams are UDP with fixed 5-tuples: one firewall session per stream; sessions are kept alive by the continuous traffic (Check Point UDP virtual session timeout 40 s by default).
• Change management: adding a VLAN under test touches the Leaf AP port (service instance), possibly a firewall rule and the trexctl inventory - all low risk. The TX switches never need production VLAN changes.
• Failure modes: a TX switch or fibre failure only affects measurements, never production traffic. A TRex server failure is detected by Grafana (no data) and by systemd restart.

5. Bill of materials

5.1 TRex server

5.2 TX switches and attachment links

6. Physical design and cabling

6.1 Port assignment sw-rzk-TX1 (RZK)

Port names assume a 5950 4-slot chassis with a 24-port SFP+ module in slot 1 and a QSFP28 module in slot 2 CONFIRM. Adjust if the module layout differs.

6.2 Port assignment sw-rzc-TX1 (RZC)

6.3 Cabling and labelling rules

• Label both ends of every AP link with the AP code (e.g. TRX K-L11 / sw-rzk-L11 XGE1/3/1). Record the labels in the trexctl inventory (device_port field) - the inventory is the single source of truth and is also used to generate switch configuration snippets.
• Use one IRF member for the AP link as decided (member 1 ports in the proposals above). If a Leaf IRF member is rebooted during maintenance, the AP goes down and the monitoring shows this as "path down" for that AP - a desired effect, but it must be understood when interpreting results.
• Keep the TRex-to-TX cabling order identical to the TRex port numbering (p0 = lowest PCI address of NIC1). Verify with dpdk_setup_ports.py -s and link LEDs before the first test.
• The inter-DC TX link carries only S-VLANs; its MTU does not need adjustment (5950 default jumbo frame limit 9416 bytes). If a DWDM transponder with a lower MTU is used, keep the TRex packet size below the transponder limit minus 8 bytes.

7. Switch and firewall configuration

All Comware commands in this section follow the 5950 R630x documentation and the configuration style found in the diagnostic files. Commands marked VALIDATED use syntax already present on the customer's 5950s or documented in the cited guides; values marked CONFIRM depend on the physical port layout of the spare units or on live state.

7.1 TX switch sw-rzk-TX1

7.1.1 Preparation

1. Factory-reset the spare 5950 (restore factory-default, reboot) and upgrade to R6301P03 (same release as the Border Leafs) using the HPE release notes HPE5950CMW710R6301P03RN.pdf. CONFIRM file names on the flash.
2. Confirm the module layout (display device) and derive the interface names. The configuration below assumes a 24-port SFP+ module in slot 1 (Ten-GigabitEthernet1/1/x) and a QSFP28 module in slot 2 (HundredGigE1/2/x).
3. Connect the management port to the switch management network and apply the base configuration via console.

7.1.2 Base configuration

# --- system -------------------------------------------------------------
system-view
 sysname sw-rzk-TX1
 clock timezone COLOGNE add 01:00:00
 clock summer-time MEST 02:00:00 March last Sunday 02:00:00 October last Sunday 01:00:00
 clock protocol ntp
# management VRF, same pattern as the production switches
ip vpn-instance MGMT
 route-distinguisher 10:1
 description MGMT/Monitoring
 vpn-target 10:1 import-extcommunity
 vpn-target 10:1 export-extcommunity
quit
interface M-GigabitEthernet0/0/0
 description mgmt_oobm
 ip binding vpn-instance MGMT
 ip address <MGMT-IP-TX-K> <MASK>
quit
 ip route-static vpn-instance MGMT 0.0.0.0 0 <MGMT-GW> description mgmt-DEFAULT-GATEWAY
 ntp-service enable
 ntp-service source M-GigabitEthernet0/0/0
 ntp-service unicast-server 10.0.31.21 vpn-instance MGMT
 ntp-service unicast-server 10.0.31.23 vpn-instance MGMT
 info-center loghost 10.0.61.10 vpn-instance MGMT
# SNMPv3 / SSH / local users: copy the LVR standard from an existing Leaf
# (prefer SNMPv3 as recommended in the switch analysis)
 ssh server enable
 lldp global enable
# --- no spanning tree on the TX switch (see 4.2) -------------------------
 undo stp global enable
# --- S-VLANs: one per attachment point, plus the stretched FW zone -------
 vlan 3101 to 3110
 vlan 3201 to 3210
# (bulk creation stays in system view; naming requires VLAN view)
vlan 3101
 name TRX-AP-K-L11
 quit
vlan 3102
 name TRX-AP-K-L12
 quit
vlan 3103
 name TRX-AP-K-L21
 quit
vlan 3104
 name TRX-AP-K-L31
 quit
vlan 3105
 name TRX-AP-K-L41
 quit
vlan 3110
 name TRX-AP-K-BL
 quit
vlan 3300
 name TRX-AP-FW-ZONE-stretched
 quit

QinQ, STP and VLAN commands: 5950 Layer 2 Configuration Guide R630x, PDF pp. 219-222 (QinQ), p. 306 (undo stp global enable); VPN/NTP/mgmt pattern identical to sw-rzk-L11 running configuration. VALIDATED

7.1.3 Attachment-point ports (QinQ access)

# one isolated S-VLAN per attachment point; QinQ pushes the PVID on ingress and pops it on egress
interface Ten-GigabitEthernet1/1/1
 description TRX AP K-L11 --> sw-rzk-L11 XGE1/3/1
 port link-type access
 port access vlan 3101
 qinq enable
quit
interface Ten-GigabitEthernet1/1/2
 description TRX AP K-L12 --> sw-rzk-L12 <PORT>
 port link-type access
 port access vlan 3102
 qinq enable
quit
interface Ten-GigabitEthernet1/1/3
 description TRX AP K-L21 --> sw-rzk-L21 <PORT>
 port link-type access
 port access vlan 3103
 qinq enable
quit
interface Ten-GigabitEthernet1/1/4
 description TRX AP K-L31 --> sw-rzk-L31 <PORT>
 port link-type access
 port access vlan 3104
 qinq enable
quit
interface Ten-GigabitEthernet1/1/5
 description TRX AP K-L41 --> sw-rzk-L41 <PORT>
 port link-type access
 port access vlan 3105
 qinq enable
quit
interface Ten-GigabitEthernet1/1/6
 description TRX AP K-BL --> sw-rzk-BL XGE1/3/6
 port link-type access
 port access vlan 3110
 qinq enable
quit
interface Ten-GigabitEthernet1/1/7
 description TRX AP FW-ZONE --> CPF-ZV01 <FW-PORT>
 port link-type access
 port access vlan 3300
 qinq enable
quit
# unused ports administratively down
interface range Ten-GigabitEthernet1/1/8 Ten-GigabitEthernet1/1/13 to Ten-GigabitEthernet1/1/20
 shutdown
quit

7.1.4 TRex-facing ports

# multiplexed monitoring ports: plain 802.1Q trunk, frames stay double-tagged towards TRex
interface Ten-GigabitEthernet1/1/21
 description TRX p0 trex-mon (double-tagged, all APs)
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 3101 to 3110 3201 to 3210 3300
quit
interface Ten-GigabitEthernet1/1/22
 description TRX p1 trex-mon (double-tagged, all APs)
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 3101 to 3110 3201 to 3210 3300
quit
interface Ten-GigabitEthernet1/1/23
 description TRX p2 trex-mon reserve
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 3101 to 3110 3201 to 3210 3300
 shutdown
quit
interface Ten-GigabitEthernet1/1/24
 description TRX p3 trex-mon reserve
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 3101 to 3110 3201 to 3210 3300
 shutdown
quit
# pinned lab ports: QinQ access, PVID selects the attachment point; TRex sends single-tagged frames
interface Ten-GigabitEthernet1/1/9
 description TRX p4 trex-lab pinned [K-L11]
 port link-type access
 port access vlan 3101
 qinq enable
quit
interface Ten-GigabitEthernet1/1/10
 description TRX p5 trex-lab pinned [K-BL]
 port link-type access
 port access vlan 3110
 qinq enable
quit
interface Ten-GigabitEthernet1/1/11
 description TRX p6 trex-lab pinned [FW-ZONE]
 port link-type access
 port access vlan 3300
 qinq enable
quit
interface Ten-GigabitEthernet1/1/12
 description TRX p7 trex-lab pinned [C-L21]
 port link-type access
 port access vlan 3204
 qinq enable
quit

7.1.5 Inter-DC trunk

interface HundredGigE1/2/1
 description TRX inter-DC --> sw-rzc-TX1 HGE1/2/1 (dark fibre)
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 3201 to 3210 3300
quit
save force

If the inter-DC path is 10G, use a Ten-GigabitEthernet port with the same VLAN configuration. If 40G is used on a QSFP28 port, no breakout is needed (FortyGigE transceiver in a 100G port is supported on the 8-port QSFP28 module; confirm with the installation guide for the module in use CONFIRM).

7.1.6 Variant: spare units are 5950 32QSFP28 (JH321A)

Split 100G ports into four 10G breakout ports with a QSFP28-to-4xSFP+ breakout cable (Layer 2 Configuration Guide R630x, "Splitting a 100-GE interface and combining 10-GE breakout interfaces", PDF pp. 37-38). The resulting interfaces are named Ten-GigabitEthernet1/0/N:1 ... :4 and support the same QinQ configuration. VALIDATED (syntax) / CONFIRM (cable type per installation guide).

interface HundredGigE1/0/1
 using tengige
quit
interface Ten-GigabitEthernet1/0/1:1
 description TRX AP K-L11 --> sw-rzk-L11 XGE1/3/1
 port link-type access
 port access vlan 3101
 qinq enable
quit

7.2 TX switch sw-rzc-TX1

Identical base configuration with sysname sw-rzc-TX1, its own management address and the RZC S-VLANs:

 vlan 3201 to 3210
vlan 3201
 name TRX-AP-C-L11
 quit
vlan 3202
 name TRX-AP-C-L12
 quit
vlan 3203
 name TRX-AP-C-L13
 quit
vlan 3204
 name TRX-AP-C-L21
 quit
vlan 3205
 name TRX-AP-C-L22
 quit
vlan 3206
 name TRX-AP-C-L23
 quit
vlan 3210
 name TRX-AP-C-BL
 quit
vlan 3300
 name TRX-AP-FW-ZONE-stretched
 quit
interface Ten-GigabitEthernet1/1/1
 description TRX AP C-L11 --> sw-rzc-L11 <PORT>
 port link-type access
 port access vlan 3201
 qinq enable
quit
# ... XGE1/1/2-6 analogously with 3202-3206 ...
interface Ten-GigabitEthernet1/1/4
 description TRX AP C-L21 --> SW-RZC-L21 XGE1/4/4
 port link-type access
 port access vlan 3204
 qinq enable
quit
interface Ten-GigabitEthernet1/1/7
 description TRX AP C-BL --> sw-rzc-BL XGE1/3/14
 port link-type access
 port access vlan 3210
 qinq enable
quit
interface Ten-GigabitEthernet1/1/8
 description TRX AP FW-ZONE --> CPF-ZV02 <FW-PORT>
 port link-type access
 port access vlan 3300
 qinq enable
quit
interface HundredGigE1/2/1
 description TRX inter-DC --> sw-rzk-TX1 HGE1/2/1 (dark fibre)
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 3201 to 3210 3300
quit
interface range Ten-GigabitEthernet1/1/9 to Ten-GigabitEthernet1/1/24
 shutdown
quit
save force

7.3 Leaf attachment ports

The Leaf port is configured exactly like an existing server trunk: 802.1Q trunk, one Ethernet service instance per VLAN under test, cross-connected to the existing VSI. The pattern is taken from sw-rzk-L11 Ten-GigabitEthernet1/3/3 (running configuration 2025-02-17). VALIDATED Only the VLAN list differs per AP and is generated by trexctl inventory render-leaf (section 9.6).

7.3.1 Template

interface <LEAF-PORT>
 port link-mode bridge
 description TRX AP <AP-CODE> --> <TX-SWITCH> <TX-PORT> (TRex attachment point)
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 1314 to 1315 <PROD-VLAN-A> <PROD-VLAN-B>
 stp edged-port
 #
 service-instance 1314
  encapsulation s-vid 1314
  xconnect vsi VPN-1314
 #
 service-instance 1315
  encapsulation s-vid 1315
  xconnect vsi VPN-1315
 #
 service-instance <PROD-VLAN-A>
  encapsulation s-vid <PROD-VLAN-A>
  xconnect vsi VPN-<PROD-VLAN-A>
 #
 service-instance <PROD-VLAN-B>
  encapsulation s-vid <PROD-VLAN-B>
  xconnect vsi VPN-<PROD-VLAN-B>

stp edged-port (L2 Configuration Guide PDF p. 294) makes the port forward immediately in the PVST-enabled VLANs. It is not used on the existing server ports; if the team prefers to stay consistent, omit it and accept a 30 s forwarding delay after link-up. Do not enable BPDU guard on these ports unless the BPDU behaviour of the TX switch has been confirmed (section 4.2).

7.3.2 Example: sw-rzk-L11 (AP K-L11)

system-view
interface Ten-GigabitEthernet1/3/1
 port link-mode bridge
 description TRX AP K-L11 --> sw-rzk-TX1 XGE1/1/1 (TRex attachment point)
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 1314 to 1315
 stp edged-port
 #
 service-instance 1314
  encapsulation s-vid 1314
  xconnect vsi VPN-1314
 #
 service-instance 1315
  encapsulation s-vid 1315
  xconnect vsi VPN-1315
quit
save force

Prerequisite check before configuring: the VSIs exist on the Leaf (display l2vpn vsi name VPN-1314 verbose) and the port is really unused (display interface Ten-GigabitEthernet1/3/1, display current-configuration interface Ten-GigabitEthernet1/3/1). CONFIRM

7.3.3 Example: SW-RZC-L21 (AP C-L21)

interface Ten-GigabitEthernet1/4/4
 port link-mode bridge
 description TRX AP C-L21 --> sw-rzc-TX1 XGE1/1/4 (TRex attachment point)
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 1314 to 1315
 stp edged-port
 #
 service-instance 1314
  encapsulation s-vid 1314
  xconnect vsi VPN-1314
 #
 service-instance 1315
  encapsulation s-vid 1315
  xconnect vsi VPN-1315
quit
save force

SW-RZC-L21 runs R6205P03. The commands are identical on that release (its own running configuration uses the same service-instance/xconnect vsi syntax), but the switch analysis recommends upgrading it to R6301P03 anyway.

7.4 Border Leaf attachment ports and optional routes

7.4.1 sw-rzk-BL (AP K-BL)

interface Ten-GigabitEthernet1/3/6
 port link-mode bridge
 description TRX AP K-BL --> sw-rzk-TX1 XGE1/1/6 (TRex attachment point)
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 1314 to 1315
 stp edged-port
 #
 service-instance 1314
  encapsulation s-vid 1314
  xconnect vsi VPN-1314
 #
 service-instance 1315
  encapsulation s-vid 1315
  xconnect vsi VPN-1315
quit

sw-rzk-BL already uses this access style (e.g. BAGG11 "CPF-ZV01 - MAIN - DMZ" with service instances 125, 137, ...). VALIDATED Note that the BL runs hardware-resource vxlan border; adding two more service instances is far below any resource limit.

7.4.2 sw-rzc-BL (AP C-BL)

interface Ten-GigabitEthernet1/3/14
 port link-mode bridge
 description TRX AP C-BL --> sw-rzc-TX1 XGE1/1/7 (TRex attachment point)
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 1314 to 1315
 stp edged-port
 #
 service-instance 1314
  encapsulation s-vid 1314
  xconnect vsi VPN-1314
 #
 service-instance 1315
  encapsulation s-vid 1315
  xconnect vsi VPN-1315
quit

7.4.3 Optional: static routes for scalable ASTF tests (sw-rzk-BL only)

Only required for the stateful test variant with routed client ranges (section 11.7, option B). The TRex ASTF client port (lab port p4 pinned to a VLAN-1314 AP) gets 10.3.14.151; the client range 10.9.92.0/24 is routed to it. OSPF 1 already redistributes static routes (import-route static), so the Enterprise firewall learns the return path automatically.

 ip route-static vpn-instance VPN-CORE 10.9.92.0 24 10.3.14.151 description TRex-ASTF-clients

Syntax identical to the existing VPN-CORE static routes on both Border Leafs. VALIDATED Remove the route after the test campaign (undo ip route-static vpn-instance VPN-CORE 10.9.92.0 24 10.3.14.151).

7.5 Firewall requirements (Check Point)

Firewall configuration is performed by the LVR firewall team in SmartConsole/Gaia; this section lists the required objects and rules. Object names are proposals.

7.5.1 Enterprise firewall cluster CPF-ZV01/ZV02 - TRex zone

7.5.2 DC firewall cluster CPF-ZV05/ZV06 - inter-VLAN streams

Class L3-DCFW needs rules between the TRex pools of the selected production VLANs: TRX-mon-<VLAN-A> <-> TRX-mon-<VLAN-B>, services TRX-udp and ICMP, Accept, Log; no NAT. No interface changes are needed on the DC firewall.

7.6 Verification commands

7.7 Rollback

• Leaf/BL: undo service-instance N for each instance, then default on the interface (or restore the saved pre-change interface block), save force. The VSIs themselves are untouched.
• TX switches: stand-alone devices; power-off is a complete rollback for the fabric.
• Firewall: disable the TRex rules; remove the zone interface.
• Always take display current-configuration snapshots before and after a change and store them in the project archive (/data/trexctl/archive/).

8. TRex server installation

Target: Rocky Linux 9 / RHEL 9 (minimal), TRex v3.08, two TRex instances, Python 3.12 virtual environment for trexctl. Commands are shown for Rocky 9; RHEL 9 is identical apart from subscription handling. Everything in this section runs as root unless stated otherwise.

8.1 Firmware and BIOS (iLO / RBSU)

8.2 Operating system installation

• Rocky Linux 9 minimal ISO, language English, time zone Europe/Berlin, hostname trex-rzk-01.
• Partitioning: RAID 1 SSD pair 1: /boot/efi 1 GB, /boot 2 GB, / 120 GB (xfs), swap 16 GB; RAID 1 SSD pair 2: /data (xfs, whole device) for InfluxDB, TRex releases, logs and archives.
• Network: management NIC only (static <MGMT-IP-SERVER>); the X710 ports are left unconfigured (they will be bound to DPDK).
• Create an administrative user in group wheel; root login via SSH disabled.

8.3 Base configuration

# time, NTP (same servers as the switches), tuned profile for low latency
dnf -y update
dnf -y install chrony tuned pciutils numactl tmux git wget tar bzip2 vim-enhanced \
    python3.12 python3.12-pip kernel-devel-$(uname -r) kernel-headers gcc make \
    elfutils-libelf-devel zlib-devel firewalld policycoreutils-python-utils
sed -i 's/^pool .*/server 10.0.31.21 iburst\nserver 10.0.31.23 iburst/' /etc/chrony.conf
systemctl enable --now chronyd
tuned-adm profile network-latency

# firewall: management access only (adjust the management prefix)
firewall-cmd --permanent --new-zone=mgmt
firewall-cmd --permanent --zone=mgmt --add-source=<MGMT-PREFIX>
firewall-cmd --permanent --zone=mgmt --add-service=ssh --add-service=https
firewall-cmd --permanent --zone=public --remove-service=ssh --remove-service=cockpit --remove-service=dhcpv6-client
firewall-cmd --reload

# kernel parameters: IOMMU passthrough + 8 GB of 2 MB hugepages (TRex allocates from them)
grubby --update-kernel=ALL --args="intel_iommu=on iommu=pt default_hugepagesz=2M hugepagesz=2M hugepages=4096"
# optional CPU isolation for the TRex data-plane threads (threads 6-15 and 28-31 in the example layout of 8.5)
grubby --update-kernel=ALL --args="isolcpus=6-15,28-31 nohz_full=6-15,28-31 rcu_nocbs=6-15,28-31"
reboot

# after reboot: verify
grep -E 'HugePages_(Total|Free)' /proc/meminfo
cat /proc/cmdline
dmesg | grep -i -E 'DMAR|IOMMU' | head
lspci -nn | grep -i ethernet
for d in $(lspci -n | awk '/8086:1572/{print $1}'); do
    echo "$d numa=$(cat /sys/bus/pci/devices/0000:$d/numa_node)"
done

Intel X710 devices show PCI ID 8086:1572 (SFP+). All eight ports should report the same NUMA node (or -1 on a single-socket system).

8.4 TRex installation

mkdir -p /data/trex /opt/trex
cd /data/trex
# the TRex download server uses a certificate chain that some systems do not trust;
# if --no-check-certificate is needed, verify the archive against the checksum on the release page
wget --no-check-certificate https://trex-tgn.cisco.com/trex/release/v3.08.tar.gz
tar xzf v3.08.tar.gz
ln -sfn /data/trex/v3.08 /opt/trex/current
cd /opt/trex/current
./t-rex-64 --help | head -5          # must print the usage text
./dpdk_setup_ports.py -s             # shows NICs, drivers, NUMA, MAC addresses

8.4.1 Binding the X710 ports to DPDK

Run the interactive generator once to let TRex detect the NICs and create a baseline configuration, then split it into the two instance files of section 8.5. Select all eight X710 ports (IDs from the table), keep the default IP suggestions (they are not used in stateless mode), and let the script save to /etc/trex_cfg.yaml.

./dpdk_setup_ports.py -i
# ... choose the 8 X710 interfaces, accept defaults, save to /etc/trex_cfg.yaml
cat /etc/trex_cfg.yaml                # baseline: interfaces list and platform/thread layout
./dpdk_setup_ports.py -s              # X710 ports must now show driver vfio-pci (or igb_uio)

With IOMMU enabled the script binds the ports to vfio-pci; without IOMMU it compiles and loads igb_uio from ko/src (needs kernel-devel, fails with Secure Boot). The binding is re-applied automatically by t-rex-64 at every start.

8.5 TRex configuration files for the two instances

Replace the PCI addresses with the values from dpdk_setup_ports.py -s (NIC1 = four consecutive functions .0-.3 of the first X710, NIC2 = the second X710). Thread numbers assume a 16-core / 32-thread single socket; threads 0-5 and 16-27 remain for the OS, InfluxDB, Grafana and trexctl (c = data-plane threads per port pair; the threads list of each dual_if entry must contain at least c entries).

### /etc/trex/trex_mon.yaml - monitoring instance (NIC1, software mode, EMU)
- version: 2
  interfaces: ['31:00.0', '31:00.1', '31:00.2', '31:00.3']
  port_limit: 4
  prefix: mon
  zmq_pub_port: 4500
  zmq_rpc_port: 4501
  limit_memory: 3072
  port_bandwidth_gb: 10
  c: 2
  services_core: 5
  port_info:
    - ip: 1.1.1.1
      default_gw: 1.1.1.2
    - ip: 1.1.1.2
      default_gw: 1.1.1.1
    - ip: 1.1.2.1
      default_gw: 1.1.2.2
    - ip: 1.1.2.2
      default_gw: 1.1.2.1
  platform:
    master_thread_id: 4
    latency_thread_id: 5
    dual_if:
      - socket: 0
        threads: [6, 7]
      - socket: 0
        threads: [8, 9]

### /etc/trex/trex_lab.yaml - lab instance (NIC2, hardware mode or ASTF)
- version: 2
  interfaces: ['98:00.0', '98:00.1', '98:00.2', '98:00.3']
  port_limit: 4
  prefix: lab
  zmq_pub_port: 4600
  zmq_rpc_port: 4601
  limit_memory: 3072
  port_bandwidth_gb: 10
  c: 4
  port_info:
    - ip: 2.1.1.1
      default_gw: 2.1.1.2
    - ip: 2.1.1.2
      default_gw: 2.1.1.1
    - ip: 2.1.2.1
      default_gw: 2.1.2.2
    - ip: 2.1.2.2
      default_gw: 2.1.2.1
  platform:
    master_thread_id: 10
    latency_thread_id: 11
    dual_if:
      - socket: 0
        threads: [12, 13, 28, 29]
      - socket: 0
        threads: [14, 15, 30, 31]

The port_info entries are placeholders: in stateless mode the real client addresses are configured by EMU (monitoring) or by the trexctl test runner (lab); in ASTF mode the lab entries are overwritten by the ASTF profile run (section 11.7). Keys prefix, zmq_pub_port, zmq_rpc_port, limit_memory, services_core, c and the platform section are documented in the TRex manual (chapter "Platform configuration"). VALIDATED

8.6 systemd units

mkdir -p /etc/trex /var/log/trex
cat >/etc/systemd/system/trex-mon.service <<'EOF'
[Unit]
Description=TRex monitoring instance (stateless, software mode, EMU)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/trex/current
ExecStart=/opt/trex/current/t-rex-64 -i --software --emu --cfg /etc/trex/trex_mon.yaml --iom 0
Restart=on-failure
RestartSec=15
LimitMEMLOCK=infinity
LimitNOFILE=65536
StandardOutput=append:/var/log/trex/trex-mon.log
StandardError=append:/var/log/trex/trex-mon.log

[Install]
WantedBy=multi-user.target
EOF

cat >/etc/systemd/system/trex-lab.service <<'EOF'
[Unit]
Description=TRex lab instance (stateless, hardware flow statistics)
After=network-online.target
Conflicts=trex-lab-astf.service

[Service]
Type=simple
User=root
WorkingDirectory=/opt/trex/current
ExecStart=/opt/trex/current/t-rex-64 -i --cfg /etc/trex/trex_lab.yaml --iom 0
Restart=no
LimitMEMLOCK=infinity
StandardOutput=append:/var/log/trex/trex-lab.log
StandardError=append:/var/log/trex/trex-lab.log
EOF

cat >/etc/systemd/system/trex-lab-astf.service <<'EOF'
[Unit]
Description=TRex lab instance (advanced stateful, ASTF)
After=network-online.target
Conflicts=trex-lab.service

[Service]
Type=simple
User=root
WorkingDirectory=/opt/trex/current
ExecStart=/opt/trex/current/t-rex-64 -i --astf --cfg /etc/trex/trex_lab.yaml --iom 0
Restart=no
LimitMEMLOCK=infinity
StandardOutput=append:/var/log/trex/trex-lab.log
StandardError=append:/var/log/trex/trex-lab.log
EOF

systemctl daemon-reload
systemctl enable --now trex-mon.service
sleep 20 && tail -n 30 /var/log/trex/trex-mon.log

The lab units are started on demand (systemctl start trex-lab or trex-lab-astf); Conflicts= guarantees that only one of them owns NIC2. t-rex-64 is a wrapper script that prepares hugepages and DPDK binding before launching _t-rex-64; the --iom 0 flag suppresses the interactive text UI output in the log.

8.7 Python environment for trexctl and the console

python3.12 -m venv /opt/trexctl/venv
/opt/trexctl/venv/bin/pip install --upgrade pip
/opt/trexctl/venv/bin/pip install pyyaml scapy influxdb-client rich jsonschema pyzmq simplejson
# make the TRex client libraries importable (STL, EMU, ASTF APIs ship with the TRex release)
echo "/opt/trex/current/automation/trex_control_plane/interactive" \
   > /opt/trexctl/venv/lib/python3.12/site-packages/trex.pth
/opt/trexctl/venv/bin/python -c "from trex.stl.api import STLClient; from trex.emu.api import EMUClient; print('ok')"

TRex v3.08 added Python 3.12 support on the client side and notes that older Python versions may not work with that release; hence the dedicated 3.12 environment. Console access: cd /opt/trex/current && ./trex-console -p 4501 --async_port 4500 --emu (monitoring) or -p 4601 --async_port 4600 (lab).

8.8 Initial self-test (before connecting to the TX switch)

1. Connect NIC1 port 2 and port 3 (p2, p3) with a DAC (loopback). Start trex-mon, open the console and run a short stateless test:

   cd /opt/trex/current && ./trex-console -p 4501 --async_port 4500
   trex> portattr -a --prom on
   trex> start -f stl/udp_1pkt_simple.py -m 1kpps --port 2
   trex> tui            # RX on port 3 must equal TX on port 2; q to quit
   trex> stop -a

2. Repeat with the latency example stl/flow_stats_latency.py and check stats --ls: latency in the single-digit microsecond range confirms the server-side baseline (DPDK, no switch).
3. Start trex-lab with NIC2 p6-p7 looped and run start -f stl/imix.py -m 100% for 60 s: no drops and no "queue full" counters confirm hardware mode line rate.
4. Record the results in the acceptance protocol (section 12.3).

8.9 Installation troubleshooting

9. Orchestration tooling: trexctl specification

trexctl is a project deliverable: a Python 3.12 command-line tool and a small daemon that turn a declarative description of the environment (inventory) and of the wanted traffic (profiles) into TRex streams and EMU clients, run them, and export statistics to InfluxDB. It replaces the proprietary trex_client.py of the reference design. This section is its functional specification; implementation effort is estimated in section 12.

9.1 Architecture

9.2 Repository and file layout

/opt/trexctl/                         # git checkout (LVR internal repository)
  venv/                               # Python 3.12 virtual environment (8.7)
  trexctl/                            # package: cli.py inventory.py profile.py emu.py streams.py exporter.py tests/
  schemas/inventory.schema.json       # JSON schema for validation
  schemas/profile.schema.json
  dashboards/*.json                   # Grafana dashboards (provisioned)
  alerts/*.yaml                       # Grafana alert rules (provisioned)
/etc/trexctl/
  inventory.yaml                      # environment description (single source of truth)
  profiles/monitoring-default.yaml    # permanent profile
  profiles/dscp-check.yaml
  profiles/lab-*.yaml                 # lab test definitions
  trexctl.yaml                        # tool settings: TRex endpoints, InfluxDB URL/token, sampling interval, budget
  pgid-map.json                       # persisted pg_id allocation (auto-managed)
  secrets.env                         # INFLUX_TOKEN (mode 0600)
/data/trexctl/
  reports/                            # test reports (JSON + Markdown)
  archive/                            # switch config snapshots, pcaps
  state/                              # exporter state
/var/log/trexctl/

9.3 Inventory (inventory.yaml)

version: 1
site:
  name: LVR RZK/RZC
  mac_oui: "02:54:52"            # locally administered prefix for TRex clients
trex:
  mon:  { server: 127.0.0.1, rpc_port: 4501, pub_port: 4500, emu_port: 4510, mode: stl-software-emu }
  lab:  { server: 127.0.0.1, rpc_port: 4601, pub_port: 4600, mode: stl-hw }
ports:                                 # TRex port roles
  mon:
    0: { type: multiplexed, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/21, default_for: [rzk, fw] }
    1: { type: multiplexed, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/22, default_for: [rzc] }
  lab:
    4: { type: pinned, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/9,  ap: K-L11 }
    5: { type: pinned, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/10, ap: K-BL }
    6: { type: pinned, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/11, ap: FW }
    7: { type: pinned, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/12, ap: C-L21 }
aps:
  K-L11: { index: 1,  dc: rzk, device: sw-rzk-L11, device_port: Ten-GigabitEthernet1/3/1, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/1, svlan: 3101, vlans: [1314, 1315] }
  K-L12: { index: 2,  dc: rzk, device: sw-rzk-L12, device_port: TBD, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/2, svlan: 3102, vlans: [1314, 1315] }
  K-L21: { index: 3,  dc: rzk, device: sw-rzk-L21, device_port: TBD, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/3, svlan: 3103, vlans: [1314, 1315] }
  K-L31: { index: 4,  dc: rzk, device: sw-rzk-L31, device_port: TBD, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/4, svlan: 3104, vlans: [1314, 1315] }
  K-L41: { index: 5,  dc: rzk, device: sw-rzk-L41, device_port: TBD, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/5, svlan: 3105, vlans: [1314, 1315] }
  K-BL:  { index: 6,  dc: rzk, device: sw-rzk-BL,  device_port: Ten-GigabitEthernet1/3/6, tx_switch: sw-rzk-TX1, tx_port: XGE1/1/6, svlan: 3110, vlans: [1314, 1315] }
  C-L11: { index: 7,  dc: rzc, device: sw-rzc-L11, device_port: TBD, tx_switch: sw-rzc-TX1, tx_port: XGE1/1/1, svlan: 3201, vlans: [1314, 1315] }
  C-L12: { index: 8,  dc: rzc, device: sw-rzc-L12, device_port: TBD, tx_switch: sw-rzc-TX1, tx_port: XGE1/1/2, svlan: 3202, vlans: [1314, 1315] }
  C-L13: { index: 9,  dc: rzc, device: sw-rzc-L13, device_port: TBD, tx_switch: sw-rzc-TX1, tx_port: XGE1/1/3, svlan: 3203, vlans: [1314, 1315] }
  C-L21: { index: 10, dc: rzc, device: SW-RZC-L21, device_port: Ten-GigabitEthernet1/4/4, tx_switch: sw-rzc-TX1, tx_port: XGE1/1/4, svlan: 3204, vlans: [1314, 1315] }
  C-L22: { index: 11, dc: rzc, device: sw-rzc-L22, device_port: TBD, tx_switch: sw-rzc-TX1, tx_port: XGE1/1/5, svlan: 3205, vlans: [1314, 1315] }
  C-L23: { index: 12, dc: rzc, device: sw-rzc-L23, device_port: TBD, tx_switch: sw-rzc-TX1, tx_port: XGE1/1/6, svlan: 3206, vlans: [1314, 1315] }
  C-BL:  { index: 13, dc: rzc, device: sw-rzc-BL,  device_port: Ten-GigabitEthernet1/3/14, tx_switch: sw-rzc-TX1, tx_port: XGE1/1/7, svlan: 3210, vlans: [1314, 1315] }
  FW:    { index: 14, dc: both, device: CPF-ZV01/ZV02, device_port: TBD, tx_switch: both, tx_port: XGE1/1/7+XGE1/1/8, svlan: 3300, vlans: [1420] }
vlans:
  1314: { vsi: VPN-1314, prefix: 10.3.14.0/24, gateway: 10.3.14.1, gateway_owner: sw-rzk-BL/VPN-CORE, pool_start: 10.3.14.101, vrf: VPN-CORE }
  1315: { vsi: VPN-1315, prefix: 10.3.15.0/24, gateway: 10.3.15.1, gateway_owner: sw-rzk-BL/VPN-CORE, pool_start: 10.3.15.101, vrf: VPN-CORE }
  1420: { prefix: 10.9.90.0/24, gateway: 10.9.90.1, gateway_owner: ENT-FW, pool_start: 10.9.90.101, vrf: FW-ZONE }
  <PROD-VLAN-A>: { vsi: VPN-A, prefix: <SUBNET-A>, gateway: <GW-A>, gateway_owner: DC-FW, pool_start: <POOL-A>, vrf: DC-FW }
budget: { max_streams: 600, max_pps: 50000, max_bps: 50000000, max_pps_per_fw_stream: 100 }

Rules enforced by the validator: S-VLANs unique; AP indexes unique and 1-49 (lab clients use index+50); every VLAN referenced by an AP exists; pool size (16) does not overlap the gateway; per VLAN the pool stays inside the prefix.

9.4 Profiles (profiles/*.yaml)

version: 1
name: monitoring-default
instance: mon
defaults: { pps: 200, size: 128, tos: 0, ttl: 64, latency: true, bidir: true, dport_base: 32000, sport_base: 40000 }
groups:
  - name: l2-intra-rzk
    class: L2                          # L2: same VLAN, dst MAC = remote client MAC
    vlans: [1314, 1315]
    pairs: [[K-L11, K-BL], [K-L12, K-BL], [K-L21, K-BL], [K-L31, K-BL], [K-L41, K-BL]]
  - name: l2-intra-rzc
    class: L2
    vlans: [1314, 1315]
    pairs: [[C-L11, C-BL], [C-L12, C-BL], [C-L13, C-BL], [C-L21, C-BL], [C-L22, C-BL], [C-L23, C-BL]]
  - name: l2-inter-dc
    class: L2
    vlans: [1314, 1315]
    pairs: [[K-L11, C-L11], [K-L12, C-L12], [K-L21, C-L21], [K-L31, C-L22], [K-L41, C-L23], [K-BL, C-BL]]
  - name: l3-bl
    class: L3                          # L3: dst MAC = resolved gateway MAC of the source VLAN
    src_vlan: 1314
    dst_vlan: 1315
    pairs: [[K-L11, K-L21], [K-L31, K-BL], [C-L21, C-L22], [C-L13, C-BL]]
  - name: l3-dcfw
    class: L3
    src_vlan: <PROD-VLAN-A>
    dst_vlan: <PROD-VLAN-B>
    pps: 50
    pairs: [[K-L11, K-L41], [K-L21, K-L31], [C-L12, C-L23], [C-L11, C-L22]]
  - name: l3-entfw
    class: L3
    src_vlan: 1314
    dst_vlan: 1420
    pps: 50
    pairs: [[K-L11, FW], [K-BL, FW], [C-L21, FW], [C-BL, FW]]
  - name: dscp-l2-interdc
    class: L2
    vlans: [1314]
    pps: 100
    size: 256
    tos_set: [0xB8, 0x88, 0x20]        # one stream per TOS value
    pairs: [[K-L11, C-L21]]
  - name: dscp-entfw
    class: L3
    src_vlan: 1314
    dst_vlan: 1420
    pps: 50
    size: 256
    tos_set: [0xB8, 0x88, 0x20]
    pairs: [[K-L11, FW]]
tags: { environment: prod, owner: netzwerk }

Expansion rules: each pair and each VLAN (or VLAN pair) and each TOS value produces one stream; bidir: true adds the reverse stream. Class L2 requires both APs to carry the VLAN; class L3 requires a gateway for the source VLAN and resolves it per source AP. The class name, DC pair (rzk-rzk, rzk-rzc, rzc-rzc, x-fw), VLANs, APs and TOS become InfluxDB tags.

9.5 Stream generation algorithm

1. Load and validate inventory and profile (JSON schema + semantic rules); compute the stream list; abort if the budget is exceeded (unless --force).
2. Allocate pg_ids: key = class|src_ap|src_vlan|dst_ap|dst_vlan|tos; look up in pgid-map.json, otherwise assign the lowest free id in 1-899 and persist. Stable ids keep Grafana history continuous.
3. Provision EMU (monitoring instance): for every (TRex port, S-VLAN, C-VLAN) create a namespace with tci=[svlan, cvlan], tpid=[0x8100, 0x8100]; add one client per needed IP with its deterministic MAC, ipv4_dg = VLAN gateway and plugins arp, icmp. Set the TRex ports to promiscuous + multicast and service mode filtered for EMU. Wait until every client that needs a gateway reports dgw.resolve = true; retry three times, then report unresolved clients and continue with the resolvable streams only.
4. Build packets: Ether(src=client_mac, dst=dst_mac)/Dot1Q(vlan=svlan)/Dot1Q(vlan=cvlan)/IP(src, dst, tos, ttl)/UDP(sport, dport)/Raw(pad), padded to size. For class L2 dst_mac is the remote TRex client MAC; for class L3 it is the resolved gateway MAC. Pinned lab ports omit the outer Dot1Q.
5. Create streams: STLStream(packet=STLPktBuilder(pkt), mode=STLTXCont(pps=pps), flow_stats=STLFlowLatencyStats(pg_id=pg_id), name=key); group by TRex port; add_streams() per port; start(ports, force=True). Streams are added with c.add_streams() individually so that a single stream can be stopped/updated via its stream id without disturbing the others (TRex supports per-stream pause/update through profiles per port; trexctl uses one profile per class to allow partial restarts).
6. Record the applied stream table (/data/trexctl/state/applied-<profile>.json) with pg_id, ports, addresses, tags - the exporter reads this file to tag the statistics.

9.6 Command-line interface

9.7 Exporter and InfluxDB data model

Interval: 2 s by default (1 s possible for convergence tests, 5 s for long-term). Each poll calls get_pgid_stats() (flow and latency statistics), get_stats() (ports, global). TX/RX packet counters are converted to per-interval deltas; drops per interval = TX delta - RX delta (clamped at 0 and compensated for in-flight packets by using the TRex rx_delay convention); loss is also kept as cumulative counters for exact outage accounting.

Series cardinality: about 180 streams x ~25 fields - trivial for InfluxDB. Writes use the v2 HTTP API with a write token scoped to the trex bucket; batched every interval; on InfluxDB unavailability the exporter buffers up to 10 minutes in memory and logs a warning.

9.8 Test runners

• throughput (lab instance, pinned ports): for each frame size run 60 s trials with non-latency streams (STLFlowStats, hardware counters) plus one low-rate latency stream; binary search between 1% and 100% of line rate until the no-drop rate (loss < 0.001%) is found within 0.5% resolution; record NDR, latency at NDR and at 50% load. Optional bidirectional mode (both pinned ports transmit).
• soak: fixed rate for a long duration; loss and latency time series into trex_test.
• convergence: raises the pps of selected monitoring groups to 1000 (1 ms resolution), records cumulative drops per stream before/after the planned event, computes outage duration = drops / pps and produces a table per stream and per AP pair; restores the original rates afterwards.
• astf: starts trex-lab-astf (via systemctl, sudo rule), loads an ASTF Python profile with the chosen client/server ranges and VLAN topology, runs for the duration, collects m_active_flows, m_est_flows, tcps_connattempt, tcps_connects, errors, goodput, latency histograms; restores trex-lab (stateless) afterwards.

9.9 Reference implementation sketches

The following excerpts illustrate the TRex API usage the implementation relies on. They follow the TRex STL and EMU API documentation (v3.x).

# EMU profile: one namespace per (port, S-VLAN, C-VLAN), one client per TRex IP
from trex.emu.api import *

def build_emu_profile(clients):
    ns_by_key = {}
    for c in clients:                                   # c: port, svlan, cvlan, mac, ip, gw
        key = (c.port, c.svlan, c.cvlan)
        if key not in ns_by_key:
            tci  = [c.svlan, c.cvlan] if c.svlan else [c.cvlan, 0]
            tpid = [0x8100, 0x8100] if c.svlan else [0x8100, 0]
            ns_key = EMUNamespaceKey(vport=c.port, tci=tci, tpid=tpid)
            ns_by_key[key] = EMUNamespaceObj(ns_key=ns_key,
                                             def_c_plugs={'arp': {'enable': True}, 'icmp': {}})
        ns_by_key[key].add_clients(EMUClientObj(mac=Mac(c.mac).V(),
                                                ipv4=Ipv4(c.ip).V(),
                                                ipv4_dg=Ipv4(c.gw).V()))
    return EMUProfile(ns=list(ns_by_key.values()), def_ns_plugs=None)

emu = EMUClient(server='127.0.0.1', sync_port=4510)
emu.connect()
emu.load_profile(profile=build_emu_profile(clients), max_rate=512, tunables=[])
# then poll emu.get_info_clients(...) until dgw.resolve is True and read dgw.rmac

# stateless streams with latency statistics, double-tagged for multiplexed ports
from trex.stl.api import *
from scapy.layers.l2 import Ether, Dot1Q
from scapy.layers.inet import IP, UDP

def build_stream(s):                                     # s: stream definition (dataclass)
    pkt = Ether(src=s.src_mac, dst=s.dst_mac)
    if s.svlan:
        pkt = pkt / Dot1Q(vlan=s.svlan)
    pkt = pkt / Dot1Q(vlan=s.cvlan) / IP(src=s.src_ip, dst=s.dst_ip, tos=s.tos, ttl=s.ttl) \
              / UDP(sport=s.sport, dport=s.dport)
    pad = max(0, s.size - len(pkt)) * b'\x00'
    return STLStream(name=s.key,
                     packet=STLPktBuilder(pkt=pkt / pad),
                     mode=STLTXCont(pps=s.pps),
                     flow_stats=STLFlowLatencyStats(pg_id=s.pg_id))

c = STLClient(server='127.0.0.1', sync_port=4501, async_port=4500)
c.connect()
c.acquire(ports=[0, 1], force=True)
c.set_port_attr(ports=[0, 1], promiscuous=True, multicast=True)
# punt only EMU protocols to the RX core (console equivalent: service --emu)
c.set_service_mode(ports=[0, 1], enabled=False, filtered=True, mask=EMU_MASK)
for port, streams in streams_by_port.items():
    c.add_streams([build_stream(s) for s in streams], ports=[port])
c.start(ports=list(streams_by_port), force=True)

# exporter poll loop (simplified)
import time
from influxdb_client import InfluxDBClient, Point, WritePrecision

influx = InfluxDBClient(url="http://127.0.0.1:8086", token=TOKEN, org="lvr")
write  = influx.write_api()
prev = {}
while True:
    st  = c.get_pgid_stats()                            # {'flow_stats': {...}, 'latency': {...}}
    now = time.time_ns()
    points = []
    for pg_id, meta in applied.items():                 # meta = tags from the applied stream table
        fs  = st['flow_stats'].get(pg_id, {})
        lat = st['latency'].get(pg_id, {}).get('latency', {})
        tx  = sum(fs.get('tx_pkts', {}).values())
        rx  = sum(fs.get('rx_pkts', {}).values())
        dtx = tx - prev.get((pg_id, 'tx'), tx)
        drx = rx - prev.get((pg_id, 'rx'), rx)
        prev[(pg_id, 'tx')] = tx
        prev[(pg_id, 'rx')] = rx
        p = Point("trex_stream").time(now, WritePrecision.NS)
        for k, v in meta.tags.items(): p = p.tag(k, v)
        p = (p.field("tx_pkts", tx).field("rx_pkts", rx)
               .field("drop_pkts", max(0, dtx - drx))
               .field("drop_pkts_total", max(0, tx - rx))
               .field("tx_pps", dtx / INTERVAL).field("rx_pps", drx / INTERVAL)
               .field("lat_avg_us", lat.get('average', 0.0))
               .field("lat_max_us", lat.get('total_max', 0))
               .field("lat_last_max_us", lat.get('last_max', 0))
               .field("lat_min_us", lat.get('total_min', 0))
               .field("jitter_us", lat.get('jitter', 0)))
        points.append(p)
    write.write(bucket="trex", record=points)
    time.sleep(INTERVAL)

Latency dictionary keys (average, total_max, total_min, last_max, jitter, histogram) are documented in the TRex STL Python API (get_pgid_stats). VALIDATED

9.10 Service integration, permissions and logging

cat >/etc/systemd/system/trexctl-exporter.service <<'EOF'
[Unit]
Description=trexctl statistics exporter (TRex -> InfluxDB)
After=trex-mon.service influxdb.service
Requires=trex-mon.service

[Service]
Type=simple
User=trexctl
Group=trexctl
EnvironmentFile=/etc/trexctl/secrets.env
ExecStart=/opt/trexctl/venv/bin/trexctl exporter run
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF
useradd -r -s /sbin/nologin -d /data/trexctl trexctl
# trexctl needs to start/stop the lab units for ASTF tests:
cat >/etc/sudoers.d/trexctl <<'EOF'
trexctl ALL=(root) NOPASSWD: /usr/bin/systemctl start trex-lab.service, \
    /usr/bin/systemctl stop trex-lab.service, \
    /usr/bin/systemctl start trex-lab-astf.service, \
    /usr/bin/systemctl stop trex-lab-astf.service
EOF
chmod 440 /etc/sudoers.d/trexctl
systemctl daemon-reload && systemctl enable --now trexctl-exporter.service

• Logs: /var/log/trexctl/trexctl.log (rotating), JSON lines for machine processing; important events (apply, stop, unresolved clients, budget violations, InfluxDB write failures) are also written to trex_event.
• A monitoring profile re-apply after a TRex restart is automatic: the exporter detects a server restart (TRex uptime reset) and triggers trexctl apply -p <active profile>.
• Configuration files are kept under version control (git) with a pre-commit hook running trexctl inventory validate and trexctl plan.

9.11 Development and acceptance of trexctl

• Unit tests for inventory/profile expansion (pytest, no TRex needed); integration tests against a TRex instance in loopback (two DAC-connected ports) and against the TRex "stl-sim" simulator for packet building.
• Deliverables: source repository, README, this specification, example inventory/profiles, Grafana dashboards and alert rules, systemd units.
• Acceptance: trexctl apply -p monitoring-default brings up all planned streams with resolved gateways within 60 s; trexctl status matches the TRex console (stats --ls); Grafana shows all streams within two sampling intervals; trexctl test throughput in loopback reports line rate for 1518-byte frames.

10. InfluxDB and Grafana

10.1 InfluxDB OSS 2.x installation

cat >/etc/yum.repos.d/influxdata.repo <<'EOF'
[influxdata]
name = InfluxData Repository - Stable
baseurl = https://repos.influxdata.com/stable/x86_64/main
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdata-archive.key
EOF
dnf -y install influxdb2 influxdb2-cli
# keep the data on /data
mkdir -p /data/influxdb && chown influxdb:influxdb /data/influxdb
cat >/etc/influxdb/config.toml <<'EOF'
bolt-path = "/data/influxdb/influxd.bolt"
engine-path = "/data/influxdb/engine"
http-bind-address = "127.0.0.1:8086"
query-memory-bytes = 2147483648
storage-cache-max-memory-size = 2147483648
EOF
systemctl enable --now influxdb
# initial setup: organisation lvr, bucket trex with 90-day retention;
# the admin token is printed once - store it in the password safe
influx setup --org lvr --bucket trex --retention 90d --username influxadmin \
    --password '<STRONG-PASSWORD>' --force
influx bucket create --org lvr --name trex_longterm --retention 730d
# scoped tokens (bucket IDs from 'influx bucket list')
TREX_ID=$(influx bucket list --org lvr --name trex --hide-headers | awk '{print $1}')
LT_ID=$(influx bucket list --org lvr --name trex_longterm --hide-headers | awk '{print $1}')
influx auth create --org lvr --description "trexctl exporter (write trex)" --write-bucket "$TREX_ID"
influx auth create --org lvr --description "grafana (read)" --read-bucket "$TREX_ID" --read-bucket "$LT_ID"

Pin the package version in the repository configuration (e.g. influxdb2-2.9.x) - InfluxData moved its "latest" tags to InfluxDB 3 Core in September 2026, which has a different storage engine and API. InfluxDB 2.x is still maintained (2.9.1 at the time of writing) and remains the recommended version for this deployment; a later migration to InfluxDB 3 Enterprise/Core is possible but out of scope.

10.2 Downsampling task

Raw 2-second data is kept for 90 days in trex. A Flux task writes 1-minute aggregates (mean/max/sum per stream) into trex_longterm for two years:

option task = {name: "trex_downsample_1m", every: 10m, offset: 1m}

from(bucket: "trex")
  |> range(start: -20m)
  |> filter(fn: (r) => r._measurement == "trex_stream")
  |> filter(fn: (r) => r._field == "lat_avg_us" or r._field == "jitter_us"
                       or r._field == "rx_pps" or r._field == "tx_pps")
  |> aggregateWindow(every: 1m, fn: mean, createEmpty: false)
  |> to(bucket: "trex_longterm", org: "lvr")

from(bucket: "trex")
  |> range(start: -20m)
  |> filter(fn: (r) => r._measurement == "trex_stream")
  |> filter(fn: (r) => r._field == "lat_max_us" or r._field == "lat_last_max_us")
  |> aggregateWindow(every: 1m, fn: max, createEmpty: false)
  |> to(bucket: "trex_longterm", org: "lvr")

from(bucket: "trex")
  |> range(start: -20m)
  |> filter(fn: (r) => r._measurement == "trex_stream" and r._field == "drop_pkts")
  |> aggregateWindow(every: 1m, fn: sum, createEmpty: false)
  |> to(bucket: "trex_longterm", org: "lvr")

Create it with influx task create --org lvr -f /opt/trexctl/influx/downsample.flux. Disk estimate: 180 streams x 25 fields x 0.5 points/s is about 2 250 points/s raw; with InfluxDB compression this is in the order of 1-2 GB per month, so 90 days of raw data need well under 10 GB.

10.3 Grafana installation

cat >/etc/yum.repos.d/grafana.repo <<'EOF'
[grafana]
name=grafana
baseurl=https://rpm.grafana.com
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://rpm.grafana.com/gpg.key
EOF
dnf -y install grafana            # pin a version if required by LVR policy, e.g. grafana-12.4.x
# TLS: certificate from the LVR PKI (CN trex-rzk-01.<domain>)
install -o root -g grafana -m 640 /path/trex-rzk-01.crt /etc/grafana/tls.crt
install -o root -g grafana -m 640 /path/trex-rzk-01.key /etc/grafana/tls.key
# grafana.ini changes
sed -i -e 's/^;protocol = http/protocol = https/' \
       -e 's/^;http_port = 3000/http_port = 443/' \
       -e 's#^;cert_file =#cert_file = /etc/grafana/tls.crt#' \
       -e 's#^;cert_key =#cert_key = /etc/grafana/tls.key#' \
       -e 's/^;root_url = .*/root_url = https:\/\/trex-rzk-01.<DOMAIN>\//' \
       -e 's/^;min_refresh_interval = 5s/min_refresh_interval = 1s/' /etc/grafana/grafana.ini
# allow the grafana service user to bind port 443 (systemd drop-in survives package upgrades)
mkdir -p /etc/systemd/system/grafana-server.service.d
cat >/etc/systemd/system/grafana-server.service.d/bind443.conf <<'EOF'
[Service]
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
EOF
systemctl daemon-reload
systemctl enable --now grafana-server

Data sources (provisioned)

# /etc/grafana/provisioning/datasources/influx.yaml
apiVersion: 1
datasources:
  - name: InfluxDB-trex
    type: influxdb
    access: proxy
    url: http://127.0.0.1:8086
    isDefault: true
    jsonData: { version: Flux, organization: lvr, defaultBucket: trex, tlsSkipVerify: true }
    secureJsonData: { token: "<GRAFANA-READ-TOKEN>" }
  - name: InfluxDB-trex-longterm
    type: influxdb
    access: proxy
    url: http://127.0.0.1:8086
    jsonData: { version: Flux, organization: lvr, defaultBucket: trex_longterm }
    secureJsonData: { token: "<GRAFANA-READ-TOKEN>" }

Authentication

Local admin plus Active Directory via LDAP ([auth.ldap] enabled = true, config_file = /etc/grafana/ldap.toml with the LVR domain controllers, group mapping: network team = Editor, others = Viewer). Anonymous access disabled. Dashboards and alert rules are provisioned from /opt/trexctl/dashboards and /opt/trexctl/alerts so that the installation is reproducible.

10.4 Dashboards

Example Flux queries

// Latency matrix cell: mean latency per src/dst AP for one class
from(bucket: "trex")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "trex_stream" and r._field == "lat_avg_us")
  |> filter(fn: (r) => r.class == "${class}" and r.src_vlan == "${vlan}")
  |> group(columns: ["src_ap", "dst_ap"])
  |> mean()
  |> pivot(rowKey: ["src_ap"], columnKey: ["dst_ap"], valueColumn: "_value")

// Drops per stream over time
from(bucket: "trex")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "trex_stream" and r._field == "drop_pkts")
  |> filter(fn: (r) => r.dc_pair =~ /${dc_pair:regex}/)
  |> aggregateWindow(every: v.windowPeriod, fn: sum, createEmpty: false)

// Number of streams currently dropping (last 2 intervals)
from(bucket: "trex")
  |> range(start: -10s)
  |> filter(fn: (r) => r._measurement == "trex_stream" and r._field == "drop_pkts" and r._value > 0)
  |> group(columns: ["pg_id"]) |> count() |> group() |> count()

// Outage table: drops / pps = outage seconds per stream in the selected range
drops = from(bucket: "trex") |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "trex_stream" and r._field == "drop_pkts")
  |> group(columns: ["name", "src_ap", "dst_ap", "class"]) |> sum() |> rename(columns: {_value: "drops"})
pps = from(bucket: "trex") |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "trex_stream" and r._field == "tx_pps")
  |> group(columns: ["name", "src_ap", "dst_ap", "class"]) |> mean() |> rename(columns: {_value: "pps"})
join(tables: {d: drops, p: pps}, on: ["name", "src_ap", "dst_ap", "class"])
  |> map(fn: (r) => ({r with outage_s: float(v: r.drops) / r.pps}))
  |> sort(columns: ["outage_s"], desc: true)

Usage hints carried over from the reference design: keep the auto-refresh at 1-5 s and the time window at 5-30 minutes for live observation; disable auto-refresh when zooming or hovering; prefer "all streams" and filter by tags (class, DC pair, AP) rather than by individual stream ids; use the long-term data source for windows above 24 hours.

10.5 Alert rules (Grafana unified alerting)

Notification channels: e-mail to the network team distribution list; optionally the existing monitoring system (IMC/syslog webhook). Maintenance windows are modelled with Grafana mute timings plus trexctl annotate events.

10.6 Backup and housekeeping

• Nightly influx backup /data/backup/influx-$(date +%F) --org lvr (bucket metadata + data), 14 days kept locally, copied to the LVR backup target.
• Grafana: /var/lib/grafana/grafana.db (users, alert state) in the nightly backup; dashboards and alerts are in git anyway.
• Retention is handled by the bucket policies; monitor free space on /data (node exporter or Grafana server dashboard).
• TLS certificate renewal: yearly, same path, systemctl restart grafana-server.

11. Operations and test procedures

11.1 Roles and daily operation

Daily: check the Overview dashboard (no path-down streams, no dropping streams, latency within baseline, TRex CPU < 50%); review alert history. Weekly: check /data usage, exporter log for InfluxDB write errors, unresolved EMU clients. Monthly: review baseline thresholds and the monitoring matrix.

11.2 Permanent delay monitoring: bring-up and baseline

1. Prerequisites complete: TX switches configured (7.1/7.2), Leaf/BL AP ports configured for VLAN 1314/1315 (7.3/7.4), firewall zone and rules active (7.5), TRex server installed with trex-mon running (8), InfluxDB and Grafana running (10), inventory validated (trexctl inventory validate).
2. Dry run: trexctl plan -p monitoring-default - review the stream table, pg_ids and budget.
3. Apply: trexctl apply -p monitoring-default. Expected output: EMU clients created (e.g. 14 APs x 2 VLANs = 28 clients + FW zone), gateways resolved for all L3 streams, N streams started on ports 0 and 1.
4. Verify on TRex: trexctl status shows RX pps = TX pps for every stream after a few seconds; latency values plausible (intra-DC fabric tens of microseconds, inter-DC 100-300 us depending on fibre length, firewall paths 50-500 us).
5. Verify on the network: display l2vpn mac-address vsi VPN-1314 on a Leaf shows the local TRex MAC on the AP port and the remote TRex MACs via VXLAN tunnels; display arp vpn-instance VPN-CORE on sw-rzk-BL shows the TRex IPs; display mac-address vlan 3101 on sw-rzk-TX1 shows MACs on both the AP port and the TRex trunk port.
6. Verify Grafana: Overview shows all streams, no drops; Latency matrix populated.
7. Baseline phase (7 days): let the system run; then derive per-class latency thresholds (e.g. p99 of the 7-day average x 1.5) and set the alert rules (10.5). Document the baseline table in the operations handbook (template below).
8. Enable alert notifications.

11.3 Generic test procedure (applies to 11.4-11.10)

1. Define: purpose, class, APs, VLANs, rate, duration, expected result; create a change/test ticket if production VLANs or firewalls are involved.
2. Pre-checks: trexctl status (monitoring healthy), target AP links up (display interface brief on TX switch), firewall rules in place, no concurrent maintenance.
3. Annotate: trexctl annotate "TEST <id>: <description> start".
4. Execute with trexctl test ... (or raw TRex console for ad-hoc cases).
5. Observe in Grafana (Lab tests / Loss dashboards) and in the switch counters (display counters rate inbound interface on TX switch and AP ports).
6. Post-checks: lab streams stopped (trexctl test stops automatically; verify with the console stats), monitoring unaffected, firewall sessions cleared.
7. Report: trexctl report --test <id> --format md; attach to the ticket; annotate "end".

11.4 Test: Layer 2 inside one VLAN

Goal: loss and delay between two APs in the same VLAN (e.g. VLAN 1314, K-L11 to C-L21, inter-DC). Where: monitoring instance (already running permanently); for a dedicated high-rate run use the lab instance.

1. Permanent view: Grafana "Class detail", class L2, src_ap K-L11, dst_ap C-L21 - read latency/jitter/loss for the time range.
2. Higher resolution for a limited time: trexctl update --group l2-inter-dc --pps 1000 (1 ms resolution; budget check applies); revert with --pps 200.
3. Line-rate variant: pin lab ports p4 to K-L11 and p7 to C-L21 (already the default pins), then trexctl test throughput --src K-L11 --dst C-L21 --vlan 1314 --sizes 128,512,1518 --duration 60 --bidir. The runner creates lab EMU-less clients (index+50 addresses), resolves nothing (L2: destination MAC = remote lab client MAC), runs the binary search and reports the NDR per frame size plus latency at NDR. Note that the inter-DC TX fibre carries 10G in each direction in this test; with a 10G inter-DC TX link, bidirectional line rate is the maximum.

Raw TRex alternative (console on the lab instance, ports p4/p7 pinned, single-tagged):

cd /opt/trex/current && ./trex-console -p 4601 --async_port 4600
trex> portattr -a --prom on
trex> start -f /opt/trexctl/profiles/raw/udp_vlan_latency.py --port 4 -t cvlan=1314,smac=02:54:52:33:05:22,dmac=02:54:52:3c:05:22,sip=10.3.14.151,dip=10.3.14.160,pps=1000
trex> start -f /opt/trexctl/profiles/raw/udp_vlan_latency.py --port 7 -t cvlan=1314,smac=02:54:52:3c:05:22,dmac=02:54:52:33:05:22,sip=10.3.14.160,dip=10.3.14.151,pps=1000,pg=901
trex> stats --ls          # latency per pg_id
trex> tui                 # live view, 'l' for latency; q to quit
trex> stop -a

# /opt/trexctl/profiles/raw/udp_vlan_latency.py - minimal single/double-tagged UDP latency stream with tunables
from trex.stl.api import *

class STLS1(object):
    def get_streams(self, direction=0, svlan=0, cvlan=1314, smac='02:54:52:33:05:22', dmac='02:54:52:3c:05:22',
                    sip='10.3.14.151', dip='10.3.14.160', pps=200, size=128, tos=0, pg=900, **kwargs):
        pkt = Ether(src=smac, dst=dmac)
        if int(svlan):
            pkt = pkt / Dot1Q(vlan=int(svlan))
        pkt = pkt / Dot1Q(vlan=int(cvlan)) / IP(src=sip, dst=dip, tos=int(tos), ttl=64) / UDP(sport=40000 + int(pg), dport=32009)
        pad = max(0, int(size) - len(pkt)) * 'x'
        return [STLStream(packet=STLPktBuilder(pkt=pkt / pad),
                          mode=STLTXCont(pps=int(pps)),
                          flow_stats=STLFlowLatencyStats(pg_id=int(pg)))]

def register():
    return STLS1()

For L2 streams no ARP is needed, but the remote switch must have learned the destination MAC: the reverse stream (or a gratuitous ARP from the lab client via service mode arp) takes care of that; otherwise the first packets are flooded in the VSI (still delivered).

11.5 Test: Layer 3 within VPN-CORE (Border Leaf routing)

Goal: delay added by the centralised gateway, asymmetry RZK vs RZC. Pairs: VLAN 1314 at AP X to VLAN 1315 at AP Y.

1. Permanent: class L3-BL in Grafana (K-L11 to K-L21 and C-L21 to C-L22). Compare RZC-sourced vs RZK-sourced latency: the difference is the double DCI crossing to the gateway in Kalk.
2. Ad-hoc with the monitoring instance: add a temporary group with a new profile file (copy monitoring-default.yaml, keep only an l3-bl group with the wanted pairs) and trexctl apply -p l3bl-adhoc; EMU resolves 10.3.14.1 / 10.3.15.1 for each source AP; verify with trexctl emu show (resolved MAC = sw-rzk-BL Vsi-interface MAC). Remove with trexctl stop -p l3bl-adhoc.
3. Line-rate: pin p4 to K-L11 (VLAN 1314 side) and p5 to K-BL (VLAN 1315 side): trexctl test throughput --src K-L11 --src-vlan 1314 --dst K-BL --dst-vlan 1315 --sizes 512,1518 --duration 60. The runner puts the lab ports in L3 service mode (set_l3_mode with VLAN) to resolve the gateways, then builds streams with the gateway MACs. Expect line rate on the 5950 (hardware routing); any loss points to the single 10G AP link being the bottleneck (both directions share it in bidirectional mode).

Switch-side checks during the test: display counters rate inbound interface Ten-GigabitEthernet1/3/1 on sw-rzk-L11; display arp vpn-instance VPN-CORE on sw-rzk-BL; display interface Vsi-interface914 and Vsi-interface915 for packet counters. VALIDATED

11.6 Test: Layer 3 through the DC firewall (production VLANs)

Goal: inter-VLAN latency and loss through CPF-ZV05/06 between two production VLANs whose gateway is the DC firewall. Prerequisites: TRex pools reserved in both VLANs, Leaf AP ports carry both VLANs, DC-FW rule (7.5.2) active.

1. Permanent: group l3-dcfw (50 pps). Grafana "Class detail", class L3-DCFW. A step change in latency usually indicates a cluster failover (active member moved to the other DC); confirm in the firewall logs and annotate.
2. Ad-hoc profile: as in 11.5 with class: L3, src_vlan: <A>, dst_vlan: <B>; keep pps <= 100 per stream through firewalls (budget rule).
3. Load test (coordinate with the firewall team): pin two lab ports to two APs that carry VLAN A and B, trexctl test soak --src K-L11 --src-vlan <A> --dst K-L41 --dst-vlan <B> --rate 1gbps --duration 600. Watch firewall CPU and connection table in SmartView at the same time. Increase the rate stepwise; stop when loss appears or the firewall team requests it.

11.7 Test: VPN-CORE through the Enterprise firewall (firewall zone) and stateful tests

Goal: inter-VRF/firewall path latency (stateless) and firewall performance (stateful).

Stateless (monitoring and lab)

1. Permanent: group l3-entfw (VLAN 1314 at four APs to the FW zone, 50 pps). The FW-zone client (10.9.90.101 on p0) resolves the VIP 10.9.90.1 via EMU; the inside clients resolve 10.3.14.1 on sw-rzk-BL. Verify the OSPF route for 10.9.90.0/24 on both Border Leafs first (display ip routing-table vpn-instance VPN-CORE 10.9.90.0 24).
2. Load test: pin p4 to K-L11 (VLAN 1314) and p6 to FW (VLAN 1420): trexctl test throughput --src K-L11 --src-vlan 1314 --dst FW --dst-vlan 1420 --sizes 512,1518 --duration 60. Stateless UDP through a Check Point gateway creates one session per 5-tuple; the runner uses a small number of 5-tuples (default 16 per direction) so the test measures forwarding, not session setup.

Stateful (ASTF) - option A: port addresses only (no routing changes)

1. trexctl test astf --profile lab-fw-http.yaml --mult 200 --duration 300. The runner stops trex-lab, starts trex-lab-astf, loads an ASTF profile where the client range is the single lab client IP 10.3.14.151 (p4, VLAN 1314 via ASTF topology VIF with vlan=1314) and the server range the single IP 10.9.90.151 (p6, VLAN 1420). Gateways are resolved from trex_cfg port_info (default_gw 10.3.14.1 / 10.9.90.1) - the runner writes a temporary lab configuration with these values. With one client IP the connection rate is limited by source-port reuse (~60 000 concurrent flows); sufficient for throughput and latency, not for CPS scalability.

Stateful (ASTF) - option B: routed ranges (scalable)

1. Switch team: ip route-static vpn-instance VPN-CORE 10.9.92.0 24 10.3.14.151 description TRex-ASTF-clients on sw-rzk-BL (7.4.3). Firewall team: static route 10.9.93.0/24 via 10.9.90.151 on the Enterprise firewall and rule TRX-astf-clients to TRX-astf-servers TCP 80/443.
2. Profile lab-fw-http-routed.yaml: client range 10.9.92.1-10.9.92.254, server range 10.9.93.1-10.9.93.254, templates http_simple/http_https (TRex astf/ examples) or a pcap-based L7 profile; --mult scales connections per second.
3. Run as in option A; read CPS, established flows, TCP retransmissions and goodput from the ASTF dashboard; correlate with firewall CPU. Remove the static route afterwards.

# ASTF profile skeleton used by trexctl (derived from the TRex astf/http_simple.py example)
from trex.astf.api import *

class Prof1():
    def get_profile(self, tunables, **kwargs):
        ip_gen_c = ASTFIPGenDist(ip_range=["10.9.92.1", "10.9.92.254"], distribution="seq")
        ip_gen_s = ASTFIPGenDist(ip_range=["10.9.93.1", "10.9.93.254"], distribution="seq")
        ip_gen = ASTFIPGen(glob=ASTFIPGenGlobal(ip_offset="1.0.0.0"), dist_client=ip_gen_c, dist_server=ip_gen_s)
        return ASTFProfile(default_ip_gen=ip_gen,
                           cap_list=[ASTFCapInfo(file="../avl/delay_10_http_browsing_0.pcap", cps=10)])

def register():
    return Prof1()

ASTF VLAN handling: the client side uses an ASTF topology VIF with the VLAN tag; the server side learns VLAN and MAC from the incoming SYN (TRex ASTF guide, "Topology"). Because the lab ports are pinned (single-tagged), no QinQ is involved in ASTF tests.

11.8 Test: DSCP / QoS verification

1. Permanent: groups dscp-l2-interdc and dscp-entfw; Grafana "DSCP" dashboard compares EF/AF41/CS1/BE side by side for the same pair. In an unloaded fabric all four must be identical within jitter.
2. Under load: run a lab soak at 8-9 Gbit/s on the same AP pair with TOS 0 while the DSCP monitoring streams keep running; a divergence between EF and BE latency/loss shows whether any device in the path differentiates (fabric: expected no; firewall: depends on configuration).
3. Mark verification: trexctl capture --port 1 --filter "vlan 1314 and udp port 32005" --count 20 (wrapper around the TRex capture API) and inspect the TOS byte at the receiving end to detect remarking.

11.9 Test: throughput / performance (stateless, lab instance)

1. Pin the two lab ports to the wanted APs (7.1.4), update inventory.yaml ports.lab, run trexctl inventory validate.
2. Decide frame sizes (64, 128, 256, 512, 1024, 1518, 9000 or IMIX) and whether bidirectional.
3. trexctl test throughput --src <AP> --dst <AP> --vlan <V> [--src-vlan/--dst-vlan for L3] --sizes 64,512,1518 --duration 60 [--bidir] [--max-rate 9gbps].
4. Result: table NDR (Mbit/s, pps) per frame size, latency avg/max at NDR and at 50%, drops at line rate; stored under /data/trexctl/reports/ and in trex_test.
5. Interpretation: the 5950 forwards at line rate for all sizes; loss at 64-byte line rate (14.88 Mpps) usually means the TRex RX side (software counters) or the X710 at small frames - check trex_port.ierrors and queue_full; use 128 bytes as the smallest meaningful size unless hardware flow statistics are confirmed to keep up.

11.10 Test: convergence during planned maintenance

1. Before the window: trexctl test convergence --group l2-inter-dc,l3-bl --pps 1000 --duration 3600 raises the selected groups to 1000 pps (1 ms resolution) and starts recording.
2. trexctl annotate "Maintenance: sw-rzk-L21 IRF member 2 reboot".
3. Perform the maintenance (IRF member reboot, link failover, ISSU, firewall failover ...).
4. After the window the runner restores the rates and prints the outage table: per stream and per AP pair the number of lost packets and the derived outage (drops/pps) plus the time of first and last loss. Streams through the rebooted member's AP show the full downtime (expected); other streams reveal fabric convergence time (EVPN/VXLAN re-route, LACP on servers is not covered because APs use single links).
5. Grafana "Loss and outages" with the annotation marks the event for later review.

11.11 Change procedure: adding a VLAN under test or an attachment point

1. IP address management reserves 16 addresses in the VLAN; firewall team adds the pool object to the TRex rules (if the VLAN's gateway is a firewall or if L3 classes will use it).
2. Edit /etc/trexctl/inventory.yaml: add the VLAN (vsi, prefix, gateway, pool_start, vrf) and append it to the vlans list of the APs that should carry it; trexctl inventory validate.
3. Render and apply the Leaf/BL configuration: trexctl inventory render-leaf --ap K-L11 prints port trunk permit vlan ... and the service-instance/xconnect vsi block; apply on the switch, save force.
4. Extend the profile (new groups or VLAN list), trexctl plan, trexctl apply.
5. Verify RX = TX for the new streams; EMU clients resolved; Grafana shows the new VLAN tag.
6. New AP: additionally patch the fibre, configure the TX switch AP port (QinQ access, new S-VLAN; add the S-VLAN to the trunk permit lists and, for RZC, to the inter-DC trunk), configure the device port, add the AP to the inventory with the next free index.
7. Commit inventory/profile changes to git with the ticket number.

11.12 Maintenance of the platform

• TRex upgrade: download the new release to /data/trex/vX.YY, run ./t-rex-64 --help and the loopback self-test (8.8) with the new version using a test configuration, then switch the /opt/trex/current symlink and restart trex-mon (monitoring gap of about 60 s; the exporter re-applies the active profile). Keep the previous release for rollback. Check the TRex release notes for Python client compatibility.
• OS patching: kernel updates require the DPDK driver to rebind (automatic with vfio-pci); reboot in a maintenance window; verify hugepages and trex-mon after reboot.
• InfluxDB/Grafana upgrades: minor versions via dnf; backup first (10.6); re-check the provisioned data sources and dashboards.
• TX switch: no routine changes; firmware stays at R6301P03 (device out of support).
• Capacity: if more than ~600 streams are needed, split the monitoring profile across p0-p3 (two more multiplexed ports) or add a third X710 for 1:1 pinned APs.

11.13 Troubleshooting matrix

12. Implementation plan

12.1 Phases and work packages

12.2 Indicative effort

12.3 Acceptance criteria

1. Server self-test: loopback latency < 20 us average, 0 drops at 1 kpps; lab instance 0 drops at 1518-byte line rate for 60 s in loopback.
2. QinQ path: for every AP, a stream from the monitoring instance to a client at that AP shows RX = TX for 10 minutes; display mac-address vlan <svlan> on the TX switch shows exactly the expected MACs.
3. Monitoring profile: all planned streams started, all L3 gateways resolved, exporter writing, Grafana Overview without path-down streams, 24 h without false alerts after baseline tuning.
4. Firewall classes: L3-DCFW and L3-ENTFW streams RX = TX; a firewall cluster failover is visible as a latency step and logged by an annotation; no loss beyond the failover window.
5. Test runners: throughput report for K-L11 to C-L21 (VLAN 1314) at 512 and 1518 bytes; convergence report from one maintenance event; ASTF option A report with established flows > 0 and zero TCP errors at the chosen rate.
6. Operations: handbook (this document v1.0) with the baseline table; runbook procedures 11.2-11.12 executed once by the LVR team.

12.4 Risks and mitigations

12.5 Open items

Appendix A - Validated command reference and citations

Appendix B - Source documents

• Big Picture-Rechenzentren-RZKC.pdf (Visio export, 15.05.2024, A3, 1 page) - topology, loopbacks, management addresses, VRF/VNI table.
• diag_sw-rzk-S1/S2, diag_sw-rzk-L11, diag_sw-rzk-BL, diag_SW-RZC-S1/S2, diag_SW-RZC-L21, diag_sw-rzc-BL, diag_sw-rzd-BL (17./18.02.2025) - running configurations, interface tables, routing and EVPN state.
• LVR-InfoKom - Comware Switch Analysis v1.0.xlsx - models, releases, recommendations, EoS/EoSL dates.
• man_trex_documentation_v1.3.docx (MAN Truck & Bus, German) - reference design: TRex switch with dot1q tunnels, trex_client.py concept, InfluxDB/Grafana integration, Grafana usage notes.
• simplified trex topology.png - customer's target picture.
• HPE FlexFabric 5950 Switch Series documentation, Release 630x (Layer 2 - LAN Switching CG/CR, VXLAN CG/CR, EVPN CG, Layer 3 IP Routing CR, Layer 3 IP Services CR), HPE5950CMW710R6301P03RN.pdf.
• Cisco TRex documentation (manual, stateless, ASTF, EMU, STL API), v3.x, October 2026.

Appendix C - Glossary