Skip to content

Appendix D - Open decisions and the plan's history

This appendix collects everything that is not yet settled, and it is the only place in the course that describes the history of the plan document and the points still open in it. Section D.1 lists the open design decisions with their options, consequences and owners; section D.2 lists the facts that must be confirmed on the devices or on the test bench before the corresponding configuration is used. Section D.3 records how the plan changed from the draft v0.9 (12 October 2026) to v0.91 (13 October 2026), section D.4 lists the points raised against plan v0.91 and their status, and section D.5 records the consolidation from v0.91 to v0.92 (14 October 2026), the version this course follows. The chapters themselves simply present the design.

D.1 Open decisions

Open decision: how the firewall test zone is attached - option B (recommended) or option A

The zone (10.9.90.0/24, VIP 10.9.90.1, members .2/.3) is the destination of class L3-ENTFW (VPN-CORE -> sw-rzk-BL -> Vsi40 -> ENT-FW -> TRex zone). The plan recommends option B and uses it as its default in every configuration, profile and address; the decision itself is still pending with the LVR network and firewall teams (plan 2.3 "RE-DECIDE", 4.3.5, open item 11).

Option What it means Consequences
B - existing transfer VSI 4003 (recommended, the plan's default) the firewall adds a VLAN 4003 sub-interface with the VIP on its existing inside bond; the free transfer VSI 4003 ("Transfer ENTFW *frei*") is already trunked to both members on BAGG10; TRex reaches it through the Border Leaf APs K-BL and C-BL (service instance 4003 on sw-rzk-BL XGE1/3/6 and sw-rzc-BL XGE1/3/14); zone clients 10.9.90.106 (K-BL) and 10.9.90.113 (C-BL), lab zone client 10.9.90.156 (p6 pinned to FWZ) no new firewall ports or optics, existing redundancy, no dependency of the cluster on the TX switches or the TX fibre; not a physical firewall link from the TX switch; the zone leg also crosses the fabric (a few tens of microseconds, identical for all zone streams); VSI 4003 had no gateway and no learned MACs in February 2025 - check that it is still unused
A - new physical firewall interface (alternative, the customer's original wish) one new 10G port per ClusterXL member (CPF-ZV01, CPF-ZV02), cabled to sw-rzk-TX1 XGE1/1/7 and sw-rzc-TX1 XGE1/1/8 - both ports are already configured as option-A ports (PVID 3300, QinQ) but shut down, and S-VLAN 3300 is already defined and permitted on the TX-to-TX link and the monitoring trunks; sub-interface VLAN 1420 with the VIP; AP FW (index 14), zone client 10.9.90.114, lab zone client 10.9.90.164 (p6 re-pinned to PVID 3300, same port MAC) matches the original wish for a physical firewall link from the TX switch; on the TX switches only cabling and undo shutdown are missing; ClusterXL monitors the new interface, so a TX switch reboot, a fibre cut, an optic or a TRex port failure can trigger a failover or a loss of redundancy - unless the firewall team makes the interface non-critical for failover (supported way for their Check Point version) or formally accepts the risk; needs two Check Point certified SFP+ SR optics and one more SR optic per TX switch

Both options use the same subnet, the same address formula, the same firewall objects and rules and the same profile group (l3-entfw); only the inventory entry of the zone and the TX switch and Border Leaf configuration differ. Owner: LVR network team together with the LVR firewall team. Needed for: class L3-ENTFW, the DSCP firewall group dscp-entfw, ASTF firewall tests, the Border Leaf AP ports (option B) or the cabling and undo shutdown of the pre-configured TX switch ports XGE1/1/7-8 in S-VLAN 3300 (option A), the inventory. Covered in: Chapter 15, B.5, B.6.

Open decision: which production VLANs are monitored, and their address pools

Option Consequences
Start with VLAN 1314/1315 only no IPAM or firewall changes; class L3-DCFW is not possible (it needs two VLANs with DC-FW gateway)
Add 2 test-oriented VLANs (candidates in the configurations: 176 "DC-Testnetz RZ Deutz/RZ-Chorweiler", 171, 724 "Test Server Netz", 1395/1396 "t3_139x_ngfw-test") low risk; enables L3-DCFW; needs 16-address pools, DHCP exclusions, DC-FW rules and the VLANs on the AP trunks
Add 2-4 business VLANs most representative measurement; more coordination with the application owners

Each production VLAN adds 34 loss-only flow-stat streams (3 400 pps) to the group l2-coverage-prod of the default profile, and the VLAN has to be added to the trunk of every AP that should carry it. Owner: LVR network team / IP address management. Covered in: Chapter 16, B.7, B.8.

Open decision: approval of the zone subnet 10.9.90.0/24 and the ASTF ranges 10.9.92.0/24 and 10.9.93.0/24

Option Consequences
Approve as proposed the zone subnet 10.9.90.0/24 - with the existing VLAN 4003 (option B) or a new VLAN 1420 (option A) - and 10.9.92.0/24 / 10.9.93.0/24 for the routed ASTF variant; the three subnets and VLAN 1420 are unused in all nine configurations of February 2025; zone VIP 10.9.90.1, members .2/.3, TRex monitoring clients .101-.116, lab clients .151-.166
Allocate alternatives inventory, firewall objects, static routes and the profile change accordingly

The routed ASTF variant also needs ip route-static vpn-instance VPN-CORE 10.9.92.0 24 10.3.14.151 on sw-rzk-BL and a firewall route for 10.9.93.0/24 via the zone lab client 10.9.90.156 (option A: 10.9.90.164). Owner: LVR IP address management with the firewall team. Covered in: Chapter 15, Chapter 22, B.6.

Open decision: Threat Prevention inspection of TRex traffic

Option Consequences
Exclude the TRex objects from inspection the streams measure firewall forwarding; lower firewall load
Inspect like production traffic the streams show what inspected traffic experiences, including inspection delay; more firewall load
Exclude for monitoring, separate inspected profile for lab tests both views, more rules to maintain

Owner: LVR firewall team. Covered in: Chapter 15.

Open decision: a second monitoring instance if more than 128 latency streams are ever needed

Option Consequences
Stay with one instance 74 latency streams used, budget 120; further pairs are monitored with loss-only flow-stat streams
Second monitoring instance on p2/p3 with its own 128 latency streams p2/p3 leave trex_mon.yaml (a NIC port belongs to one instance only); own prefix, ZMQ ports, memory and threads; the EMU helper is a single named service on port 4510 (scripts/dpdk_setup_ports.py), so how EMU is shared must be validated on the bench

Owner: platform administrator with Secured Networks. Covered in: Chapter 16, Chapter 11.

Open decision: speed and optics of the TX-to-TX link once the distance is known

Option Consequences
10G enough for monitoring (about 26 Mbit/s); shared by monitoring and lab traffic of all RZC APs - lab tests near 10G across it disturb the monitoring
40G or 100G lab tests above 10 Gbit/s aggregate towards RZC possible; needs matching QSFP optics on both TX switches
DWDM channel depends on the transponder client ports and their MTU

The DCI between the Border Leafs uses 40G ER4 optics (40 km class), which suggests that the RZK-RZC distance is more than 10 km - choose the reach class from the measured distance. The fibre adds about 5 us per km to every RZC stream. Owner: LVR network team (fibre / DWDM provider). Covered in: Chapter 11, Chapter 14, B.4.

Open decision: host protection of the TRex control ports

Every start of t-rex-64 runs trex-cfg, which stops firewalld on Red Hat-family systems (scripts/trex-cfg). The TRex control ports (RPC 4501/4601, publisher 4500/4600, EMU 4510, Scapy server 4507) listen on all addresses and have no authentication: whoever reaches them can start traffic into production VLANs. The plan (4.8, 8.3) therefore disables and removes firewalld, protects the server with the native nftables service and an allow-list for the management hosts, and adds ACLs on the management network; this proposal, the ACLs and the list of management hosts still need the agreement of the LVR Linux and security teams (plan open item 13).

Option Consequences
nftables service with an allow-list for the management hosts instead of firewalld (the plan's proposal; reference in B.2) unaffected by TRex starts and upgrades; departs from the RHEL default tooling; the Linux team maintains the ruleset
Keep firewalld and start it again after every TRex start (systemd drop-in) RHEL default tooling; a short unprotected window at every TRex start
Patch trex-cfg minimal change; modifies a vendor file and must be repeated after every TRex upgrade
ACLs on the management network protect independently of the host; useful in addition to any of the options above

Whatever is chosen, the acceptance test includes "TRex restarted - host protection still active". Owner: LVR Linux / security team with the platform administrator (ACLs: LVR network team). Needed for: go-live of trex-rzk-01. Covered in: Chapter 17, Chapter 20, B.2.

Open decision: ClusterXL virtual MAC or stream rebuild after a firewall failover

In February 2025 sw-rzk-BL resolved 10.9.0.22 (ENT-FW) and 10.9.0.35 (DC-FW) to MACs of the RZC members: the clusters answered with member MACs, no virtual MAC was seen. A stream whose destination MAC is a firewall MAC - all L3-DCFW streams, the zone-side L3-ENTFW and DSCP streams - keeps the old MAC after a failover and loses packets until it is rebuilt with the newly resolved MAC.

Option Consequences
Member MACs as today; the streams are rebuilt no firewall change; the gateway-MAC tracker of trexctl polls the resolved gateway MAC (dgw.rmac) every 10 s, rebuilds the affected streams (same pg_ids, counters restart) and writes the event gateway_mac_changed (Grafana alert "Gateway MAC changed"); until trexctl exists this is a manual step (Chapter 23, incident I3); the loss of these streams includes the detection time, which depends on how ClusterXL announces the new MAC - the plan's 15-s target is still to be confirmed (D.2 item 19, D.4 item 1)
ClusterXL virtual MAC the streams survive a failover and every firewall stream measures the failover gap; a configuration change on production firewall clusters, to be assessed by the firewall team for their Check Point version

Test the behaviour during the first planned failover. Owner: LVR firewall team with the platform administrator (plan open item 12). Needed for: classes L3-DCFW and L3-ENTFW, the DSCP firewall group, convergence measurements of firewall failovers. Covered in: Chapter 15, Chapter 18, Chapter 23, B.6.

Open decision: how the per-stream latency thresholds (baseline) are stored

Option Consequences
Rolling baseline in the alert rule (7-day median from trex_longterm at every evaluation) no extra data; new streams are covered after a few days; a lasting degradation becomes the new normal once it covers half of the window; a 7-day query at every evaluation costs CPU
Frozen baseline stored once (per-stream thresholds written with to() into a small measurement, for example trex_baseline) stable, explicit, reviewable, cheap; an extra measurement and a procedure to re-run after accepted changes; new streams have no threshold until it is re-run
Fixed thresholds per class and pair type, written into the rules simplest, fully visible in git; coarse - one threshold for all streams of a class, including the longer intra-RZC paths

Owner: platform administrator with the LVR network team. Needed for: enabling notifications after the 7-day baseline (scenario S1). Covered in: Chapter 19, Chapter 21.

D.2 Items to confirm on the devices or on the bench

confirm on device - each item is plausible and consistent with the documentation, but must be confirmed before the related configuration is applied.

# Item and why it matters How to confirm Owner
1 Free AP ports. sw-rzk-L11 XGE1/3/1, sw-rzk-BL XGE1/3/6, SW-RZC-L21 XGE1/4/4 and sw-rzc-BL XGE1/3/14 were in default state and down on 17/18 Feb 2025; the ports on sw-rzk-L12/L21/L31/L41 and sw-rzc-L11/L12/L13/L22/L23 are unknown (no diag files). An AP needs an unused SFP+ port; the slot 2/4 modules are 10GBASE-T display interface brief, display current-configuration interface <port>, on-site check LVR network
2 Module layout of the spare 5950s. The TX switch configuration assumes JH404A with 24 x SFP+ in slot 1 and QSFP28 in slot 2; a JH321A needs the breakout variant display device, display device manuinfo LVR network
3 ClusterXL behaviour. Evidence for one stretched ENT-FW cluster: single OSPF neighbour 10.9.0.22, Check Point MAC 001c-7f65-8ec8 local on sw-rzk-BL and 001c-7f65-ff14 via sw-rzc-BL in VSI 4002, stretched sync VLAN 123; same assumption for the DC-FW (sync VLAN 196). If option A is chosen, its production risk depends on how the new interface is monitored Firewall team: cluster status and monitored interfaces for their Check Point version LVR firewall
4 Use of VLAN 1314/1315 and the DC-FW mapping. The VLANs (t3_1314_webserv_win, t3_1315_winserv) are mapped to BAGG31 "Vlan-all" on both Border Leafs, so broadcasts and unknown unicast reach the DC firewall; in February 2025 no MACs were learned LVR confirms that the VLANs are reserved for tests and that the DC-FW has no interface in them; display l2vpn mac-address vsi VPN-1314 LVR network, firewall
5 DHCP exclusions and lab addresses. Vsi914/915 relay DHCP to 10.10.252.252 and 10.10.254.254, so the scopes must exclude the TRex pools (.101-.116). Lab clients use .151-.166 of every VLAN used for lab tests (lab IP = .151 + AP index - 1 of the pinned AP; their MAC is the fixed lab-port MAC) and need a second reservation and DHCP exclusion DHCP team checks the scopes; IPAM reserves .101-.116 and .151-.166 LVR network / IPAM
6 BPDU handling on the QinQ ports. With STP globally disabled on the TX switch, any BPDU arriving from a Leaf is expected to be flooded inside that S-VLAN to the TRex ports only. The examined Leafs and Border Leafs have spanning tree globally disabled (display stp: "Protocol status: Disabled") and send no BPDUs of their own; a Leaf that runs PVST would send them to 01-00-0c-cc-cc-cd (L2CG p. 272) Bench: capture on a TRex port (-f ether dst 01:00:0c:cc:cc:cd or ether dst 01:80:c2:00:00:00), display stp on the Leaf; then decide on bpdu-drop any bench (Secured Networks / LVR)
7 Direction of unicast-suppression. The command limits unknown unicast "on an interface" (L2CR p. 98); it must be known whether it acts on received or sent traffic before it is used on lab ports Bench: lab stream to an unknown MAC, rates on the monitoring trunks bench
8 Service-mode mask with latency streams. EMU needs filtered service mode; latency and flow-stat streams must still be counted with the chosen mask (NO_TCP_UDP_MASK or service --emu) Bench: loopback p2-p3, latency stream plus EMU client, compare stats -l / stats -s per mode bench
9 Fibre distance RZK-RZC and the TX-to-TX path. The DCI uses 40G ER4 optics; the distance decides the TX-to-TX optics and the fibre offset (about 5 us per km) Fibre or DWDM documentation, OTDR; display transceiver diagnosis interface LVR network
10 X710 optics and DACs. The X710 operates only with Intel SFP+ optics (TRex manual, Table 6); passive DAC usually works Bench: dpdk_setup_ports.py -t, dmesg | grep i40e while the kernel driver is still bound Secured Networks / LVR Linux
11 Shared helper services. The Scapy server (4507) and the EMU server (4510) are single named daemons per host. An interactive stateless start restarts the Scapy server unless --no-scapy-server is given, so both lab units carry the flag; with it a lab unit neither restarts the helper at its start nor stops it at its exit (should_scapy_server_run(), cleanup_servers() in scripts/dpdk_setup_ports.py). A second instance started with --emu would restart the EMU server of trex-mon, so the lab units never use --emu (the firewalld stop of trex-cfg is part of the host-protection decision in D.1) Bench: start and stop trex-lab and trex-lab-astf while trex-mon runs; the process listening on port 4507 keeps its PID (ss -ltnp), and the EMU clients and streams of trex-mon are unaffected LVR Linux / Secured Networks
12 CPU threads of the two instances. services_core on the master thread 4, not on the busy-polling RX thread 5; OS, InfluxDB and Grafana on threads 0-3 and 16-19; isolcpus=4-15,20-31; the layout assumes siblings n and n+16 Bench: lscpu -e=CPU,CORE, latency of a loopback stream with and without load on the siblings Secured Networks / LVR Linux
13 Queue mode of trex-mon. In software mode with c: 1 TRex must select ONE_QUE, the mode with accurate latency; MULTI_QUE (with c >= 2) makes the latency values inaccurate (src/trex_modes.h) After every start: grep 'queues mode' /var/log/trex/trex-mon.log shows "set dpdk queues mode to ONE_QUE"; record it in the acceptance protocol Secured Networks / platform administrator
14 Gateway ARP from a Leaf AP (centralised gateway). With a centralised EVPN gateway HPE requires ARP flood suppression on the VTEPs, because a gateway with remote ARP learning disabled does not answer ARP requests for its MAC that arrive through tunnels (EVPN Configuration Guide R630x, PDF p. 28). LVR has no arp suppression enable and remote ARP learning at its default; in February 2025 sw-rzk-BL held ARP entries learned through the DCI tunnel, none through the RZK Leaf tunnels Rollout phase 3: an EMU client behind K-L11 (and one behind an RZC Leaf AP) resolves 10.3.14.1 (emu_show_client_info, display arp vpn-instance VPN-CORE). If not: decide between arp suppression enable in VSI view on the Leafs (VXCR pp. 5-6, a production change) and the L3 classes from the BL APs LVR network / bench
15 RZK gateway MAC on the RZC Leafs. In February 2025 SW-RZC-L21 had no EVPN MAC entry for ec9b-8b67-37b8 (gateway MAC of Vsi914/915), while sw-rzk-L11 and sw-rzc-BL had one. Without it, routed streams from an RZC AP are flooded as unknown unicast and arrive as duplicates at the other RZC APs Commissioning: display l2vpn mac-address vsi VPN-1314 and display evpn route mac on every RZC Leaf with an AP LVR network
16 SW-RZC-L21 XGE1/4/4. The port was down and in default state, but an HPE transceiver is inserted - the port may be reserved On-site check with the LVR network team; display transceiver interface Ten-GigabitEthernet1/4/4 (FCR p. 291) LVR network
17 VSI presence per Leaf. Not every VSI exists on every Leaf (VPN-724 and VPN-171 exist on sw-rzk-L11, not on SW-RZC-L21); a VLAN can only be added to an AP whose Leaf has its VSI Before every VLAN addition: display l2vpn vsi name VPN-<VLAN> on the Leaf of each AP LVR network
18 TOS bit 0 and UDP checksum 0 through the firewalls. TRex sets bit 0 of the TOS byte of statistics packets (0xB8 leaves as 0xB9, ECN field 01), and latency streams carry UDP checksum 0; the firewalls must forward both Bench and first firewall tests: capture at the receiving TRex port (TOS, checksum), firewall logs for drops of the TRex objects bench, LVR firewall
19 Recovery time after a firewall failover. The plan's acceptance target: the streams towards the firewall MAC recover within 15 s (plan 12.3 item 4). EMU learns a new gateway MAC at once from an ARP request, gratuitous requests included; an ARP reply sent to the broadcast address is dropped (counter pktRxErrNoBroadcast); otherwise the client's next periodic query is needed, every 60 s by default (trex-emu src/emu/plugins/arp/arp.go). How ClusterXL announces the new active member therefore decides whether 15 s are reached; a shorter ARP timer for the tracked VLANs ('arp': {'timer': 10}) or a forced query (emu_arp_cmd_query ... --garp off) shortens the gap, a virtual MAC removes it First planned failover, with the firewall team: emu_show_client_info ... --json (dgw.rmac) of the zone client at K-BL (-p 0 --vlan 3110 4003 --mac 02:54:52:06:0f:a3) and emu_arp_show_counters for its namespace before and after; the loss gap of the firewall-class streams in Grafana; the firewall team explains how the cluster announced the change LVR firewall / platform administrator
20 --unbind-unused-ports for the two-port ASTF configuration. trex-lab-astf uses only p4 and p6 of NIC2. If another port of the same X710 is still bound to the Linux i40e driver, TRex stops with an error; with --unbind-unused-ports it unbinds that port instead (src/main_dpdk.cpp help text "i40e only", scripts/dpdk_setup_ports.py) Bench, after a reboot: start trex-lab-astf first (log, dpdk_setup_ports.py -s for the drivers of p5 and p7), stop it, then start trex-lab and check that all four lab ports come up Secured Networks / bench

D.3 Changes from plan v0.9 to v0.91

Plan v0.91 (13 October 2026) adopted all of the following points of the draft v0.9 (12 October 2026); the list is kept for traceability. Each item names what v0.9 said, what v0.91 and this course say, and where the topic is covered.

  1. Latency capacity of trex-mon

    • Plan v0.9: up to 1023 latency streams; every monitoring stream a latency stream; budget 600 streams, about 31 kpps / 35 Mbit/s; pg_ids 1-899.
    • Plan v0.91 and this course: at most 128 latency streams per instance (src/trex_defs.h), in addition to the flow-stat streams (1023 with --software); default profile 74 latency + 102 loss-only streams (176 streams, about 20.6 kpps / 22 Mbit/s); budgets 120 / 600; pg_ids latency 1-127, loss-only 1000-1999, lab 2000+.
    • Covered in: Ch. 5, Ch. 16, B.8.

  2. Firewall zone

    • Plan v0.9: direct firewall link through the TX switches decided; "a TX switch or fibre failure only affects measurements"; its options A/B meant "stretched zone / two zones".
    • Plan v0.91 and this course: the ClusterXL risk of a monitored, non-redundant firewall interface is described; option B (existing transfer VSI 4003) is added, recommended and used as the default, option A (new firewall ports on the TX switches) is the alternative; the decision is reopened.
    • Covered in: Ch. 15, B.6, D.1.

  3. VLAN 1314/1315

    • Plan v0.9: "not mapped to any access port", described as "Performance-Test VLAN 1314/1315".
    • Plan v0.91 and this course: also mapped to the DC firewall on BAGG31 "Vlan-all" on both Border Leafs; VLAN names t3_1314_webserv_win / t3_1315_winserv ("Performance-Test" is the description of Vsi-interface914/915); their reservation for tests is to be confirmed.
    • Covered in: Ch. 10, Ch. 24, D.2.

  4. Fibre offset and TX-to-TX optics

    • Plan v0.9: "every RZC stream includes the TX-to-TX fibre once"; 10G LR/ER or 40G/100G LR4.
    • Plan v0.91 and this course: RZK->RZC once, RZC->RZK once, RZC->RZC twice (intra-RZC minus intra-RZK = 2 x one-way fibre delay); the DCI's 40G ER4 optics suggest more than 10 km, so the TX-to-TX optics follow the measured distance.
    • Covered in: Ch. 14, Ch. 16, D.1.

  5. Flooding from lab ports

    • Plan v0.9: not addressed.
    • Plan v0.91 and this course: unknown-unicast flooding of lab traffic into trex-mon and the production VSI; mitigations: pre-learning, static MAC entries on the lab ports, rate ramps, test VLANs (this course also discusses unknown-unicast suppression, to be validated on the bench).
    • Covered in: Ch. 24, B.3.

  6. API and console names

    • Plan v0.9: EMU_MASK in the Python sketch; stats --ls in the self-test, the tests and the acceptance criteria.
    • Plan v0.91 and this course: no EMU_MASK - the masks are NO_TCP_UDP_MASK, BGP_MASK, DHCP_MASK, TRANSPORT_MASK, MDNS_MASK and ALL_MASK, and service --emu sets DHCP + MDNS + NO_TCP_UDP + TRANSPORT (trex/common/trex_client.py); stats -s and stats -l instead of stats --ls.
    • Covered in: Ch. 5, Ch. 6, A.3.

  7. Reachability of the zone

    • Plan v0.9: a new subnet becomes reachable once the firewall advertises it.
    • Plan v0.91 and this course: VPN-CORE already has a default route 0.0.0.0/0 (O_ASE1) via 10.9.0.22, so the zone is reachable without switch changes; an explicit OSPF advertisement by the firewall is still recommended.
    • Covered in: Ch. 10, Ch. 15.

  8. Thread comment in trex_mon.yaml

    • Plan v0.9: "threads 0-5 and 16-27 remain for the OS".
    • Plan v0.91 and this course: threads 4 and 5 are the trex-mon master and latency threads; the OS, InfluxDB and Grafana use threads 0-3 and 16-19 (physical cores 0-3); isolcpus=4-15,20-31 keeps the TRex threads and their hyper-thread siblings free.
    • Covered in: Ch. 17, B.1.

  9. DCI crossings of inter-DC L2 streams

    • Plan v0.9: "intra-DC + 2 x DCI + fibre".
    • Plan v0.91 and this course: one DCI crossing per direction; each direction is its own stream.
    • Covered in: Ch. 14.

  10. Latency statistics

    • Plan v0.9: the exporter stores total_max (as lat_max_us) and the polled last_max; average used as the interval average.
    • Plan v0.91 and this course: last_max covers only the last 0.5 s window and average is low-pass filtered; lat_max_us is the interval maximum derived from histogram differences, lat_last_max_us keeps last_max for reference only (TRex STL Python API reference, get_pgid_stats).
    • Covered in: Ch. 5, Ch. 18, B.10.

  11. Port numbers on the lab console

    • Plan v0.9: lab examples with --port 4 and --port 7 on the trex-lab console.
    • Plan v0.91 and this course: port numbers are instance-local: on the trex-lab console p4-p7 are ports 0-3 (p4 = --port 0, p7 = --port 3; TRex manual 6.2.1).
    • Covered in: Ch. 3, Ch. 22, A.1.

  12. ASTF client and server ports

    • Plan v0.9: trex-lab-astf starts with trex_lab.yaml (p4-p7); p4 is the client side and p6 the server side.
    • Plan v0.91 and this course: ASTF roles are fixed - even TRex ports client side, odd ports server side - so with p4-p7 p6 would be port 2, a client port; trex-lab-astf uses its own trex_lab_astf.yaml with the interface list [p4, p6] (port 0 client, port 1 server) and starts with --unbind-unused-ports (TRex ASTF guide; batch options --astf-server-only, --astf-client-mask).
    • Covered in: Ch. 7, Ch. 22, B.1, B.2.

  13. Lab client addresses

    • Plan v0.9: lab examples with 10.3.14.151 and 10.3.14.160; reservation and DHCP exclusion only for the monitoring pools (.101-.116).
    • Plan v0.91 and this course: lab clients use .151-.166 of every VLAN used for lab tests (lab IP = .151 + AP index - 1 of the pinned AP, i.e. index + 50 relative to the monitoring pool); each such VLAN needs a second IPAM reservation and DHCP exclusion. v0.91 also derived a per-AP lab MAC from the same formula; since v0.92 a lab client uses the fixed MAC of its lab port instead (see D.5).
    • Covered in: Ch. 22, Ch. 24, D.2.

  14. Port selection of start

    • Plan v0.9: the lab self-test with "NIC2 p6-p7 looped" runs start -f stl/imix.py -m 100% without a port list.
    • Plan v0.91 and this course: the v3.08 console start has no -a option and without -p/--port uses all acquired ports - in this self-test also p4 and p5 - so the port is named explicitly (--port 2 = p6 on the lab console; start_line() in trex/stl/trex_stl_client.py).
    • Covered in: Ch. 4, Ch. 17, A.3.

  15. IPv4 ID of statistics streams

    • Plan v0.9: "every stream carries a TRex packet group id (pg_id) in the IPv4 identification field (TRex reserves 0xff00-0xffff)".
    • Plan v0.91 and this course: the documentation states 0xff00-0xffff, the v3.x source reserves from IP_ID_RESERVE_BASE = 0xFC00: a flow-stat stream carries 0xFC00 + an internal hw_id (not the pg_id), every latency stream 0xFFFF (src/stx/stl/trex_stl_fs.cpp). The plan's glossary entry still differs - see D.4 item 2.
    • Covered in: Ch. 5, A.1.

  16. TOS byte on the wire

    • Plan v0.9: the DSCP test inspects the TOS byte at the receiving end to detect re-marking.
    • Plan v0.91 and this course: TRex sets bit 0 of the TOS byte of every statistics stream (TOS_GO_TO_CPU in src/bp_sim.h): 0xB8 (EF) leaves as 0xB9, the DSCP is unchanged and the ECN field reads 01; received values are compared with this, not with the configured byte.
    • Covered in: Ch. 5, Ch. 14.

  17. UDP checksum of latency streams

    • Plan v0.9: latency streams built with the default (computed) UDP checksum.
    • Plan v0.91 and this course: latency streams are built with UDP checksum 0, because the TX core writes the signature into every packet without recalculating the checksum; devices that verify UDP checksums (firewalls) would otherwise drop them.
    • Covered in: Ch. 5, Ch. 21, B.8.

  18. Gateway ARP through VXLAN tunnels

    • Plan v0.9: the clients at all APs resolve 10.3.14.1 on sw-rzk-BL.
    • Plan v0.91 and this course: with a centralised EVPN gateway HPE requires ARP flood suppression on the VTEPs (EVPN Configuration Guide R630x, PDF p. 28); LVR has none, so the resolution from a Leaf AP is a commissioning check in rollout phase 3, with arp suppression enable on the Leaf VSIs or L3 classes from the BL APs as fallbacks.
    • Covered in: Ch. 6, Ch. 21, D.2.

  19. Host firewall

    • Plan v0.9: firewalld with a management zone protects the server.
    • Plan v0.91 and this course: trex-cfg stops firewalld at every TRex start (scripts/trex-cfg); together with the unauthenticated TRex control ports this makes host protection an explicit part of the design: nftables allow-list instead of firewalld, plus management-network ACLs (details open).
    • Covered in: Ch. 17, Ch. 20, B.2, D.1.

  20. services_core of trex-mon

    • Plan v0.9: 5 (the latency thread).
    • Plan v0.91 and this course: 4 (the master thread, TRex default), because the RX thread busy-polls while latency streams are active (hot_state_loop() in src/stx/common/trex_rx_core.cpp).
    • Covered in: Ch. 17, B.1.

  21. Firewall failover and the stream MACs

    • Plan v0.9: a firewall cluster failover is visible as a latency step, with no loss beyond the failover window.
    • Plan v0.91 and this course: the clusters answered with member MACs in February 2025; streams with a firewall destination MAC (L3-DCFW, the zone side of L3-ENTFW and DSCP) lose packets until they are rebuilt with the newly resolved MAC by the gateway-MAC tracker of trexctl (event gateway_mac_changed), unless a ClusterXL virtual MAC is used. How fast the rebuild can happen is still open (D.4 item 1).
    • Covered in: Ch. 15, Ch. 23, B.6, D.1.

  22. Address of the zone client

    • Plan v0.9: the zone client is 10.9.90.101.
    • Plan v0.91 and this course: the zone clients follow the client formula (IP = pool start + AP index - 1): with option B, the default, 10.9.90.106 at K-BL (index 6) and 10.9.90.113 at C-BL (index 13); with option A 10.9.90.114 (AP FW, index 14).
    • Covered in: Ch. 14, Ch. 15.

  23. Syslog host on the TX switches

    • Plan v0.9: info-center loghost 10.0.61.10 vpn-instance MGMT.
    • Plan v0.91 and this course: info-center loghost vpn-instance MGMT 10.0.61.10 - the vpn-instance keyword stands before the address (Network Management and Monitoring Command Reference R630x, PDF p. 440).
    • Covered in: Ch. 12, B.3.

  24. BE reference in the DSCP group

    • Plan v0.9: DSCP copies with EF, AF41 and CS1 (12 streams), while the DSCP test compares EF/AF41/CS1/BE side by side.
    • Plan v0.91 and this course: the DSCP groups include a BE (0x00) copy of the same pair, size and rate: 16 DSCP latency streams instead of 12, 74 latency streams in total instead of 70.
    • Covered in: Ch. 14, Ch. 16, B.8.

  25. --no-ht

    • Plan v0.9: Hyper-Threading enabled; --no-ht not discussed.
    • Plan v0.91 and this course: Hyper-Threading stays enabled; the thread layout is written explicitly into the configuration files and isolcpus keeps the siblings of the TRex threads free. --no-ht is an option of dpdk_setup_ports.py that only affects the configuration files it generates, not a t-rex-64 flag (TRex manual, Table 12).
    • Covered in: Ch. 17, A.2.

  26. Queue mode of trex-mon

    • Plan v0.9: c: 2 with dual_if threads [6, 7] and [8, 9].
    • Plan v0.91 and this course: c: 1 with threads [6] and [8]: in stateless software mode c: 1 selects ONE_QUE (accurate latency), c: 2 selects MULTI_QUE, where "latency won't be accurate" (src/trex_modes.h, choose_mode()); the start-up log line "set dpdk queues mode to ONE_QUE" is an acceptance criterion.
    • Covered in: Ch. 3, Ch. 17, B.1.

  27. Address of the lab zone client

    • Plan v0.9: the ASTF server side and the firewall route for 10.9.93.0/24 use 10.9.90.151.
    • Plan v0.91 and this course: the lab zone client follows the lab IP formula (.151 + AP index - 1): on p6, pinned to FWZ (K-BL, index 6), it is 10.9.90.156; with option A (AP FW, index 14) 10.9.90.164. v0.91 also gave it a per-AP lab MAC (02:54:52:38:0f:a3; option A 02:54:52:40:05:8c); v0.92 replaced these with p6's fixed port MAC 02:54:52:f0:00:06 in both options (see D.5).
    • Covered in: Ch. 7, Ch. 22, B.1.

  28. The bundled NDR tool

    • Plan v0.9: not addressed.
    • Plan v0.91 and this course: never run the bundled ./ndr tool on trex-rzk-01: it connects to the default RPC port 4501 (= trex-mon) and resets all ports (trex/examples/stl/stl_ndr_bench_tool.py); no-drop-rate searches use own scripts or trexctl on port 4601.
    • Covered in: Ch. 22, Ch. 24.

  29. Packet counters in the exporter

    • Plan v0.9: the exporter sketch summed all values of the tx_pkts / rx_pkts dictionaries.
    • Plan v0.91 and this course: the per-port dictionaries of get_pgid_stats() also contain the key 'total'; the counters are read from ['total'] - summing all values double-counts.
    • Covered in: Ch. 5, Ch. 18.

Further details of the reference configuration in Appendix B:

Topic Plan v0.9 Plan v0.91 This course (Appendix B)
stp edged-port on AP ports configured on every AP port resolved: still in the AP port blocks, but explained as optional (not used on the existing server ports) optional; no effect on the examined switches, where spanning tree is globally disabled
Downsampling task aggregates lat_max_us (then the lifetime total_max) and lat_last_max_us with max; no named results lat_max_us is now the interval maximum; lat_last_max_us is also kept; the per-interval hist_* counts are not downsampled; still no named results - an open point, see D.4 item 6 aggregates only the interval maximum lat_max_us with max (the 1-minute maximum already covers every packet); sums drop_pkts and hist_*, so the long-term bucket keeps the latency distribution; one named yield() per pipeline
InfluxDB repository key fetched by URL unchanged - a minor difference that remains official procedure with a check of the key fingerprint (InfluxDB documentation, "Install InfluxDB")
dpdk_setup_ports.py -i after -i the X710 ports show vfio-pci resolved: -i only generates the configuration file; the ports are bound at every TRex start or with dpdk_setup_ports.py --cfg <file> same
BIOS workload profile "Low Latency (or Custom with the settings below)" resolved: Custom profile, because the predefined "Low Latency" profile disables VT-d and VT-x same; the predefined profile also disables SR-IOV and Turbo Boost (UEFI System Utilities User Guide for HPE Compute Gen11 servers, Table 1, PDF p. 65)
Grafana version "12.4.x LTS" resolved: 12.4.x or 13.x; Grafana has no official LTS label same

D.4 Open points raised against plan v0.91

The following points were raised against plan v0.91: a target that cannot be guaranteed as written, places where the text does not yet match the design that the plan describes elsewhere or the device data, and small technical details. Item 4 was resolved in plan v0.92 (D.5); the others remain open in v0.92, and this course already teaches the form described under This course. Item 1 matters most - it affects the acceptance test.

  1. Recovery within 15 s after a firewall failover

    • Plan v0.91: acceptance criterion 12.3 item 4 - the streams towards the firewall MAC are rebuilt automatically (event gateway_mac_changed) "and recover within 15 s of the failover (or immediately if a virtual MAC is used)". The gateway-MAC tracker (9.5 step 6) polls dgw.rmac every 10 s, forces a re-resolution every 60 s and gives the detection time as "worst case ~10 s + rebuild".
    • This course: the 15 s are a target to be confirmed in the first planned failover confirm on device. EMU learns a new gateway MAC at once from an ARP request, gratuitous requests included; an ARP reply sent to the broadcast address is dropped; otherwise the client's next periodic query is needed, every 60 s by default - and the tracker's re-resolution every 60 s does not bring this below 15 s either. A shorter ARP timer for the clients of the tracked VLANs ('arp': {'timer': 10}, one ARP request per client every 10 s) and/or a forced query when firewall-class streams lose packets (EMUClient.arp.cmd_query(c_key, garp=False), console emu_arp_cmd_query ... --garp off) close the gap; a ClusterXL virtual MAC removes it.
    • Why: trex-emu src/emu/plugins/arp/arp.go (HandleRxArpPacket(), onTimerUpdate(), NewArpClient()) and emu_plugin_arp.py (client default {'arp': {'timer': 60, 'timer_disable': False}}, cmd_query()).
    • Covered in: Ch. 6, Ch. 15, Ch. 18, Ch. 23, D.2 item 19.

  2. Glossary entry for the pg_id

    • Plan v0.91: Appendix C, "Flow statistics / pg_id": "TRex per-stream TX/RX counting identified by a packet group id carried in the IPv4 identification field".
    • This course: the user-chosen pg_id identifies the statistics of a stream; inside the packets TRex uses an internal hw_id - flow-stat streams carry 0xFC00 + hw_id in the IPv4 ID, latency streams carry 0xFFFF there and the hw_id in their 16-byte signature.
    • Why: src/stx/stl/trex_stl_fs.h, src/stx/stl/trex_stl_fs.cpp; the plan's own section 4.4 describes it the same way.
    • Covered in: Ch. 5, Appendix C ("hw_id", "IP ID", "pg_id").

  3. Open item 6 names VLAN 1420 only

    • Plan v0.91: open item 6 asks for the "approval of VLAN 1420 and subnets 10.9.90.0/24 (zone), 10.9.92.0/24 and 10.9.93.0/24 (ASTF) or alternatives".
    • This course: the approval covers the zone subnet 10.9.90.0/24 - with the existing VLAN 4003 (option B) or a new VLAN 1420 (option A) - and the ASTF ranges 10.9.92.0/24 and 10.9.93.0/24.
    • Why: the plan's default is option B (4.3.5), which needs no new VLAN.
    • Covered in: D.1, Ch. 15.

  4. TX switch configuration blocks still configure option A - resolved in plan v0.92

    • Plan v0.91: the configuration blocks of sections 7.1.2-7.2 configure option A without marking it: vlan 3300; sw-rzk-TX1 XGE1/1/7 and sw-rzc-TX1 XGE1/1/8 as "FW-ZONE" access ports with PVID 3300; 3300 in the permit lists of the monitoring trunks and of the TX-to-TX trunk; lab port p6 (XGE1/1/11) "pinned [FW-ZONE]" with PVID 3300. Its port tables (6.1, 6.2), the inventory (9.3) and trex_lab_astf.yaml (8.5) use option B.
    • Plan v0.92 and this course: the TX switch configurations show option B as the default and pre-provision option A: VLAN 3300 is defined on both switches, XGE1/1/7 (rzk) and XGE1/1/8 (rzc) are configured as the option-A firewall ports (PVID 3300, QinQ) and shutdown, the monitoring trunks permit 3101 to 3110 3201 to 3210 3300 and the TX-to-TX trunk 3201 to 3210 3300 (harmless with option B, required with option A), and p6 is port access vlan 3110 with the description TRX p6 trex-lab pinned [FW-ZONE via K-BL, VLAN 4003] - followed by a marked block of what option A adds (cabling, undo shutdown, re-pin of p6).
    • Why: a switch built from the unmarked v0.91 blocks put p6 into S-VLAN 3300, which contains no firewall port with option B: the default ASTF run and the zone load test would have found no server side (10.9.90.1 not resolved, no established flows).
    • Covered in: Ch. 12, B.3, B.4.

  5. Kinds of trex_event

    • Plan v0.91: the data model (9.7) lists the kinds "annotation, apply, stop, test", although the tracker (9.5 step 6) writes gateway_mac_changed events and the alert rule "Gateway MAC changed" (10.5) filters on that kind.
    • This course: trex_event kind = annotation, apply, stop, test or gateway_mac_changed.
    • Why: the data model must contain every value that the alert rules use.
    • Covered in: Ch. 8, Ch. 18, Ch. 19.

  6. Downsampling task without named results

    • Plan v0.91: the Flux task of 10.2 contains three from() ... |> to(bucket: "trex_longterm") pipelines; none ends with yield().
    • This course: each pipeline ends with its own named result (yield(name: "mean"), yield(name: "max"), yield(name: "sum")); after influx task create, influx task run list and influx task log list show whether the runs succeed.
    • Why: Flux documentation, yield(): a script with several results must name each of them, and the names must be unique.
    • Covered in: Ch. 8, B.10.

  7. Values of the class tag

    • Plan v0.91: the profile schema (9.4) gives every group class: L2 or class: L3 (the forwarding type) and turns the class name into an InfluxDB tag, while the dashboards, alert rules and procedures (10.4, 10.5, 11.x) filter on L2, L3-BL, L3-DCFW, L3-ENTFW and DSCP.
    • This course: class: carries the tag value (L2, L3-BL, L3-DCFW, L3-ENTFW, DSCP), and the DSCP groups add path: L2|L3-ENTFW for the forwarding type; Chapter 18 notes that the plan's sketch derives the same tag from the group. The dc_pair value x-fw, which with option A followed from the AP FW ("dc: both"), is defined explicitly: streams to or from the firewall zone (VLAN 4003); with option B their APs alone would give rzk-rzk or rzc-rzc.
    • Why: with the schema as written, the tag could only take the values L2 and L3.
    • Covered in: Ch. 16, Ch. 18, B.8.

  8. Spanning tree on the fabric switches

    • Plan v0.91: section 3.2 "PVST is enabled; STP is disabled for most server VLANs"; 7.3.1 "stp edged-port ... makes the port forward immediately in the PVST-enabled VLANs ... omit it and accept a 30 s forwarding delay after link-up".
    • This course: spanning tree is globally disabled on all four examined switches; their stp mode pvst and undo stp vlan ... enable lines are configured but inactive. A new AP forwards immediately in every VLAN, stp edged-port has no effect there, and these switches send no BPDUs of their own. For the Leafs without diag files, display stp brief before the rollout shows whether they behave the same confirm on device.
    • Why: display stp and display stp brief in the diag files of 17/18 February 2025 report "Protocol status: Disabled" on sw-rzk-BL, sw-rzc-BL, sw-rzk-L11 and SW-RZC-L21; the configurations contain no stp global enable.
    • Covered in: Ch. 10, Ch. 12, Ch. 13.

  9. ICMPv6 in the nftables ruleset (optional)

    • Plan v0.91: the ruleset of 8.3 accepts ICMP for IPv4 only (ip protocol icmp accept).
    • This course: if the management network uses IPv6, accept ICMPv6 as well (meta l4proto ipv6-icmp accept, or meta l4proto { icmp, ipv6-icmp } accept for both); if IPv6 is not used there, a comment saying so is enough.
    • Why: IPv6 neighbour discovery uses ICMPv6, which the input policy drop would otherwise discard.
    • Covered in: Ch. 17, B.2.

D.5 Changes from plan v0.91 to v0.92 (14 October 2026)

Plan v0.92 (14 October 2026) is a consolidation of v0.91 after a full read-through; no section numbers changed. This course teaches v0.92: every configuration, address, pin and procedure in the chapters and in Appendices A-C follows it. The four items below are the plan's own change list (its Appendix D, rows 15-18) in the words of this course; the items of D.3 and the open points of D.4 (except item 4) are unaffected.

  1. Firewall-zone option B carried through the TX-switch configuration and the lab pins

    • Plan v0.91: the TX-switch configuration blocks still built option A unmarked (D.4 item 4); the lab pin default for p5 was K-BL, so p5 and p6 shared S-VLAN 3110.
    • Plan v0.92 and this course: option B is the default everywhere, and option A is pre-provisioned: VLAN 3300 (name TRX-AP-FW-ZONE-stretched) exists on both TX switches; sw-rzk-TX1 XGE1/1/7 and sw-rzc-TX1 XGE1/1/8 are configured as the option-A firewall ports (access, PVID 3300, qinq enable) and shutdown; the monitoring trunks permit 3101 to 3110 3201 to 3210 3300, the TX-to-TX trunk 3201 to 3210 3300 - harmless with option B (no active member port), required with option A, which therefore needs only the firewall cabling and undo shutdown. Lab pin defaults: p4 K-L11 (3101), p5 K-L41 (3105), p6 FWZ = K-BL (3110, VLAN 4003; option A 3300), p7 C-L21 (3204); p5 no longer duplicates p6. The inventory alias FWZ, the profile pairs of l3-entfw and dscp-entfw, the bill of materials and the optics counts follow option B.
    • Covered in: Ch. 11, Ch. 12, Ch. 15, Ch. 22, B.3, B.4, B.7.

  2. Fixed lab-port MACs

    • Plan v0.91: lab clients used the per-AP MAC formula with index + 50 (02:54:52:33:05:22 for the K-L11 lab client in VLAN 1314, 02:54:52:38:0f:a3 for the zone lab client at K-BL, and so on), so a lab port's MAC changed with every re-pin and with the VLAN.
    • Plan v0.92 and this course: every lab port has one fixed MAC - p4 02:54:52:f0:00:04, p5 02:54:52:f0:00:05, p6 02:54:52:f0:00:06, p7 02:54:52:f0:00:07 - set as src_mac in the port_info of trex_lab.yaml (four ports) and trex_lab_astf.yaml (p4, p6) and recorded as mac: in the inventory ports.lab. Only the lab IP follows the pinned AP (lab pool start .151 + AP index - 1). Reason: TRex answers ARP for a lab port and runs ASTF with the port MAC, and pins change; the fixed MAC keeps the static MAC entries on the TX switch (one per lab port in the S-VLAN of its pin) and the firewall objects stable. The monitoring clients (EMU) keep the per-AP formula.
    • Covered in: Ch. 4, Ch. 7, Ch. 13, Ch. 24, A.1, B.1, B.3.

  3. Inventory coverage, pin defaults, state files, apply and supervisor

    • Plan v0.91: Leaf APs carried only 1314 and 1315 in the inventory; ports.mon.0 was the default for [rzk, fw]; pgid-map.json lived in /etc/trexctl/; what happens to the running streams at the end of trexctl apply and after a trex-mon restart was not specified.
    • Plan v0.92 and this course: every Leaf and Border Leaf AP carries all VLANs under test (vlans_under_test: &utv [1314, 1315, <PROD-VLAN-A>, <PROD-VLAN-B>], Leaf APs vlans: *utv, K-BL and C-BL the same list plus 4003), as the loss-only coverage groups and L3-DCFW need them at every AP; ports.mon.0 default_for: [rzk], ports.mon.1 default_for: [rzc]; lab pin defaults as in item 1; pgid-map.json lives in /data/trexctl/state/ with applied-<profile>.json and the exporter state (nightly backup, copy into the configuration repository when it changes); trexctl apply ends with disconnect(stop_traffic=False, release_ports=True), so the streams keep running without an owner and a later apply re-acquires the ports with force=True; the exporter's supervisor thread detects a trex-mon restart (uptime reset, or active_streams at zero while a profile is recorded as active) and re-applies the active profile and restarts the gateway-MAC tracker. The validator also checks that the lab-port MACs are unique and equal to the src_mac entries of the TRex configuration files.
    • Covered in: Ch. 16, Ch. 18, Ch. 20, Appendix C ("Supervisor thread"), B.2, B.7.

  4. Cross-references and wording

    • Plan v0.91: "two" TRex configuration files and systemd units in places; inconsistent cross-references to the using tengige breakout variant and to the stateful test variants ("ASTF option A/B"); the DCI crossings of L3-BL streams were not stated; the second IPAM range per VLAN (lab pool) was missing from 2.3 and 11.11.
    • Plan v0.92 and this course: three TRex configuration files (trex_mon.yaml, trex_lab.yaml, trex_lab_astf.yaml) and three systemd units; the breakout variant is section 7.1.6; the stateful tests are "ASTF variant 1" (port addresses only) and "variant 2" (routed ranges); an L3-BL stream between two RZC APs crosses the DCI twice, between an RZC and an RZK AP once (the only 1314/1315 gateway is on sw-rzk-BL); every VLAN used for lab tests needs two 16-address IPAM ranges (monitoring .101-.116, lab .151-.166). The plan's effort estimate for trexctl rose to 12-14 days because of the gateway-MAC tracker.
    • Covered in: Ch. 14, Ch. 17, Ch. 22, A.1, B.1, B.2.