Appendix A - Cheat sheets¶

This appendix collects the commands and numbers you need most often when you operate, test or troubleshoot the platform: the TRex server and console, the Python APIs, the Linux side of trex-rzk-01, the Comware verification commands for the TX switches and the fabric, InfluxDB/Flux and Grafana. Every command was checked against the source named in its section - the TRex documentation or the trex-core/trex-emu source for TRex, the HPE FlexFabric 5950 R630x guides or the LVR running configurations for Comware, the official InfluxDB and Grafana documentation for the data pipeline. Commands of the planned orchestrator trexctl are deliberately absent: it does not exist yet, so everything here uses the native tools that work on day one. The explanations behind the commands are in the chapters linked from each section; full source references are in Appendix E.

Before you type anything on trex-mon

trex-mon is a production monitoring system. Open its console read-only (-r) unless you deliberately take it over. Never run reset, stop, service --off, disconnect or a forced acquire (-f, acquire --force) on trex-mon while it is monitoring: reset force-acquires the ports, stops the traffic and removes all streams (TRex STL Python API reference, reset()), and disconnect in a read-write session stops the traffic on the owned ports (disconnect(stop_traffic=True) is the default; quit leaves the traffic running). Both end the monitoring until the profile is applied again. emu_load_profile first removes every EMU client and namespace - plan EMU changes as incremental additions. See Chapter 20 and Chapter 24.

A.1 The platform in numbers¶

The two TRex instances¶

trex-lab runs either as the stateless unit or as the ASTF unit, never both; the ASTF unit has its own configuration file with only two ports.

	`trex-mon` (monitoring)	`trex-lab` (tests, stateless)	`trex-lab` in ASTF mode
systemd unit	`trex-mon.service`	`trex-lab.service`	`trex-lab-astf.service` (`Conflicts=` with `trex-lab.service`)
Configuration file	`/etc/trex/trex_mon.yaml`	`/etc/trex/trex_lab.yaml`	`/etc/trex/trex_lab_astf.yaml`
Server ports	NIC1: p0-p3 (p0/p1 active, p2/p3 reserve)	NIC2: p4-p7	p4 and p6
TRex port IDs inside the instance	0-3 = p0-p3	0-3 = p4-p7 (p4 = 0 ... p7 = 3)	0 = p4 (client side), 1 = p6 (server side)
`prefix`	`mon`	`lab`	`lab`
RPC (sync) / publisher (async) port	4501 / 4500	4601 / 4600	4601 / 4600
EMU RPC port	4510	- (no EMU)	- (no EMU)
Server flags	`-i --software --emu`	`-i --no-scapy-server` (hardware flow statistics)	`-i --astf --no-scapy-server --unbind-unused-ports`
`c` (data-plane threads per port pair)	1 - queue mode `ONE_QUE`, accurate software latency	4	4
Threads: master / latency / `dual_if`	4 / 5 / [6], [8]	10 / 11 / [12, 13, 28, 29], [14, 15, 30, 31]	10 / 11 / [12, 13, 28, 29]
Frames towards the TX switch	double-tagged (S-VLAN + C-VLAN)	single-tagged (C-VLAN); the TX switch adds the S-tag	single-tagged

TRex numbers its ports per instance in the order of the interfaces list of the configuration file: the first entry becomes port 0 (TRex manual, section 6.2.1 "Basic Configuration"). Because trex_lab.yaml lists only the four NIC2 interfaces, the server port p4 is port 0 in the trex-lab console and API. ASTF roles are fixed - even ports are client side, odd ports server side - so trex_lab_astf.yaml lists only p4 and p6, in this order. Details: Chapter 3, Chapter 11 and B.1.

Limits and budgets¶

verified

Item	Value	Source / chapter
Latency streams (`STLFlowLatencyStats`) per TRex instance	128 concurrent; budget 120	TRex stateless guide, "Per stream latency/jitter/packet errors - Limitations"; `src/trex_defs.h` `MAX_FLOW_STATS_PAYLOAD 128`
Flow-stat streams (`STLFlowStats`) with `--software`	1023; budget 600	Stateless guide Table 14; `MAX_FLOW_STATS 1023`
Flow-stat streams, X710 hardware mode	127	Table 14; `MAX_FLOW_STATS_X710 127`
QinQ (two tags) with flow statistics	only with `--software` (since v2.21)	Stateless guide, "Per stream statistics - Limitations"
X710 hardware mode	packet counters only, no RX byte counters; `--no-hw-flow-stat` counts in software	Stateless guide, after Table 14
Latency stream payload	at least 16 bytes; the last 16 bytes carry the TRex signature	Stateless guide, latency section
Total rate of all latency streams	keep far below 5 Mpps	Stateless guide, latency section
Multiplier (`-m`, `update`)	does not change latency streams: to change their rate, remove and re-add them (their counters restart)	Stateless guide, latency section
Latency accuracy in software mode	only with `c: 1` (queue mode `ONE_QUE`; start-up log "set dpdk queues mode to ONE_QUE"); with `c` >= 2 software mode uses `MULTI_QUE`, where latency is not accurate	`src/trex_modes.h`, `choose_mode()` in `src/trex_modes.cpp`
IPv4 ID of statistics streams	documented reserved range 0xff00-0xffff; the v3.x source uses `IP_ID_RESERVE_BASE` = 0xFC00: flow-stat stream = 0xFC00 + internal hw_id (not the pg_id), every latency stream 0xFFFF	Stateless guide, "Per stream statistics"; `src/stx/stl/trex_stl_fs.cpp`
TOS byte of statistics streams	TRex sets bit 0 (`TOS_GO_TO_CPU`): 0xB8 leaves as 0xB9, DSCP unchanged	`src/bp_sim.h`, `src/stx/stl/trex_stl_fs.cpp`
UDP checksum of latency streams	build with `chksum=0`: the per-packet signature is written without recalculating the checksum; confirm on device that the firewalls pass checksum-0 UDP	Chapter 5
Default monitoring profile	74 latency streams (about 10 400 pps) + 102 flow-stat streams (about 10 200 pps) = 176 streams, about 20.6 kpps, about 22 Mbit/s at Layer 2 (about 26 Mbit/s on the wire)	Chapter 16, B.8
Guard rails	at most 120 latency streams, 600 flow-stat streams, 50 kpps in total and 100 pps per stream through a firewall	Chapter 16
Loss resolution = 1 / pps	1000 pps = 1 ms, 200 pps = 5 ms, 100 pps = 10 ms, 50 pps = 20 ms	Chapter 1
Outage duration	lost packets / pps	Chapter 23

Addressing and identifiers¶

Item	Rule	Example
Monitoring client IP	pool start + (AP index - 1)	K-L11 (index 1), VLAN 1314: 10.3.14.101
Monitoring client MAC	`02:54:52:<AP index hex>:<VLAN high byte>:<VLAN low byte>`	K-L11, VLAN 1314 (0x0522): `02:54:52:01:05:22`
Lab client MAC	one fixed MAC per lab port (`src_mac` in the TRex configuration), whatever the port is pinned to: p4 `02:54:52:f0:00:04`, p5 `:05`, p6 `:06`, p7 `:07` - TRex answers ARP for a lab port with the port MAC, so the MAC cannot depend on the AP	p4: `02:54:52:f0:00:04` (Comware `0254-52f0-0004`)
Lab client IP	lab pool start .151 + (AP index - 1) of the pinned AP (index + 50 relative to the monitoring pool); addresses .151-.166 of a VLAN - a second IPAM reservation and DHCP exclusion	p4 at K-L11 (index 1), VLAN 1314: 10.3.14.151; p7 at C-L21 (index 10): 10.3.14.160
Firewall-zone clients (option B, the plan's default)	index of the Border Leaf AP; VLAN 4003 (0x0FA3)	K-BL (6) on p0 {S 3110, C 4003}: 10.9.90.106, `02:54:52:06:0f:a3`; C-BL (13) on p1 {S 3210, C 4003}: 10.9.90.113, `02:54:52:0d:0f:a3`; lab zone client (p6 = `FWZ`, K-BL index 6): 10.9.90.156, p6's port MAC `02:54:52:f0:00:06`
Firewall-zone clients if option A is chosen	AP FW, index 14; VLAN 1420 (0x058C)	p0 {S 3300, C 1420}: 10.9.90.114, `02:54:52:0e:05:8c`; lab zone client (p6, PVID 3300): 10.9.90.164, p6's port MAC `02:54:52:f0:00:06`
UDP destination port	32000 + class	32001 L2, 32002 L3-BL, 32003 L3-DCFW, 32004 L3-ENTFW, 32005 DSCP, 32009 LAB
UDP source port	40000 + stream index	stable 5-tuple per stream
pg_id ranges (planned)	latency 1-127, flow-stat 1000-1999, lab 2000 and above	keep stable for Grafana history
TOS byte	BE 0x00 (reference), EF 0xB8, AF41 0x88, CS1 0x20; on the wire with bit 0 set: 0x01, 0xB9, 0x89, 0x21	DSCP class
Frame size	128 bytes without FCS (DSCP class 256 bytes), TTL 64

Attachment points¶

AP	Device and port	TX switch port	S-VLAN	AP index	Default trex-mon port
K-L11	sw-rzk-L11 `XGE1/3/1`	sw-rzk-TX1 `XGE1/1/1`	3101	1	p0
K-L12	sw-rzk-L12 (TBD)	sw-rzk-TX1 `XGE1/1/2`	3102	2	p0
K-L21	sw-rzk-L21 (TBD)	sw-rzk-TX1 `XGE1/1/3`	3103	3	p0
K-L31	sw-rzk-L31 (TBD)	sw-rzk-TX1 `XGE1/1/4`	3104	4	p0
K-L41	sw-rzk-L41 (TBD)	sw-rzk-TX1 `XGE1/1/5`	3105	5	p0
K-BL	sw-rzk-BL `XGE1/3/6`	sw-rzk-TX1 `XGE1/1/6`	3110	6	p0
C-L11	sw-rzc-L11 (TBD)	sw-rzc-TX1 `XGE1/1/1`	3201	7	p1
C-L12	sw-rzc-L12 (TBD)	sw-rzc-TX1 `XGE1/1/2`	3202	8	p1
C-L13	sw-rzc-L13 (TBD)	sw-rzc-TX1 `XGE1/1/3`	3203	9	p1
C-L21	SW-RZC-L21 `XGE1/4/4`	sw-rzc-TX1 `XGE1/1/4`	3204	10	p1
C-L22	sw-rzc-L22 (TBD)	sw-rzc-TX1 `XGE1/1/5`	3205	11	p1
C-L23	sw-rzc-L23 (TBD)	sw-rzc-TX1 `XGE1/1/6`	3206	12	p1
C-BL	sw-rzc-BL `XGE1/3/14`	sw-rzc-TX1 `XGE1/1/7`	3210	13	p1
FW (option A only)	ENT-FW zone ports of CPF-ZV01 / CPF-ZV02 decision	rzk `XGE1/1/7` + rzc `XGE1/1/8` (configured as option-A ports, PVID 3300 + QinQ, but `shutdown` with option B)	3300	14	p0

With firewall-zone option B, the plan's default, there is no FW AP: the K-BL and C-BL APs also carry VLAN 4003, the firewall zone. S-VLAN 3300 is nevertheless defined on both TX switches and permitted on the monitoring trunks and the TX-to-TX trunk, without an active member port; option A later needs only the firewall cabling and undo shutdown. Lab ports on sw-rzk-TX1: XGE1/1/9-12 = p4-p7, default pins K-L11 (3101), K-L41 (3105), FWZ (3110) and C-L21 (3204). FWZ is the inventory alias of the zone (K-BL with VLAN 4003), so p6 has PVID 3110 and TRex tags VLAN 4003 (option A: AP FW, PVID 3300, VLAN 1420). Monitoring trunks: XGE1/1/21-24 = p0-p3. TX-to-TX link: HGE1/2/1 (module layout confirm on device). The four named Leaf and Border Leaf ports were free and down in February 2025; SW-RZC-L21 XGE1/4/4 has an HPE transceiver inserted and may be reserved confirm on device. Details: Chapter 12 and Chapter 13.

A.2 TRex server: `t-rex-64`¶

Flag	Meaning	Used by
`-i`	interactive mode: the server waits for the console and the Python API	both instances
`--cfg FILE`	platform configuration file (default `/etc/trex_cfg.yaml`)	both instances
`--software`	no hardware filters, every received packet is handled in software; needed for QinQ flow statistics and for 1023 flow-stat streams	`trex-mon`
`--emu`	starts the TRex-EMU server and its ZMQ channel to the RX core	`trex-mon`
`--astf`	advanced stateful mode	`trex-lab-astf`
`--iom 0`	I/O mode silent (0 silent, 1 normal, 2 short) - keeps the log files readable	all units
`--no-hw-flow-stat`	X710 stateless only: count flow statistics in software (adds RX bytes, lower maximum rate)	`trex-lab`, optional
`-c N`	data-plane threads per port pair (overrides `c` in the configuration file); `trex-mon` must run with 1 (queue mode `ONE_QUE`)	not used (set in the files)
`--prefix NAME`	DPDK file prefix (overrides `prefix`)	not used (set in the files)
`--no-scapy-server`	do not (re)start the Scapy helper server (port 4507): it is one named daemon per host, and every interactive stateless start restarts it unless this flag is given - so a lab start never restarts the helper of `trex-mon`	`trex-lab`, `trex-lab-astf`
`--unbind-unused-ports`	i40e (X710) only: unbind unused ports of the same NIC that are still bound to the Linux driver, instead of exiting with an error	`trex-lab-astf` (uses only p4 and p6 of NIC2) confirm on device
`-v N`	debug output at start-up (1 = start-up information)	troubleshooting

Sources: TRex manual, section 6.3 "Command line options"; --emu: TRex EMU guide, "Start TRex as a server"; --prefix, --no-scapy-server, --unbind-unused-ports: option table and help text in src/main_dpdk.cpp; start rule of the Scapy server: should_scapy_server_run() and run_servers() in scripts/dpdk_setup_ports.py verified.

The three server command lines (flags as in the systemd units, Appendix B.2)

# trex-mon
/opt/trex/current/t-rex-64 -i --software --emu \
    --cfg /etc/trex/trex_mon.yaml --iom 0
# trex-lab (stateless, hardware mode)
/opt/trex/current/t-rex-64 -i --cfg /etc/trex/trex_lab.yaml --iom 0 --no-scapy-server
# trex-lab-astf (own configuration: p4 = port 0, p6 = port 1)
/opt/trex/current/t-rex-64 -i --astf --no-scapy-server --unbind-unused-ports \
    --cfg /etc/trex/trex_lab_astf.yaml --iom 0

Four things about the wrapper

t-rex-64 must run as root and from the TRex directory: it calls ./trex-cfg (port binding, hugepages, helper daemons) before it starts the real binary _t-rex-64 (scripts/t-rex-64). That is why the units set WorkingDirectory=/opt/trex/current.
On Red Hat-family systems trex-cfg also runs systemctl stop firewalld.service - at every call of the wrapper, --help included (see A.5).
--no-ht is not a server flag. It belongs to dpdk_setup_ports.py and only affects the configuration file that script generates (TRex manual, Table 12).
The port binding is re-applied at every start, so a kernel update does not need a manual re-bind.

After every start of trex-mon the log must contain the line set dpdk queues mode to ONE_QUE (grep 'queues mode' /var/log/trex/trex-mon.log); MULTI_QUE means that c is not 1 and the latency values are not accurate (choose_mode() in src/trex_modes.cpp) verified in the source, confirm on device in the first start-up log.

A.3 trex-console¶

Connecting¶

trex-mon (monitoring)trex-lab (tests)

cd /opt/trex/current
# read-only: look, do not touch
./trex-console -p 4501 --async_port 4500 --emu -r
# acquires the ports - only when nothing else owns them
./trex-console -p 4501 --async_port 4500 --emu

cd /opt/trex/current
# works for trex-lab and trex-lab-astf, whichever unit runs;
# port numbers are instance-local: trex-lab 0-3 = p4-p7, trex-lab-astf 0 = p4, 1 = p6
./trex-console -p 4601 --async_port 4600

Option	Meaning	Default
`-s`, `--server HOST`	TRex server address	localhost
`-p`, `--port PORT`	RPC (sync) port	4501
`--async_port PORT`	publisher (async) port	4500
`--emu`	also connect to the EMU server; enables the `emu_*` commands	off
`--emu-server-port PORT`	EMU RPC port	4510
`-r`, `--readonly`	do not acquire any port (view only)	off
`-a`, `--acquire PORTS`	acquire only these ports	all ports
`-f`, `--force`	force-acquire ports owned by another user	off
`-u`, `--user NAME`	user name recorded as port owner	login name
`--batch FILE`	run the console commands in FILE	-
`-t`, `--tui`	start directly in the text UI	-
`-q`, `--quiet`	suppress output	-

Source: setParserOptions() in trex/console/trex_console.py verified.

Traffic commands (stateless)¶

Command	What it does	Notes
`start -f FILE [-p P ...] [-m MULT] [-d TIME] [--force] [-n] [-t TUNABLES]`	removes the streams on the ports, loads the profile, starts	`-t` must be the last option - everything after it goes to the profile: `-t fsize=128,pg_id=7` (old style) or `-t --fsize 128`; `--force` stops active traffic first; `-n` dry run. There is no `-a`: without `-p`, `start` uses all acquired ports
`stop [-p P ...] [-a] [--remove]`	stops traffic	`--remove` also removes the streams
`pause [-p P] [-i ID ...]`, `resume [-p P] [-i ID ...]`	pauses / resumes a port, or selected stream IDs on one port
`update [-p P] -m MULT [-i ID ...] [--force]`	changes the rate of running traffic; `+`/`-` allowed (`-m 10%+`)	latency streams keep their rate - remove and re-add them instead
`streams [-p P] [-i ID ...] [--code [FILE]]`	lists the loaded streams; `--code` prints Python that recreates them	read-only
`profiles [-p P]`	lists the loaded profiles	read-only
`acquire [-p P] [--force]`, `release [-p P]`	takes / returns port ownership
`reset [-p P] [-r]`	force-acquires, stops, removes all streams, clears this client's statistics view; `-r` also restarts the NIC	never on trex-mon in operation
`clear [-p P]`	clears the cached statistics of this console session	does not reset server counters
`disconnect`	disconnects with `stop_traffic=True` (the default of `disconnect()`): stops the traffic on the owned ports	never in a read-write session on trex-mon; leave with `quit`
`events`	shows the event log (port and server events)	read-only

Multiplier formats (-m): 10 (factor), 1kpps, 10mpps, 10mbps (L2 incl. FCS), 10mbpsl1 (L1), 50% (of L1 line rate). Duration (-d): 100 seconds, 10m, 1h. Source: match_multiplier_help and match_time_unit in trex/utils/parsing_opts.py; commands in trex/stl/trex_stl_client.py verified. Explained in Chapter 4.

Statistics: `stats`¶

Option	Shows
(none)	global statistics and port statistics
`-g`	global statistics only (CPU, RX CPU, total TX/RX, drop rate, queue full)
`-p`	port statistics (packets, bytes, errors per port)
`--ps`	port status (link, speed, promiscuous, service mode, VLAN, NUMA, PCI address)
`-s`	per-stream statistics per pg_id (TX/RX packets, rates) - at most 4 pg_ids, latency pg_ids first
`-l`	latency per pg_id (min/avg/max, jitter, errors) - at most 5 latency pg_ids
`--lh`	latency histogram per pg_id - at most 5 latency pg_ids
`-c`	CPU utilisation per data-plane thread
`-m`	mbuf (packet buffer) utilisation
`-x` / `--xz`	NIC extended counters (xstats) without / with zero lines
`--port P ...` / `-a`	restrict to ports / all ports

There is no stats --ls: use stats -s (packet counters) and stats -l (latency). The console slices the pg_id list (_show_streams_stats(), _show_flow_latency_stats() in trex/stl/trex_stl_client.py), so with the about 176 monitoring streams it is a spot-check tool; the complete table comes from get_pgid_stats() (A.4) or Grafana. In the stats command -p means "port statistics"; the port list is given with --port (later options win, conflict_handler='resolve' in parsing_opts.py). In ASTF mode stats -a shows the ASTF counters (--za including zero lines). Explained in Chapter 5 and Chapter 20.

TUI keys (`tui`, `tui -l` locked legend)¶

Where	Key	Action
everywhere	`D` / `S` / `L` / `U`	dashboard / streams (pg_id) / latency / utilisation
everywhere	`Q`	quit the TUI; `Esc` returns to the console
ASTF mode	`T`	ASTF counters (`L` latency, `S` streams)
dashboard	`0` ... `3`	toggle a port in the view; `A` all ports, `O` owned ports, `N` reset view
dashboard	`C`	clear the view (local)
dashboard	`P` / `R`	pause / resume the traffic of the owned ports
latency view	`H`	toggle the histogram

Source: trex/console/trex_tui.py verified. A read-only console owns no ports, so P cannot pause anything there - one more reason to open trex-mon with -r. The streams view uses the same function as stats -s (at most 4 pg_ids), the latency view the same as stats -l / --lh (at most 5).

Ports, service mode and capture¶

Command	Effect
`portattr`	shows the port status (same as `stats --ps`)
`portattr -a --prom on`	promiscuous mode on all acquired ports (needed for EMU clients and foreign destination MACs)
`portattr -a --mult on`	receive multicast (EMU)
`portattr --port 2 --link down`	administratively takes the link down (LED: `--led on`)
`portattr --supp`	which attributes this NIC supports

Console	Python (`set_service_mode`)	Mask value	Packets handed to the service path
`service`	`enabled=True`	-	all received packets (full service mode)
`service --off`	`enabled=False`	-	none (normal mode)
`service --no-tcp-udp`	`filtered=True, mask=NO_TCP_UDP_MASK`	1	everything that is not TCP/UDP (ARP, ICMP, ...)
`service --bgp`	`BGP_MASK`	2	BGP
`service --dhcp`	`DHCP_MASK`	4	DHCPv4/v6
`service --tran`	`TRANSPORT_MASK`	8	TCP/UDP generated by EMU
`service --mdns`	`MDNS_MASK`	16	mDNS
`service --emu`	`DHCP_MASK + MDNS_MASK + NO_TCP_UDP_MASK + TRANSPORT_MASK`	29	all EMU services
`service --all`	`ALL_MASK`	255	everything that can be filtered

There is no constant EMU_MASK. In Python, enabled and filtered are mutually exclusive and a mask requires filtered=True - so write enabled=False, filtered=True, mask=.... Whether latency and flow-stat streams are still counted with the chosen mask must be confirmed on the bench confirm on device. Sources: _get_service_params() and set_service_mode_base() in trex/common/trex_client.py. Explained in Chapter 6.

Command	Effect
`capture`	lists active captures (ID, matched, packets, BPF filter)
`capture monitor start --rx 0 -v [-f FILTER]`	live view of received packets (`-v` full decode, `-p` Wireshark pipe); stop with `capture monitor stop`
`capture record start --rx 0 --limit 200 [-f FILTER]`	records into a buffer; prints the capture ID
`capture record stop --id N -o /tmp/rx0.pcap`	stops capture N and writes the PCAP
`capture record start --rx 0 --limit 0 -f udp and dst port 32001`	`--limit 0`: counts matching packets only ("Matched" column)
`capture clear`	removes all captures

Captures need service mode (stateless guide, section "Packet capturing"). The filter is tcpdump/BPF syntax; put -f last, because every following word belongs to the filter. For QinQ frames each vlan keyword moves one tag inwards, e.g. -f vlan 3101 and vlan 1314 and udp confirm on device (behaviour of the TRex BPF engine with two tags, and what a capture sees in the filtered service mode of trex-mon).

EMU commands (console started with `--emu`)¶

Command	Purpose
`emu_load_profile -f FILE [-m MAX_RATE] [-t TUNABLES]`	replaces the loaded EMU profile: first removes all namespaces and clients (every emulated host disappears briefly), then loads the file (`-t` must be last). On `trex-mon` add or remove clients incrementally instead
`emu_remove_profile`	removes all namespaces and clients
`emu_show_all [--max-ns N] [--max-clients N]`	namespaces and clients with IPv4 and default gateway
`emu_show_ns_info -p 0 --vlan 3101 1314`	one namespace: port, tags, TPIDs, number of clients
`emu_show_client_info -p 0 --vlan 3101 1314 --mac 02:54:52:01:05:22 --json`	one client, incl. `dgw` (`resolve`, `rmac` = resolved gateway MAC)
`emu_arp_show_cache -p 0 --vlan 3101 1314` (or `--all-ns`)	ARP cache of a namespace (shared by its clients)
`emu_arp_cmd_query -p 0 --vlan 3101 1314 --mac 02:54:52:01:05:22 --garp on`	sends a gratuitous ARP for the client (`--garp off`: an ARP query for its gateway - forces an immediate re-resolution, for example after a firewall failover)
`emu_icmp_ping -p 0 --vlan 3101 1314 --mac 02:54:52:01:05:22 [--dst 10.3.14.1] [--amount 5]`	ping from an emulated client; default destination = its gateway
`emu_show_counters`, `emu_arp_show_counters -p 0 --vlan 3101 1314`	framework / ARP counters

--vlan lists the tags outer first: S-VLAN, then C-VLAN (tci[0] is the first tag after the MAC header, trex-emu src/emu/core/parser.go). If no --tpid is given, 0x8100 is used for every tag (DEF_TPID, src/emu/core/thread_ctx.go). Sources: TRex EMU guide, console section; trex/emu/trex_emu_client.py and the emu_plugins verified. Explained in Chapter 6.

ASTF (unit `trex-lab-astf`)¶

Command	Purpose
`start -f astf/http_simple.py -m 100 -d 60 [-l 1000]`	starts an ASTF profile; `-m` is a plain number (scales the connection rate), `-l` adds latency streams at the given pps
`stats -a` / `stats --za`	ASTF counters (client and server columns) without / with zero lines
`tui`, then `T` / `L`	live ASTF counters / latency
`update -m 200`	changes the multiplier of the running profile
`stop`	stops the profile
`topo load -f FILE`, `topo show`, `topo resolve`, `topo clear`	ASTF topology (virtual interfaces with VLAN, gateways)

In ASTF mode service cannot drop --no-tcp-udp ("Cannot set NO_TCP_UDP off in ASTF!"). ASTF port roles are fixed: even TRex ports are client side, odd ports server side - in trex-lab-astf port 0 = p4 (client, VLAN 1314 at K-L11) and port 1 = p6 (server, firewall zone: 10.9.90.156 in VLAN 4003; option A: 10.9.90.164 in VLAN 1420). Source: TRex ASTF guide ("Running TRex with this profile interactive"; 2.7.2 "Interactive mode" for the topology commands); trex/astf/trex_astf_client.py verified. Explained in Chapter 7.

A.4 Python API essentials¶

The client libraries ship with TRex under automation/trex_control_plane/interactive. Use the Python 3.12 virtual environment (TRex v3.08 release notes: "Python 3.12 support (client side), older python might not work").

STL (stateless)EMU

sketch - validate on the bench

from trex.stl.api import *            # STLClient, STLStream, Ether, IP, UDP ...
# not exported by trex.stl.api: the service-mode masks and Dot1Q
from trex.common.trex_client import NO_TCP_UDP_MASK
from scapy.layers.l2 import Dot1Q

# trex-mon; for trex-lab use sync_port=4601, async_port=4600
c = STLClient(server='127.0.0.1', sync_port=4501, async_port=4500)
c.connect()
c.acquire(ports=[0, 1], force=False)  # (1)!
c.set_port_attr(ports=[0, 1], promiscuous=True, multicast=True)
c.set_service_mode(ports=[0, 1], enabled=False, filtered=True,
                   mask=NO_TCP_UDP_MASK)

# K-L11 -> K-BL in VLAN 1314, outer tag (S-VLAN) first
pkt = (Ether(src='02:54:52:01:05:22', dst='02:54:52:06:05:22')
       / Dot1Q(vlan=3101) / Dot1Q(vlan=1314)
       / IP(src='10.3.14.101', dst='10.3.14.106', ttl=64)
       / UDP(sport=40001, dport=32001, chksum=0))      # (2)!
pkt = pkt / (b'\x00' * max(0, 128 - len(pkt)))   # 128 bytes without FCS

s = STLStream(name='l2-K-L11-K-BL-1314',
              packet=STLPktBuilder(pkt=pkt),
              mode=STLTXCont(pps=200),
              flow_stats=STLFlowLatencyStats(pg_id=1))
c.add_streams([s], ports=[0])
c.start(ports=[0])

st = c.get_pgid_stats(pgid_list=[1])    # at most 1024 pg_ids per call
fs = st['flow_stats'][1]                # per-port dicts plus 'total'
lat = st['latency'][1]['latency']
print(fs['tx_pkts']['total'], fs['rx_pkts']['total'])
print(lat['average'], lat['total_max'], lat['jitter'])

c.stop(ports=[0])
c.disconnect()                          # (3)!

On trex-mon the ports belong to the monitoring tooling. Run sketches like this one on a bench instance started with --software (double-tagged frames with flow statistics need software mode), not against the production monitoring instance. On trex-lab (hardware mode, ports 0-3 = p4-p7) the lab ports send single-tagged frames: drop the S-VLAN tag there.
Latency streams are built with UDP checksum 0, because TRex writes the latency signature into every packet without recalculating the checksum.
The default stop_traffic=True stops the traffic on the ports this client owns. A read-only client (exporter, status script) never acquires ports and disconnects with disconnect(stop_traffic=False). The planned trexctl apply ends the same way but also releases the ports (disconnect(stop_traffic=False, release_ports=True)): the streams keep running without an owner until the next apply re-acquires them.

sketch - validate on the bench

from trex.emu.api import *

emu = EMUClient(server='127.0.0.1', sync_port=4510)
emu.connect()

# tci[0] is the OUTER tag (S-VLAN of the AP), tci[1] the C-VLAN
ns_key = EMUNamespaceKey(vport=0, tci=[3101, 1314], tpid=[0x8100, 0x8100])
ns = EMUNamespaceObj(ns_key=ns_key,
                     def_c_plugs={'arp': {'enable': True}, 'icmp': {}})
ns.add_clients(EMUClientObj(mac=Mac('02:54:52:01:05:22').V(),
                            ipv4=Ipv4('10.3.14.101').V(),
                            ipv4_dg=Ipv4('10.3.14.1').V()))
emu.load_profile(profile=EMUProfile(ns=[ns], def_ns_plugs=None),
                 max_rate=512, tunables=[])        # removes ALL existing namespaces and clients first

key = EMUClientKey(ns_key, Mac('02:54:52:01:05:22').V())
info = emu.get_info_clients([key])
print(info[0]['dgw'])    # 'resolve' and 'rmac' = the resolved gateway MAC

API calls you will meet¶

Call	Purpose
`STLClient(server, sync_port=4501, async_port=4500)`, `connect()`, `disconnect(stop_traffic=True, release_ports=True)`	session; the default `stop_traffic=True` stops the traffic on the ports this client owns - read-only tools use `disconnect(stop_traffic=False)`; the planned `trexctl apply` leaves with `disconnect(stop_traffic=False, release_ports=True)` so that the streams run on without an owner
`acquire(ports, force)`, `release(ports)`, `reset(ports)`	ownership (`reset` = force acquire + stop + remove streams + clear this client's statistics view)
`add_streams(streams, ports)`, `remove_all_streams(ports)`	stream management
`start(ports, mult, duration, force)`, `stop(ports, rx_delay_ms)`, `update(ports, mult)`, `pause(ports)`, `resume(ports)`	traffic control; `update` and `mult` do not change latency streams
`wait_on_traffic(ports, timeout, rx_delay_ms)`	wait for bursts to end
`get_stats()`, `get_pgid_stats(pgid_list)`, `get_active_pgids()`, `clear_stats()`	statistics; `clear_stats()` only moves this client's reference point
`set_port_attr(ports, promiscuous=, multicast=, link_up=)`	port attributes
`set_service_mode(ports, enabled, filtered, mask)`	service mode
`set_l2_mode(port, dst_mac)`, `set_l3_mode(port, src_ipv4, dst_ipv4, vlan=None)`, `set_vlan(ports, vlan)`, `resolve(ports, retries, vlan)`, `ping_ip(src_port, dst_ip, ..., vlan)`	port-level L2/L3 (lab ports); `set_vlan` tags only the port's own ARP / gratuitous ARP - stream packets carry their tags in the Scapy packet
`start_capture(rx_ports=, limit=, bpf_filter=)`, `stop_capture(capture_id, output)`	capture
`STLStream(name, packet, mode, flow_stats)`, `STLTXCont(pps=)`, `STLTXSingleBurst(total_pkts=)`, `STLFlowStats(pg_id)`, `STLFlowLatencyStats(pg_id)`	streams

get_pgid_stats() returns flow_stats[pg_id] with tx_pkts, rx_pkts (and bytes) as dictionaries per port plus 'total', and latency[pg_id] with latency (average, total_min, total_max, last_max, jitter, histogram) and err_cntrs (dropped, dup, out_of_order, seq_too_high, seq_too_low). Remember: average is low-pass filtered (previous/2 + last period/2), last_max covers only the last 0.5 s window, histogram keys are the lower bounds of logarithmic buckets in microseconds (TRex STL Python API reference, get_pgid_stats). Explained in Chapter 5 and Chapter 18.

A.5 Linux and DPDK on trex-rzk-01¶

verified dpdk_setup_ports.py options (trex-core scripts/dpdk_setup_ports.py) · confirm on device PCI addresses, unit names and paths as installed

NICs and DPDKHugepages, IOMMU, CPUServices and logs

cd /opt/trex/current
./dpdk_setup_ports.py -s    # status: DPDK-bound and kernel-bound ports
./dpdk_setup_ports.py -t    # table: PCI address, NUMA node, MAC, driver
./dpdk_setup_ports.py -m    # memory banks (channels per NUMA node)
./dpdk_setup_ports.py -i    # interactive creation of a configuration file (binds nothing)
# non-interactive draft of a configuration file
./dpdk_setup_ports.py -c 31:00.0 31:00.1 31:00.2 31:00.3 --no-ht --prefix mon \
    --zmq-pub-port 4500 --zmq-rpc-port 4501 -o /tmp/trex_mon_draft.yaml
# bind the ports of a configuration without starting TRex (TRex does this at every start)
./dpdk_setup_ports.py --cfg /etc/trex/trex_mon.yaml
# return all DPDK ports to the Linux driver (both TRex instances stopped)
./dpdk_setup_ports.py -l
lspci -nn -d 8086:1572      # the Intel X710 SFP+ ports
cat /sys/bus/pci/devices/0000:31:00.0/numa_node

grep -E 'HugePages_(Total|Free)|Hugepagesize' /proc/meminfo
cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
cat /proc/cmdline   # IOMMU and hugepage arguments (Chapter 17)
dmesg | grep -i -E 'DMAR|IOMMU' | head
lsmod | grep vfio
lscpu -e=CPU,CORE,SOCKET,NODE   # which CPUs share a physical core (siblings)
numactl --hardware
tuned-adm active

systemctl status trex-mon
systemctl restart trex-mon      # monitoring gap; the exporter's supervisor thread re-applies the active profile (trexctl apply)
grep 'queues mode' /var/log/trex/trex-mon.log | tail -n 1   # after every start: ONE_QUE
systemctl start trex-lab        # or trex-lab-astf (Conflicts= stops the other)
systemctl stop trex-lab
systemctl list-units 'trex*'
tail -f /var/log/trex/trex-mon.log   # TRex output (StandardOutput=append:)
journalctl -u trex-mon -b            # start/stop events of the unit only
ss -ltnp | grep -E ':(4500|4501|4507|4510|4600|4601) '   # TRex listeners (all addresses)
systemctl is-active firewalld nftables                    # host protection, see below

Check	Why
`dpdk_setup_ports.py -s` shows all eight X710 ports on a DPDK driver (`vfio-pci` expected)	TRex cannot use ports bound to the kernel driver `i40e`
`HugePages_Total` = 4096 (2 MB pages)	both instances together use `limit_memory` 2 x 3072 MB; the wrapper only raises the count if fewer than 2048 pages per NUMA node are configured (`config_hugepages()` in `dpdk_setup_ports.py`)
`iommu=` present in `/proc/cmdline`	`dpdk_setup_ports.py` uses `vfio-pci` only if `iommu=` is on the kernel command line; otherwise it falls back to `uio_pci_generic` or a freshly compiled `igb_uio` (unsigned module - Secure Boot must be off). A pre-built `igb_uio` for the running kernel, if present, is tried first (`do_run()` in the script)
all X710 ports on the NUMA node of the TRex threads	otherwise packets cross the socket interconnect (single socket: `-1` or `0`)
hyper-thread siblings of the TRex threads	`lscpu -e` shows the `CORE` of every CPU; with siblings n and n+16 the OS uses threads 0-3 and 16-19 and `isolcpus=4-15,20-31` keeps the TRex threads and their siblings free - see B.1
`set dpdk queues mode to ONE_QUE` in `/var/log/trex/trex-mon.log`	`trex-mon` runs `--software` with `c: 1`; `MULTI_QUE` would make the latency values inaccurate

Known issue confirm on device: TRex v3.06 and later ship their own libstdc++.so.6; if ./t-rex-64 --help aborts with GLIBCXX_3.4.30 not found, point TRex to the system library: cd so/x86_64 && mv libstdc++.so.6 libstdc++.so.6.trex && ln -s /usr/lib64/libstdc++.so.6 libstdc++.so.6. Explained in Chapter 17.

TRex stops firewalld at every start

On Red Hat-family systems the wrapper script trex-cfg, which t-rex-64 runs before every start, executes systemctl stop firewalld.service (trex-core scripts/trex-cfg). After a boot or a restart of trex-mon, a host firewall built on firewalld is therefore no longer active - while the TRex control ports (4500/4501, 4600/4601, EMU 4510, Scapy 4507) listen on all addresses without authentication. The plan therefore replaces firewalld with the nftables service and an allow-list for the management hosts, plus ACLs on the management network; the details are still an open decision decision - see B.2 and Appendix D. After every TRex start check that the chosen protection is active (systemctl is-active firewalld nftables).

A.6 Comware verification commands¶

All commands are display commands (no configuration change). Page numbers are PDF pages of the HPE FlexFabric 5950 Release 630x guides: L2CR = Layer 2 - LAN Switching Command Reference, VXCR = VXLAN Command Reference, EVCR = EVPN Command Reference, L3S = Layer 3 - IP Services Command Reference, L3R = Layer 3 - IP Routing Command Reference, FCR = Fundamentals Command Reference. verified syntax for R630x.

TX switches (sw-rzk-TX1, sw-rzc-TX1)¶

Command	What you should see	Reference
`display device manuinfo`	the module in each slot (24 x SFP+ in slot 1 and a QSFP28 module in slot 2 assumed)	FCR p. 273
`display qinq`	every AP port and every lab port; the trunks are not listed	L2CR p. 277
`display vlan 3101`	`XGE1/1/1` untagged member (AP), `XGE1/1/21-24` tagged members (trunks), a lab port if pinned to 3101	L2CR p. 193
`display mac-address vlan 3101`	TRex client MACs (`0254-5201-....`) on the trunks, the fixed lab-port MAC `0254-52f0-0004` on `XGE1/1/9` while p4 is pinned to K-L11, Leaf-side MACs (gateway, remote clients) on `XGE1/1/1`	L2CR p. 11
`display mac-address mac-move`	no entries; an entry means one MAC on two ports (duplicate client, loop)	L2CR p. 14
`display interface brief`	AP ports, trunks and the TX-to-TX link up at the expected speed	L2CR p. 47
`display counters rate inbound interface`	per-port rates; compare with the profile (about 20.6 kpps for `trex-mon`)	L2CR p. 43
`display lldp neighbor-information list`	the Leaf/BL on each AP port; nothing on TRex ports	L2CR p. 355
`display stp`	spanning tree globally disabled	L2CR p. 294
`display transceiver diagnosis interface HundredGigE1/2/1`	optical levels of the TX-to-TX link	FCR p. 290
`display current-configuration interface Ten-GigabitEthernet1/1/9`	the current pin (`port access vlan`) of lab port p4	FCR p. 179

Leafs and Border Leafs¶

Command	What you should see	Reference
`display interface Ten-GigabitEthernet1/3/1`	AP port up, 10 Gbit/s, no input errors	L2CR p. 47
`display current-configuration interface Ten-GigabitEthernet1/3/1`	trunk and service instances as in B.5	FCR p. 179
`display l2vpn service-instance interface Ten-GigabitEthernet1/3/1 verbose`	the service instances of the AP port and their VSIs	VXCR p. 15
`display l2vpn vsi name VPN-1314 verbose`	VSI state and its local attachments; before adding a VLAN to an AP check that its VSI exists on that Leaf (not every VSI exists on every Leaf)	VXCR p. 17
`display l2vpn mac-address vsi VPN-1314`	local TRex MAC on the AP port, remote TRex MACs via VXLAN tunnels; on RZC Leafs also the gateway MAC `ec9b-8b67-37b8` of Vsi-interface914/915 (commissioning check: missing on SW-RZC-L21 in February 2025)	VXCR p. 13
`display l2vpn mac-address vsi VPN-4003` (both Border Leafs; firewall zone, option B)	the zone clients `0254-5206-0fa3` (local on `XGE1/3/6` of sw-rzk-BL) and `0254-520d-0fa3` (local on `XGE1/3/14` of sw-rzc-BL) and the MAC of the active firewall member; before the zone is set up the VSI should still be empty (no MACs in February 2025)	VXCR p. 13
`display evpn route mac`	EVPN MAC routes for the TRex client MACs	EVCR p. 23
`display vxlan tunnel`	VXLAN tunnels to the remote VTEPs	VXCR p. 21
`display arp vpn-instance VPN-CORE` (sw-rzk-BL)	ARP entries 10.3.14.1xx / 10.3.15.1xx - entries of clients behind Leaf APs show that the centralised gateway answers ARP through the tunnels confirm on device	L3S p. 33
`display interface Vsi-interface914`	packet counters of the VLAN 1314 gateway	VXCR p. 59
`display ip routing-table vpn-instance VPN-CORE 10.9.90.0 24`	route to the firewall zone via 10.9.0.22 (or, until the firewall advertises it, the default route)	L3R pp. 25-26
`display ospf 1 peer`	neighbour 10.9.0.22 (ENT-FW) Full	L3R p. 181
`display mac-address mac-move`	no entries	L2CR p. 14
`display counters rate inbound interface Ten-GigabitEthernet1/3/1`	the AP rate during a test	L2CR p. 43

Small changes you may need¶

Re-pin lab port p4 to AP C-L21 (sw-rzk-TX1)

system-view
interface Ten-GigabitEthernet1/1/9
 port access vlan 3204
 description TRX p4 trex-lab pinned [C-L21]
quit

verified syntax (port access vlan, L2CR p. 202); port numbers confirm on device. The TRex side keeps sending single-tagged frames; only the S-VLAN behind the port changes - the port MAC (02:54:52:f0:00:04) and the TRex configuration stay as they are, only the lab IP follows the new AP (10.3.14.160 at C-L21). If static MAC entries protect the lab port (B.3), move them to the new S-VLAN as well. Explained in Chapter 12 and Chapter 22.

A.7 InfluxDB: influx CLI and Flux¶

Task	Command
Is InfluxDB up? (no token needed)	`influx ping`
Create an active CLI connection profile	`influx config create -n lvr -u http://127.0.0.1:8086 -o lvr -t <TOKEN> -a`
Buckets and retention	`influx bucket list -o lvr`
Change a retention period	`influx bucket update -i <BUCKET-ID> -r 120d`
Tokens	`influx auth list` (since 2.9.0 tokens are stored hashed: copy a new token when it is created)
Tasks	`influx task list`, `influx task run list --task-id <ID>`, `influx task log list --task-id <ID>`
Run a query from a file	`influx query -f /tmp/q.flux`
Back up	`influx backup /data/backup/influx-$(date +%F)` (one bucket: `-b trex`)

Sources: InfluxDB OSS v2 documentation, influx CLI reference pages; token hashing: "Manage API tokens - Token hashing" verified. A backup of a 2.9 instance contains no plaintext operator token: a full restore needs influx restore --operator-token - keep the operator token in the password safe.

Flux snippets for the planned data model (measurement trex_stream, tags such as name, class, dc_pair, kind; fields such as drop_pkts, rx_pps, lat_max_us) sketch - validate in the Data Explorer:

Drops per class and DC pairStreams without RX (path down)Top 10 interval maximaOutage seconds per stream

from(bucket: "trex")
  |> range(start: -15m)
  |> filter(fn: (r) => r._measurement == "trex_stream" and r._field == "drop_pkts")
  |> group(columns: ["class", "dc_pair"])
  |> sum()

from(bucket: "trex")
  |> range(start: -1m)
  |> filter(fn: (r) => r._measurement == "trex_stream" and r._field == "rx_pps")
  |> group(columns: ["name"])
  |> max()
  |> filter(fn: (r) => float(v: r._value) == 0.0)

from(bucket: "trex")
  |> range(start: -1h)
  |> filter(fn: (r) => r._measurement == "trex_stream" and r._field == "lat_max_us")
  |> group(columns: ["name"])
  |> max()
  |> group()
  |> sort(columns: ["_value"], desc: true)
  |> limit(n: 10)

drops = from(bucket: "trex")
  |> range(start: -24h)
  |> filter(fn: (r) => r._measurement == "trex_stream" and r._field == "drop_pkts")
  |> group(columns: ["name"])
  |> sum()
  |> set(key: "_field", value: "drops")
pps = from(bucket: "trex")
  |> range(start: -24h)
  |> filter(fn: (r) => r._measurement == "trex_stream" and r._field == "tx_pps")
  |> group(columns: ["name"])
  |> mean()
  |> set(key: "_field", value: "pps")
union(tables: [drops, pps])
  |> pivot(rowKey: ["name"], columnKey: ["_field"], valueColumn: "_value")
  |> map(fn: (r) => ({r with outage_s: float(v: r.drops) / r.pps}))
  |> sort(columns: ["outage_s"], desc: true)

For time ranges longer than the 90-day raw retention use bucket trex_longterm (1-minute aggregates). Explained in Chapter 8 and Chapter 18.

A.8 Grafana quick tips¶

Topic	Tip
Keyboard	`?` lists all shortcuts; `Ctrl`+`S` save, `D` `S` dashboard settings, `D` `K` kiosk mode, `E` edit / `V` full screen for the panel under the mouse, `Esc` back
Flux macros	`v.timeRangeStart`, `v.timeRangeStop`, `v.windowPeriod`, `v.defaultBucket`, `v.organization`
Variables	`${class}` inside expressions; multi-value variables in a regex filter: `r.dc_pair =~ /${dc_pair:regex}/` (`:pipe` gives `a\|b`)
Refresh	live views: 5-30 minute window with a few seconds refresh; the minimum refresh is `[dashboards] min_refresh_interval` (default 5 s); switch auto-refresh off while zooming
Long ranges	use the `trex_longterm` data source above 24 hours
Filtering	filter by tags (class, DC pair, AP) rather than by individual pg_ids
Maintenance	recurring windows: mute timings; one-off windows: silences - both stop notifications, not alert evaluation
Provisioning	data sources and dashboards come from files under `/etc/grafana/provisioning/`; edit the file, not the UI (see B.11)

Sources: Grafana documentation - "Use dashboards" (keyboard shortcuts), "InfluxDB query editor" (macros), "Variable syntax", "Configure Grafana" (min_refresh_interval), "Configure mute timings", "Provision Grafana" verified. Explained in Chapter 9 and Chapter 19.