6. TRex EMU - emulated hosts

In this chapter

You will learn

• why a TRex port on its own cannot represent the dozens of hosts that the LVR monitoring needs, and what TRex EMU adds
• how the EMU server is built: a separate Go process, a ZMQ packet channel to the RX core, JSON-RPC on TCP port 4510, and the thread / namespace / client / plugin model
• how the namespace key {vport, tci, tpid} maps a double-tagged LVR frame to the right emulated host, and why the tag order matters
• what the ARP and ICMP plugins put on the wire, minute by minute, and how one gateway entry is shared per namespace
• how EMU and the stateless streams share a port in filtered service mode, and what each of them contributes
• how to load, inspect and test EMU clients from the console and from Python without disturbing the running monitoring, and what the fabric sees of them
• how to diagnose the typical faults: unresolved gateways, wrong tag order or TPID, wrong service mask, duplicate MACs, stale gateway MACs after a firewall failover

Prerequisites: Chapter 3 (ports, RX core, configuration file) and Chapter 4 (streams). Chapter 12 explains the QinQ design in depth.

Every monitoring stream of the LVR platform starts at a host and ends at a host. For the fabric these hosts must look real: they need a MAC address that the switches learn at the right attachment point, an IP address that the Border Leaf gateway or a firewall can resolve with ARP, and they should answer a ping when an operator checks them. TRex EMU is the component that creates these hosts. If EMU does not work, no routed stream can be delivered and no latency value appears in Grafana, so as the support person you need to understand it down to the frame.

The problem EMU solves

What a TRex port can do on its own

Without EMU a TRex port is exactly one network identity. The configuration file gives it one source MAC (the NIC MAC or src_mac), at most one IPv4 address (ip), one destination (dest_mac, or the MAC resolved from default_gw) and one VLAN setting (vlan) in port_info (TRex manual, "Platform YAML"). The stateless guide describes the same as two port modes: L2 mode (only a destination MAC) and L3 mode (one source IPv4 and one destination IPv4; TRex resolves the destination by ARP and sends gratuitous ARP for its own address). The port VLAN can be a single tag or one QinQ pair (set_vlan(ports, vlan) accepts a list of up to two tags), but there is only one such setting per port, and it only tags the port's own ARP and gratuitous ARP - stream packets always carry their tags in the packet itself. A port answers ARP and ping only for its own address, and only in service mode.

This is ideal for a lab - one port, one host, one VLAN - but not for permanent monitoring.

What the LVR monitoring needs

The monitoring instance trex-mon sends all its traffic through two active ports, p0 and p1, to the TX switch sw-rzk-TX1. Those ports are trunks that carry every S-VLAN, so behind one physical port sit all attachment points (APs) of both data centres, and behind every AP sit several production VLANs (C-VLANs). Each combination of AP and VLAN needs its own host:

Requirement	Why	Port L3 mode	EMU
One MAC per AP and VLAN	The fabric must learn the source at the right Leaf, in the right VSI	One MAC per port	One MAC per client
One IPv4 per AP and VLAN, from the reserved pool (e.g. 10.3.14.101-116)	Firewall rules and dashboards identify the source by address	One IPv4 per port	One IPv4 per client
Two tags per host: S-VLAN (AP) + C-VLAN	QinQ multiplexing on the trunk to sw-rzk-TX1	One VLAN setting per port	Per namespace
Answer ARP from sw-rzk-BL, the DC-FW and the ENT-FW	Routed streams are delivered by the last router, which must resolve the destination	Only for the port address	For every client
Resolve its own default gateway	L3 streams are sent to the gateway MAC	One destination per port	Per client, shared per namespace
Answer ping	Operators and firewall teams check reachability	Only in service mode	ICMP plugin
Stay alive in MAC and ARP tables	No flooding, no ARP failure after aging	Gratuitous ARP for the port	ARP keepalive per client

With the default monitoring profile (VLANs 1314 and 1315 plus two production VLANs at 13 APs, plus the two firewall-zone clients in VLAN 4003 - 10.9.90.106 at K-BL and 10.9.90.113 at C-BL) this adds up to roughly 50-60 emulated hosts on p0/p1 illustrative values.

Why EMU and not more ports or Linux namespaces?

More ports do not scale: eight 10G ports against 13 APs with several VLANs each. TRex can also attach Linux network namespaces to a port (stateless guide, "Linux network namespace"), but each node takes about 100 ms to create and STL ports must stay in full service mode. EMU was built for this job: a user-space server that creates thousands of clients per second, shares one ARP entry per gateway, supports two VLAN tags per namespace and runs beside high-rate streams in filtered service mode.

The EMU architecture

A separate process with two channels

TRex-EMU is a separate program written in Go (trex_emu/trex-emu in the TRex package). You do not start it yourself: when t-rex-64 is started with --emu, the TRex start-up script launches it as a daemon called Emu on the services core and stops it again when TRex exits. The services core is services_core from the configuration file or, if that is not set, the core of master_thread_id; trex-mon sets services_core: 4, its master thread (Chapter 17). The EMU log file is /var/log/trex/emu_daemon_server.log. EMU only works in software mode: the TRex server refuses --emu without --software with the message "emu server should work in software mode". That is one reason why trex-mon runs with --software --emu.

EMU is a second process. Packets travel between the RX core and EMU over ZMQ; the console or trexctl talks to TRex on RPC port 4501 and to EMU on RPC port 4510.

EMU has two channels to the outside world:

1. Packet channel. The RX core of TRex forwards every received packet that matches the EMU filter to EMU over a ZeroMQ (ZMQ) channel. EMU sends its own packets (ARP replies, gratuitous ARP, ICMP replies) back over the same channel, and TRex puts them directly onto the physical port. When both run on the same machine the channel uses ZMQ IPC (files) by default; --emu-zmq-tcp switches it to TCP.
2. Control channel. EMU is configured through JSON-RPC over ZMQ on TCP port 4510. The TRex console with --emu, the Python class EMUClient and later trexctl all use this port. The stateless part of TRex keeps its own RPC port (4501 by default), so a script that uses both talks to two servers.

Threads, namespaces, clients and plugins

The EMU guide defines four objects:

Thread

The EMU execution context. It owns the RPC server, the timer wheel, the event bus, the ZMQ "veth" for packets, the packet parser, the packet buffer pool and the counters, and it manages a set of namespaces.

Namespace

A separate network segment, identified by a unique key {physical port, dot1q, QinQ}. All clients of a namespace share the same port and the same VLAN tags.

Client

One emulated host inside a namespace, identified by its MAC address. A client has exactly one IPv4 address (plus optional IPv6 addresses) and one default gateway. There is no routing table: all routed traffic of a client goes to its default gateway.

Plugin

The implementation of one protocol (ARP, ICMP, DHCP, ...). A plugin can attach its own state to a thread, a namespace and a client. ARP, for example, keeps the shared ARP cache at namespace level and the per-host timers at client level.

Plugins talk to each other through an event bus: a DHCP client that receives a new address publishes it, and the ARP plugin, which subscribes to "IPv4 address changed" and "default gateway changed", re-attaches the client to the new gateway. With the static LVR addresses the bus is almost idle.

The guide quotes about 3 000 client creations per second and 3-5 Mpps per thread; the LVR monitoring needs about 50 clients and a few packets per second.

Namespaces: matching the VLAN tags

The namespace key has three parts: vport (the TRex port number in this instance), tci (two VLAN IDs) and tpid (two tag protocol identifiers). For the LVR monitoring the key always looks like this verified (constructor EMUNamespaceKey(vport, tci, tpid) in trex/emu/trex_emu_profile.py):

EMUNamespaceKey(vport=0, tci=[3101, 1314], tpid=[0x8100, 0x8100])

tci[0] is the tag that comes first on the wire (the outer S-VLAN), tci[1] the inner C-VLAN. Every AP and VLAN combination gets its own namespace with one client.

Rules that follow from the EMU source code

The EMU guide only says that the key is "physical port, dot1q, QinQ". The details matter for troubleshooting, so here is what the trex-emu source code does verified (trex-emu src/emu/core/parser.go, thread_ctx.go, client_ctx.go):

• Order = wire order. When a frame arrives, the parser stores the first tag after the source MAC in position 0 and the second in position 1. When EMU builds a frame, it writes position 0 first. So tci[0] is always the outer tag - at LVR the S-VLAN of the AP - and tci[1] the inner tag, the production VLAN.
• Only the VLAN ID is compared. The parser masks out the priority (PCP) and DEI bits; EMU sends its own frames with priority 0.
• TPID is part of the key. The lookup compares TPID and VLAN ID together. A frame with outer TPID 0x88a8 does not match a namespace defined with 0x8100. If you give a tci without tpid, 0x8100 is used.
• At most two tags. A frame with a third tag is dropped by the parser (counter errToManyDot1q).

One namespace per attachment point and VLAN

Because the key contains the S-VLAN, a client at AP K-L11 (S-VLAN 3101) and a client at AP K-BL (S-VLAN 3110) live in different namespaces even if both are in VLAN 1314. The EMU guide warns that clients behind the same physical port cannot talk at Layer 2, because a switch never sends a frame back out of its ingress port. At LVR this never bites: an L2 stream from K-L11 to K-BL leaves p0 in S-VLAN 3101 and returns to p0 in S-VLAN 3110 after a trip through the fabric - two different VLANs for the TX switch.

In the LVR platform

One namespace per (TRex port, S-VLAN, C-VLAN), one client per namespace, plugins arp and icmp. The client addresses follow the deterministic scheme: IP = pool start + (AP index - 1), MAC = 02:54:52:<AP index>:<VLAN high>:<VLAN low>. AP K-L11 (index 1) in VLAN 1314 (0x0522) is therefore 10.3.14.101 with MAC 02:54:52:01:05:22. Every MAC seen in any switch table tells you at a glance which AP and which VLAN it belongs to.

Clients and their plugins

Client fields

A client is created with a small JSON object (Go structure CClientCmd in the EMU guide). The fields relevant for LVR:

Field	Meaning
mac	Key of the client, cannot change during its lifetime
ipv4	Source IPv4 of the client
ipv4_dg	Default gateway, resolved by the ARP plugin
ipv4_mtu	MTU, 1500 by default
ipv4_force_dg, ipv4_force_mac	Use a fixed gateway MAC instead of ARP
plugs	Plugins of this client and their initial configuration

When you read a client back, EMU adds the field dgw with two values: resolve (true when the gateway MAC is known) and rmac (the resolved gateway MAC). These two fields are what the stream builder needs.

The plugins

EMU ships plugins for ARP, CDP, DHCPv4 (client and server), DHCPv6, DNS, 802.1X (DOT1X), ICMP, IGMP, IPv6 (ND, MLD), LLDP, mDNS, Netflow/IPFIX, a user-space TCP/UDP transport and Cisco telemetry (TDL) (EMU guide, "High level functionality"). The LVR monitoring uses only two of them: arp and icmp.

Plugins are enabled at three levels - def_ns_plugs (all namespaces), def_c_plugs (all clients of a namespace) and plugs (one client) - and the more specific setting wins.

The ARP plugin in detail

The EMU guide describes ARP only briefly. The behaviour below is taken from the plugin source (src/emu/plugins/arp/arp.go) verified; it is exactly what you will see in a capture or on the gateway:

1. At creation, as soon as a client has an IPv4 address and a default gateway, it sends one gratuitous ARP (broadcast, sender and target IP = its own address) and one ARP request for its gateway.
2. It answers ARP requests for its own IPv4 with a unicast ARP reply from its own MAC.
3. It learns sender MAC and IP from every ARP request it receives in the namespace - gratuitous requests included - and from every unicast ARP reply, and updates the entry at once. An ARP reply sent to the Ethernet broadcast address is dropped (counter pktRxErrNoBroadcast).
4. Every timer seconds (default 60) it sends another ARP request for its gateway. The source code calls this timer "query and keep the client alive from DUT". It can be changed with {'arp': {'timer': N}} or switched off with {'arp': {'timer_disable': True}}. A query can also be forced at any moment: console emu_arp_cmd_query ... --garp off, Python EMUClient.arp.cmd_query(c_key, garp=False) (garp=True sends a gratuitous ARP instead).

What one EMU client and its gateway exchange. The right column shows the state of the shared ARP entry for the gateway inside EMU.

Shared gateway resolution

The ARP cache lives at namespace level, not per client. All clients of a namespace that use the same gateway point to one cache entry, so the gateway is resolved once (EMU guide, "ARP Plugin": one entry in the ARP cache, each client holds a pointer to it). The entry moves through four states, which emu_arp_show_cache displays:

State	Meaning	Timers (from the source)
Incomplete	Gateway asked, no answer yet	Retries after 1, 1, 1, 3, 5, 7 s, then every 17 s
Complete	Gateway MAC known, dgw.resolve = true	Valid 10 min, renewed by every accepted ARP packet from the gateway (not by broadcast replies)
Refresh	A 10-min period without any ARP from the gateway	Re-query; after three unanswered queries back to Incomplete
Learned	Entry learned from a received ARP packet that no client uses as gateway	Removed after 1 min without use

Because every client asks for its gateway every 60 seconds, a healthy entry never leaves the Complete state. If the gateway stops answering, dgw.resolve only turns false 10 to 20 minutes later (at the first 10-minute check without any ARP from the gateway). It is a configuration check, not an outage detector - the stream counters show an outage immediately.

The ICMP plugin

The ICMP plugin answers echo (and timestamp) requests addressed to the client's IPv4 and MAC. It swaps the MAC and IP addresses of the received frame, so the reply goes back to the MAC that delivered the request - normally the gateway - with the same VLAN tags; broadcast pings are ignored. It can also send pings (emu_icmp_ping); since a client has no routing table, they always go via the default gateway, even to a destination in the same subnet.

EMU and the stateless streams on one port

Service modes and the filter mask

A TRex port in stateless mode normally ignores received packets apart from counting them, so that the RX path stays fast. Service mode changes this (stateless guide, "Port service mode"):

Mode	What reaches the control plane	Use
Off (normal)	Nothing; TRex only counts packets and measures latency	Pure traffic tests
On (service)	Everything; the port answers ARP and ping for its own address	Short setup tasks, captures
Filtered (service --emu or other flags)	Only the protocol classes in the mask, e.g. ARP and ICMP for EMU	EMU beside running streams

Filtered mode exists since TRex v2.66 and is relevant only in software mode. The guide states that the performance is as good as with service mode off. The mask is a bit field defined in the TRex Python client (trex/common/trex_client.py):

Constant	Value	Packets forwarded
NO_MASK	0	none
NO_TCP_UDP_MASK	1	everything that is not TCP or UDP: ARP, ICMP, ...
BGP_MASK	2	BGP (for the BIRD integration)
DHCP_MASK	4	DHCPv4 and DHCPv6
TRANSPORT_MASK	8	TCP and UDP with source or destination port 0xFF00-0xFFFF (EMU transport plugin)
MDNS_MASK	16	mDNS
ALL_MASK	255	all of the above

There is no constant called EMU_MASK. The console flag service --emu sets DHCP_MASK | MDNS_MASK | NO_TCP_UDP_MASK | TRANSPORT_MASK. For an ARP/ICMP-only setup such as LVR the minimum is NO_TCP_UDP_MASK; in Python verified (names from trex_client.py, set_service_mode() in trex_stl_client.py):

from trex.common.trex_client import NO_TCP_UDP_MASK
# stl = STLClient(...), connected and with the ports acquired
stl.set_service_mode(ports=[0, 1], enabled=False, filtered=True, mask=NO_TCP_UDP_MASK)

enabled and filtered are mutually exclusive, and a mask is required exactly when filtered=True.

On the RX core every received packet first passes the statistics and latency handling and is only then checked by the EMU filter (RXPortManager::handle_pkt in src/stx/common/trex_rx_port_mngr.cpp). A packet can therefore be counted and copied to EMU, and the UDP monitoring packets (destination ports 32000-32009, source ports from 40000) are counted but never sent to EMU. Confirm on the bench that flow-stat and latency counters behave identically with the chosen mask confirm on device.

Keep stream ports below 0xFF00

The transport filter forwards any TCP or UDP packet with a port of 65280 or higher to EMU. LVR streams use destination ports 32000-32009 and source ports 40000 + stream index, far below that range. Do not move them up.

Promiscuous and multicast mode

EMU clients have their own MAC addresses. A NIC port drops unicast frames for other MACs unless it is promiscuous, so the gateway's ARP reply to 02:54:52:01:05:22 would never reach EMU. The trex-mon ports must therefore be promiscuous, with multicast enabled as the EMU guide asks (console portattr -a --prom on and --mult on, API set_port_attr(ports, promiscuous=True, multicast=True)). The stream counters need it too, because the monitoring streams are addressed to client MACs.

Who does what

EMU and the stateless streams are independent. EMU never sends test traffic and never counts packets; the stream engine never answers ARP for the client addresses. They meet in two places only: the MAC address and the gateway MAC.

EMU supplies host presence and the resolved gateway MAC; the orchestrator copies the MAC into the stream; the DP and RX cores send and count.

The animation shows the sequence for one client in namespace {p0, S-VLAN 3101, C-VLAN 1314}. Watch for three things: the ARP request for the gateway leaves TRex with both tags and reaches the gateway on sw-rzk-BL; the gateway's unicast reply comes back to the client MAC (remember that it only gets to EMU because the port is promiscuous and the mask lets ARP through); and when the gateway asks for the client, EMU answers on its behalf. Only then does the stream start, with the gateway MAC as destination, and from that moment EMU is no longer involved in the data path.

For a stream of class L3-BL from K-L11 (VLAN 1314) to K-BL (VLAN 1315) the orchestrator builds this packet sketch - validate on the bench:

pkt = (Ether(src='02:54:52:01:05:22', dst=gw_mac) /   # gw_mac = dgw.rmac from EMU
       Dot1Q(vlan=3101) / Dot1Q(vlan=1314) /          # same tags as the namespace
       IP(src='10.3.14.101', dst='10.3.15.106', ttl=64) /
       UDP(sport=40001, dport=32002, chksum=0))       # 32002 = class L3-BL; checksum 0 for a latency stream

The UDP checksum is 0 because L3-BL streams are latency streams: the TX core writes the latency signature into every packet without recomputing the checksum (Chapter 5).

For class L2 the destination MAC is the MAC of the remote TRex client, which is known from the deterministic scheme; no ARP is needed. EMU still matters here: the remote client's gratuitous ARP and its 60-second ARP keepalive keep its MAC learned in every switch of the path, so the first packets of an L2 stream are not flooded as unknown unicast.

Two rules for consistent streams

1. A client's streams leave through the port that hosts its namespace. If client 02:54:52:01:05:22 lives in a namespace on vport 0, its streams must also be sent from p0. Otherwise sw-rzk-TX1 sees the same source MAC on p0 (EMU's ARP) and on p1 (the stream) in S-VLAN 3101 and the MAC flaps between the two ports.
2. If the gateway MAC changes, rebuild the stream. A stream that is already running keeps the destination MAC it was built with, while EMU updates dgw.rmac by itself from the next ARP packet of the gateway that it accepts (see the ARP plugin in detail). For the gateways on sw-rzk-BL a change should practically never happen. For the firewalls it is expected: in February 2025 sw-rzk-BL resolved the ENT-FW (10.9.0.22) and the DC-FW (10.9.0.35) to MACs of their RZC members - the clusters answered with member MACs, no virtual MAC was seen. After a failover the new active member answers with its own MAC, so every stream whose destination MAC is a firewall - both directions of L3-DCFW, and the streams of the zone clients 10.9.90.106 and 10.9.90.113, whose gateway is the zone VIP 10.9.90.1 - keeps sending to the former active member and shows loss until it is rebuilt. trexctl's gateway-MAC tracker (Chapter 18) polls dgw.rmac every 10 s and rebuilds the affected streams (same pg_ids, counters restart; event gateway_mac_changed). How fast EMU learns the new MAC depends on how ClusterXL announces it: a gratuitous ARP request is learned at once, a broadcast ARP reply is ignored and the next periodic query (every 60 s by default) is needed. The plan's recovery target of 15 s therefore needs a shorter ARP timer for the firewall VLANs (for example {'arp': {'timer': 10}}) or a forced query confirm on device; a ClusterXL virtual MAC would avoid the issue altogether. Test the behaviour with the firewall team during a planned failover (Chapter 15).

Production impact

Only trex-mon may run with --emu. The TRex start-up script manages EMU as a single daemon per host, identified by its RPC port 4510: an instance started with --emu restarts whatever already listens on that port, and stops EMU again when it exits. A lab instance started with --emu on the same server would therefore remove all monitoring hosts, and a second monitoring instance with its own EMU (an option if more than 128 latency streams are ever needed) has to be validated on the bench first. The Scapy server (port 4507, used by the TRex GUI to build packets) is a single per-host daemon in the same way: every interactive stateless instance restarts it when it starts and stops it when it exits - unless it runs with --no-scapy-server. Both lab units carry that option, so starting or stopping a lab unit leaves the Scapy server of trex-mon alone. Also remember that EMU keeps no state on disk: after every restart of trex-mon the EMU profile has to be loaded again - on the LVR server the exporter's supervisor thread does this by re-running trexctl apply for the active profile once the server accepts connections again (chapter 18).

Using EMU from the console

Starting

Connect the console to the monitoring instance with EMU support (default ports 4501/4500 for trex-mon) verified (console options in trex/console/trex_console.py):

./trex-console -p 4501 --async_port 4500 --emu

The console then also connects to the EMU server on port 4510 (--emu-server and --emu-server-port change address and port) and loads the EMU command set. In a console that was started without --emu, plugins load emu loads it later. Note that in the EMU commands -p means the TRex port, not the RPC port.

A session step by step

The following sequence prepares the ports and loads an EMU profile by hand - what trexctl apply will automate. The profile file is the one from the LVR example below sketch - validate on the bench (options from trex/utils/parsing_opts.py):

trex>portattr -a --prom on
trex>portattr -a --mult on
trex>service --emu
trex(service-filtered)>emu_load_profile -f /etc/trex/emu/lvr_mon_emu.py -t --vport 0
trex(service-filtered)>emu_show_all
trex(service-filtered)>emu_show_client_info -p 0 --vlan 3101 1314 --mac 02:54:52:01:05:22 --json
trex(service-filtered)>emu_arp_show_cache -p 0 --vlan 3101 1314
trex(service-filtered)>emu_icmp_ping -p 0 --vlan 3101 1314 --mac 02:54:52:01:05:22 --amount 3

Command	Purpose
emu_load_profile -f FILE [-m RATE] -t TUNABLES	First removes all namespaces and clients (exactly like emu_remove_profile), then loads the new profile; -t must be the last option, everything after it goes to the profile
emu_remove_profile	Removes all namespaces and clients
emu_show_all [--max-ns N] [--max-clients N]	All namespaces (port, VLAN tags, TPIDs, plugins) and their clients
emu_show_ns_info -p P [--vlan V V] [--tpid T T]	One namespace
emu_show_client_info -p P [--vlan V V] --mac MAC [--json]	One client, including dgw.resolve and dgw.rmac
emu_arp_show_cache -p P [--vlan V V] or --all-ns	Shared ARP cache of a namespace with state and reference count
emu_arp_show_counters -p P [--vlan V V]	ARP counters of a namespace (queries, replies, GARPs)
emu_arp_cmd_query -p P [--vlan V V] --mac MAC --garp on/off	Send a gratuitous ARP (on) or a gateway query (off) now
emu_icmp_ping -p P [--vlan V V] --mac MAC [--dst IP] [--amount N] [--pace PPS] [--size B]	Ping from a client; default destination is its gateway
emu_icmp_show_counters -p P [--vlan V V]	ICMP counters of a namespace
emu_show_counters	Framework counters: context, buffers, timers, the ZMQ channel and the packet parser

--vlan takes one or two values in wire order (outer first). Without --tpid the console assumes 0x8100, which is what LVR uses.

Production impact: reloading the EMU profile

emu_load_profile - and load_profile() in Python - always starts with a complete remove_profile: every namespace and client of trex-mon disappears and is created again. For that moment no host answers ARP or ping for the monitoring addresses, and every gateway has to be resolved again. On the running platform make EMU changes incrementally instead - add_ns() and add_clients() for a new AP or VLAN, remove_clients() and remove_ns() for a removed one (Python API, trex/emu/trex_emu_client.py) - and keep a full reload for a maintenance window or a restart of trex-mon.

Reading client information

The EMU guide shows the client information of its first tutorial right after loading from TRex docs (EMU guide, 3.1 "Tutorial: Load TRex server, One Client with ARP/ICMP plugin"):

emu_show_client_info --mac 00:00:00:70:00:01 -p 0 --json

{
    "plug_names": ["arp", "icmp"],
    "dhcp_ipv6": "::",
    "mac": "00:00:00:70:00:01",
    "ipv4_force_dg": false,
    "ipv6_force_mac": "00:00:00:00:00:00",
    "dg_ipv6": "::",
    "ipv4_mtu": 1500,
    "ipv4_force_mac": "00:00:00:00:00:00",
    "ipv6_dgw": "None",
    "ipv6": "1234::1000",
    "ipv6_force_dg": false,
    "ipv4_dg": "1.1.5.1",
    "ipv6_local": "fe80::200:ff:fe70:1",
    "dgw": {
        "rmac": "00:00:00:00:00:00",
        "resolve": false
    },
    "ipv6_router": "None",
    "ipv4": "1.1.5.2",
    "ipv6_slaac": "::"
}

Here the gateway 1.1.5.1 is not (yet) resolved. A resolved client shows, for example, "dgw": {"resolve": true, "rmac": "22:55:f6:f0:e8:70"} (EMU guide, 3.4.2 "DHCPv4 Server"). In the same first tutorial the guide also prints the ARP counters of the namespace from TRex docs:

emu_arp_show_counters -p 0

Arp Counters
        name          | value
----------------------+------
timerEventIncomplete  | 43
addIncomplete         | 1
pktTxArpQuery         | 54
pktTxGArp             | 1
tblActive             | 1
tblAdd                | 1
associateWithClient   | 1

Learn to read this pattern: one gratuitous ARP, 54 gateway queries, 43 retry timer events in the Incomplete state, and no pktRxArpReply line. Counters with value zero are not printed, so this namespace has not received a single ARP reply - the gateway has not answered. In a healthy LVR namespace pktRxArpReply grows by about one per minute and client (the answers to the keepalive queries), and pktTxReply grows whenever the gateway asks for a client.

Using EMU from Python

trexctl will use the Python API. The sketch below shows the calls involved; it uses only names verified in the EMU guide and the TRex client source (trex/emu/trex_emu_client.py, trex_emu_profile.py) sketch - validate on the bench.

import time
from trex.emu.api import *          # EMUClient, EMUNamespaceKey, EMUClientKey, Mac, ...
from trex.stl.api import STLClient
from trex.common.trex_client import NO_TCP_UDP_MASK

# 1. prepare the TRex ports of trex-mon (STL RPC 4501)
stl = STLClient(server='127.0.0.1', sync_port=4501, async_port=4500)
stl.connect()
stl.acquire(ports=[0, 1], force=True)
stl.set_port_attr(ports=[0, 1], promiscuous=True, multicast=True)
stl.set_service_mode(ports=[0, 1], enabled=False, filtered=True, mask=NO_TCP_UDP_MASK)

# 2. load the EMU profile (EMU RPC 4510) - removes all existing namespaces and clients first
emu = EMUClient(server='127.0.0.1', sync_port=4510)
emu.connect()
emu.load_profile(profile='/etc/trex/emu/lvr_mon_emu.py', max_rate=100,
                 tunables=['--vport', '0'])

# 3. wait until the gateway of one client is resolved and read its MAC
ns_key = EMUNamespaceKey(vport=0, tci=[3101, 1314], tpid=[0x8100, 0x8100])
c_key = EMUClientKey(ns_key, Mac('02:54:52:01:05:22').V())
gw_mac = None
for _ in range(10):
    info = emu.get_info_clients([c_key])[0]                # dict, see the JSON above
    dgw = info.get('dgw')
    if isinstance(dgw, dict) and dgw.get('resolve'):
        gw_mac = dgw['rmac']
        break
    time.sleep(1)
if gw_mac is None:
    print('gateway 10.3.14.1 not resolved for 02:54:52:01:05:22 - check the AP')

load_profile accepts a file name or an EMUProfile object; max_rate limits the client creation rate (clients per second). Like the console command, it replaces everything that EMU holds. get_all_ns_and_clients() returns all namespaces with their clients in one call - the way to check every gateway at once.

An LVR EMU profile: two APs x two VLANs

The following profile creates the four clients from the namespace figure: APs K-L11 (index 1, S-VLAN 3101) and K-BL (index 6, S-VLAN 3110), VLANs 1314 and 1315 sketch - validate on the bench. It follows the structure of the official examples (scripts/emu/simple_emu.py): a class with get_profile(tuneables) and a register() function.

/etc/trex/emu/lvr_mon_emu.py

from trex.emu.api import *
import argparse

APS = {'K-L11': (1, 3101), 'K-BL': (6, 3110)}            # AP: (AP index, S-VLAN)
VLANS = {1314: ('10.3.14.101', '10.3.14.1'),             # VLAN: (pool start, gateway)
         1315: ('10.3.15.101', '10.3.15.1')}

def client_mac(ap_index, vlan):              # 02:54:52:<AP>:<VLAN high>:<VLAN low>
    return '02:54:52:%02x:%02x:%02x' % (ap_index, vlan >> 8, vlan & 0xff)

class LvrMonEmu():
    def __init__(self):
        self.def_ns_plugs = None
        self.def_c_plugs = {'arp': {'timer': 60}, 'icmp': {}}   # (1)!

    def create_profile(self, vport):
        ns_list = []
        for ap, (index, svlan) in APS.items():
            for vlan, (pool_start, gw) in VLANS.items():
                ns_key = EMUNamespaceKey(vport=vport,
                                         tci=[svlan, vlan],          # (2)!
                                         tpid=[0x8100, 0x8100])
                ns = EMUNamespaceObj(ns_key=ns_key, def_c_plugs=self.def_c_plugs)
                ip = Ipv4(pool_start)[index - 1]                     # (3)!
                ns.add_clients(EMUClientObj(mac=Mac(client_mac(index, vlan)).V(),
                                            ipv4=ip.V(),
                                            ipv4_dg=Ipv4(gw).V()))
                ns_list.append(ns)
        return EMUProfile(ns=ns_list, def_ns_plugs=self.def_ns_plugs)

    def get_profile(self, tuneables):
        parser = argparse.ArgumentParser(description='LVR monitoring clients')
        parser.add_argument('--vport', type=int, default=0, help='trex-mon port')
        args = parser.parse_args(tuneables)
        return self.create_profile(args.vport)

def register():
    return LvrMonEmu()

1. Every client gets ARP (60-second keepalive, the default written out for clarity) and ICMP.
2. Outer tag first: S-VLAN of the AP, then the production VLAN. Swapping the two is an easy mistake to make.
3. Ipv4(start)[n] is the address n steps after start, as in the official examples: index 1 gives .101, index 6 gives .106.

The resulting clients:

AP	Namespace (vport, tci)	MAC	IPv4	Gateway (device)
K-L11	0, [3101, 1314]	02:54:52:01:05:22	10.3.14.101	10.3.14.1 (sw-rzk-BL Vsi-interface914)
K-L11	0, [3101, 1315]	02:54:52:01:05:23	10.3.15.101	10.3.15.1 (sw-rzk-BL Vsi-interface915)
K-BL	0, [3110, 1314]	02:54:52:06:05:22	10.3.14.106	10.3.14.1
K-BL	0, [3110, 1315]	02:54:52:06:05:23	10.3.15.106	10.3.15.1

In the LVR platform

In production trexctl will generate this profile from the inventory: one namespace per (TRex port, S-VLAN, C-VLAN) that appears in the stream table, one client per required address, gateway = gateway of the VLAN, plugins arp and icmp; then it waits until every client that needs a gateway reports dgw.resolve = true, reports the unresolved ones, and its gateway-MAC tracker keeps watching dgw.rmac in the VLANs whose gateway is a firewall cluster, so that it can rebuild streams after a gateway MAC change. Until trexctl exists, the file above loaded with emu_load_profile does the same job - at the first start, or in a window, because every load removes all clients first. The lab instance trex-lab does not use EMU: it runs in hardware mode, where EMU is not available, and its ports are pinned to one AP each and use the port L3 mode.

What the fabric sees of an EMU client

From the outside an EMU client is an ordinary host. Following its first gratuitous ARP through the fabric shows where it leaves traces - and which commands show them.

sw-rzk-TX1. The frame arrives on the trunk to p0 with outer tag 3101. The TX switch learns 02:54:52:01:05:22 in VLAN 3101 on that port and floods the broadcast within S-VLAN 3101: to the AP port towards K-L11 (where the S-tag is removed) and to the other TRex ports. It knows nothing about VLAN 1314 - all C-VLANs of an AP share the MAC table of its S-VLAN. Check with display mac-address vlan 3101.

sw-rzk-L11. The frame arrives single-tagged with VLAN 1314 on the AP port, matches service-instance 1314 and enters VSI VPN-1314. The Leaf learns the MAC locally and advertises it to the other VTEPs through MP-BGP in an EVPN MAC/IP advertisement route, type 2 (HPE FlexFabric 5950 EVPN Configuration Guide R630x, PDF p. 8); the broadcast itself is flooded to the other VTEPs of the VSI. Check with display l2vpn mac-address vsi VPN-1314 and display evpn route mac verified.

sw-rzk-BL (gateway). The gateway answers the client's ARP request for 10.3.14.1 - for a client behind a Leaf AP such as K-L11 this still has to be confirmed in rollout phase 3, see the centralised-gateway caveat below - and the client appears in its ARP table at the latest when the gateway resolves 10.3.14.101 for return traffic. Check with display arp vpn-instance VPN-CORE and test the ICMP plugin with a ping from the gateway verified (syntax: Network Management and Monitoring Configuration Guide R630x, PDF p. 16):

ping -vpn-instance VPN-CORE 10.3.14.101

Everybody else in the VLAN. Gratuitous ARP and the 60-second ARP requests are broadcasts that reach every port of the VLAN, including the DC firewall (VLANs 1314/1315 are mapped on BAGG31 "Vlan-all" on both Border Leafs). Fifty clients produce less than one broadcast per second - negligible, but visible in captures.

Aging. The default MAC aging time on the 5950 is 300 seconds (Layer 2 - LAN Switching Command Reference R630x, PDF p. 28) and no LVR configuration changes it. The 60-second ARP keepalive refreshes the client MAC on the whole path well within that time, even if no stream runs.

EVPN details worth knowing. A MAC that appears behind another VTEP is a MAC move: the new VTEP advertises the route with a higher sequence number and the old one withdraws its route (EVPN Configuration Guide, PDF p. 17). ARP flood suppression would let a VTEP answer ARP requests on behalf of known hosts from a table with a fixed 25-minute aging (PDF pp. 16 and 42). It is disabled by default and not configured in the LVR VSIs (only vxlan tunnel mac-learning disable is set). If it is ever enabled - it is one of the fallbacks below - EMU no longer sees every ARP request for its clients, and a VTEP can keep answering for a removed client for up to 25 minutes.

Centralised gateway and ARP - confirm in rollout phase 3

All switch gateways of LVR are centralised on the Border Leafs. The HPE FlexFabric 5950 EVPN Configuration Guide R630x (PDF p. 28) states that with a centralised EVPN gateway ARP flood suppression must be enabled on the VTEPs, because when remote ARP learning is disabled, the gateway does not answer ARP requests for its MAC that arrive through VXLAN tunnels. LVR has no arp suppression enable in any VSI and no vxlan tunnel arp-learning disable, so remote ARP learning is at its default, enabled (PDF p. 36). In February 2025 the ARP table of sw-rzk-BL held dynamic entries learned through the DCI tunnel to sw-rzc-BL - hosts in RZC do resolve RZK gateways through VXLAN - but none through the tunnels to the RZK Leafs. Whether an EMU client behind a Leaf AP such as K-L11 resolves 10.3.14.1 is therefore confirmed in rollout phase 3, before any L3 stream depends on it confirm on device. The symptom if it does not: the clients at the BL APs resolve their gateway, those at the Leaf APs stay unresolved. Fallbacks: arp suppression enable in the VSI view of the Leafs (VXLAN Command Reference R630x, PDF pp. 5-6; disabled by default - a production change on the Leaf VSIs), or run the L3 classes from the Border Leaf APs only.

A related commissioning check: in February 2025 SW-RZC-L21 had no EVPN MAC entry for ec9b-8b67-37b8, the gateway MAC of Vsi-interface914/915 on sw-rzk-BL, while sw-rzk-L11 and sw-rzc-BL had it. An RZC Leaf without this entry floods the routed TRex streams of its AP as unknown unicast, which also creates duplicates at the other RZC APs. Check display evpn route mac on every Leaf with an AP before the L3 classes start.

Production impact

EMU clients are real hosts for the network. A client with an address that is not reserved, or that is also handed out by DHCP (Vsi-interface914/915 relay DHCP), steals the address from a production host: the gateway's ARP entry flips between the two MACs. Use only reserved pool addresses and make sure the DHCP scopes exclude them.

Troubleshooting EMU

Most EMU problems show up as "client not resolved" or "routed streams show 100 % loss". Work from the inside out: TRex port, EMU namespace, TX switch, Leaf, gateway.

Symptom	Likely cause	How to check	Fix
Console cannot reach port 4510, no emu_ commands	trex-mon started without --emu, or without --software, or EMU daemon restarted or stopped by another instance started with --emu	/var/log/trex/emu_daemon_server.log; console started with --emu	Start trex-mon with --software --emu; never use --emu elsewhere
All clients unresolved, pktTxArpQuery grows, no pktRxArpReply	Ports not promiscuous, or not in filtered service mode, or mask without NO_TCP_UDP_MASK	portattr (line "promiscuous"), console prompt shows (service-filtered)	portattr -a --prom on, service --emu
Clients of one AP unresolved	S-VLAN not permitted on the TRex trunk, wrong PVID on the AP port, VLAN or service instance missing on the Leaf AP port	display mac-address vlan 3101 on the TX switch; display l2vpn mac-address vsi VPN-1314 on the Leaf	Correct the AP configuration (Chapter 13)
Clients at all Leaf APs unresolved, clients at the BL APs fine	Centralised-gateway caveat: sw-rzk-BL does not answer ARP requests that arrive through the Leaf VXLAN tunnels	Client MACs learned on TX switch and Leaf, but no ARP entry for them in display arp vpn-instance VPN-CORE on sw-rzk-BL; emu_arp_show_counters shows queries without replies	Rollout decision: arp suppression enable on the Leaf VSIs (production change) or L3 classes from the BL APs only
Clients of one VLAN unresolved at all APs	Wrong ipv4_dg, gateway down, firewall interface or rule missing (DC-FW / ENT-FW gateways)	emu_show_client_info ... --json (ipv4_dg), display arp vpn-instance VPN-CORE	Correct the gateway or involve the firewall team
Namespace exists, but nothing is ever received	Tag order swapped (tci=[1314, 3101]) or TPID 0x88a8	emu_show_all (columns "Vlan tags", "Tpids")	Outer tag first, TPID 0x8100
ARP fine, routed stream shows 100 % loss	Stream built with an old gateway MAC (for example after a firewall failover) or with a source MAC that is not the client MAC	Compare the stream's DA/SA with dgw.rmac and the client MAC	Rebuild the stream; if dgw.rmac still shows the old MAC, force a gateway query first (emu_arp_cmd_query ... --garp off)
MAC flapping on sw-rzk-TX1, EVPN MAC moves	Same client MAC in namespaces on two TRex ports, or streams sent from another port than the namespace	display mac-address mac-move, display evpn route mac	One port per client, streams from that port
Ping from the gateway fails although ARP works	icmp plugin not enabled for the client, or a firewall blocks ICMP	emu_icmp_show_counters, plug_names in the client info	Enable icmp; firewall rule for ICMP

Two more tools: emu_arp_cmd_query ... --garp on sends a gratuitous ARP immediately, which re-teaches the MAC to the fabric after a change on the AP; and in emu_show_counters the parser counters arpPkts and icmpPkts prove that packets reach EMU at all, while errToManyDot1q reveals frames with more than two tags. A TRex capture (capture monitor start --rx 0 -v) needs service mode; whether it shows everything in filtered mode has to be checked on the bench confirm on device.

Key takeaways

• A TRex port alone is one host with one VLAN setting. EMU adds many hosts per port: at LVR one namespace per (port, S-VLAN, C-VLAN) with one client each.
• EMU is a separate Go process, started by t-rex-64 --software --emu, connected to the RX core by a ZMQ packet channel and controlled by JSON-RPC on port 4510.
• The namespace key is {vport, tci[2], tpid[2]}; tci[0] is the outer tag on the wire (the S-VLAN of the AP), tci[1] the production VLAN. Only the VLAN ID counts, the TPID must match, at most two tags.
• The ARP plugin sends a gratuitous ARP and a gateway query at creation, answers ARP for the client, and repeats the gateway query every 60 seconds; the gateway entry is shared per namespace and exposed as dgw.resolve / dgw.rmac. It learns a changed MAC at once from any ARP request (gratuitous ones included) or unicast reply, ignores ARP replies sent to the broadcast address, and can be made to query at once (emu_arp_cmd_query ... --garp off).
• EMU and the streams are independent. The ports must be promiscuous and in filtered service mode (service --emu, or NO_TCP_UDP_MASK from Python); there is no EMU_MASK constant.
• The stream builder copies the client MAC (source) and the resolved gateway MAC (destination, L3 classes) into the packet; a client's streams must leave through the port of its namespace, and streams must be rebuilt when the gateway MAC changes - expected after a firewall failover, because the LVR clusters answered with member MACs. trexctl's gateway-MAC tracker does this (event gateway_mac_changed); whether streams recover within the plan's 15 s depends on how ClusterXL announces the new MAC and on the ARP timer of the firewall VLANs confirm on device.
• emu_load_profile / load_profile() first removes every namespace and client: on the running trex-mon change EMU incrementally or in a window. EMU (4510) and the Scapy server (4507) exist once per host, so only trex-mon uses --emu, and the lab units run with --no-scapy-server.
• For the fabric an EMU client is a real host (MAC per S-VLAN on the TX switch, per VSI on the Leaf, EVPN type-2 route, ARP entry on the gateway), so unreserved addresses or duplicate MACs affect production. Whether the centralised gateway answers clients behind Leaf APs is confirmed in rollout phase 3.

Check your understanding

1. A colleague wrote tci=[1314, 3101] for the K-L11 client in VLAN 1314. What happens?

EMU sends the client's ARP frames with 1314 as the outer tag. sw-rzk-TX1 has no S-VLAN 1314, so the frames go nowhere. Incoming frames from the AP carry 3101 outside and 1314 inside and do not match the namespace either. The client stays unresolved; emu_show_all shows the swapped tags and emu_arp_show_counters shows queries but no replies.

2. Which mask do you pass to set_service_mode for the LVR monitoring, and why does the console flag --emu also work?

NO_TCP_UDP_MASK is the minimum: it forwards ARP, ICMP and all other non-TCP/UDP packets to EMU. service --emu sets DHCP_MASK | MDNS_MASK | NO_TCP_UDP_MASK | TRANSPORT_MASK, which includes it. The extra classes do not hurt because LVR streams use neither DHCP nor mDNS nor ports at or above 0xFF00. There is no EMU_MASK constant.

3. The gateway of VLAN 1314 is reachable, but dgw.resolve of all K-L11 clients is false while those at K-BL are fine. Where do you look first?

The problem is specific to one AP, so look at that AP's path: the TRex trunk must permit S-VLAN 3101, the TX switch AP port must be an access port in VLAN 3101 with QinQ enabled, and the Leaf port XGE1/3/1 must permit VLAN 1314 with service-instance 1314 mapped to VPN-1314. display mac-address vlan 3101 on the TX switch shows whether the client MAC and the gateway MAC are learned at all. If the client MAC is learned on the TX switch and on the Leaf and the clients of the other Leaf APs are unresolved as well, think of the centralised-gateway caveat: K-BL clients reach sw-rzk-BL locally, K-L11 clients only through a VXLAN tunnel - exactly what rollout phase 3 confirms.

4. Why do the L2 streams not suffer from unknown-unicast flooding, although nobody ARPs for the destination client?

The destination client is an EMU client too. Its gratuitous ARP at creation and its gateway query every 60 seconds are broadcasts that teach its MAC to every switch on the path. With a MAC aging time of 300 seconds the entry never ages out while EMU runs, and the reverse stream refreshes it as well.

5. The Enterprise firewall fails over. What happens to the two directions of the L3-ENTFW streams?

The streams from VLAN 1314 to the zone are sent to the gateway MAC of sw-rzk-BL, which does not change: after a short loss burst (and possibly a latency step) they recover by themselves. The streams of the zone clients 10.9.90.106 (K-BL) and 10.9.90.113 (C-BL) are sent to the MAC that answered for the zone VIP 10.9.90.1 - in February 2025 the clusters answered with member MACs. EMU learns the new member's MAC at once if ClusterXL announces it with a gratuitous ARP request; a broadcast ARP reply is ignored, and then the next periodic query (every 60 s by default) brings it. Even then the running streams still carry the old MAC, so the dashboard shows loss until trexctl's gateway-MAC tracker has rebuilt them (event gateway_mac_changed). Recovery within the plan's 15 s therefore needs a shorter ARP timer or a forced query and is confirmed in a planned failover; a ClusterXL virtual MAC would avoid the gap.

6. Someone wants to start the lab instance with --emu as well. What do you answer?

No. The start-up script treats EMU as one daemon per host, identified by port 4510, and restarts it when an instance with --emu starts (and stops it when that instance exits), so the monitoring hosts would disappear. The lab instance also runs in hardware mode, where EMU is not available; its pinned ports use the port L3 mode instead. For the same reason the lab units run with --no-scapy-server: the Scapy helper is a per-host daemon as well.

7. A colleague wants to add one VLAN to the monitoring and plans to run emu_load_profile with the extended file on the running trex-mon. What do you point out?

emu_load_profile first removes every namespace and client and then creates the whole profile again: for a moment no monitoring host answers ARP or ping and every gateway is resolved again - a change to all hosts, not just to the new VLAN. Either add only the new namespaces and clients (add_ns() / add_clients() in Python, the way trexctl will do it) or schedule the reload in a window.

References

• TRex EMU guide, https://trex-tgn.cisco.com/trex/doc/trex_emu.html - 2.1 "High level functionality" (plugins, performance), 2.2 "TRex Architecture with TRex-EMU", 2.3 "Internal Emulation Server Architecture" (shared ARP entry), 2.4 "EMU main objects", 2.6 "EMU Namespace", 2.7 "EMU Client", 3.1 "Tutorial: Load TRex server, One Client with ARP/ICMP plugin" (console output), 3.2 "Tutorial: Ping", 3.4.2 "DHCPv4 Server" (resolved dgw), 3.12 "Python API", 3.13 "Tutorial: Debug tools", 3.14 "Tutorial: ZMQ transport layer", 8 "Network Topologies"
• TRex stateless guide, https://trex-tgn.cisco.com/trex/doc/trex_stateless.html - 2.7.8 "Tutorial: Port layer mode configuration", 2.8 "Port service mode", 2.10.4 "Linux network namespace"
• TRex ASTF guide, https://trex-tgn.cisco.com/trex/doc/trex_astf.html - 2.5.2 "Filters for EMU"
• TRex manual, https://trex-tgn.cisco.com/trex/doc/trex_manual.html - 6.2 "Platform YAML (--cfg argument)" (port_info, services_core)
• trex-emu source, https://github.com/cisco-system-traffic-generator/trex-emu (master): src/emu/core/parser.go (ParsePacket), src/emu/core/thread_ctx.go (CTunnelKey.SetJson, DEF_TPID), src/emu/core/client_ctx.go (GetL2Header), src/emu/plugins/arp/arp.go (HandleRxArpPacket: requests learned at once, broadcast replies dropped with pktRxErrNoBroadcast; defaultCompleteTimer), src/emu/plugins/icmp/icmp.go
• trex-core source, https://github.com/cisco-system-traffic-generator/trex-core (master): scripts/automation/trex_control_plane/interactive/trex/common/trex_client.py (masks, set_service_mode_base, _get_service_params), .../trex/utils/parsing_opts.py (service and EMU console options), .../trex/emu/trex_emu_client.py (load_profile, remove_profile, add_ns, add_clients, remove_clients, remove_ns), .../trex/emu/trex_emu_profile.py, .../trex/emu/emu_plugins/emu_plugin_arp.py (cmd_query, RPC arp_c_cmd_query), .../trex/console/trex_console.py, scripts/dpdk_setup_ports.py (get_core_for_services, run_servers, should_scapy_server_run, cleanup_servers), scripts/general_daemon_server (daemons found by their port: Scapy 4507, Emu 4510), scripts/trex-emu, scripts/emu/simple_emu.py, src/main_dpdk.cpp ("emu server should work in software mode"), src/stx/common/trex_rx_port_mngr.cpp (handle_pkt, is_emu_filter)
• HPE FlexFabric 5950 EVPN Configuration Guide R630x, PDF p. 8 (MAC learning), p. 16 (ARP flood suppression), p. 17 (MAC mobility), p. 28 (centralised EVPN gateway), p. 36 (remote MAC/ARP learning), p. 42 (enabling ARP flood suppression)
• HPE FlexFabric 5950 VXLAN Command Reference R630x, PDF pp. 5-6 (arp suppression enable, VSI view, default disabled)
• HPE FlexFabric 5950 Layer 2 - LAN Switching Command Reference R630x, PDF p. 28 (mac-address timer, default 300 s)
• HPE FlexFabric 5950 Network Management and Monitoring Configuration Guide R630x, PDF p. 16 (ping -vpn-instance)
• LVR running configurations (February 2025): sw-rzk-BL (vsi VPN-1314, interface Vsi-interface914), sw-rzk-L11 (vsi VPN-1314, vxlan tunnel mac-learning disable)
• LVR diag files (17/18 February 2025): sw-rzk-BL ARP table of VPN-CORE (entries via the DCI tunnel, none via the RZK Leaf tunnels; 10.9.0.22 and 10.9.0.35 resolved to RZC member MACs), display evpn route mac on SW-RZC-L21, sw-rzk-L11 and sw-rzc-BL (gateway MAC ec9b-8b67-37b8)