vRouter Command Line Utilities ============================== :date: 2020-12-07  Overview -------- vRouter is the component that takes packets from VMs and forwards them to their destinations. In this effort, vRouter depends on the vRouter agent to make sense of the overall topology, understand the various policies that govern the communication between VMs, and program them in vRouter in a way vRouter understands. vRouter has a few fundamental data structures that abstracts out the various communication paths. There is "interface," "flow," "route," and "nexthop" that enables vRouter to push packets to their eventual destinations. In addition, vRouter also has good statistics that can help understand and debug packet paths. Various command line utilities provided by the vRouter can be used to display these data structures and better understand the behavior that one sees in a compute node. This section describes the shell prompt utilities available for examining the state of the vRouter kernel module in OpenSDN. The most useful commands for inspecting the OpenSDN vRouter module are summarized in the following table. +---------------+-----------------------------------------------------+ | Command | Description | +===============+=====================================================+ | ``vif`` | Inspect vRouter interfaces associated with the | | | vRouter module. | +---------------+-----------------------------------------------------+ | ``flow`` | Display active flows in a system. | +---------------+-----------------------------------------------------+ | ``vrfstats`` | Display next hop statistics for a particular VRF. | +---------------+-----------------------------------------------------+ | ``rt`` | Display routes in a VRF. | +---------------+-----------------------------------------------------+ | ``dropstats`` | Inspect packet drop counters in the vRouter. | +---------------+-----------------------------------------------------+ | ``mpls`` | Display the input label map programmed into the | | | vRouter. | +---------------+-----------------------------------------------------+ | ``mirror`` | Display the mirror table entries. | +---------------+-----------------------------------------------------+ | ``vxlan`` | Display the VXLAN table entries. | +---------------+-----------------------------------------------------+ | ``nh`` | Display the next hops that the vRouter knows. | +---------------+-----------------------------------------------------+ | ``--help`` | Display all command options available for the | | | current command. | +---------------+-----------------------------------------------------+ | dpdkinfo | Displays internal data structure details of a DPDK | | | enabled vRouter. | +---------------+-----------------------------------------------------+ | dpdkconf | Use this command to add or delete a DDP profile. | +---------------+-----------------------------------------------------+ The following sections describe each of the vRouter utilities in detail. vif Command ----------- The vRouter requires vRouter interfaces (``vif``) to forward traffic. Use the ``vif`` command to see the interfaces that are known by the vRouter. .. note:: Having interfaces only in the OS (Linux) is not sufficient for forwarding. The relevant interfaces must be added to vRouter. Typically, the set up of interfaces is handled by components like ``nova-compute`` or vRouter agent. The ``vif`` command can be used to see the interfaces that the vRouter is aware of by including the ``--list`` option. **Example: vif --list** :: bash$ vif --list Vrouter Interface Table Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2 D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged Mnp=No MAC Proxy vif0/0 OS: eth0 (Speed 1000, Duplex 1) Type:Physical HWaddr:00:25:90:c3:08:68 IPaddr:0 Vrf:0 Flags:L3L2Vp MTU:1514 Ref:22 RX packets:2664341 bytes:702708970 errors:0 TX packets:1141456 bytes:234609942 errors:0 vif0/1 OS: vhost0 Type:Host HWaddr:00:25:90:c3:08:68 IPaddr:0 Vrf:0 Flags:L3L2 MTU:1514 Ref:3 RX packets:716612 bytes:155442906 errors:0 TX packets:2248399 bytes:552491888 errors:0 vif0/2 OS: pkt0 Type:Agent HWaddr:00:00:5e:00:01:00 IPaddr:0 Vrf:65535 Flags:L3 MTU:1514 Ref:2 RX packets:450524 bytes:94618532 errors:0 TX packets:437968 bytes:66753290 errors:0 vif0/3 OS: tap519615d8-a2 Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0 Vrf:1 Flags:PL3L2 MTU:9160 Ref:6 RX packets:134 bytes:15697 errors:0 TX packets:8568 bytes:945944 errors:0 Table 1: vif Fields .. list-table:: :header-rows: 1 * - Release - Description * - vif Output Field - Description * - ``vif0/X`` - The vRouter assigned name, where 0 is the router ID and X is the index allocated to the interface within the vRouter. * - ``OS: pkt0`` - The ``pkt0`` (in this case) is the name of the actual OS (Linux) visible interface name. For physical interfaces, the speed and the duplex settings are also displayed. * - ``Type:xxxxx`` - ``Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0`` The type of interface and its IP address, as defined by vRouter. The values can be different from what is seen in the OS. Types defined by vRouter include: * Virtual – Interface of a virtual machine (VM). * Physical – Physical interface (NIC) in the system. * Host – An interface toward the host. * Agent – An interface used to trap packets to the vRouter agent when decisions need to be made for the forwarding path. * - ``Vrf:xxxxx`` - ``Vrf:65535 Flags:L3 MTU:1514 Ref:2`` The identifier of the vrf to which the interface is assigned, the flags set on the interface, the MTU as understood by vRouter, and a reference count of how many individual entities actually hold reference to the interface (mainly of debugging value). Flag options identify that the following are enabled for the interface: * P - ​Policy. All traffic that comes to vRouter from this interface are subjected to policy. * L3 - ​Layer 3 forwarding. * L2 - ​Layer 2 bridging. * X - Cross connect mode, only set on physical and host interfaces, indicating that packets are moved between physical and host directly, with minimal intervention by vRouter. Typically set when the agent is not alive or not in good shape. * M - Mirroring transmit direction. All packets that egresses this interface are mirrored. * Mr - Mirroring receive direction​. All packets that ingresses this interface will be mirrored. * Tc - ​Checksum offload on the transmit side. Valid only on the physical interface. * - ``Rx`` - ``RX packets:60 bytes:4873 errors:0`` Packets received by vRouter from this interface. * - ``Tx`` - ``TX packets:21 bytes:2158 errors:0`` Packets transmitted out by vRouter on this interface. vif Options ~~~~~~~~~~~ Use\ ``vif –-help`` to display all options available for the vif command. Following is a brief description of each option. .. note:: It is not recommended to use the following options unless you are very experienced with the system utilities. :: # vif --help Usage: vif [--create --mac < --mac ] [--add > --mac --vrf --type [vhost|agent|physical|virtual|monitoring] --transport [eth|pmd|virtual|socket] --xconnect --policy, --vhost-phys, --dhcp-enable] --vif --id --pmd --pci] [--delete |] [--get ][--kernel] [--set --vlan --vrf ] [--list][--core ][--rate] [--sock-dir ] [--clear][--id ][--core ] [--help} +--------------+------------------------------------------------------+ | Option | Description | +==============+======================================================+ | ``--create`` | Creates a “host” interface with name | | | ````\ and mac ````\ on the host | | | kernel. The ``vhost0`` interface that you see on | | | Linux is a typical example of invocation of this | | | command. | +--------------+------------------------------------------------------+ | ``--add`` | Adds the existing interfaces in the host OS to | | | vRouter, with type and flag options. | +--------------+------------------------------------------------------+ | ``--delete`` | Deletes the interface from vRouter. The | | | `` i`` is the vRouter interface ID as given | | | by ``vif0/X``, where ``X`` is the ID. So, in | | | ``vif0/1``, ``1`` is the interface index of that vif | | | inside the vRouter module. | +--------------+------------------------------------------------------+ | ``--get`` | Displays a specific interface. The ````\ is | | | the vRouter interface ID, unless the command is | | | appended by the ``—kernel`` option, in which case | | | the ID is the kernel ID. | +--------------+------------------------------------------------------+ | ``--set`` | Set working parameters of an interface. The ones | | | supported are the ``vlan id`` and the ``vrf``. The | | | ``vlan id`` as understood by vRouter differs from | | | what one typically expects and is relevant for | | | interfaces of service instances. | +--------------+------------------------------------------------------+ | ``--list`` | Display all of the interfaces of which the vRouter | | | is aware. | +--------------+------------------------------------------------------+ | ``--help`` | Display all options available for the current | | | command. | +--------------+------------------------------------------------------+ | ``--clear`` | Clears statistics for all interfaces on all cores. | | | For more information, see clear | | | Command | +--------------+------------------------------------------------------+ clear Command ------------- OpenSDN Release 2008 supports clearing of vif statistics counters for all interfaces by using the ``--clear`` command. Table 2: clear Command Options +----------------------------------+----------------------------------+ | Option | Description | +==================================+==================================+ | ``--clear`` | Clears statistics for all | | | interfaces on all cores. | +----------------------------------+----------------------------------+ | ``--clear --id `` | Clears statistics for a specific | | | interface. | +----------------------------------+----------------------------------+ | ``--clear --core `` | Clears statistics on a specific | | | core for all interfaces. | +----------------------------------+----------------------------------+ | ``--clear | Clears statistics for a specific | | --id --core `` | interface on a specific core. | +----------------------------------+----------------------------------+ flow Command ------------ Use the ``flow`` command to display all active flows in a system. Example: flow -l ~~~~~~~~~~~~~~~~ Use ``-l``\ to list everything in the flow table. The -1 is the only relevant debugging option. :: # flow –l Flow table Index Source:Port Destination:Port Proto(V) ------------------------------------------------------------------------------------------------- 263484 1.1.1.252:1203 1.1.1.253:0 1 (3) (Action:F, S(nh):91, Statistics:22/1848) 379480 1.1.1.253:1203 1.1.1.252:0 1 (3) (Action:F, S(nh):75, Statistics:22/1848) ​Each record in the flow table listing displays the index of the record, the source IP: source port, the destination ip: destination port, the inet protocol, and the source VRF (V) to which the flow belongs. Each new flow has to be approved by the vRouter agent. The agent does this by setting actions for each flow. There are three main actions associated with a flow table entry: Forward (‘F’), Drop (‘D’), and Nat (‘N’). For NAT, there are additional flags indicating the type of NAT to which the flow is subject, including: SNAT (S), DNAT (D), source port translation (Ps), and destination port translation (Pd). S(nh) indicates the source nexthop index used for the RPF check to validate that the traffic is from a known source. If the packet must go to an ECMP destination, E:X is also displayed, where ‘X’ indicates the destination to be used through the index within the ECMP next hop. The Statistics field indicates the Packets/Bytes that hit this flow entry. There is a Mirror Index field if the traffic is mirrored, listing the indices into the mirror table (which can be dumped by using ``mirror –-dump``). If there is an explicit association between the forward and the reverse flows, as is the case with NAT, you will see a double arrow in each of the records with either side of the arrow displaying the flow index for that direction. Example: flow -r ~~~~~~~~~~~~~~~~ Use ``-r`` to view all of the flow setup rates. :: # flow –r New = 2, Flow setup rate = 3 flows/sec, Flow rate = 3 flows/sec, for last 548 ms New = 2, Flow setup rate = 3 flows/sec, Flow rate = 3 flows/sec, for last 543 ms New = -2, Flow setup rate = -3 flows/sec, Flow rate = -3 flows/sec, for last 541 ms New = 2, Flow setup rate = 3 flows/sec, Flow rate = 3 flows/sec, for last 544 ms New = -2, Flow setup rate = -3 flows/sec, Flow rate = -3 flows/sec, for last 542 ms Example: flow --help ~~~~~~~~~~~~~~~~~~~~ Use ``--help`` to display all options available for the flow command. :: # flow –-help Usage:flow [-f flow_index][-d flow_index][-i flow_index] [--mirror=mirror table index] [-l] -f Set forward action for flow at flow_index -d Set drop action for flow at flow_index -i Invalidate flow at flow_index --mirror mirror index to mirror to -l List all flows -r Start dumping flow setup rate --help Print this help vrfstats Command ---------------- Use ``vrfstats`` to display statistics per next hop for a ``vrf``. It is typically used to determine if packets are hitting the expected next hop. Example: vrfstats --dump ~~~~~~~~~~~~~~~~~~~~~~~~ The ``—dump`` option displays the statistics for all VRFs that have seen traffic. In the following example, there was traffic only in ``Vrf 0``\ (the public VRF). ``Receives`` shows the number of packets that came in the fabric destined to this location. ``Encaps`` shows the number of packets destined to the fabric. If there is VM traffic going out on the fabric, the respective tunnel counters will increment. ​ :: # vrfstats --dump Vrf: 0 Discards 414, Resolves 3, Receives 165334 Ecmp Composites 0, L3 Mcast Composites 0, L2 Mcast Composites 0, Fabric Composites 0, Multi Proto Composites 0 Udp Tunnels 0, Udp Mpls Tunnels 0, Gre Mpls Tunnels 0 L2 Encaps 0, Encaps 130955 Example: vrfstats --get 0​ ~~~~~~~~~~~~~~~~~~~~~~~~~~ Use ``--get 0`` to retrieve statistics for a particular ``vrf``. :: # vrfstats --get 0 Vrf: 0 Discards 418, Resolves 3, Receives 166929 Ecmp Composites 0, L3 Mcast Composites 0, L2 Mcast Composites 0, Fabric Composites 0, Multi Proto Composites 0 Udp Tunnels 0, Udp Mpls Tunnels 0, Gre Mpls Tunnels 0 L2 Encaps 0, Encaps 132179 ​Example: ​vrfstats --help ~~~~~~~~~~~~~~~~~~~~~~~~~~ :: Usage: vrfstats --get --dump --help --get Displays packet statistics for the vrf --dump Displays packet statistics for all vrfs --help Displays this help message rt Command ---------- Use the rt command to display all routes in a VRF. Example: rt --dump ~~~~~~~~~~~~~~~~~~ The following example displays ``inet`` family routes for ``vrf 0``. :: # rt --dump 0 Kernel IP routing table 0/0/unicast Destination PPL Flags Label Nexthop 0.0.0.0/8 0 - 5 1.0.0.0/8 0 - 5 2.0.0.0/8 0 - 5 3.0.0.0/8 0 - 5 4.0.0.0/8 0 - 5 5.0.0.0/8 0 - 5 In this example output, the first line displays the routing table that is being dumped. In ``0/0/unicast``, the first 0 is for the router ID, the next 0 is for the VRF ID, and unicast identifies the unicast table. The vRouter maintains separate tables for unicast and multicast routes. ​ By default, if the ``—table``\ option is not specified, only the unicast table is dumped. Each record in the table output specifies the destination prefix length, the parent route prefix length from which this route has been expanded, the flags for the route, the MPLS label if the destination is a VM in another location, and the next hop ID. To understand the second field “PPL”, it is good to keep in mind that the unicast routing table is internally implemented as an ‘mtrie’. The ``Flags`` field can have two values. ``L`` indicates that the label field is valid, and ``H`` indicates that ``vroute`` should proxy arp for this IP. The ``Nexthop`` field indicates the next hop ID to which the route points. Example: rt --dump --table mcst ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To dump the multicast table, use the ``—table`` option with ``mcst`` as the argument. :: # rt --dump 0 --table mcst Kernel IP routing table 0/0/multicast (Src,Group) Nexthop 0.0.0.0,255.255.255.255 dropstats Command ----------------- Use the dropstats command to see packet drop counters in vRouter. Use the dropstats --debug command to view the Cloned Original counters. Example: dropstats ~~~~~~~~~~~~~~~~~~ :: (vrouter-agent-dpdk)[root@nodec56 /]$ dropstats Invalid IF 0 Trap No IF 0 IF TX Discard 0 IF Drop 0 IF RX Discard 0 Flow Unusable 0 Flow No Memory 0 Flow Table Full 0 Flow NAT no rflow 0 Flow Action Drop 0 Flow Action Invalid 0 Flow Invalid Protocol 0 Flow Queue Limit Exceeded 0 New Flow Drops 0 Flow Unusable (Eviction) 0 Original Packet Trapped 0 Discards 0 TTL Exceeded 0 Mcast Clone Fail 0 Invalid NH 2 Invalid Label 0 Invalid Protocol 0 Etree Leaf to Leaf 0 Bmac/ISID Mismatch 0 Rewrite Fail 0 Invalid Mcast Source 0 Packet Loop 0 Push Fails 0 Pull Fails 0 Duplicated 0 Head Alloc Fails 0 PCOW fails 0 Invalid Packets 0 Misc 0 Nowhere to go 0 Checksum errors 0 No Fmd 0 Invalid VNID 0 Fragment errors 0 Invalid Source 0 Jumbo Mcast Pkt with DF Bit 0 No L2 Route 0 Memory Failures 0 Fragment Queueing Failures 0 No Encrypt Path Failures 0 Invalid HBS received packet 0 VLAN fwd intf failed TX 0 VLAN fwd intf failed enq 0 (vrouter-agent-dpdk)[root@nodec56 /]$ dropstats --debug Cloned Original 0 .. note:: Cloned Original drops are still included in the Drops section in the output of the vif --list command. dropstats ARP Block ~~~~~~~~~~~~~~~~~~~ GARP packets from VMs are dropped by vRouter, an expected behavior. In the example output, the first counter GARP indicates how many packets were dropped. ARP requests that are not handled by vRouter are dropped, for example, requests for a system that is not a host. These drops are counted by\ ``ARP notme``\ counters. The ``Invalid ARPs`` counter is incremented when the Ethernet protocol is ARP, but the ARP operation was neither a request nor a response. dropstats Interface Block ~~~~~~~~~~~~~~~~~~~~~~~~~ ``Invalid IF`` counters are incremented normally during transient conditions, and should not be a concern. ``Trap No IF`` counters are incremented when vRouter is not able to find the interface to trap the packets to vRouter agent, and should not happen in a working system. ``IF TX Discard`` and ``IF RX Discard`` counters are incremented when vRouter is not in a state to transmit and receive packets, and typically happens when vRouter goes through a reset state or when the module is unloaded. ``IF Drop``\ counters indicate packets that are dropped in the interface layer. The increase can typically happen when interface settings are wrong. dropstats Flow Block ~~~~~~~~~~~~~~~~~~~~ When packets go through flow processing, the first packet in a flow is cached and the vRouter agent is notified so it can take actions on the packet according to the policies configured. If more packets arrive after the first packet but before the agent makes a decision on the first packet, then those new packets are dropped. The dropped packets are tracked by the Flow unusable counter. The ``Flow No Memory`` counter increments when the flow block doesn't have enough memory to perform internal operations. The ``Flow Table Full`` counter increments when the vRouter cannot install a new flow due to lack of available slots. A particular flow can only go in certain slots, and if all those slots are occupied, packets are dropped. It is possible that the flow table is not full, but the counter might increment. The ``Flow NAT no rflow`` counter tracks packets that are dropped when there is no reverse flow associated with a forward flow that had action set as NAT. For NAT, the vRouter needs both forward and reverse flows to be set properly. If they are not set, packets are dropped. The ``Flow Action Drop`` counter tracks packets that are dropped due to policies that prohibit a flow. The ``Flow Action Invalid`` counter usually does not increment in the normal course of time, and can be ignored. The ``Flow Invalid Protocol`` usually does not increment in the normal course of time, and can be ignored. The ``Flow Queue Limit Exceeded`` usually does not increment in the normal course of time, and can be ignored. dropstats Miscellaneous Operational Block ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``Discard`` counter tracks packets that hit a discard next hop. For various reasons interpreted by the agent and during some transient conditions, a route can point to a discard next hop. When packets hit that route, they are dropped. The ``TTL Exceeded`` counter increments when the MPLS time-to-live goes to zero. The ``Mcast Clone Fail`` happens when the vRouter is not able to replicate a packet for flooding. The ``Cloned Original``\ is an internal tracking counter. It is harmless and can be ignored. The ``Invalid NH``\ counter tracks the number of packets that hit a next hop that was not in a state to be used (usually in transient conditions) or a next hop that was not expected, or no next hops when there was a next hop expected. Such increments happen rarely, and should not continuously increment. The ``Invalid Label``\ counter tracks packets with an MPLS label unusable by vRouter because the value is not in the expected range. The ``Invalid Protocol``\ ​typically increments when the IP header is corrupt. The ``Rewrite Fail``\ counter tracks the number of times vRouter was not able to write next hop rewrite data to the packet. The ``Invalid Mcast Source`` tracks the multicast packets that came from an unknown or unexpected source and thus were dropped. The ``Duplicated`` counter tracks the number of duplicate packets that are created after dropping the original packets. An original packet is duplicated when generic send offload (GSO) is enabled in the vRouter or the original packet is unable to include the header information of the vRouter agent. The ``Invalid Source``\ counter tracks the number of packets that came from an invalid or unexpected source and thus were dropped. The remaining counters are of value only to developers. mpls Command ------------ The ``mpls`` utility command displays the input label map that has been programmed in the vRouter. Example: mpls --dump ~~~~~~~~~~~~~~~~~~~~ The ``—dump`` command dumps the complete label map. The output is divided into two columns. The first field is the label and the second is the next hop corresponding to the label. When an MPLS packet with the specified label arrives in the vRouter, it uses the next hop corresponding to the label to forward the packet. :: # mpls –dump MPLS Input Label Map Label NextHop ---------------------- 16 9 17 11 You can inspect the operation on ``nh 9``\ as follows: :: # nh --get 9 Id:009 Type:Encap Fmly: AF_INET Flags:Valid, Policy, Rid:0 Ref_cnt:4 EncapFmly:0806 Oif:3 Len:14 Data:02 d0 60 aa 50 57 00 25 90 c3 08 69 08 00 The nh output shows that the next hop directs the packet to go out on the interface with index 3 (``Oif:3``) with the given rewrite data. To check the index of 3, use the following: :: # vif –get 3 vif0/3 OS: tapd060aa50-57 Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0 Vrf:1 Flags:PL3L2 MTU:9160 Ref:6 RX packets:1056 bytes:103471 errors:0 TX packets:1041 bytes:102372 errors:0 The\ ``-get 3`` output shows that the index of 3 corresponds to a tap interface that goes to a VM. You can also dump individual entries in the map using the ``—get`` option, as follows: :: # mpls –get 16 MPLS Input Label Map Label NextHop ----------------------- 16 9 Example: mpls -help ~~~~~~~~~~~~~~~~~~~ :: # mpls –help Usage: mpls --dump mpls --get