Novell is now a part of Micro Focus

My Favorites

Close

Please to see your favorites.

Bandwidth rate shaping with tc stops working after upgrade to SLES 10 SP2

This document (7002555) is provided subject to the disclaimer at the end of this document.

Environment

Novell SUSE Linux Enterprise Server 10 Service Pack 2
Novell SUSE Linux Enterprise Server 10

Situation

Bandwidth rate shaping with the Linux kernel Traffic Control utility (/usr/sbin/tc) stops working after upgrading from SUSE Linux Enterprise Server (SLES) 10 and earlier versions to SLES 10 Service Pack (SP) 2.

For example a policy to limit the outgoing bandwidth for eth0 to approximately 430 KB/s (kilobytes) could be implemented like this :
# tc qdisc add dev eth0 root handle 1:0 cbq bandwidth 1000Mbit avpkt 1000 cell 8
# tc class add dev eth0 parent 1:0 classid 1:1 cbq bandwidth 1000Mbit rate 3482kbit prio 1 avpkt 1000 bounded isolated
# tc filter add dev eth0 parent 1:0 protocol ip u32 match ip src 192.168.3.4 flowid 1:1
The reason for this that TCP Segmentation Offload (TSO) has been enabled by default in the newer network cards. And while SLES 9 and earlier SLES 10 kernels did disable this feature by default (since it did not work on most hardware), this setting is now left enabled in the newer kernels.

TCP segmentation offload is used to reduce the CPU overhead of TCP/IP on fast networks. TSO breaks down large groups of data sent over a network into smaller segments that pass through all the network elements between the source and destination. This type of offload relies on the network interface controller (NIC) to segment the data and then add the TCP, IP and data link layer protocol headers to each segment. The NIC must support TSO, which is also called large segment offload (LSO).

Since TSO enables transmission of data "outside" the linux kernel to optimize the throughput without bugging the CPU, it then breaks the traffic control shaping because the kernel only sees the "tips of the icebergs" going through the NIC.

Resolution

To identify whether TSO is enabled on a NIC, use a command like
# ethtool -k eth0
This should produce output like the following :
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on

The solution to the problem is to disable TSO for the interface, which can be done by following these steps as a test :
  1. Remove any active tc settings from the interface with a command like
    # tc qdisc del root dev eth0
  2. Change the TSO setting to off for the interface. If it is eth0, the command would be
    # ethtool -K eth0 tso off
    (note the capital K)
  3. Verify that it has been disabled:
    # ethtool -k eth0
  4. Reapply the tc settings - if using the example above :
    # tc qdisc add dev eth0 root handle 1:0 cbq bandwidth 1000Mbit avpkt 1000 cell 8
    # tc class add dev eth0 parent 1:0 classid 1:1 cbq bandwidth 1000Mbit rate 3482kbit prio 1 avpkt 1000 bounded isolated
    # tc filter add dev eth0 parent 1:0 protocol ip u32 match ip src 192.168.3.4 flowid 1:1
Now verify that the bandwidth gets limited as expected - e.g. by copying a large file with scp. Note that it takes a little while before the rate settles around the rate that has been defined.
If it works, then the next step is to implement a permanent solution which disables TSO for the network card(s) in question during system boot :
  1. Open the /etc/sysconfig/network/ifcfg- file that coresponds with the card in an editor
  2. Add the following setting to the ETHTOOL_OPTIONS parameter :
    -K iface tso off 
    so that the line e.g. reads :
    ETHTOOL_OPTIONS='-K iface tso off'
  3. Save the file
  4. Repeat steps 1-3 for each card that should have TSO disabled
  5. Reboot the system to implement the new settings
  6. Verify that TSO has been disabled for each interface with 
    # ethtool -k ethX

Disclaimer

This Support Knowledgebase provides a valuable tool for NetIQ/Novell/SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:7002555
  • Creation Date:05-FEB-09
  • Modified Date:27-APR-12
    • SUSESUSE Linux Enterprise Server

Did this document solve your problem? Provide Feedback