I’ve recently been working on a project to deploy a couple of Pure Storage Flash Array //M10‘s, and rather than using Fiber Channel we opted for the 10Gb Ethernet (admittedly for reasons of cost) and using iSCSI as the transport mechanism.
Whenever you read up on iSCSI (and NFS for that matter) there inevitably ends up being a discussion around the MTU size. MY thinking here is that if your network has sufficient bandwidth to handle the Jumbo Frames and large MTU sizes, then it should be done.
Now I’m not going to ramble on about enabling Jumbo Frames exactly, but I am going to focus on the MTU size.
What is MTU?
MTU stands for Message Transport Unit. It defines the maximum size of a network frame that you can send in a single data transmission across the network. The default MTU size is 1500. Whether that be Red Hat Enterprise Linux, , Fedora, Slackware, Ubuntu, Microsoft Windows (pick a version), Cisco IOS and Juniper’s JunOS it has in my experience always been 1500 (though that’s not to say that some specialist providers may change this default value for black box solutions.
So what is a Jumbo Frame?
The internet is pretty much unified on the idea that any packet or frame which is above the 1500 byte default, can be considered a jumbo frame. Typically you would want to enable this for specific needs such as NFS and iSCSI and the bandwidth is at least 1Gbps or better 10Gbps.
MTU sizing
A lot of what I had ready in the early days about this topic suggests that you should set the MTU to 9000 bytes, so what should you be mindful of when doing so?
Well, lets take an example, you have a requirement where you need to enable jumbo frames and you have set an MTU size of 9000 across your entire environment;
- virtual machine interfaces
- physical network interfaces
- fabric interconnects
- and core switches
So you enable an MTU of 9000 everywhere, and you then test your shiny new jumbo frame enabled network by way of a large ping;
Linux
$ ping -s 9000 -M do 192.168.1.1
Windows
> ping -l 9000 -f -t 192.168.1.1
Both of the above perform the same job. They will attempt to send an ICMP ping;
- To our chosen destination – 192.168.1.1
- With a packet size of 9000 bytes (option -l 9000 or -s 9000), remember the default is 1500 so this is definitely a Jumbo packet
- Where the request is not fragmented, thus ensuring that a packet of such a size can actually reach the intended destination without being reduced
The key to the above examples is the “-f” (Windows) and “-M do” (Linux). This will enforce the requirement that the packet can be sent from your server/workstation to its intended destination without the size of the packet being messed with aka fragmented (as that would negate the whole point of using jumbo frames).
If you do not receive a normal ping response back which states its size as being 9000 then something is not configured correctly.
The error might look like the following;
ping: local error: Message too long, mtu=1500
ping: local error: Message too long, mtu=1500
The above error is highlighting the fact that we are attempting to send a packet which is bigger than the local NIC is configured to handle. It is telling us the MTU is set at 1500 bytes. In this instance we would need to reconfigure our network card to handle the jumbo sized packets.
Now lets take a look at what happens with the ICMP ping request and it’s size. As a test I have pinged the localhost interface on my machine and I get the following;
[toby@testbox ~]$ ping -s 9000 -M do localhost
PING localhost(localhost (::1)) 9000 data bytes
9008 bytes from localhost (::1): icmp_seq=1 ttl=64 time=0.142 ms
9008 bytes from localhost (::1): icmp_seq=2 ttl=64 time=0.148 ms
9008 bytes from localhost (::1): icmp_seq=3 ttl=64 time=0.145 ms
^C
--- localhost ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2085ms
rtt min/avg/max/mdev = 0.142/0.145/0.148/0.002 ms
Firstly notice the size of each request. The initial request may have been 9000 however that doesn’t take into account the need for the header to be added to the packet, so that it can be correctly sent over your network or the Internet. Secondly notice that the packet was received without any fragmentation (note I used the “-M do” option to ensure fragmentation couldn’t take place). In this instance the loopback interface is configured with a massive MTU of 65536 bytes and so all worked swimmingly.
Note that the final packet size is actually 9008 bytes.
The packet size increased by 8 bytes due to the addition of the ICMP header mentioned above, making the total 9008 bytes.
My example above stated that the MTU had been set to 9000 on ALL devices. In this instance the packets will never get to their intended destination without being fragmented as 9008 bytes is bigger than 9000 bytes (stating the obvious I know).
The resolution
The intermediary devices (routers, bridges, switches and firewalls) will need an MTU size that is bigger than 9000 and be size sufficiently to accept the desired packet size. A standard ethernet frame (according to Cisco) would require an additional 18 bytes on top of the 9000 for the payload. And it would be wise to actually specify a bit higher. So, an MTU size of 9216 bytes would be better as it would allow enough headroom for everything to pass through nicely.
Focusing on the available options in a Windows world
And here is the real reason for this post. Microsoft with all their wisdom, provide you with a drop down box to select the required predefined MTU size for your NICs. With Windows 2012 R2 (possibly slightly earlier versions too), the nearest size you can set via the network card configuration GUI is 9014. This would result in the packet being fragmented or in the case of iSCSI it would potentially result in very poor performance. The MTU 9014 isn’t going to work if the rest of the network or the destination device are set at 9000.
The lesson here is make sure that both source and destination machines have an MTU of equal size and that anything in between must be able to support a higher MTU size than 9000. And given that Microsoft have hardcoded the GUI with a specific number of options, you will probably want to configure your environment to handle this slightly higher size.
Note. 1Gbps Ethernet only supported a maximum MTU size of 9000, so although Jumbo Frames can be enabled you would need to reduce the MTU size slightly on the source and destination servers, with everything in between set at 9000.
Featured image credit; TaylorHerring. As bike frames go, the Penny Farthing could well be considered to have a jumbo frame.
