Environment
Internet Small Computer Systems Interface - iSCSI
Novell NetWare 6.5 Support Pack 5
Situation
Many customers today are finding interest in an affordable
storage solution over the higher cost
FC (fiber channel) SAN solutions. There are many third
party vendors delivering these types of
storage solutions and many more on the horizon. iSCSI
has even began to gain support among some
SAN vendors which may have an iSCSI interface available on
their SAN's. Along with this new found
interest comes a host of questions concerns and "alphabet
soup". We hope to address all of these within
this document.
First, it's important for us to understand what iSCSI is and
the main difference there is between a SAN.
iSCSI - Internet SCSI (small computer systems interface)
provides the means of accessing external storage
arrays via Ethernet or LAN based connectivity. Legacy
systems such as SAN's are generally accessible via
fiber channel. SAN connectivity can involve multiple
fiber connection from a host (server) to a FC switch
which then has a FC connection to the SAN. Historically
faster, this is due to the 4 Gbps fiber connection
versus 1 Gbps Ethernet and the limitation of iSCSI drives
being SATA only. With the fast approach of
10 Gbps Ethernet and the expected availability of
iSCSI/SAS drives iSCSI is becoming more and more a viable
solution for businesses small to large in size.
Resolution
To address the here and now we will discuss iSCSI optimization
to provide the best possible throughput
for most customers. Keeping in mind that infrastructure,
hardware, and configuration can drastically affect
iSCSI performance as well as the iSCSI delivery
implementation, meaning the end services provided using
iSCSI.
Let us start with connectivity ( NIC's, Switches, varying
technologies) which is the cornerstone for a successful
iSCSI implementation. Contrary to some reports and
beliefs high network traffic and/or "collisions" does not
result in data corruption, it does however result in poor
throughput and performance. In a properly configured switch
environment this shouldn't be an issue. For the best
possible performance we recommend using Jumbo frames (Ethernet
enhancement that allows for up to 9k frames as opposed to 1.5k
frames) especially in an environment where high iSCSI traffic is to
be expected ex. Web servers, mail servers, backups etc.
The use of jumbo frames we drastically reduce the number
of packets traversing the iSCSI channel/link. The
reduction of packet traffic will also reduce the overhead
associated to packet transfer ex. checksum operations,
Acknowledgements and the overall operation of packet
formatting which greatly affects the CPU. So in a"nutshell" - A larger payload will obviously decrease the amount of
traffic on the wire and the amount of work
both the iSCSI initiator and target would have to
perform.
When implementing jumbo frames verify that your NIC's
have the latest drivers and BIOS available.
Switches should have the latest firmware available as well as
any IOS specific to the switch. And of course,
jumbo frames can only be implemented on devices (NIC's and
Switches) that support jumbo frames. On
NetWare we recommend setting the following parameters -
Set
maximum physical receive packet size = 18000
This
parameter is specific to the frame size and in the case
aboveit is double
the size of the expected frame which is 9k. Had the jumbo
size specified by the NIC been 4500 then the above setting would be
at least 9000.
We also recommend using a higher Minimum and Maximum number of
packet receive buffers. These values can be set at your own
discretion.
At the onset of iSCSI development there was initially an issue
where during data transfer the NetWare
initiator could only achieve an 11K maximum window size.
This greatly affected the overall performance of
iSCSI and caused high demand Apps and data transfer to
become exceedingly slow. This drawback has since
been addressed and corrected in the latest Win-sock code
available as well as implementing the set parameter listed
below which will increase the window size during the
initial connection establishment -
Set
TCP Maximum Initial Receive Window = 64
This will
cause the initial receive window that is advertised by TCP to be
64K. In most cases the window
starts out at
a smaller size and grows to 65k if possible.
***IMPORTANT
***
Failure to
set this parameter may result in a "stuck" window size of"0". This is because the default setting
is
for 6k and a window size greater than 6k is attempted before
the connection has been established. In this
state the
window can become frozen/stuck. This will be corrected in a
post SP6 TCPIP.
Another important factor to consider are the load line
parameters associated with the NIC. Many provide offloading (
Checksum, segmentation, etc.) to optimize throughput by moving
these processes from the machines CPU to a processor directly on
the NIC. Take extreme care and test before implementing some
of these load line parameters in a production environment.
Many have been found to cause some problems and anomalies within a
NetWare environment. We recommend setting flow control and
duplex manually on each device or port, we do not recommend using"auto". We have not done any testing with TOE (TCP Offloading
Engine) which is touted to be an exceeding improvement for higher
throughput. TOE accomplishes this by offloading many CPU
intensive processes that are TCP specific such as Checksum and
Sequence number calculations, handling Ack's, Windowing (sliding
window/data receive size capabilities), and even connection
establishment and termination not excluding encapsulation. We
believe that TOE along with 10Gbps Ethernet (once both
technologies mature and become more available) users can see a
tremendous improvement in throughput which could eventually
exceed that of fiber channel .
NetWare 6.5 specific recommendation irrespective to iSCSI
target being used -
1. Net Ware server should be running support pack 5 or later
(this is due to TCP updates and optimization that directly affects
iSCSI performance.
2. Server should be running the latest available version of
Win-sock which at the time of this writing SP6 is due for release
and that is the recommended version. Since Win-sock provides
the interface between iSCSI and TCP it is imperative that this
piece is updated. Many enhancements and fixes have been
provided to the later versions of Win-sock and many of these are
directly specific to iSCSI.
3. Server should have sufficient memory available and
memory should be optimally tuned as
well.
See TID 3920657
4. It is not recommended to run mission critical apps across
iSCSI due to the delays that can very well
occur during a high utilization period.
5. It is not recommended to attempt backups during production
times or periods of high user activity.
Continuing work is being done to improve the overall
performance and reliability of iSCSI.
Some of these include improving TCP/Win-sock/iSCSI interaction
with specifics to windowing
initial connection negotiation and packet ACK'ing. At
this time only fail-over is supported for
multiple NIC usage but due to customer request we are
considering adding support for MPIO
(multi-path I/O) to create a larger pipe, 2 NIC's running 1Gig
Ethernet in a round robin config to
theoretically achieve a 2 Gig pipe which would provide to
distinctive paths to a volume. This would
improve overall performance and throughput greatly.
Also, there has been much interest in SAN boot
capability which could be achieved through HBA support on
iSCSI. This is also a considered future
enhancement .