FreeNAS® uses FreeBSD's lagg(4) interface to provide link aggregation and link failover. The lagg interface allows aggregation of multiple network interfaces into a single virtual lagg interface, providing fault-tolerance and high-speed multi-link throughput. The aggregation protocols supported by lagg determine which ports are used for outgoing traffic and whether a specific port accepts incoming traffic. The link state of the lagg interface is used to validate if the port is active or not.
Aggregation works best on switches supporting LACP, which distributes traffic bi-directionally while responding to failure of individual links. FreeNAS® also supports active/passive failover between pairs of links.
The LACP, FEC and load-balance modes select the output interface using a hash that includes the Ethernet source and destination address, VLAN tag (if available), IP source and destination address, and flow label (IPv6 only). The benefit can only be observed when multiple clients are transferring files from your NAS. The flow entering into your NAS depends on the Ethernet switch load-balance algorithm.
The lagg driver supports the following aggregation protocols:
Failover: the default protocol. Sends traffic only through the active port. If the master port becomes unavailable, the next active port is used. The first interface added is the master port; any interfaces added after that are used as failover devices. By default, received traffic is only accepted when received through the active port. This constraint can be relaxed, which is useful for certain bridged network setups, by setting net.link.lagg.failover_rx_all to a non-zero value in System → Sysctls → Add Sysctl.
FEC: supports Cisco EtherChannel on older Cisco switches. This is a static setup and does not negotiate aggregation with the peer or exchange frames to monitor the link.
LACP: supports the IEEE 802.3ad Link Aggregation Control Protocol (LACP) and the Marker Protocol. LACP will negotiate a set of aggregable links with the peer into one or more link aggregated groups (LAGs). Each LAG is composed of ports of the same speed, set to full-duplex operation. The traffic will be balanced across the ports in the LAG with the greatest total speed; in most cases there will only be one LAG which contains all ports. In the event of changes in physical connectivity, link aggregation will quickly converge to a new configuration. LACP must be configured on the switch as well.
Load Balance: balances outgoing traffic across the active ports based on hashed protocol header information and accepts incoming traffic from any active port. The hash includes the Ethernet source and destination address, VLAN tag (if available), and IP source and destination addresses. This is a static setup and does not negotiate aggregation with the peer or exchange frames to monitor the link. Requires a switch which supports IEEE 802.3ad static link aggregation.
Round Robin: distributes outgoing traffic using a round-robin scheduler through all active ports and accepts incoming traffic from any active port. This mode can cause unordered packet arrival at the client. This has a side effect of limiting throughput as reordering packets can be CPU intensive on the client. Requires a switch which supports IEEE 802.3ad static link aggregation.
None: this protocol disables any traffic without disabling the lagg interface itself.
NOTE: the FreeNAS® system must be rebooted after configuring the lagg device and TCP access will be lost during reboot. Do not configure the interfaces used in the lagg device before creating the lagg device.
Considerations When Using LACP, MPIO, NFS, or ESXi
LACP bonds Ethernet connections in order to improve bandwidth. For example, four physical interfaces can be used to create one mega interface. However, it cannot increase the bandwidth for a single conversation. It is designed to increase bandwidth when multiple clients are simultaneously accessing the same system. It also assumes that quality Ethernet hardware is used and it will not make much difference when using inferior Ethernet chipsets such as a Realtek.
LACP reads the sender and receiver IP addresses and, if they are deemed to belong to the same TCP connection, always sends the packet over the same interface to ensure that TCP does not need to reorder packets. This makes LACP ideal for load balancing many simultaneous TCP connections, but does nothing for increasing the speed over one TCP connection.
MPIO operates at the iSCSI protocol level. For example, if you create four IP addresses and there are four simultaneous TCP connections, MPIO will send the data over all available links. When configuring MPIO, make sure that the IP addresses on the interfaces are configured to be on separate subnets with non-overlapping netmasks or configure static routes to do point-to-point communication. Otherwise, all packets will pass through one interface.
LACP and other forms of link aggregation generally do not work well with virtualization solutions. In a virtualized environment, consider the use of iSCSI MPIO through the creation of an iSCSI Portal. This allows an iSCSI initiator to recognize multiple links to a target, utilizing them for increased bandwidth or redundancy. This how-to contains instructions for configuring MPIO on ESXi.
NFS does not understand MPIO. Therefore, you will need one fast interface since creating an iSCSI portal will not improve bandwidth when using NFS. LACP does not work well to increase the bandwidth for point-to-point NFS (one server and one client). LACP is a good solution for link redundancy or for one server and many clients.
Creating a Link Aggregation
Before creating a link aggregation, double-check that no interfaces have been manually configured in Network → Interfaces → View Interfaces. If any configured interfaces exist, delete them as lagg creation will fail if any interfaces are manually configured.
Figure 5.4a shows the configuration options when adding a lagg interface using Network → Link Aggregations → Create Link Aggregation.
Figure 5.4a: Creating a lagg Interface
NOTE: if interfaces are installed but do not appear in the Physical NICs in the LAGG list, check that a FreeBSD driver for the interface exists here.
Select the desired aggregation protocol, highlight the interface(s) to associate with the lagg device, and click the OK button.
Once the lagg device has been created, it will be listed in Link Aggregations under an entry which indicates the type of protocol. As seen in Figure 5.4b, it will also appear in View Link Aggregations.
Figure 5.4b: Viewing Link Aggregations
Click a link aggregation entry to see the buttons to edit that lagg interface, delete the link aggregation, or edit its member interfaces.
If you click the Edit button for a lagg, you will see the configuration screen shown in Figure 5.4c. Table 5.4a describes the options in this screen.
Figure 5.4c: Editing a lagg
Table 5.4a: Configurable Options for a lagg Member
|NIC||string||read-only as automatically assigned next available numeric ID|
|Interface Name||string||by default same as NIC field, can be changed to a more descriptive value|
|DHCP||checkbox||check if the lagg device gets its IP address info from a DHCP server|
|IPv4 Address||string||mandatory if DHCP is left unchecked|
|IPv4 Netmask||drop-down menu||mandatory if DHCP is left unchecked|
|Auto configure IPv6||checkbox||check only if DHCP server available to provide IPv6 address info|
|IPv6 Prefix Length||drop-down menu||required if input IPv6 address|
|Options||string||additional ifconfig(8) options|
This screen also allows you to configure an alias for the lagg interface. If you wish to set multiple aliases, click the "Add extra alias" link for each alias that you wish to configure.
If you click the Edit Members button, click the entry for a member, then click its Edit button, you will see the configuration screen shown in Figure 5.4d. The configurable options are summarized in Table 5.4b.
After creating the lagg interface, set the IP address manually or with DHCP and save. The connection to the web interface may be lost at this point, and if so, the system must be rebooted from the console setup menu. You may also have to change your switch settings to communicate through the new lagg interface. After reboot, if the IP address was set manually, you may also have to manually enter a default gateway from the console setup menu option in order to get access into the GUI through the new lagg interface.
Figure 5.4d: Editing a Member of a lagg
Table 5.4b: Configuring a Member Interface
|LAGG Interface group||drop-down menu||select the member interface to configure|
|LAGG Priority Number||integer||order of selected interface within the lagg; configure a failover to set the master interface to 0 and the other interfaces to 1, 2, etc.|
|LAGG Physical NIC||drop-down menu||physical interface of the selected member|
|Options||string||additional parameters from ifconfig(8)|
NOTE: options can be set at either the lagg level (using the Edit button) or the individual parent interface level (using the Edit Members button). Do not set the option at both levels as each level automatically inherits its options from the other. Typically, changes are made at the lagg level (Figure 5.4c) as each interface member will inherit from the lagg. If you instead configure the interface level (Figure 5.4d), you will have to repeat the configuration for each interface within the lagg. However, some lagg options can only be set by editing the interface. For instance, the MTU of a lagg is inherited from the interface. To set an MTU on a lagg, set all the interfaces to the same MTU.
To see if the link aggregation is load balancing properly, run the following command from Shell:
More information about this command can be found at systat(1).