An Oracle White Paper May 2010 BEST PRACTICES FOR NETWORK AVAILABILITY WITH ORACLE VM SERVER FOR SPARC - PDF

Please download to get full document.

View again

of 24
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Information Report
Category:

Genealogy

Published:

Views: 6 | Pages: 24

Extension: PDF | Download: 0

Share
Related documents
Description
An Oracle White Paper May 2010 BEST PRACTICES FOR NETWORK AVAILABILITY WITH ORACLE VM SERVER FOR SPARC Introduction... 1 Introducing Oracle VM Server for SPARC... 1 About This White Paper... 2 Network
Transcript
An Oracle White Paper May 2010 BEST PRACTICES FOR NETWORK AVAILABILITY WITH ORACLE VM SERVER FOR SPARC Introduction... 1 Introducing Oracle VM Server for SPARC... 1 About This White Paper... 2 Network Availability Overview... 3 Availability Through Redundancy... 3 IP Multipathing in Oracle Solaris... 5 IP Multipathing In Oracle VM Server for SPARC... 6 Configuring Network Availability Prerequisites for IPMP with Oracle VM Server for SPARC Configuring IPMP with Two I/O Domains Configure a Second I/O Domain Configuring IPMP in a Single I/O Domain Configure the I/O Domain Configure IPMP in the Guest Domain and Test Summary About The Author Acknowledgments References... 21 Introduction Virtually every server in a modern datacenter depends on network connectivity in order to access resources and provide services to clients. The best practice for enhancing network connectivity is to configure multiple physical paths to the network so that if a network adapter, cable, or upstream switch fails due to equipment problems or human error, the second path continues to carry traffic. Redundant physical connections must be complemented with software that can fail over between links in the event of an error. The Oracle Solaris operating system supports IP multipathing (IPMP). This allows a system to be configured to automatically fail over a group of IP addresses hosted on multiple interfaces while load sharing outbound traffic across all available links. The best practices for maximizing a physical server s connectivity to a physical network are well understood but how do these rules translate into the virtual world? What does it mean to have multiple network connections to a server when those connections and the server itself are virtual? This technical white paper is the second in a series that is designed to help map I/O best practices from the physical world into the virtual world supported by logical domains. Introducing Oracle VM Server for SPARC Oracle VM Server for SPARC (previously called Sun Logical Domains) is a virtualization technology supported on Oracle's servers with CoolThreads technology those powered by UltraSPARC T1, T2, and T2 Plus processors. Oracle VM Server for SPARC allows server resources to be partitioned and allocated to separate virtual machines. Resources can be allocated down to the CPU thread, cryptographic processor, memory, and PCI bus. The Oracle VM Server for SPARC architecture consists of four classes of virtual machines that run on a thin hypervisor layer (Figure 1): a single control domain manages the virtualized environment; one or more I/O domains own real I/O devices that service domains use to provide virtual I/O services to guest domains. Each domain runs in its own dedicated partition of the server hardware. A single domain can take on multiple roles: for example I/O and service domains are usually combined into a single domain that handles physical and virtual I/O. In many configurations, the control domain is combined with an I/O and service domain. In this paper, we refer to a combined I/O and service domain as simply an I/O domain. Logical domains allow individual PCI root nexus nodes (referred to as PCI buses in this paper) to be allocated to I/O domains so that each one owns a PCI bus and any devices connected to it. On servers with two PCI buses, two I/O domains can be configured to provide multiple paths from guest domains to I/O resources. If an I/O domain, its PCI bus, or its peripherals fail, or the domain needs 1 to be rebooted, a guest domain can continue to access its I/O resources through the second I/O domain given an appropriate configuration in the guest domain. Figure 1. Oracle VM Server for SPARC supports multiple guest domains, each with their own secure partition of server hardware. I/O is handled by I/O domains that own one or more PCI buses. The combination of logical domains and IPMP in the Oracle Solaris OS can be used to increase network availability between guest operating systems and the physical network. This combination of technologies can be used on servers with CoolThreads technology having either one or two PCI buses, or equivalent I/O devices such as Oracle s 10 Gigabit Ethernet Network Interface Unit (NIU) in the case of UltraSPARC T2 processor-based systems. Now, if a single I/O path fails, the guest OS still retains network connectivity. This technical white paper discusses the approaches and trade-offs for establishing data reliability through redundancy using the resources of the server itself, namely its own PCI bus(es), built-in disk controller, and disk drives. Oracle VM Server for SPARC opens up a realm of possibilities for implementing I/O availability and reliability techniques in the virtual world, and this paper is the first in a series to address this broad topic. Perhaps not surprisingly, the techniques discussed in this paper all have their benefits and drawbacks. The choice of which one to implement must be made after carefully comparing each solution s benefits to datacenter requirements. About This White Paper This white paper discusses the approaches and trade-offs for increasing network availability using multiple network paths between the physical network and logical domains. Network Availability Overview provides an overview of using multiple paths to the network to increase availability, and which solutions are available on which Oracle servers. Configuring Network Availability gives common background and steps for configuring either solution. Configuring IPMP with Two I/O Domains provides step-by-step instructions for configuring IPMP via two I/O domains. 2 Configuring IPMP in a Single I/O Domain provides step-by-step instructions for configuring IPMP using a single I/O domain. Network Availability Overview The way to improve network availability the ability of a real or virtual server to maintain network connectivity is to use redundant links that eliminate single points of failure. Following a path from the network to the server or guest OS, there are many points at which human error and hardware failure can cause a network link to fail, causing a loss of connectivity (Table 1). Switch ports can fail or be configured incorrectly so that they prevent the flow of network traffic. Cables can be plugged into the wrong ports, their conductors or connectors can break, and they can fail if their pattern of twists is disrupted by incorrect connector placement or though bending at a tight radius. On the server, network interfaces can fail, as can their connectors. Within the server, failures of PCI bus controllers and interfaces to processors are likely to be associated with catastrophic events that cause the server itself to fail. When virtualization is used, however, software is interposed between the physical NIC and the guest operating systems, adding another potential point of failure. With logical domains, an I/O domain owns the PCI bus to which a physical NIC is attached, and it acts as an intermediary between the hardware and the guest OS. Specifically, the I/O domain is configured with a virtual switch to which guest domains are connected by means of Virtual NICs. The guest domains themselves see a virtual network interface that can be managed similarly to a physical interface. Given that an I/O domain can fail, or it may need to be rebooted after OS updates are applied, it must be included in the list of possible points of failure. TABLE 1. NETWORK-RELATED POINTS OF FAILURE NETWORK COMPONENT REASONS FOR FAILURE Switch ports and switches Hardware failure, misconfiguration, firmware update Cabling Human error, connector or conductor failures NIC Hardware, software, or connector failure PCI Bus Hardware failure, would probably seriously impair other server functions (such as access to disks) I/O Domain OS upgrades requiring reboot, Oracle VM Server for SPARC configuration changes requiring reboot, OS failure Availability Through Redundancy The first step toward increasing network availability is establishing redundant paths between guest domains and the physical network. Multiple I/O domains provide redundant paths to the physical hardware so that an interruption in a single I/O domain s availability, however momentary, does not 3 stop the flow of network traffic. Multiple physical paths, from the PCI bus to the upstream switch ports, can protect against the more common failures including simply removing a network cable. The ways in which Oracle VM Server for SPARC can map multiple physical connections to multiple virtual devices within guest domains depends on the server architecture. For the purpose of network connectivity, a server can be configured with as many I/O domains as it has PCI buses or NIUs. Multiple virtual network devices also can be provided by a single I/O domain. So whether a server has one or two PCI buses, a guest domain can always be configured with two virtual network devices that give it some level of redundancy. The most desirable configuration is to have two physical network connections attached to a server through two different PCI buses or a PCI bus and an NIU, through two different I/O domains, and on to two different virtual network devices in a guest OS. On servers with a single PCI bus, where using an NIU is impractical (see below), two network interfaces can still be connected through a single I/O domain, however this does not remove the possibility of that I/O domain acting as a single point of failure. Server I/O Architecture Summary Table 2 summarizes the I/O architecture for all Oracle servers with CoolThreads technology as of the date of this paper. The set of recommended network and I/O domain configurations are enumerated below. TABLE 2. SERVER CONFIGURATIONS AFFECTING I/O DOMAIN CONFIGURATIONS WITH ORACLE VM SERVER FOR SPARC. NUMBER OF INTERFACES CONNECTED TO EACH PCI BUS SERVER PCI BUSES NIUS PCI BUS DISK CONTROLLERS BUILT-IN ETHERNET PORTS EXPANSION SLOTS Oracle s Sun SPARC Enterprise T5240 Server 2 0 A B Oracle s Sun SPARC Enterprise T5140 Server 2 0 A B Oracle s Sun SPARC Enterprise T5220 Server 1 1 A NIU x10 Gbps Ethernet Oracle s Sun SPARC Enterprise T5120 Server 1 1 A NIU x10 Gbps Ethernet Oracle s Sun SPARC Enterprise T2000 Server 2 0 A B Oracle s Sun SPARC Enterprise T1000 Server 2 0 A B Two of Oracle s servers, the Sun SPARC Enterprise T5120 and T5220 servers, have one PCI bus and one NIU. The NIU can be configured to replace one or two PCI Express card slots with one or two 10 Gigabit Ethernet interfaces. If you wish to use the NIU and 10 Gigabit Ethernet, you can use two I/O domains, one connected to one or more Gigabit Ethernet NICs on the PCI bus, and one connected to one or more 10 Gigabit NICs on the NIU. If you wish to use Gigabit Ethernet using the built-in interfaces or PCI Express-based interfaces, you can use them in any combination through a single I/O domain. On servers with two PCI buses, use any combination of built-in network interfaces and additional PCI Express expansion-slot NICs to establish at least one path through each of two PCI buses. Configure two I/O domains, one connected to each PCI bus, to provide two virtual network devices to guest domains. If expansion slots are not available for additional NICs, use a single I/O domain connected to two or more built-in interfaces. The Sun SPARC Enterprise T2000 server has two PCI buses and two built-in network interfaces connected to each bus. It is the only server in Oracle s current product line that can support two I/O domains and two physically independent networks without the use of an expansion slot. The Sun SPARC Enterprise T1000 Server has two PCI buses but one expansion slot. If this slot is not occupied, use a PCI Express NIC for one network connection, and one built-in interface for the second one. IP Multipathing in Oracle Solaris In the previous section we described how to configure physical links and I/O domains to best establish redundant paths to guest domains. This section describes how to use those links to increase availability. For more information on IP Multipathing, please consult the System Administration Guide: IP Services, part number , available at docs.sun.com. IP Multipathing in the Oracle Solaris OS configures two or more physical interfaces into an IPMP group. When an IPMP group is configured, the IPMP daemon (in.mpathd) monitors the health of each link and fails over the group s primary IP address if the interface currently supporting it fails. Thus the interfaces configured in an IPMP group all contribute to making one or more IP address continuously available to the outside world. IPMP can accommodate different speed links, so an IPMP group can be configured to use 10 Gigabit Ethernet on an NIU along with built-in and PCI Express-based Gigabit Ethernet interfaces. IPMP uses a public address on each interface that fails over if network connectivity is lost on one of them. Typical use is an active-active configuration where inbound traffic arrives over the interface supporting one of the public IP addresses. Outbound traffic is spread across the available interfaces to increase the IPMP group s throughput. This is an excellent strategy for services such as Web servers, 5 where a small incoming request results in a much larger response. An alternative approach is to load balance incoming traffic across each of the IPMP group s public addresses. Although two interfaces are most often configured into an IPMP group, a larger number can be configured for higher availability and throughput. The IPMP daemon monitors the health of the IPMP group s interfaces in two ways. It monitors the link status, and it sends periodic ICMP probes from each interface to the default router address. When configured from inside a logical domain, link-based failure detection does not operate because the virtual network device is always connected to a virtual switch. Instead, the IPMP daemon uses probebased detection to recognize a failure. IP Multipathing In Oracle VM Server for SPARC There are three ways in which IPMP can be configured with logical domains. The preferred approaches configure IPMP in the guest OS, with virtual devices supported through one or two I/O domains. IPMP With Two I/O Domains Two I/O domains can be used on servers with two PCI buses or with one PCI bus and an NIU. It can be used with a built-in interface and a NIC or XAUI card for the internal NIU in an expansion slot, or it can be used with two NICs in expansion slots. This solution is preferred on servers with two PCI buses as the use of two I/O domains allows one I/O domain to be rebooted without affecting network I/O. This configuration is illustrated in Figure 2. Each of the two I/O domains owns one PCI bus and connects a virtual switch to the physical device. Each virtual switch provides a virtual network interface to guest domains. 6 Figure 2. IPMP can be configured inside of a guest domain using virtual network devices that are supported by two I/O domains. The IPMP group is configured from within the Oracle Solaris OS instance, and probe-based failure detection is used. The guest OS sees two virtual network devices just as in the previous example: from the perspective of the guest OS, this and the single I/O domain configuration are indistinguishable. This solution has no single point of failure except for the server and the hypervisor, making it the preferred approach for high availability on servers with two PCI buses. IPMP With a Single I/O Domain A single I/O domain can be used to connect to two physical interfaces on servers with either one or two PCI buses. This is the preferred solution for servers with a single PCI bus. This solution also can be used when IPMP is implemented more for throughput than availability, as the two I/O domain solution provides more availability. This configuration is illustrated in Figure 3. Each physical device is attached to its own virtual switch in the I/O domain. Each virtual switch is connected to a virtual network device in the guest domain. The 7 IPMP group is configured from within the guest s Oracle Solaris OS instance. Probe-based failure detection is used to initiate failover between virtual network devices in the event of a failure. Figure 3. IPMP can be configured inside of a guest domain using virtual network devices that are supported by a single I/O domain. This solution eliminates the low-hanging fruit of single points of failure, namely the NIC, cable, and upstream switch ports. The I/O domain is a single point of failure, and rebooting the I/O domain will cause network connectivity to be interrupted temporarily. IPMP Inside an I/O Domain IPMP can be implemented entirely within an I/O domain, hiding the implementation from the guest domain. This solution can simplify guest domain configuration, and it can be used on servers with one or two PCI buses. There are two drawbacks to this approach: the single I/O domain is a single point of failure for the guest; and the I/O domain must route packets between the virtual switch and the IPMP group, adding TCP/IP stack overhead to each packet transmission. For these reasons, we do not recommend this configuration. 8 This configuration is illustrated for a server with two PCI buses. A single virtual network device in the guest domain connects to a virtual switch in the I/O domain. The I/O domain is configured as a router, and packets are routed from the virtual switch to the IPMP group. Figure 4 illustrates an IPMP configuration using one built-in Ethernet interface, and one interface on a separate PCI Express NIC. This solution eliminates the NIC, network cables, and upstream switch ports as single points of failure, however the I/O domain remains as a single point of failure. Figure 4. IPMP can be configured inside of an I/O domain, but this approach adds routing overhead to the virtual network, and the single I/O domain remains as a single point of failure. Solution Summary The three IPMP configurations can be supported on Oracle servers with CoolThreads technology as summarized in Table 3. The table serves as a reminder that single I/O domain solutions are viable for all servers, although the single I/O domain remains as a single point of failure. The table is also a reminder that two PCI buses or a PCI bus and an NIU are required for two I/O domain solutions. 9 Note that more than one solution can be used on the same server. For example, one guest domain can be supported with IPMP through a single I/O domain, while another guest domain can be supported with IPMP through two I/O domains. TABLE 3. SERVER CONFIGURATIONS AND HOW THEY SUPPORT THE THREE IPMP SOLUTIONS. APPLICABLE SOLUTION SERVER PCI BUSES SINGLE I/O DOMAIN TWO I/O DOMAINS INSIDE I/O DOMAIN Oracle s Sun SPARC Enterprise T5240 Server 2 Oracle s Sun SPARC Enterprise T5140 Server 2 Oracle s Sun SPARC Enterprise T5220 Server 1 (With NIU) Oracle s Sun SPARC Enterprise T5120 Server 1 (With NIU) Oracle s Sun SPARC Enterprise T2000 Server 2 Oracle s Sun SPARC Enterprise T1000 Server 2 Configuring Network Availability This section discusses the common background for configuring IPMP with two I/O domains or with a single I/O domain. The first technical white paper in this series, Best Practices for Data Reliability with Oracle VM Server for SPARC, provides background material and resources for understanding logical domains, installing the Oracle Solaris OS, patching server firmware, installing the Oracle VM Server for SPARC Manager (SUNWldm) package, and creating your first control and I/O domain. This, and the following sections, assumes that you are familiar with this material and that you have an I/O and guest domain to work with. 10 Prerequisites for IPMP with Oracle VM Server for SPARC There are several prerequisites for configuring either of the two solutions described in the following two sections. Default Route Should Be External It is certainly possible to configur
Recommended
View more...
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks
SAVE OUR EARTH

We need your sign to support Project to invent "SMART AND CONTROLLABLE REFLECTIVE BALLOONS" to cover the Sun and Save Our Earth.

More details...

Sign Now!

We are very appreciated for your Prompt Action!

x