hp小型機 mc集群介紹

1、,高可用性系統介紹(MC/ServiceGuard),HP 小型機培訓,,HA (High Availability)定義,A system is highly available if a single component or resource failure interrupts the system for only a brief time,What cause a system to go down,planned reas

2、ons: reconfigure the kernal apply patchs perform hardware and software upgrades perform full system backups perform system maintenance,unplanned reasons: hardware failures: CPU, Memory, Di

3、sk drives , LAN Card, Cable,Disk Controller cards etc system panics application errors power failures user errors,% of Failures,Hardware,,High Availability Terms,Downtime: any amount of ti

4、me when the application is unavailable (planned or unplanned) planned: customer plans to bring down the system unplanned: due to an unplanned event or outage,High available

5、: A system that can be recover quickly from all or most resource failures. The application may become unavailable, but only for a short period of time. DownTime: 5 min 50 min

6、 8.8 hours 12 hours 24 hours 3.6 days 7.2 days 10.8 days Availability: 99.999% 99.99% 99.9% 99.86% 99.73% 99.0% 98% 97%,Outage: an occurrence that renders

7、 an application unavailable when it is expected to be available (Hardware,software,user,environmental Problem),Availability: The time that application is can be used during times when when it is

8、 expected to be useble. Availability igored planned or scheduled downtime and is expressed as a igored planned or scheduled downtime and is expressed as a percentag

9、e,Fault tolerant: These system protect against hardware failures by providing totally redundant hardware in a single system,Standard reliability: A system that relies only on basic har

10、dware;there are no additional precautions taken to protect against an outage. (97-98%),,SPOF(Single Points of Failure),,,,,SPOF Solution,CPU

11、Memory Cluster,Disk Mirror and RAID,Interface Cards Mirror and PV Links,LAN, NICs

12、 Redundant LANs and LANIC,Power UPS,SPU,LAN,Power,CPU,Memory,NIC,Disk,SCSI Controller,,,root,root mirror,,High Availability Solution,,Continuously Available Systems

13、 future HP products,Highly Available System MC/ServiceGuard MC/LockManager

14、 OnLine JFS Process Resource Manager ClusterView,Protected Data

15、 MirrorDisk/UX HP DiskArray/EMC DiskArray JFS,Reliable system

16、 HP9000 systems HP peripherals HP-UX,Cluster(群集),c

17、luster is a networked group of nodes (hosts) which monitor each other in order to ensure that interruptions to the availability of application running on these nodes are kept small .,,,Pkg A,,,Pkg B,,,,,,,,,,,,,,,,,,,r

18、oot,root,PrimaryLAN Card,PrimaryLAN Card,Standby LAN Cards,Pkg A Disks,Pkg B Disks,,,,,Dedicated Heatbeat LAN,Primary Lan :Heatbeat/Data,,,Standby LAN :Heatbeat/Data,,Standby LAN :Heatbeat/Data,,Node 1,Node 2,,,,cli

19、ent,,,Pkg A,,,,Pkg B,,,,,,,,,,,,,,,,,,,root,root,PrimaryLAN Card,PrimaryLAN Card,Standby LAN Cards,Pkg A Disks,Pkg B Disks,,,,,Dedicated Heatbeat LAN,Primary Lan :Heatbeat/Data,,,Standby LAN :Heatbeat/Data,,Standby

20、 LAN :Heatbeat/Data,,Node 1,Node 2,,,Sample cluster (two-nodes),,cmcld,Package概念,Package: an application along with its programs and resources (volume group, target node, Network address, control Script

21、and services) Floating IP: application IP address(attach to host NIC). Client connect to host through the floating IP Original node: adoptive node : a pac

22、kage can have several adoptive nodes,LVM,PV links: dual links(hardware paths) to the same disk such that if one link fails, LVM automaticlly rerouteds the I/O to an alternate path MC/SG VG: if a VG

23、 is a part of an MC/SG, only one node will be allowed to access the VG at a time Exclusive Mode Activation: in general, you must provide at least one volume group for each package,Sample cluster (8 nodes cluster)

24、 (Max 16 nodes),,,,,,,,,,,,,,,,,,,,,,,,,,,,,,WAN,,,client,DiskArray,standby,EMC symmetrix,HP XP256,Cluster reformation,,System B,,Pkg 3,System C,Pkg 4,,,clusterReformation,,,System A leave,S

25、ystem A join,,clusterReformation,Lock Disk概念,The cluster lock is a disk located in a volume group shared by all nodes in the cluster,required for 2-nodes clusteroptional for 3 or 4 nodes clusternot supported for 5 nod

26、e or more cluster,,,Pkg A,,,,Pkg B,,,,,,,,,,,root,root,PrimaryLAN Card,PrimaryLAN Card,Standby LAN Cards,Pkg A Disks,Pkg B Disks,,,,,Dedicated Heatbeat LAN,Node 1,Node 2,,,,Lock Disk,X,,,,,,,model 10, mode20, model3

27、0,FC60等DiskArray 需要單獨另配一塊鎖盤AutoRaid12H:其中的一個物理卷可用作鎖盤 不需單獨另配一塊鎖盤,,MC處理的失效類型,Node(host) failover : SPU (CPU, Memory, disk I/O, Power)LA

28、N failover: LAN Card, LAN link,,Pkg A(float IP_A),,,,Pkg B(float IP_B),,,,,,,,,,,,,,,,,,,root,root,PrimaryLAN Card,PrimaryLAN Card,Standby LAN Cards,Pkg A Disks,Pkg B Disks,,,,,Dedicated Heatbeat

29、 LAN,Primary Lan :Heatbeat/Data,,,Standby LAN :Heatbeat/Data,,Standby LAN :Heatbeat/Data,,Node 1,Node 2,Pkg A(float IP_A),X,,,,Client,Application Switch Demo(SPU Failure),Pkg A client,,,,,Pkg A,,,,Pkg B,,,,,,,,,,,,,

30、,,,,,,root,root,PrimaryLAN Card,PrimaryLAN Card,Standby LAN Cards,Pkg A Disks,Pkg B Disks,,,,,Dedicated Heatbeat LAN,Primary Lan :Heatbeat/Data,,,Standby LAN :Heatbeat/Data,,Standby LAN :Heatbeat/Data,,Node 1,Node 2

31、,Pkg A(float IP),X,,,,Client,Application Switch Demo(SPU Failure),Pkg A client,,,,Pkg A,,,,Pkg B,,,,,,,,,,,,,,,,,,,root,root,PrimaryLAN Card,PrimaryLAN Card,Standby LAN Cards,Pkg A Disks,Pkg B Disks,,,,,Dedicated H

32、eatbeat LAN,Primary Lan :Heatbeat/Data,,,Standby LAN :Heatbeat/Data,,Standby LAN :Heatbeat/Data,,Node 1,Node 2,Pkg A,,,,,,,,Application Switch Demo(LAN Failure),Client,X,,應用切換時間,activate_volume_group,,,Pkg A,,,,Pkg

33、B,,,,,,,,,,,root,root,PrimaryLAN Card,PrimaryLAN Card,Standby LAN Cards,Pkg A Disks,Pkg B Disks,,,,,Dedicated Heatbeat LAN,Node 1,Node 2,Pkg A,X,,,,umount_fs,remove_ip_address,customer_defined_halt_cmds,halt_services

34、,deactivate_volume_group,check_and_mount,add_ip_address,customer_defined_run_cmds,start_services,MC管理命令(1): Cluster startup,1. Automatic->/etc/rc.config.d/cmcluster AUTOSTART_CMCLD=1 2. Manual: cmr

35、uncl 3. Single-node: cmruncl -n hostname,MC管理命令(2): cluster view:,CLUSTER STATUS cluster1 up NODE STATUS STATE systemA

36、 up running PACKAGE STATUS STATE PKG_SWITCH NODE pkg_A up running enabled systemA pkg_B

37、 up running enabled systemB NODE STATUS STATE systemB up running,cmviewcl,MC管理命令(3): cluster stop:

38、,cmhaltcl [-f] forcely close database and applicationcmviewcl CLUSTER STATUS cluster1 down,MC管理命令(4): node stop & join,node stop: cmhaltn

39、ode [-f] -n systemBCLUSTER STATUS cluster1 upNODE STATUS STATE systemA up running

40、 PACKAGE STATUS STATE PKG_SWITCH NODE pkg_A up running enabled systemA pkg_B up running enabled

41、 systemA NODE STATUS STATE systemB down haltednode start : cmrunnode systemBCLUSTER STATUS

42、 cluster1 up NODE STATUS STATE systemA up running PACKAGE STATUS STATE PKG_SW

43、ITCH NODE pkg_A up running enabled systemA pkg_B up running enabled systemA NODE

44、 STATUS STATE systemB up running,MC管理命令(5): package stop,PACKAGE STATUS STATE PKG_SWITCH NODE pkg_A

45、 up running enabled systemA pkg_B up running enabled systemBcmhaltpkg pkg_BPACKAGE STATUS STATE PKG_SWIT

46、CH NODE pkg_A up running enabled systemA pkg_B down unowned disabled unowned,MC管理命令(6): package status chang

47、e & start,PACKAGE STATUS STATE PKG_SWITCH NODE pkg_A up running enabled systemA pkg_B down unowned

48、 disabled unownedcmrunpkg -n systemB pkg_B --------> not successfulcmrunnode systemBcmrunpkg -n systemA pkg_B PACKAGE STATUS STATE PKG_SWITCH NODE

49、 pkg_A up running enabled systemA pkg_B up running disabled systemAcmmodpkg -e pkg_B PACKAGE

50、 STATUS STATE PKG_SWITCH NODE pkg_A up running enabled systemA pkg_B up running enabled sys

51、temA,,MC測試方法,MC/ServiceGuard軟件安裝: swlist －> B3935BA B.11.00MC/ServiceGuard運行: cmruncl cmviewcl手工切換包: cmhaltpkg pkg_name cmrunpkg pkg_name手工停止節(jié)點: cmhaltnode -f [node_na

52、me]操作系統故障: shutdown -r -y 0,注意事項：電源連接,,,,,,,,,,,,,,,,,,,,,,,,,,,,,N,L,G,,,,,,,,,,,,,,,,,,UPS,N,N,專用地線,輸入端,G,L,G,電源箱,G,N,L,G,N,L,G:：地線N：零線L：火線,,,,220v,< 1.0 v,電阻小于1歐姆,,,,,,L,,,,15A,15A,15A,零線與地線不能接在一起地線要求直接接地,,,,

53、,,Standby LAN Card,注意事項：心跳線網絡連接(switch),PrimaryLAN Card,,,,,Pkg A,,,,Pkg B,,,,,,,,root,root,HeartBeat LAN Cards,Pkg A Disks,Pkg B Disks,,,,,Node 1,Node 2,,,,,,,,,,,,,,Pkg A,,,,Pkg B,,,,,,,,root,root,HeartBeat LAN

54、 Cards,Pkg A Disks,Pkg B Disks,,,,,Node 1,Node 2,,,,,,,,,,,,,,,,12345678,12345678,,,,,,,,,1---32---6,Directconnect,SPOF,,注意事項,1.應用穩(wěn)定: MC不能保護應用程序本身的缺陷、OS的bug等等。 應用在單機上穩(wěn)定運行后再配置MC系統 2.

55、數據可靠性: MC不能保證數據的可用性。 采用適合的磁盤技術保護數據。3.應用系統整體可靠性: MC只保證主機系統的高可靠性。 整個應用系統的可靠性需要考慮到各方面的單點故障SPOF 如

56、采用可靠性的網絡，中間件產品，客戶端程序等。4.主機處理能力：考慮MC系統切換后，一臺主機運行多個應用的處理能力。5. 應用設計考慮：分解應用均衡負載(active/active模式 －>避免active/standby模式) 一個應用一個卷組 (根據應用劃分磁盤陣列的空間) 客戶端程序用 flo

57、ating IP 進行連接，不要用固定的主機地址。 數據一致性：保證MC卷組對各節(jié)點同步。 (vgexport vgimport命令) 不要改變MC配置文件 ： /.rhosts /etc/hosts /etc/cmcluster/

58、cmclnodelist /etc/cmcluster/* 網絡服務,,MC系統切換后的措施,假設2節(jié)點Cluster ,主機名為host1 、host2， 主機host1出現故障：確認應用切換并且可用： 在主機host2上執(zhí)行： cmviewcl [pkg_name的狀態(tài)應為running] ps -ef

59、| grep ora ping float_IP查找故障： log文件 : /var/adm/syslog/syslog.log /etc/cmcluster/pkg??/control.sh.log修復： HP響應中心：記錄主機序列號 (010)656

60、43888 接好Modem及電話線，Key=>Service狀態(tài) 恢復應用：主機host1修復啟動后，cmrunnode host1恢復應用在原主機運行： cmhaltpkg pkg_name [此命令將中斷應用] cmmodpkg -e -n host1 -n host2 pkg_name

61、 cmrunpkg -n host1 pkg_name cmviewcl,MC/ServiceGuard與MC/LockManager的區(qū)別,ServiceGuard LockManager Multiple applications ea

62、ch running exclusively on one nodealmost any application oracle OPS(Oracle Parallel Server) ONLYRaw volumes, HFS,JFS OPS DB: raw vol

63、umes ONLY Applications reconnects to the same IP addressEach application has its own all node accesses the same OPS disk volume groupsdisk volume groupsapplic

64、ation scaling dependent upon potential increase in application scaling dependent onperformance of single SPU database partitioningapplication is not available