doctor:faults
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
doctor:faults [2015/02/02 12:46] Gerald Kunzmann [Alternative list of faults] |
doctor:faults [2015/02/17 15:24] (current) Gerald Kunzmann [Initial list of faults] |
||
|---|---|---|---|
| Line 3: | Line 3: | ||
| ===== Initial list of faults ===== | ===== Initial list of faults ===== | ||
| - | Faults can be gathered by enabling SNMP and installing some opensource tool to catch and poll SNMP. When using for example Zabbix one can also put agent running on host to catch any other fault. Here is some initial list of high level faults and how they can be caught. List assumes that one enables usage of SNMP and then would use tool like Zabbix. There is also Pacemaker mentioned if used. Usage of that is limited to number of nodes, so it works better only for controller nodes. | + | Faults in the listed elements need to be immediately notified to the VNFM in order to perform an immediate action like live migration or switch to a hot standby entity. In addition, a maintenance action should be triggered to, e.g., reboot the server or replace a defect hardware element. |
| + | |||
| + | Faults can be of different severity, i.e. critical, warning, maintenance, or info. Critical faults require immediate action as a severe degradation of the system has happened or is expected. Warnings indicate that the system performance is going down: related actions include closer (e.g. more frequent) monitoring of that part of the system or preparation for a cold migration to a backup VM. Type maintenance may trigger maintenance actions like a re-boot of the server or replacement of a faulty, but redundant HW. Info messages do not require any action. | ||
| + | |||
| + | Faults can be gathered by, e.g., enabling SNMP and installing some open source tools to catch and poll SNMP. When using for example Zabbix one can also put an agent running on the hosts to catch any other fault. Table 1 provides a list of high level faults that are considered within the scope of the Doctor project requiring immediate action by the VNFM. | ||
| + | |||
| + | |||
| + | === High level list of faults: === | ||
| + | |||
| + | ^ Service ^ Fault ^ Severity ^ How to detect? ^ Comment ^ Action to recover ^ | ||
| + | | **Compute Hardware** | Processor/CPU failure, CPU condition not ok | Critical | Zabbix | | Switch to hot standby | | ||
| + | | ::: | Memory failure / Memory condition not ok | Critical | Zabbix (IPMI) | | Switch to hot standby | | ||
| + | | ::: | Network card failure, e.g. Network adapter connectivity lost | Critical | Zabbix / Ceilometer | | Switch to hot standby | | ||
| + | | ::: | Disk crash | Info | RAID monitoring | Network storage is very redundant (e.g. RAID system) and can guarantee high availability. | Inform OAM | | ||
| + | | ::: | Disk aging | Info | S.M.A.R.T (IPMI or OS) | | Inform OAM | | ||
| + | | ::: | Storage controller | Critical | Zabbix (IMPI) | | Live migration if storage is still accessible; otherwise Hot Standby | | ||
| + | | ::: | PDU/power failure, power off, server reset | Critical | Zabbix / Ceilometer | | Switch to hot standby | | ||
| + | | ::: | Power degradation, Power redundancy lost, Power threshold exceeded | Warning | SNMP | | Live migration | | ||
| + | | ::: | Chassis problem (e.g. fan degraded/failed, chassis power degraded), CPU fan problem, Temperature/thermal condition not okay | Warning | SNMP | | Live migration | | ||
| + | | ::: | Mainboard failure | Critical | Zabbix (IPMI) | | Switch to hot standby | | ||
| + | | ::: | OS crash (e.g. kernel panic) | Critical | Zabbix | | Switch to hot standby | | ||
| + | | **Hypervisor** | System has restarted | Critical | Zabbix | | Switch to hot standby | | ||
| + | | ::: | Hypervisor failure | Warning / Critical | Zabbix / Ceilometer | | Migration / switch to hot standby | | ||
| + | | ::: | Zabbix / Ceilometer is unreachable | Warning | ? | | Live migration | | ||
| + | | **Networking** | SDN/OpenFlow Switch/Controller degraded/failed | Critical | ? | | Switch to hot standby or reconfigure virtual network topology | | ||
| + | | ::: | HW failure of physical switch/router | Warning | SNMP | Redundancy of physical infrastructure is reduced or no longer available. | Inform OAM to replace defect HW and configure new HW | | ||
| + | |||
| - | Faults can be of different **severity**, i.e. critical, warning, maintanance, or info. //Critical// faults require immediate action as a severe degradation of the system has happened or is expected. //Warnings// indicate that the system performance is going down: related actions include closer (e.g. more frequent) monitoring of that part of the system or preparation for a cold migration to a backup VM. Type //maintenance// may trigger maintenance actions like a re-boot of the server or replacement of a faulty, but redundant HW. //Info// messages do not require any action. | ||
| - | ^ Service ^ Fault ^ Severity ^ How to detect? ^ Comment ^ Action to recover ^ | ||
| - | | **Compute Hardware** | Processor/CPU failure, CPU condition not ok | Critical | Zabbix | | switch to hot standby | | ||
| - | | ::: | Processor load too high | Warning | Zabbix / Ceilometer | | scale virtual machine | | ||
| - | | ::: | Memory failure / Memory condition not ok | Critical | Zabbix (IPMI) | | switch to hot standby | | ||
| - | | ::: | Available memory too low / overload | Warning | Zabbix / Ceilometer | | scale virtual machine | | ||
| - | | ::: | Network card failure, e.g. Network adapter connectivity lost | Critical | Zabbix / Ceilometer | | switch to hot standby | | ||
| - | | **Storage (SAN)** | Disk crash | Info | RAID monitoring | Network storage is very redundant (e.g. RAID system) and can guarantee high availability. | Inform OAM | | ||
| - | | ::: | Disk aging | Info | S.M.A.R.T (IPMI or OS) | | Inform OAM | | ||
| - | | ::: | Disk full, Free FS space is less than 10% on volume {#FSNAME} | Warning | Zabbix / Ceilometer | | Scale storage | | ||
| - | | ::: | Storage controller | Critical | Zabbix (IMPI) | | Live migration if storage is still accessible; otherwise Hot Standby | | ||
| - | | **Server** | PDU/power failure, power off, server reset | Critical | Zabbix / Ceilometer | | switch to hot standby | | ||
| - | | ::: | Power degradation, Power redundancy lost, Power threshold exceeded | Warning | | | live migration | | ||
| - | | ::: | chassis problem (e.g. fan degraded/failed, chassis power degraded), CPU fan problem, Temperature/thermal condition not okay | Warning | SNMP / Zabbix / Ceilometer | | live migration; trigger maintenance action for PM | | ||
| - | | ::: | Mainboard failure | Critical | Zabbix (IPMI) | | switch to hot standby | | ||
| - | | ::: | OS crash (e.g. kernel panic) | Critical | Zabbix | | switch to hot standby | | ||
| - | | **Hypervisor** | //t.b.d.// | | | | | | ||
| - | | ::: | | | | | | | ||
| - | | **Networking** | //t.b.d.// | | | | | | ||
| - | | ::: | | | | | | | ||
doctor/faults.1422881160.txt.gz · Last modified: 2015/02/02 12:46 by Gerald Kunzmann
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: Creative Commons Attribution 4.0 International License
