doctor:faults
This is an old revision of the document!
Table of Contents
Faults
Initial list of faults
Faults can be gathered by enabling SNMP and installing some opensource tool to catch and poll SNMP. When using for example Zabbix one can also put agent running on host to catch any other fault. Here is some initial list of high level faults and how they can be caught. List assumes that one enables usage of SNMP and then would use tool like Zabbix. There is also Pacemaker mentioned if used. Usage of that is limited to number of nodes, so it works better only for controller nodes.
| Raw faults | How to detect | Effected virtual resource | Actions to recover | ||
|---|---|---|---|---|---|
| Where | Description | Method | Comment | ||
| chassis | Blade not present | SNMP | |||
| Chassis fan degraded | SNMP | ||||
| Chassis fan failed | SNMP | ||||
| Chassis fan not present | SNMP | ||||
| Chassis manager degraded | SNMP | ||||
| Chassis manager failed | SNMP | ||||
| Chassis manager not present | SNMP | ||||
| Chassis power degraded | SNMP | ||||
| Chassis power failed | SNMP | ||||
| Chassis power input line status error | SNMP | ||||
| Chassis power not present | SNMP | ||||
| Chassis removal | SNMP | ||||
| Network connector not present | SNMP | ||||
| disk array | Disk array error | SNMP | |||
| libvirt | State of a virtual machine has changed | SNMP | |||
| openstack | Openstack service is in failed state such as nova-compute | zabbix agent | |||
| Openstack status | zabbix agent | ||||
| Openvswitch daemon is not in active state | zabbix agent | ||||
| Openvswitch status | zabbix agent | ||||
| os | Available memory too low | zabbix agent | |||
| Free FS space is less than 10% on volume {#FSNAME} | zabbix agent | ||||
| Host information has changed | zabbix agent | ||||
| Processor load too high | zabbix agent | ||||
| System has restarted | zabbix agent | ||||
| Zabbix agent is unreachable | |||||
| pacemaker | Corosync is not in active state | SNMP | controller node, as limited? | ||
| Pacemaker is not in active state | SNMP | controller node, as limited? | |||
| Pacemaker node {#NODENAME} status has changed on {HOST.NAME} | SNMP | controller node, as limited? | |||
| Pacemaker PCS daemon is not in active state | SNMP | controller node, as limited? | |||
| Pacemaker resource {#RESOURCENAME} status has changed on {HOST.NAME} | SNMP | controller node, as limited? | |||
| server | Cold start | SNMP | |||
| Cpu condition not ok | SNMP | ||||
| Fan degraded | SNMP | ||||
| Fan failed | SNMP | ||||
| Fan not present | SNMP | ||||
| Fan redundancy lost | SNMP | ||||
| HW SNMP agent authentication failure | SNMP | ||||
| Network adapter connectivity lost | SNMP | ||||
| Memory condition not ok | SNMP | ||||
| POST error | SNMP | ||||
| Power degraded | SNMP | ||||
| Power failed | SNMP | ||||
| Power not present | SNMP | ||||
| Power redundancy lost | SNMP | ||||
| Power threshold exceeded | SNMP | ||||
| security override engaged | SNMP | ||||
| self test error | SNMP | ||||
| Server power off | SNMP | ||||
| Server power on | SNMP | ||||
| Server power on failure | SNMP | ||||
| Server reset | SNMP | ||||
| Temperature status degraded | SNMP | ||||
| Thermal condition not ok | SNMP | ||||
| Thermal confirmation | SNMP | Up again after thermal shudown | |||
| switch | Link down | SNMP | |||
| Link up | SNMP | ||||
Describing faults
Many of the faults needs to be configurable while others not. Hardware faults especially might need different triggers in different HW while some OpenStack internal fault will always be caught the same way.
doctor/faults.1419385820.txt.gz · Last modified: 2014/12/24 01:50 by Ryota Mibu
Except where otherwise noted, content on this wiki is licensed under the following license: Creative Commons Attribution 4.0 International License
