Novell Cluster Services on OES Linux Trace and Debug Commands

  • 7005837
  • 26-Apr-2010
  • 18-Nov-2019

Environment

Novell Cluster Services (NCS)
Novell Open Enterprise Server 2 (OES 2) Linux
Novell Open Enterprise Server 11 (OES 11) Linux
Open Enterprise Server 2015 (OES 2015) Linux
Open Enterprise Server 2018 (OES 2018) Linux

Situation

How to trace, debug, troubleshoot Cluster Services on OES Linux

Resolution

  • Edit /opt/novell/ncs/bin/ldncs and uncomment the appropriate lines on each node:
# echo -n "TRACE ON"> /proc/ncs/vll
# echo -n "TRACE SBD ON"> /proc/ncs/vll
# echo -n "TRACE GIPC ON"> /proc/ncs/vll
# echo -n "TRACE MCAST ON"> /proc/ncs/vll
# echo -n "TRACE CVB ON"> /proc/ncs/cluster
# echo -n "TRACE CSS ON"> /proc/ncs/cluster
# echo -n "TRACE CRM ON"> /proc/ncs/cluster
# echo -n "TRACE CMA ON"> /proc/ncs/cluster

# echo -n "debug"> /admin/adminfs.cmd
The node will then need to be restarted with
  • OES 2015 and earlier
    • /opt/novell/ncs/bin/uldncs and /opt/novell/ncs/bin/ldncs
  • OES 2018 and later
    • systemctl restart novell-ncs.service
Alternatively, trace commands can be enabled or disabled at the command line; e.g.
echo -n "TRACE CRM ON"> /proc/ncs/cluster
  • Output is to /var/log/messages
    • Also through journalctl on OES 2018 and later
  • Note that the full Event Log as seen in iManager -> Clusters -> Cluster Event Log is stored in /admin/Novell/Cluster/EventLog.xml on the Master Node
  • Each resource's load/unload/monitor script has a corresponding load.out/unload.out/monitor.out log in /var/opt/novell/log/ncs on the node where the script is being executed

Additional Information

NOTE:  This section is not intended to be a tutorial on the inner workings of Novell Cluster Services.  The information provided here is supplementary and is to help experienced support personnel target specific areas of NCS when troubleshooting.

Many of the trace commands, above, correspond to an NCS component as follows:



  • CLSTRLIB - Cluster Configuration Library
    • Interfaces with eDirectory
    • Stores own local copy of eDirectory Cluster Data
      • Synchronised via ncs-configd
      • /var/opt/novell/ncs/
    • Sends (local) eDirectory Cluster Data to other nodes
  • GIPC - Group Interprocess Communication
    • Runs Heartbeat Protocol
      • Via Network
      • Via SAN (SBD)

  • SBD - Split Brain Detector
    • Detects and notifies Split Brain conditions
    • Issues Poison Pills

  • VLL - Virtual Interface Architecture Link Layer
    • Interface Layer:  GIPC, SBD and CRM

  • VIPX - Virtual Interface Provider Library Extensions
    • Provider Library for Virtual Interface (VI) Architecture

  • CSS  - Cluster System Services
    • Generic API for cluster-aware applications
      • Distributed shared memory
      • Distributed locking

  • CRM - Cluster Resource Manager
    • Keeps track of where resources are running
    • Runs scripts (via ncs-resourced)
    • Executes failover/failback policies

  • CVB - Cluster Volume Broker
    • Manages storage events
    • Multiple Server Activation Prevention (MSAP)
    • Ensures consistent image of storage

  • CMA - Cluster Management Agent
    • Interface to Command Line Interface and iManager via /admin