Enabling SCSI verbose logging and HBA verbose logging for analysis

Please note: Enabling verbose SCSI logging will cause a huge amount of data written to log files. Depending on your system design it may happen that disk space gets filled up very quick.
As soon as all information was gathered, please revert the changes and disable verbose SCSI logging again.

There are 10 different categories of which logging verbosity can be set from 0 (off) to 7 (most verbose).

 

1)  SCSI_LOG_ERROR                 0

2)  SCSI_LOG_TIMEOUT               3

3)  SCSI_LOG_SCAN                  6

4)  SCSI_LOG_MLQUEUE               9

5)  SCSI_LOG_MLCOMPLETE            12

6)  SCSI_LOG_LLQUEUE               15

7)  SCSI_LOG_LLCOMPLETE            18

8)  SCSI_LOG_HLQUEUE               21

9)  SCSI_LOG_HLCOMPLETE            24

 

Now take the selected verbosity of categories from 10) through 1) and calculate binary numbers together.

For example to raise verbosity:

Category IOCTL HLCOMPLETE HLQUEUE LLCOMPLETE LLQUEUE MLCOMPLETE MLQUEUE SCAN TIMEOUT ERROR

Decimal  3     0          0       0          0       0          0       3    6       1

Binary   011   000        000     000        000     000        000     011  110     001

Convert binary 011000000000000000000011110001 to hexadecimal by converting groups of four digits.

In this case:

1   8    0    0    0    0    F    1

 

To set logging verbosity until next reboot, use command:

 

sysctl -w dev.scsi.logging_level=0x180000F1 or echo 0x180000F1 > /proc/sys/dev/scsi/logging_level

 

OR

 

to set scsi logging which will be valid across reboots as well.

1. Insert dev.scsi.logging_level = 0x180000F1 into /etc/sysctl.conf or edit to desired log level

2. Run sysctl -p to make this change effective immediately

Instead of using hexadecimal numbers, it is also possible to enter the referring decimal numbers.
A useful setting of the logging level without being buried in logging details is ERROR=3, SCAN=3, MLQUEUE=2, MLCOMPLETE=2, which evaluates to:


 

The process to enable verbose logging on HBAs is as follows:

 

 In case of Emulex cards, the kernel uses module called lpfc.ko:

  1) Identify parameter to make verbose logging possible

      lpfc_log_verbose:Verbose logging bit-mask (int)

  2) Add this parameter to module parameters. Edit /etc/modprobe.conf.local and add

  3) Run mkinitrd, to apply this parameter during initrd phase of booting server

  4) Reboot server, to see verbose logging of this module

 

 To find which module storage the controller uses run: