DeepSea deployment results in errors for the iSCSI Gateway

This document (7018668) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 4

Situation

When deploying SUSE Enterprise Storage using Salt/DeepSea and having an iSCSI Gateway configured, the following errors are seen when running stage 4:

:~ # salt-run state.orch ceph.stage.4

[WARNING ] Could not write out jid file for job 20170221120717480962. Retrying.

[WARNING ] Could not write out jid file for job 20170221120717480962. Retrying.

[WARNING ] Could not write out jid file for job 20170221120717480962. Retrying.

[WARNING ] Could not write out jid file for job 20170221120717480962. Retrying.

[WARNING ] Could not write out jid file for job 20170221120717480962. Retrying.

[ERROR   ] prep_jid could not store a jid after 5 tries.

[ERROR   ] Could not store job cache info. Job details for this run may be unavailable.

[ERROR   ] Run failed on minions: server.lab.suse.de

Failures:

    server.lab.suse.de:

      Name: lrbd - Function: pkg.installed - Result: Clean Started: - 12:07:27.126075 Duration: 484.869 ms

    ----------

              ID: enable lrbd

        Function: service.running

            Name: lrbd

          Result: False

         Comment: Service lrbd has been enabled, and is dead

         Started: 12:07:27.611338

        Duration: 894.987 ms

         Changes:

                  ----------

                  lrbd:

                      True

      Name: service.restart - Function: module.run - Result: Changed Started: - 12:07:28.506752 Duration: 1359.962 ms

    Summary for server.lab.suse.de

    ------------

    Succeeded: 2 (changed=2)

    Failed:    1

    ------------

    Total states run:     3

    Total run time:   2.740 s


No minions matched the target. No command was sent, no jid was assigned.

adminserver.lab.suse.de_master:

  Name: auth - Function: salt.state - Result: Changed Started: - 12:07:18.813741 Duration: 1248.03 ms

  Name: keyring - Function: salt.state - Result: Changed Started: - 12:07:20.061930 Duration: 255.429 ms

  Name: sysconfig - Function: salt.state - Result: Changed Started: - 12:07:20.317510 Duration: 358.052 ms

  Name: iscsi import - Function: salt.state - Result: Changed Started: - 12:07:20.675734 Duration: 6089.31 ms

----------

          ID: iscsi apply

    Function: salt.state

      Result: False

     Comment: Run failed on minions: server.lab.suse.de

              Failures:

                  server.lab.suse.de:

                    Name: lrbd - Function: pkg.installed - Result: Clean Started: - 12:07:27.126075 Duration: 484.869 ms

                  ----------

                            ID: enable lrbd

                      Function: service.running

                          Name: lrbd

                        Result: False

                       Comment: Service lrbd has been enabled, and is dead

                       Started: 12:07:27.611338

                      Duration: 894.987 ms

                       Changes:

                                ----------

                                lrbd:

                                    True

                    Name: service.restart - Function: module.run - Result: Changed Started: - 12:07:28.506752 Duration: 1359.962 ms

                  Summary for server.lab.suse.de

                  ------------

                  Succeeded: 2 (changed=2)

                  Failed:    1

                  ------------

                  Total states run:     3

                  Total run time:   2.740 s

     Started: 12:07:26.765193

    Duration: 3112.397 ms

     Changes:

  Name: cephfs pools - Function: salt.state - Result: Changed Started: - 12:07:29.877742 Duration: 3778.575 ms

  Name: mds auth - Function: salt.state - Result: Changed Started: - 12:07:33.656472 Duration: 1072.477 ms

  Name: mds - Function: salt.state - Result: Changed Started: - 12:07:34.729093 Duration: 525.992 ms

  Name: rgw auth - Function: salt.state - Result: Clean Started: - 12:07:35.255234 Duration: 821.704 ms

  Name: rgw - Function: salt.state - Result: Clean Started: - 12:07:36.077085 Duration: 104.533 ms

  Name: openattic auth - Function: salt.state - Result: Changed Started: - 12:07:36.181747 Duration: 502.064 ms

  Name: openattic - Function: salt.state - Result: Changed Started: - 12:07:36.683972 Duration: 37952.848 ms

  Name: openattic keyring - Function: salt.state - Result: Changed Started: - 12:08:14.636991 Duration: 517.841 ms

  Name: openattic oaconfig - Function: salt.state - Result: Changed Started: - 12:08:15.155038 Duration: 13705.437 ms


Summary for adminserver.lab.suse.de_master

-------------

Succeeded: 13 (changed=11)

Failed:     1

-------------

Total states run:     14

Total run time:   70.045 s

Resolution

The errors are cosmetic and can be ignored.

Cause

Additional Information

The errors are more often encountered when running the Admin node as a virtual machine. To verify the configuration was successful run only the iSCSI specific subset (that forms part of Stage 4) again:

salt-run state.orch ceph.stage.iscsi

This should return something like the following:

:~ # salt-run state.orch ceph.stage.iscsi

adminserver.lab.suse.de_master:

  Name: auth - Function: salt.state - Result: Changed Started: - 17:26:33.706143 Duration: 1090.991 ms

  Name: keyring - Function: salt.state - Result: Clean Started: - 17:26:34.797278 Duration: 356.39 ms

  Name: sysconfig - Function: salt.state - Result: Changed Started: - 17:26:35.153830 Duration: 252.811 ms

  Name: iscsi import - Function: salt.state - Result: Changed Started: - 17:26:35.406796 Duration: 5123.672 ms

  Name: iscsi apply - Function: salt.state - Result: Changed Started: - 17:26:40.530622 Duration: 6884.202 ms


Summary for adminserver.hwlab.suse.de_master

------------

Succeeded: 5 (changed=4)

Failed:    0

------------

Total states run:     5

Total run time:  13.708 s

Alternatively to manually verify the installation take the following steps on the node designated to run the iSCSI Gateway:

1. Verify the lrbd service is running:

:~ # systemctl status lrbd.service

● lrbd.service - configures target.service from Ceph

   Loaded: loaded (/usr/lib/systemd/system/lrbd.service; enabled; vendor preset: disabled)

   Active: active (exited) since Tue 2017-02-21 12:07:29 CET; 2h 55min ago

  Process: 32216 ExecStart=/usr/sbin/lrbd $LRBD_OPTIONS (code=exited, status=0/SUCCESS)

 Main PID: 32216 (code=exited, status=0/SUCCESS)

    Tasks: 0 (limit: 512)

   CGroup: /system.slice/lrbd.service

2. Verify the configured RBD images were mapped, by default and with the below example an image demo is created:

:~ # rbd showmapped

id pool image snap device

0  rbd  demo  -    /dev/rbd0

3. Verify the iSCSI Gateway is listening (the second option uses the default target port):

:~ # netstat -plt | grep iscsi

tcp        0      0 server.:iscsi-target *:*                     LISTEN      -

:~ # netstat -plnt | grep 3260

tcp        0      0 192.100.100.10:3260      0.0.0.0:*               LISTEN      -

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.