IBM monitoring

IBM Storage Ceph: use Ceph installation procedure

Data is obtained from IBM Storage Defender Data Protect via REST API.

• Allow access from the XorMon host to IBM Storage Defender Data Protect nodes on port 443
  
  
• Create a new user with the Viewer role on the storage
  • Login to IBM Storage Defender Data Protect UI
  • Go to Settings -> Access management
  • Add Local User and assign “Viewer” (example uses 'stor2rrd' user but use 'xormon' preferable)
    
• Add storage device to configuration in the UI:
  XorMon UI ➡ Settings icon ➡ Device ➡ Storage ➡ New ➡ Vendor:device ➡ IBM : Storage Defender
  
  
  Use Cluster loadbalancing FQDN in the device configuration
  
  
• Run "Test" for configured storage device, it must show "OK"
  
  
• Wait about 1 hour, then reload the web browser, you should see it in XorMon UI
  
  

Install the DS8000 storage API (DSCLI)

• Under root user
• Download it from here: ftp://ftp.software.ibm.com/storage/ds8000/updates/DS8K_Customer_Download_Files/CLI/
• Follow IBM doc for installation: DSCLI installation
• Installation on Linux

  mount DSCLI-7.9.41.59.iso /mnt -o loop,ro
  umask 022     
  /mnt/IMAGES/HMC/Disk1/InstData/Linux/NoVM/dsclisetup.bin -i console
    Preparing to install...
    Extracting the installation resources from the installer archive...
    Configuring the installer for this system's environment...
    Congratulations. IBM System Storage DS Command Line Interface has been
    .......
    successfully installed to: /opt/ibm/dscli
  

If you get en error "Error: Could not find or load main class com.zerog.lax.LAX" during installation start, then set up this env variable and start it again:

export JAVA_TOOL_OPTIONS="-Djdk.util.zip.disableZip64ExtraFieldValidation=true"

Create a user on the storage with restricted rights

• If you prefere to non encrypted communication with the storage then edit /opt/ibm/dscli/profile/dscli.profile file adding the string "port:1750".
  This way the DSCLI will be able to use the port on a non encrypted protocol.
  
  
• Allow access from the XorMon host to the storage on ports 1750 and 1751 (encrypted communication).
  
• Under xormon Unix user identification do following
  
  
• Make sure you have execution rights on : /opt/ibm/dscli/dscli
  
  
• Log as administrator into DSCLI
  (you might create xormon user on the storage even from the other admin api like TPC ..)
  If you get following error then it is probably a firewall issue:

  $ /opt/ibm/dscli/dscli  -hmc1 192.168.1.1 -user admin
  CMUN00018E  Unable to connect to the management console server
  

• Log as storage administrator and create user xormon under monitor role:

  $ /opt/ibm/dscli/dscli  -hmc1 192.168.1.1 -user admin
  dscli> mkuser -pw PASSWORD -group monitor xormon
  

• Store storage xormon password in DSCLI (/home/xormon/dscli/security.dat):

  $ /opt/ibm/dscli/dscli managepwfile -action add -mc1 192.168.1.1 -name xormon -pw PASSWORD
  

• Connection check, it should not ask for a password:

  $ /opt/ibm/dscli/dscli  -hmc1 <HMC1 ip addr> -user xormon ver -l
    Date/Time: December 10, 2013 3:01:43 PM CET IBM DSCLI Version: 7.7.10.317
    DS: -
    DSCLI          7.7.10.317
    StorageManager 7.7.5.0.20121203.1
    ================Version=================
    Storage Image    LMC
    ===========================
    IBM.2107-75CXXXX 7.7.10.294
  

XorMon storage configuration

• Add storage into configuration from the UI:
  Settings icon ➡ Storage ➡ New ➡ Vendor:device ➡ IBM DS8000
  
  
• You can get DEVID from DSCLI:

  $ /opt/ibm/dscli/dscli  -hmc1  192.168.1.1 -user xormon
    Date/Time: April 18, 2013 2:55:36 PM CEST IBM DSCLI Version: 7.6.31.165 DS: -
    dscli> lssi -s
     ID
     ================
     IBM.2107-75VG681
  

  When you get this error:

  dscli> lssi -s
    Date/Time: April 18, 2013 3:04:25 PM CEST IBM DSCLI Version: 7.6.31.165 DS: -
    CMUC00201E lssi:: You are logged in with an expired password and limited 
      authorization. Use the chuser command to changer password and restore normal authorization.
  

  Then change the password for xormon user

  dscli>chuser -pw PASSWORD1 xormon
  

  and refresh stored password as well

  $ /opt/ibm/dscli/dscli managepwfile  -action remove -mc1 192.168.1.1 -name xormon  
  $ /opt/ibm/dscli/dscli managepwfile  -action add -mc1 192.168.1.1 -name xormon -pw PASSWORD1
  

• Run "Test" for configured storage device, it must show "OK"
  
  
• Wait about 1 hour, then reload the web browser, you should see it in XorMon UI
  
  

XorMon collects data from Storage Scale (formerly Spectrum Scale / GPFS) and IBM Elastic Storage Server from GUI nodes using REST API.
At least one GUI node has to be installed in the cluster.
IBM Storage Scale version 5.1.0 and newer are supported.
Note there is not supported monitoring of AIX based nodes as Storage Scale does not support installation of performance monitoring package on them, read this.

Network connectivity

Open HTTPS 443 REST API port from XorMon to all Storage Scale GUI node IP addresses on the firewall.

User configuration

Perform the following steps to create a GUI user and assign user role:

• Log in to Storage Scale GUI
• Go to Services > GUI page in the IBM Storage Scale GUI
• Click Users
• Click Create User - the Create User window appears
  (example uses 'stor2rrd' user but use 'xormon' preferably)
  
  
• Type the name of the new user in the Name field
• Select Monitor as User Group
• Type the password in the Temporary password and Confirm password fields
• Click Create
  
• Login to GUI with the new user and change Temporary password to a new one
  

XorMon storage configuration

• Add storage into configuration from the UI:
  XorMon UI ➡ Settings icon ➡ Device ➡ Storage ➡ New ➡ Vendor:device ➡ IBM Storage Scale
  
  Configure only GUI nodes
  Configure GUI nodes' hostnames/IPs
  Fill in the user from previous step
  
  
• Run "Test" for configured storage device, it must show "OK"
  
  
• Wait about 1 hour, then reload the web browser, you should see it in XorMon UI
  
  

IBM Storage Virtualize (former IBM Spectrum Virtualize and IBM SVC).

IBM Spectrum Control (TPC) and XorMon coexistence

Prerequisites

• Allow access from the XorMon host to the IBM SVC / Storwize / FlashSystem storage on port 22 (ssh).
  
  

Storage access

There are 2 possibilities how to connect the storage:

• Use user/password
  Create new user "xormon" on the storage
  • Storage firmware 8.3.1.2+: use "Monitor" role

    $ ssh superuser@<storage control enclosure IP address> "svctask mkuser -name xormon -usergrp "Monitor"
    

    Using this user role requires setting statistics_frequency on attached storage to 5 minutes under superuser

    ssh superuser@<storage control enclosure IP address> "svctask startstats -interval 5"
    
    # make sure it has been done by:
    ssh superuser@<storage control enclosure IP address> "lssystem" | grep statistics
      statistics_status on
      statistics_frequency 5
    

  • Storage firmware 8+ up to 8.3.1.2: use "Restricted Administrator"

    $ ssh superuser@<storage control enclosure IP address> "svctask mkuser -name xormon -usergrp RestrictedAdmin"  
    

  • Older firmwares: use "Administrator" role.
    It is the lowest role which allows retrieve of statistics data from the storage. Read this for explanation.
    

    $ ssh superuser@<storage control enclosure IP address> "svctask mkuser -name xormon   -usergrp Administrator"
    

• SSH-keys
  
  • Create SSH-keys on XorMon host under xormon user if do not exist yet.
    Type enter instead of passphrase

    # su - xormon
    $ ls -l ~/.ssh/id_rsa.pub
    $ ssh-keygen -t rsa -N ""
      Generating public/private rsa key pair.
      Enter file in which to save the key (/home/xormon/.ssh/id_rsa):
      ...
    $ ls -l ~/.ssh/id_rsa.pub
      -rw-r--r-- 1 xormon xormon 382 Jun  1 12:47 /home/xormon/.ssh/id_rsa.pub
    

    You might need to add this into your SSH cfg in case of rsa using on OpenSSH 8.8+ especially in case ssh connection fails with: "no matching host key type found. Their offer: ssh-rsa"

    vi ~/.ssh/config
    Host *
        HostKeyAlgorithms=+ssh-rsa
        PubkeyAcceptedAlgorithms=+ssh-rsa
    

    New storage devices might require stronger key, use this one instead of above 'RSA'

    $ ssh-keygen -t ecdsa -b 521 -N ""
    

  • Copy ssh-key into the storage under storage superuser
    In case of IBM Storwize V7000 Unified use one of "Control Enclosure" IPs (definitely not "File Module" IP).
    Detailed description of IBM Storwize V7000 Unified.

    $ scp ~/.ssh/id_rsa.pub superuser@<storage control enclosure IP address>:/tmp/123tmp
    

  • Create new user "xormon" on the storage
    • Storage firmware 8.3.1.2+: use "Monitor" role

      $ ssh superuser@<storage control enclosure IP address> "svctask mkuser -name xormon -usergrp Monitor -keyfile /tmp/123tmp"  
      

      Using this user role requires setting statistics_frequency on attached storage to 5 minutes under superuser

      ssh superuser@<storage control enclosure IP address> "svctask startstats -interval 5"
      
      # make sure it has been done by:
      ssh superuser@<storage control enclosure IP address> "lssystem" | grep statistics
        statistics_status on
        statistics_frequency 5
      

    • Storage firmware 8+ up to 8.3.1.2: use "Restricted Administrator"

      $ ssh superuser@<storage control enclosure IP address> "svctask mkuser -name xormon -usergrp RestrictedAdmin -keyfile /tmp/123tmp"  
      

    • Older firmwares: use "Administrator" role.
      It is the lowest role which allows retrieve of statistics data from the storage. Read this for explanation.
      

      $ ssh superuser@<storage control enclosure IP address> "svctask mkuser -name xormon -usergrp Administrator -keyfile /tmp/123tmp"  
      

  • If xormon user already exists on the storage then assign him new ssh-keys:

    $ ssh superuser@<storage control enclosure IP address> "svctask chuser -keyfile /tmp/123tmp xormon"
    

  • Check if that works:

    $ ssh -i ~/.ssh/id_rsa xormon@<storage control enclosure IP address> "svcinfo lssystem"
      id 00000100C0906BAA
      name SVC
      location local
      partnership
      bandwidth
      ...
    

XorMon storage configuration

• Add storage into configuration from the UI:
  XorMon UI: Settings icon ➡ Device ➡ Storage ➡ New ➡ Vendor:device ➡ IBM Storwize / SVC / FlashSystem
  
  
• Run "Test" for configured storage device, it must show "OK"
  
  
• Wait about 1 hour, then reload the web browser, you should see it in XorMon UI
  
  

Troubleshooting

• Storwize 8.5.0+ and support for multi-factor user authentication (REMOTE account usage)
  There is a bug in Storwize code which does not allow to use REMOTE account for xormon ssh access.
  Upgrade to Storwize code versions 8.5.0.10 (Q3-Q4 2023) or use LOCAL account
  
  

Configure IBM FlashSystem A9000 like XIV.

Create a user on the storage and check the storage connectivity

• make sure that CIM service is running.
  It should be running by default but especially on A9000 we have seen it has been stopped.
  Use the storage UI or command line like this:

  cim_show
  cim_enable
  

• Allow access from the XorMon host to the storage on port 5989 (SMI-S).
  
  
• When you use the XIV UI then create xormon user in category readonly
  
  
• XIV command line:

  export XIV_XCLIUSER=admin
  export XIV_XCLIPASSWORD=<admin password>
  /opt/ibm/XIVUI/xcli -m <xiv ip> user_define user=xormon password=<password> password_verify=<password> category=readonly
  

XorMon storage configuration

• Add storage into configuration from the UI:
  XorMon UI:Settings icon ➡ Device ➡Storage ➡ New ➡ Vendor:device ➡ IBM XIV / A9000
  
  
• Do not use shell special characters like @#!?|$*[]\{}`"'& in the password, use rather ;:.+- instead.
  
  
• Run "Test" for configured storage device, it must show "OK"
  
  
• Wait about 1 hour, then reload the web browser, you should see it in XorMon UI
  
  

It gets data REST API of the storage. Not all however support it.
if you use older models with no REST API available, then you must use CLI method supported by STOR2RRD only so far.

Configuration

• Allow access from the XorMon host to the NetApp storage on port 8443.
  
  
• Storage user: either use pre-defined user 'monitor' which has monitor role or create 'xormon' under same role
  Change its password if necessary through the storage UI.
  
  

XorMon storage configuration

• Add storage into configuration from the UI:
  XorMon UI: Settings icon ➡ Device ➡ Storage ➡ New ➡ Vendor:device ➡ NetApp E/EF-series (use this one on purpose)
  
  
• Run "Test" for configured storage device, it must show "OK"
  
  
• Wait about 1 hour, then reload the web browser, you should see it in XorMon UI
  
  

Installation is same as for IBM SVC/Storwize.

It get data using REST API provided by the Management node.

Storage connectivity

• Allow access from the XorMon host to the IBM COSS storage on port 443 (https).
  
  
• Create read-only user
  • Login to IBM COSS Web UI
    
  • Navigate to Security ➡ Accounts ➡ Create Account
    
  • Assign read-only operator role
    
  • Assign read-only access to Manager Device
    
    
    (example uses 'stor2rrd' user but use 'xormon' preferably)
    
    

  XorMon storage configuration

  • Add storage into configuration from the UI:
    Settings icon ➡ Storage ➡ New ➡ Vendor:device ➡ IBM ➡ Cloud Object Storage
    
    
    
  • Run "Test" for configured storage device, it must show "OK"
    
    
  • Wait about 1 hour, then reload the web browser, you should see it in XorMon UI