Hitachi Troubleshooting

Hitachi CCI & Export Tool troubleshooting

Export Tool
REST API

Hitachi CCI (HORCM/raidcom)

Hitachi official docu: How To Debug CCI Issues

When the following error occurs during the connection test of the device

raidcom: [EX_EWSTOT] Timeout waiting for specified status

Solution:

Stop HORM instance, in this case number one

su -l xormon  
/HORCM/usr/bin/horcmshutdown.sh 1

Increase the timeout parameter (timeout(10ms)) in /etc/horcm1.conf. The default value should be 3000, increase the value to 10000.

# ip_address service poll(10ms) timeout(10ms)
localhost    11001   1000       3000
HORCM_CMD
# dev_name dev_name dev_name
\\.\IPCMD-192.168.1.1-31001
\\.\IPCMD-192.168.1.2-31001

# ip_address service poll(10ms) timeout(10ms)
localhost    11001   1000       10000
HORCM_CMD
# dev_name dev_name dev_name
\\.\IPCMD-192.168.1.1-31001
\\.\IPCMD-192.168.1.2-31001

Start HORM instance
```
/HORCM/usr/bin/horcmstart.sh 1
```
Try connection test of the device

This is most comon error after testing xormon connection to the storage:

$ ./bin/config_check.sh <Storage Name>

  STORAGE: SVPG01: VSPG
  TCP connection to "<SVP IP>" on port "1099" is ok
  TCP connection to "<SVP IP>" on port "51099" is ok
  TCP connection to "<SVP IP>" on port "51100" is ok
  WARNING : TCP connection to "<SVP IP>" on port "5110" has failed! It might need to be open on the firewall.
  VSPG Error: command failed: raidcom -login xormon    -I
  connection failed!!

1. Make sure you use proper user credential (user & password)
2. Make sure you do not use shell special characters like #!?|$*[]\{}`"'& in the password, use ;:.+-%@ instead

Error during starting of HORCM under xormon user
```
raidcom: [EX_ATTHOR] Can't be attached to HORC manager
```
Problem is that it is running under root! Stopt it under root and start it up under xormon
When that happen set user permission for /HORCM again as per adivse below, starting under root screw user rights.
```
# /HORCM/usr/bin/horcmshutdown.sh 1
# su - xormon  
$ /HORCM/usr/bin/horcmstart.sh 1 
```
Check if they are running
```
$ ps -ef | grep horcm
  xormon   19660912 1 0 Feb 26 - 0:03 horcmd_01
```

Make sure you are able to start HORCM instance under xormon

$ /HORCM/usr/bin/horcmstart.sh 1
$ ps -ef |grep horcmd_01
  xormon   14124     1  0 17:21 ?        00:00:00 horcmd_01

When it does not start then check file rights in /HORCM dir
Make sure that everything is executable and writeable by xormon user
Under root identification execute this:

# touch /HORCM/etc/USE_OLD_IOCTL
# chown xormon   /HORCM
# chown -R xormon   /HORCM/* /HORCM/.uds
# chmod 755 /HORCM /HORCM/usr/bin /HORCM/usr/bin/* /HORCM/log* /HORCM/etc/horcmgr /HORCM/etc/*conf /HORCM/.uds/

If you us commande device for storage communication in /etc/horcm1.conf then assure that stoar2rrd user owns it:

$ ls /dev/sd* | /HORCM/usr/bin/inqraid -CLI -fx
  DEVICE_FILE     PORT    SERIAL  LDEV CTG  H/M/12  SSID R:Group PRODUCT_ID
  sda             -            -     -   -       -     -       - Virtual disk
  sdb             -            -     -   -       -     -       - Virtual disk
  sdc             CL4-A   411711   64B   -       -  000A A:00001 OPEN-V-CM
$ ls -l /dev/sdc
  brw-rw---- 1 xormon   disk 8, 0 Oct 30 10:23 /dev/sdc

Try to log on manually, this example uses /etc/horcm1.conf conf file

$ raidcom -login xormon    -I1
$ raidcom get pool -I1
  PID  POLS U(%) SSCNT Available(MB) Capacity(MB)     Seq# Num LDEV# H(%)   FMT_CAP(MB)
  000  POLN  69    212     21376320     70428792    310878  48 12211  90             -
  001  POLN  15     13    199501344    234772440    310878 160 12514  90             -
$ raidcom -logout -I1
  raidcom : User on Serial#(310444) was removed.

Use the latest CCI, older CCI do not have to work with newer storage firmware
```
$ raidqry -h
  Model  : RAID-Manager/Linux/x64
  Ver&Rev: 01-49-03/00
```

restart HORCM instance under proper user xormon:

$ /HORCM/usr/bin/horcmshutdown.sh 1
$ /HORCM/usr/bin/horcmstart.sh 1

meta_resource was missing on the array for xormon user group in this case:

$ ./bin/config_check.sh <Storage Name>

  STORAGE: SVPG01: VSPG
  TCP connection to "<SVP IP>" on port "1099" is ok
  TCP connection to "<SVP IP>" on port "51099" is ok
  TCP connection to "<SVP IP>" on port "51100" is ok
  VSPG Error: command failed: raidcom get resource -key opt -I440983
  connection failed!!
  raidcom : User on Serial#(310444) was removed.

Hitachi Export Tool

Hitachi official docu:
Troubleshooting Export Tool for Performance Monitor
Device Manager - Storage Navigator Messages
Run Export Tool manually to see if it works, follow this link
Make sure Export Tool installation directory and all subdirectories are owned by xormon user
```
# chown -R xormon /opt/hds
```

New Export Tool might not communicate properly with the storage SVP due to used SSL:

Export tool command failed: "java -Djdk.tls.maxCertificateChainLength=20 -classpath "/opt/hds/90-09-24/lib/JSanExportLoader.jar:/opt/hds/90-09-24/lib/ccj-3.0.1.jar" -Del.tool.Xmx=536870912 -Dmd.command=/tmp/STGMPR-HPEXP8-1129-CC3226/STGMPR-HPEXP8-1129-CC3226-type-hus_vm.txt -Del.logpath=/tmp/STGMPR-HPEXP8-1129-CC3226/log -Dmd.rmitimeout=20 -Del.security=ignore -Del.secmode=cnsa sanproject.getexptool.RJElMainEx"
Loading ExportTool...
secmode=cnsa
The SSL communication to the server cannot be established: host=10.201.38.120, port=443.-
Verify that the following items are correct, and then retry the operation:
 - The specified IP address or host name
 - The root and server certificates
 - The host name or the IP address of the connected server set for Subject Alternative Name and Common Name of the certificate
 - The security mode of the connected server
 - The TLS protocol of the connected server
 - The cipher suites of the connected server
 - The minimum key length (key exchange) of the connected server
 - The signature hash algorithm of the connected server certificate
...
javax.net.ssl.SSLHandshakeException: Certificates do not conform to algorithm constraints
...
2024/09/17 10:39:13 ExportLoader fail.
2024/09/17 10:39:13 Delete temporary directory.
2024/09/17 10:39:13  delete ./lib/0AC92678
2024/09/17 10:39:13  done.
2024/09/17 10:39:13 ExportLoader finish. exitcode=101

A possible cause is that java is started with the parameter -Del.secmode=cnsa in the Export Tool script

Check /opt/hds/<microcode>/runUnix.sh or /opt/hds/microcode/runUnix.bat file and remove -Del.secmode=cnsa from the command

java -Djdk.tls.maxCertificateChainLength=20 -classpath "./lib/JSanExportLoader.jar:./lib/ccj-3.0.1.jar" -Del.tool.Xmx=536870912 -Dmd.command=command.txt -Del.logpath=log -Dmd.rmitimeout=20 -Del.security=ignore -Del.secmode=cnsa sanproject.getexptool.RJElMainEx

-->

java -Djdk.tls.maxCertificateChainLength=20 -classpath "./lib/JSanExportLoader.jar:./lib/ccj-3.0.1.jar" -Del.tool.Xmx=536870912 -Dmd.command=command.txt -Del.logpath=log -Dmd.rmitimeout=20 -Del.security=ignore sanproject.getexptool.RJElMainEx

Since XorMon version 1.6.15 is this workarounded automatically.

VSPG 5500 requires Java 11, it prints this error on its start if there is older Java

Exception in thread "main" java.lang.UnsupportedClassVersionError: (com/hitachi/sanproject/rmi/ssl/uty/SSLUtyCommunicationType) bad major version at offset=6
        at java.lang.ClassLoader.defineClassImpl(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:267)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:151)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:602)
        at java.net.URLClassLoader.access$400(URLClassLoader.java:126)
        at java.net.URLClassLoader$ClassFinder.run(URLClassLoader.java:1057)
        at java.security.AccessController.doPrivileged(AccessController.java:414)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:494)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:651)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:617)
        at java.lang.J9VMInternals.verifyImpl(Native Method)
        at java.lang.J9VMInternals.verify(J9VMInternals.java:70)
        at java.lang.J9VMInternals.initialize(J9VMInternals.java:130)
        at sanproject.getexptool.RJElMain.downloadFiles(RJElMain.java:346)
        at sanproject.getexptool.RJElMain.startLoader(RJElMain.java:108)
        at sanproject.getexptool.RJElMain.main(RJElMain.java:386)

Test Export Tool 2

$ cd /opt/hds/<firmware level> # example /opt/hds/88-03-23
$ sh ./runUnix.sh show interval -ip <ip controller> -login <user> <password>
  Interval : 5 min
  show interval command success

There may be a permission problem (MSG ID = 8007).

A role error (permission error) occurred. (URL = HITPS://172.28.7.76:443/ConfigurationManager/simple/vl1/objects/short-range-monitor-settings/instance, 
HTTP staus code = 403, details = Forbidden) [MSG ID = 8007] {"errorSource":"/ConfigurationManager/simple/vl1/objects/short-range—-monitor—settings/instance", "messageId": "KART7O007-E", 
"message":"This operation is not permited. Either appropriate permissions are not assigned or the resource group configuration was modified by other management interfaces after user authentication. 
Perform user authentication again. If this problem occurs repeatedly, verify and assign appropriate permissions."}

Execution stops.

Solution: The user must be assigned the Performance Management role (Storage Administrator(read/write)).

This is most comon error after testing xormon connection to the storage:

$ ./bin/config_check.sh <Storage Name>

  STORAGE: SVPG01: VSPG
  TCP connection to "<SVP IP>" on port "1099" is ok
  TCP connection to "<SVP IP>" on port "51099" is ok
  TCP connection to "<SVP IP>" on port "51100" is ok
  WARNING : TCP connection to "" on port "5110" has failed! It might need to be open on the firewall.
  Loading ExportTool...
  Missing ip
  2019/01/02 15:46:48 start Export Loader Version : 88-00-00
  2019/01/02 15:46:48  command file path=/opt/hds/hds_test/command2.txt
  2019/01/02 15:46:48  log file path    =/opt/hds/88-03-02/log
  2019/01/02 15:46:48  log file name    =loader0102154648.log
  2019/01/02 15:46:48  ExportLoader mode=ALL
  2019/01/02 15:46:48  download port    =51099
  2019/01/02 15:46:48 read target address from command file.
  2019/01/02 15:46:48 Missing ip
  2019/01/02 15:46:48 ExportLoader fail.
  2019/01/02 15:46:48 ExportLoader finish. exitcode=101
  Loading ExportTool...
  Export tool start [Version 88-03-01-FF/00]
  command file = /opt/hds/hds_test/command.txt
  [  1] ip <SVP IP>      ; Specifies IP adress of SVP
  [  2] dkcsn 415002           ; Specifies Serial Number of SVP
  [  3] login User = ["user"], Passwd = [****************]
  Login failed [line = 3]
  Execution stops.
  Export tool end
  ...

This could mean wrong user or password:
```
  Login failed [line = 2] 
```
Make sure you have installed Java version 8. It might not work with Java 6 especiall when you this in error output:
```
  [  3] login User = ["user"], Passwd = [****************]
```
Java version:
```
$ java -version
  openjdk version "1.8.0_131"
```
Make sure that /opt/hds and all objects inside are owned by xormon user

In the log you see the error:
```
"2019/02/04 16:03:29 . Find Exception = java.net.MalformedURLException: invalid authority: //eka_hus-vm_1.eka.tmk.group/RMIControl".
```
Export tool uses RMI and the server hostname for communication with HUS VM.
The Internet standards for protocols mandate that hostname labels may contain only the ASCII letters ‘a’ through ‘z’ (in a case-insensitive manner), the digits ‘0’ through ‘9’, and the hyphen(‘-‘).
Change the underscore (_) to hyphen(‘-‘) in hostname and issue is resolved.
In the log you see the error:
```
The error "sanproject.serverux.data.SANRmiException : RMI server error (1, 5400)" 
```
Hitachi description: A different user might open one of the following windows or run the export tool. In this case, wait until one of the users moves to a different window or exits the export tool.

Someone else is using Export Tool at the same time.

Rebooting the Service Processor (SVP) might also help
Not all hosts being reported by XorMon NG
It worked fine since some new hosts were added, then aven after a day stats for them missing, only what helped was user rights escalation on the storage (perhaps not all rights are reuired, just noting that it resolved it)

Hitachi REST API

Error: user authentication failed. The authorization header is incorrect.
Reason: User is not created.
Solution: Create user xormon on the storage, read only access