EMC Monitoring: PowerMax VMAX VPLEX ECS XtremIO PowerStore

Dell EMC² monitoring

PowerMax / VMAX
PowerStore
ECS
DataDomain / PowerProtect DD
PowerScale / Isilon
ME4/5 (WBI)
ME4/5 (SMI-S)
PowerFlex / ScaleIO
SC series (Compellent)
XtremIO
VPLEX
Metro node
Unity
VNX / CLARiiON
VNX file / Celerra
VNXe

The product uses REST API natively provided by EMC XtremIO XMS management module to get all necessary data.

Storage connectivity

Allow access from the XorMon NG host to the EMC storage on port 443 (https)
Create user xormon in storage XMS with role Read-Only

(example uses 'stor2rrd' user but use 'xormon' preferably)

XorMon NG storage configuration

Add storage into configuration from the UI:
XorMon NG UI ➡ Settings icon ➡ Device ➡ Storage ➡ New ➡ Vendor:device ➡ EMC XtremIO
You have to define each monitored cluster separately.
Wait about 1 hour, then reload the web browser, you should see it in XorMon NG UI

VPLEX monitoring is supported by our first generation tool STOR2RRD only so far.

Do you want to let us know your priorities? Vote for your missing features and device support. More votes, bigger priority for us.
List of not supported technologies yet

Metro node monitoring is supported by our first generation tool STOR2RRD only so far.

Do you want to let us know your priorities? Vote for your missing features and device support. More votes, bigger priority for us.
List of not supported technologies yet

The product uses storage CLI connected via Dell EMC Unisphere UEM CLI client to get all necessary data.

Use this version xormon-ng-1.6.29.tar.gz preferably if there is not any newer one available. There was fixed a few minor issues

Prerequisites

Dell EMC Unity Unisphere UEM CLI download
https://www.dell.com/support/home/
Download binaries is available for signed users ONLY. BTW no active support is needed.

Identifying your product is necessary to access documentation ➡ choose any Unity model ➡ fresh version www.dell.com/support/home/en-us/product-support/product/unity-300/docs#q=UnisphereCLI&sort=relevancy&f:lang=[en]
For signed-in accounts ➡ Choose any Unity model or enter SerialNumber ➡ Under “Drivers&Downloads” section www.dell.com/support/home/en-us/product-support/product/unity-300/drivers ➡ Filter with keyword “UnisphereCLI” ➡ Select Dell Unity Uninisphere UEM CLI (RedHat Linux 64-bit)
Supported platforms
- Linux RedHat
- SuSE Linux
- Solaris (x86 & Sparc)
- Debian (you must convert it via alien as per bellow)

EMC Unisphere CLI installation

Red Hat, Rocky Linux, CentOS:

# umask 022
# rpm -i UnisphereCLI-Linux-64-x86-en_US-4.0.1.1.3166-1.x86_64.rpm

Debian, Ubuntu:
Install alien when it does not exist on your system

# apt-get install alien

Convert Red Hat rpm package via "alien" cmd

# alien UnisphereCLI-Linux-64-x86-en_US-4.0.1.1.3166-1.x86_64.rpm
  UnisphereCLI-Linux-64-x86-en_US-4.0.1.1.3166-1.i386.deb generated

Install it

# umask 022
# dpkg -i  UnisphereCLI-Linux-64-x86-en_US-4.0.1.1.3166-1.i386.deb

Link uemcli.sh to /usr/bin/uemcli if it has not been done during rpm installation
```
# ln -s /opt/dellemc/uemcli/bin/uemcli.sh /usr/bin/uemcli
```

Setting security level under root user:
Note: The path has changed in recent UnisphereCLI, at least for UnisphereCLI-Linux-64-x86-en_US-5.2.1.1730628-1.x86_64.rpm:
The installation is now under "/opt/dellemc", change bellow cmds accordingly

# /opt/emc/uemcli/bin/setlevel.sh low
  Setting low verifying level.....

You can even use "medium" level, then you have to accept and store certificate at the first connection.

under root:

# /opt/emc/uemcli/bin/setlevel.sh medium
  Setting medium verifying level.....

under xormon user:

# su - xormon 
$ uemcli -d 192.168.1.1 -u Local/admin -p "XXXXXX" /user/account show
  Remote certificate:
  Issuer:     CN=EMC-Unity
  Subject:    CN=EMC-Unity
  Valid from: 2016-07-15 09:54:57 Z
  Valid to:   2019-07-15 09:54:57 Z
  Serial:     0094aea927526b466e
  Id:         2595584890
  Would you like to:
    [1] Accept the certificate for this session
    [2] Reject the certificate
    [3] Accept and store
  Please input your selection (The default selection is [1]): 3

  Storage system address: 192.168.1.1
  Storage system port: 443
  HTTPS connection
  1:    ID   = user_admin
        Name = admin
        Role = administrator
        Type = local

If the option "[3] Accept and store" is missing

$ uemcli -d 192.168.1.1 -u Local/admin -p "XXXXXX" /user/account show
  Remote certificate:
  Issuer:     CN=EMC-Unity
  Subject:    CN=EMC-Unity
  Valid from: 2016-07-15 09:54:57 Z
  Valid to:   2019-07-15 09:54:57 Z
  Serial:     0094aea927526b466e
  Id:         2595584890
  Would you like to:
    [1] Accept the certificate for this session
    [2] Reject the certificate
  Please input your selection (The default selection is [1]):

It's necessary to set SecurityLevel to low
There should be file /etc/uemcli_security.conf readable for root probably with "medium" security.

SecurityLevel = medium

Changing to low level should resolve it.

SecurityLevel = low

Then option "[3] Accept and store" should appear in the selection.

Storage connectivity

Allow access from the XorMon NG host to the Unity storage on port 443 (https)

create user xormon under operator role on the storage.
You might skip it and use already existing account with minimal rights.
Do not use shell special characters like #!?|$*[]\{}`"'& in the password, use ;:.+-%@ instead.
Cmd returns "Error code: 0x6000068" if you do not pass them.

Do not use directly /opt/emc/uemcli/bin/uemcli, use always /usr/bin/uemcli.
If the package was converted with "alien", and after installation "/usr/bin/uemcli" - which should be a softlink - could not be found, you should create it manually

ln -s /opt/dellemc/uemcli/bin/uemcli.sh /usr/bin/uemcli

$ uemcli -d 192.168.1.1 -u Local/admin -p "XXXXXX" /user/account show
  Storage system address: 192.168.1.1
  Storage system port: 443
  HTTPS connection
  1:    ID   = user_admin
        Name = admin
        Role = administrator
        Type = local
...

$ uemcli -d 192.168.1.1 -u Local/admin -p "XXXXXX" /user/account create -name xormon -role operator -type local -passwd "Test2test_"
  Storage system address: 192.168.1.1
  Storage system port: 443
  HTTPS connection

  ID = user_xormon  
  Operation completed successfully.

Test storage connection for xormon account:

$ /usr/bin/uemcli -d 192.168.1.1 -u Local/xormon -p "XXXXXX" /sys/general show -detail
  Storage system address: 192.168.1.1
  Storage system port: 443
  HTTPS connection

  1:    System name            = EMC-Unity
        Model                  = UnityVSA
        Platform type          = EMC Storage System
        System UUID            = 420FFE1D-0234-FD02-976E-9217DC6135ED
        License activation key = ELMEUV0716HN0K
        Product serial number  = VIRT162859DQXZ
        Health state           = OK (5)
        Health details         = "The system is operating normally."

XorMon NG storage configuration

Add storage into configuration from the UI:
XorMon NG UI ➡ Settings icon ➡ Device ➡ Storage ➡ New ➡ Vendor:device ➡ EMC XtremIO
Run "Test" for configured storage device, it must show "OK"
Wait about 1 hour, then reload the web browser, you should see it in XorMon NG UI

VNX monitoring is supported by our first generation tool STOR2RRD only so far.

Do you want to let us know your priorities? Vote for your missing features and device support. More votes, bigger priority for us.
List of not supported technologies yet

VNX file monitoring is supported by our first generation tool STOR2RRD only so far.

Do you want to let us know your priorities? Vote for your missing features and device support. More votes, bigger priority for us.
List of not supported technologies yet

VNXe monitoring is supported by our first generation tool STOR2RRD only so far.

Do you want to let us know your priorities? Vote for your missing features and device support. More votes, bigger priority for us.
List of not supported technologies yet

Source of performance data are not storage devices themself but REST API of one of these:

Dell Storage Enterprise Manager
Dell Storage Manager Data Collector

Connection directly to the storage will not work, use one of above options.

User rights options

Create user xormon under admin role or use already existing admin account.

Node (controller): CPU, IO, data, response time
Port: IO, data, response time

Create user xormon under Reporter role and assign him rights to all or specific:
- volume folders
- server folders
- disk folders
In this case will be reported utilization only for assigned folders and their items.

Storage connectivity

Install the Enterprise Manager or DSM Data Collector and attach there your Compellent boxes.
Create user xormon on the Enterprise Manager or DSM Data Collector under admin role.
Unfortunatelly admin role is only the role which can get all performance data from the storage.
If you use web tool Unisphere central for SC series then registration there is not enough.
You have to register storage for user in Enterprise Manager or DSM Data Collector.
Allow access from the XorMon NG host to the Enterprise Manager on port 3033 (HTTPS).
Set "Report gathering settings" of IO usage to 5 minutes.

XorMon NG storage configuration

Add storage into configuration from the UI:
XorMon NG UI ➡ Settings icon ➡ Device ➡ Storage ➡ New ➡ Vendor:device ➡ Dell SC series
On the picture you can see 3 compellent storage in the Enterprise Manager.

Configure every storage you want to monitor, use same storage name alias as you see in the Enterprise Manager.
Run "Test" for configured storage device, it must show "OK"
Wait about 1 hour, then reload the web browser, you should see it in XorMon NG UI

Storage agent uses eithre the EMC Unisphere as data source, or the storage itself (that method works at least on 5978_711/0365 storage firmware and newer, most probaly even on olders)
EMC Unispheres 1.6.x, 8.x, 9.x and 10.x are supported.
Read why TDEVs are not monitored.

Prerequisites

Allow access from the XorMon NG host to the EMC Unisphere or to the storage itself depends on mode you use on port 8443 (TCP).

EMC Configuration

Unisphere (scenario 1)
- Create user xormon under "PerfMonitor" role on Unisphere (example uses 'stor2rrd' user but use 'xormon' preferable).
- Allow statistics logging on the EMC Unisphere
  UI ➡ Performance ➡ Settings ➡ System Registrations ➡ Diagnostics ➡ On (Diagnostics interval 5 minutes)
  
  EMC Unisphere 1.6
  
  EMC Unisphere 8+
Direct storage attachment (scenario 2)
- Connect on one of both PMAX e-management port and on the wheel on the top right corner
- Click “users and groups”, then “local users”, and then the “create” button
- Enter: The account name, description and Password 2 times (warning: both PMAX and XorMon NG have forbidden special characters and they are not the same)
- Choose the “perfmonitor” role
- Keep right access as “read only”

XorMon NG storage configuration

Add storage into configuration from the UI, fill in Symmetrix ID for both scenarios
XorMon NG UI: Settings icon ➡ Device ➡ Storage ➡ New ➡ Vendor:device ➡ Dell EMC VMAX
Run "Test" for configured storage device, it must show "OK"
Wait about 1 hour, then reload the web browser, you should see it in XorMon NG UI

Storage agent uses EMC Data Domain CLI accessed via ssh to get all necessary data from the storage.
EMC DataDomain firmware versions 6.x and 7.x are supported.
EMC DataDomain firmware versions 5.5 is supported only in STOR2RRD.
EMC DataDomain 6.0.2.50+ and 6.1.2.40+ (6.2 is not affected) do not provide disk stats community.emc.com/docs/DOC-78887

Prerequisites

Allow access from the XorMon NG host to the EMC storage on port 22 (TCP).
Create user xormon under user role on the storage (example uses 'stor2rrd' user but use 'xormon' preferable)

Exchange ssh keys
Print public key:

$ cat .ssh/id_rsa.pub
  ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPpQlm0Z/x1X4txd2GgZgvg1Fm7QHeusQ6LqBp/Sb1doAoiDmXErk+lNY27aJAqb4YiGnqejioqCJ9N93NiKpffxH+L3

Log on to the storage and add above key:

$ ssh <data domain host> -l xormon
  xormon@data_domain_host> adminaccess add ssh-keys
    Enter the key and then press Control-D, or press Control-C to cancel.
    ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPpQlm0Z/x1X4txd2GgZgvg1Fm7QHeusQ6LqBp/Sb1doAoiDmXErk+lNY27aJAqb4YiGnqejioqCJ9N93NiKpffxH+L3

Test connection:

$ ssh <data domain host> -l xormon
  EMC Data Domain Virtual Edition
  Welcome to Data Domain OS 6.1.0.11-571028

XorMon NG storage configuration

Add storage into configuration from the UI:
XorMon NG UI: Settings icon ➡ Device ➡ Storage ➡ New ➡ Vendor:device ➡ Dell EMC DataDomain
Run "Test" for configured storage device, it must show "OK"
Wait about 1 hour, then reload the web browser, you should see it in XorMon NG UI

Troubleshooting

DD refuses some ssh sessions, you can find in XorMon NG log (~/xormon-ng/server-nest/log/datadomain) this:

Error: Unable to connect to remote machine (dd01) : No such file or directory
Error: Error command: ssh login failed  unable to load Perl module IO::Pty: Can't locate IO/Pty.pm ...

Increase DataDomain parameter login-max-active, xormon ssh sessions are in conjuction with normal backup sessions.

sysadmin@dd(active:0)# adminaccess option show login-max-active.
  Adminaccess option "login-max-active" set to "150".
sysadmin@dd(active:0)# adminaccess option set login-max-active 500
  Adminaccess option "login-max-active" set to "500".

Isilon v7.x, v8.x and v9.x firmwares are supported.
When you use Isilon v9.x then just use one Management IP Address from the Cluster (do not use direct Node's IPs).
The product uses EMC Isilon REST API interface natively provided by the storage to get all necessary data.

Storage connectivity

Allow access from the XorMon NG host to the EMC storage on port 8080.

Create user xormon on the storage with new read-only role called xormon.

$ ssh -l admin 192.168.1.1
isilon01-1% isi auth users create xormon --enabled yes --password-expires no --password XXXXXXX
isilon01-1% isi auth roles create --name xormon --description "Read-only role for XorMon NG"
isi auth roles modify xormon   --add-user=xormon --add-priv-ro=ISI_PRIV_EVENT --add-priv-ro=ISI_PRIV_LOGIN_PAPI --add-priv-ro=ISI_PRIV_STATISTICS --add-priv-ro=ISI_PRIV_SMARTPOOLS --add-priv-ro=ISI_PRIV_DEVICES --add-priv-ro=ISI_PRIV_QUOTA

XorMon NG storage configuration

Add storage into configuration from the UI:
XorMon NG UI: Settings icon ➡ Device ➡ Storage ➡ New ➡ Vendor:device ➡ Dell EMC Isilon
Run "Test" for configured storage device, it must show "OK"
Wait about 1 hour, then reload the web browser, you should see it in XorMon NG UI

The product uses EMC ME4 SMI-S and WBI API interfaces natively provided by the storage.

Note: storage firmware GT280R010-01 does not provide data via SMI-S (up to GT280R009-02 it is fine), it is a confirmed issue in the storage firmware (SR #14853424). Upgrade to GT280R011-01 or use WBI access method.

Storage connectivity

Allow access from the XorMon NG host to the storage on ports 80 (WBI) and 5988 (SMI-S).
If you use https and secure SMI-S then test ports 443 and 5989 instead.
If you use storage cmd line configuration like in examples below (you can use the storage UI instead) then port 22 (ssh) must be open.

Make sure that secure SMI-S (port 5989) and WBI HTTPS (port 443) protocols are enabled on the storage

CMD line:

$ ssh -l manage <storage_hostname>
# show protocols
  Service and Security Protocols
  ------------------------------
  Web Browser Interface (HTTP): Enabled
  Secure Web Browser Interface (HTTPS): Enabled
  Command Line Interface (Telnet): Disabled
  Secure Command Line Interface (SSH): Enabled
  Storage Management Initiative Specification (SMI-S): Enabled
  Unsecure Storage Management Initiative Specification (SMI-S 5988): Disabled
  ...

If secure SMI-S or WBI HTTPS are not enabled:

# set protocols smis enabled
# set protocols https enabled

Create user xormon on the storage with monitor and standard roles, allow SMI-S and WBI.
- CMD line method:
  Adjust password to your preferred one from used Test1234.
```
$ ssh -l manage <storage_hostname>
# create user interfaces smis,wbi locale English password Test1234 roles monitor,standard xormon
```
- UI:
  
  (example uses 'stor2rrd' user but use 'xormon' preferable)
Note for newer firmwares role "standard" is necessary for WBI access.

XorMon NG storage configuration

Add storage into configuration from the UI:
XorMon NG UI: Settings icon ➡ Device ➡Storage ➡ New ➡ Vendor:device ➡ Dell EMC ➡ PowerVault ME4 ➡ SMI-S

If you set https in above storage properties then this must be enabled on the storage:

$ ssh -l manage <storage_hostname>
# show protocols
  ...
  Secure Web Browser Interface (HTTPS): Enabled
  Storage Management Initiative Specification (SMI-S): Enabled
  ...

If you use http then this:

  Web Browser Interface (HTTP): Enabled
  Unsecure Storage Management Initiative Specification (SMI-S 5988): Enabled

Do not use shell special characters like @:;#!?|$*[]\{}`"'& in the password, use rather .+-=_% instead.
Run "Test" for configured storage device, it must show "OK"
Wait about 1 hour, then reload the web browser, you should see it in XorMon NG UI

Note: ECS 3.6+ is supported since this XorMon NG version xormon-ng-1.6.19.tar.gz

Data is obtained from the EMC ECS Rest API, from VDC nodes.

Allow access from the XorMon NG host to each EMC ECS VDC node on port 4443
Create Storage Management user "xormon" with System Monitor role on each monitored VDC.
Add storage into configuration from the UI:
XorMon NG UI: Settings icon ➡ Device ➡ Storage ➡ New ➡ Vendor:device ➡ Dell EMC : Elastic Cloud Storage

The addition of a VDC to ECS configuration is intended as choosing specific VDC you want to display in Stor2rrd GUI.
To monitor your chosen VDC, just set one or two nodes which are management EndPoint for a given VDC.
No need to add every node in VDC
Run "Test" for configured storage device, it must show "OK"
Wait about 1 hour, then reload the web browser, you should see it in XorMon NG UI

Note: there is a bug in the product versions 1.0.0 - 1.1.1 preventing data collection, upgrade to XorMon NG 1.1.3.

The product uses REST API natively provided by EMC PowerStore storage.

Note the issue with PowerStore firmware 3.0.0.1, upgrade to 3.2.1.0.

Storage connectivity

Allow access from the XorMon NG host to the EMC storage on port 443 (https)
Create a new user in PowerStore Manager UI with Operator role
See this: www.delltechnologies.com/en-us/collaterals/unauth/white-papers/products/storage/h18150-dell-emc-powerstore-manager-overview.pdf
(page 40: it is sufficient to give only Operator privileges to an account which is only responsible for monitoring instead of giving full privileges with the Administrator role)

XorMon NG storage configuration

Add storage into configuration from the UI:
Settings icon ➡ Storage ➡ New ➡ Vendor:device ➡ EMC PowerStore
Run "Test" for configured storage device, it must show "OK"
Wait about 1 hour, then reload the web browser, you should see it in XorMon NG UI

XorMon NG obtains preformance and configuration data from the PowerFlex Gateway using REST API.

Storage connectivity

Allow access from the XorMon NG host to the PowerFlex Gateway on port 443
Create a new user with the 'Monitor' role
- Login to the primary MDM server as 'root'
- Login to PowerFlex SCLI as administrator
```
scli --login --username admin 
```
- Create new user User will be created with a temporary password which needs to be changed during the first login
```
scli --add_user --username xormon --user_role Monitor 
```
- Login to PowerFlex SCLI with the new user credentials and set new password
```
scli --login --username xormon
scli --set_password 
```

XorMon NG storage configuration

Add storage to the configuration:
XorMon NG UI: Settings icon ➡ Device ➡ Storage ➡ New ➡ Vendor:device ➡ Dell EMC : PowerFlex
Run "Test" for configured storage device, it must show "OK"
Wait about 1 hour, then reload the web browser, you should see it in XorMon NG UI

The product uses storage WBI API interfaces natively provided by the storage.
WBI access must be used on new storage models which do not support SMI-S protocol.

Storage connectivity

Allow access from the XorMon NG host to the storage on ports 443.

Make sure that secure and WBI HTTPS (port 443) protocol is enabled on the storage

CMD line:

$ ssh -l manage <storage_hostname>
# show protocols
  Service and Security Protocols
  ------------------------------
  ...
  Secure Web Browser Interface (HTTPS): Enabled
  ...

If secure WBI HTTPS is not enabled:

# set protocols https enabled

Create user xormon on the storage with monitor and standard roles, allow WBI.
- CMD line method:
  Adjust password to your preferred one from used Test1234.
```
$ ssh -l manage <storage_hostname>
# create user interfaces wbi locale English password Test1234 roles monitor,standard xormon
```
- UI:
  
  (example uses 'stor2rrd' user but use 'xormon' preferable)
Note for some newer firmwares role "standard" is necessary for WBI access.

XorMon NG storage configuration

Add storage into configuration from the UI:
XorMon NG UI: Settings icon ➡ Device ➡ Storage ➡ New ➡ Vendor:device ➡ Dell EMC ➡ PowerVault ME4 ➡ WBI

If you set https in above storage properties then this must be enabled on the storage:
```
$ ssh -l manage <storage_hostname>
# show protocols
  ...
  Secure Web Browser Interface (HTTPS): Enabled
  ...
```
If you use http then this:
```
  Web Browser Interface (HTTP): Enabled
```
Run "Test" for configured storage device, it must show "OK"
Wait about 1 hour, then reload the web browser, you should see it in XorMon NG UI