Professional Documents
Culture Documents
ECS - ECS Installation-Software
ECS - ECS Installation-Software
Topic
ECS Installation
Selections
Installation: Software
Select ECS Version: 3.7.x.x
What installation activity will you be performing against the ECS 3.7.0.x environment?: Software
Installation Guide for EX300, EX500 and EX3000 (EX-Series) Hardware
REPORT PROBLEMS
If you find any errors in this procedure or have comments regarding this application, send email to
[email protected]
Copyright © 2022 Dell Inc. or its subsidiaries. All Rights Reserved. Dell Technologies, Dell, EMC, Dell
EMC and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be
trademarks of their respective owners.
The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of
any kind with respect to the information in this publication, and specifically disclaims implied warranties of
merchantability or fitness for a particular purpose.
Use, copying, and distribution of any software described in this publication requires an applicable
software license.
This document may contain certain words that are not consistent with Dell's current language guidelines.
Dell plans to update the document over subsequent future releases to revise these words accordingly.
This document may contain language from third party content that is not under Dell's control and is not
consistent with Dell's current guidelines for Dell's own content. When such third party content is updated
by the relevant third parties, this document will be revised accordingly.
Page 1 of 91
Contents
Preliminary Activity Tasks .......................................................................................................6
Read, understand, and perform these tasks.................................................................................................6
Installation overview................................................................................................................7
About this guide ............................................................................................................................................7
Installation scenarios and workflow ..............................................................................................................7
ECS Designer ...............................................................................................................................................7
ECS software files.........................................................................................................................................8
Understanding when to run the different ECS health checks .......................................................................9
Admin account permissions (using the sudo command) ..............................................................................9
Using copy and paste to run commands.....................................................................................................10
Multirack cabling .........................................................................................................................................11
Page 2 of 91
Configure the rack ID and static IP addresses......................................................................35
Determine whether to use the installation scripts created through ECS designer or install ECS manually35
Configure the Rack ID.................................................................................................................................35
About configuring static IP addresses for the ECS Appliance ....................................................................36
Separating networks ...................................................................................................................................37
Configure DNS............................................................................................................................................37
Configure NTP ............................................................................................................................................37
Adjust system time after configuring NTP servers ......................................................................................38
Manually configure static IP addresses ......................................................................................................38
Configure the hostname..............................................................................................................................39
Post-installation.....................................................................................................................66
Perform post-installation tasks....................................................................................................................66
Run a post-install health check from the Service Console..........................................................................66
Update to the latest xDoctor version...........................................................................................................67
Configuring xDoctor not to monitor custom FrontEnd switches ..................................................................67
Run xDoctor checks....................................................................................................................................68
Check relevant KB articles for the latest installation updates .....................................................................69
Optionally, configure SNMP to collect metrics and send traps ...................................................................69
Cleaning the nodes in the cluster................................................................................................................69
ECS Login, License, SRS, and Provision ...................................................................................................70
Log into the ECS Portal ..............................................................................................................................70
Page 3 of 91
Upload the ECS license file ........................................................................................................................71
Provision ECS resources ............................................................................................................................72
ECS geo......................................................................................................................................................72
Provisioning ECS resources .......................................................................................................................72
Check disks health after initial login, configuration, and provision..............................................................73
Connect ECS to the Secure Remote Services ...........................................................................................73
Secure Remote Services prerequisites.......................................................................................................73
After you finish ............................................................................................................................................74
Add the Secure Remote Services Gateway to ECS ...................................................................................75
Verify that Secure Remote Services call home works ................................................................................76
Restart xDoctor, and verify ECS is registered with the xDoctor portal........................................................76
Page 4 of 91
Enabling or disabling Cloud DVR..........................................................................................89
Enabling Cloud DVR ...................................................................................................................................89
Sample Output ............................................................................................................................................89
Disabling Cloud DVR ..................................................................................................................................90
Sample output.............................................................................................................................................90
Page 5 of 91
Preliminary Activity Tasks
This section may contain tasks that you must complete before performing this procedure.
Table 1 List of cautions, warnings, notes, and/or KB solutions related to this activity
2. This is a link to the top trending service topics. These topics may or not be related to this activity.
This is merely a proactive attempt to make you aware of any KB articles that may be associated with
this product.
Note: There may not be any top trending service topics for this product at any given time.
Page 6 of 91
Installation overview
This chapter includes the following information:
• Installing and configuring ECS software at a single site (stand-alone) with a single or multiple
racks.
• Installing and configuring ECS software at a multisite (geo) deployment.
1. If you are installing the ECS software on a system that is deployed on a third-party rack, install
the latest released operating system version on each node.
If you are installing ECS software on an ECS appliance that has been prepackaged and
deployed on a Dell EMC rack, check the ECS operating system version running on the
appliance. If the latest operating system is not running on the appliance, install the latest version
of the operating system. If the appliance is running the latest version of the ECS operating
system, you do not need to install the ECS operating system.
2. Configure the ECS nodes, rack ID, and static IP addresses.
3. Install the ECS Software (Hardware Abstraction Layer (HAL), fabric, and object services).
If you are using installation scripts that are created through the ECS designer, copy the ECS
software installation files, and continue to use the installation scripts to install the ECS software.
Once the software is installed continue to step 4, using the instructions provided in this guide.
4. Log in to the ECS Portal to license ECS, configure Secure Remote Services (SRS), and perform
initial provisioning operations.
ECS Designer
The ECS Designer Excel based application is a Dell EMC resource used to streamline the design and
implementation of ECS. The ECS Designer is used for the collection of initial ECS configuration
information.
The ECS Designer must be completed , and approved prior to beginning installation of ECS Software.
Once completed and approved, you may use the ECS Designer to create ECS Software Installation
Scripts which automate the ECS Installation process.
NOTE:
Obtaining ECS automated scripts, that is, completing the ECS Designer is a must for ECS installation because it makes easier for
the support team to troubleshoot the issues that may occur during the installation. For more information about obtaining ECS
automated scripts, refer to the Obtain the ECS Automated Script topic.
Page 7 of 91
For full instructions on updating, completing and submitting the ECS Designer for approval, review the
content at ECS Script Deployment Process site before completing the ECS Designer.
For more information and to download the ECS Designer go to the ECS Designer site. The ECS
Designer can also be downloaded from the SolVe Desktop. Only authorized personnel can obtain the
ECS Designer.
NOTE:
Contact your ECS customer support representative if you have trouble accessing either site.
NOTE:
The following filenames are accurate as of the publication date of this installation guide. Always check for the latest available files
posted to https://www.dell.com/support/home/en-us/.
Table 1. ECS software files The table lists the ECS Software files
Filename Description
ecs-<current_version>- • Contains the ECS software and associated files that are
production.tgz used by countries in which D@RE support is permitted
to install or upgrade ECS software.
• The file is obtained from the ECS product page before
installing or upgrading ECS.
• Do not use this file to install or upgrade in Russia, and
China as these two countries do not allow encryption.
ecs-<current_version>- 5. Contains the ECS software and associated files that are
production-no-encryption.tgz used by countries in which encryption is not permitted to
install or upgrade ECS software.
6. The file is obtained from the ECS product page before
installing or upgrading ECS.
7. Use this file to install or upgrade in Russia or China
these two countries do not allow encryption.
Page 8 of 91
and is used during ECS software installation. There are no strict
naming conventions for the file, but it is recommended to apply a
filename that is identifiable with the customer site.
Compatibility Validates hardware, network, and site Post ECS OS imaging. Run the tool before
Checker tool infrastructure requirements including: creating the topology.txt file, and installing
ECS software (HAL, Fabric, and Object
• Supported Docker version services)
installed
• ECS OS version
• Sufficient disk space available
for the install
• Environment readiness (NTP,
DNS)
Service 8. OS, Fabric, and Docker health Post-software installation. Run the checks after
Console installing HAL, Fabric, and Object services.
9. Formatted drives
10.Root FS free space
11.Nodes running the same OS
version
xDoctor Collects system-level and node-level Post-software installation. Run the tool after
diagnostics with data analysis. running the Service Console health checks.
# sudo getrackinfo
• If a command requires the root environment run sudo with the -i parameter. For example:
Page 9 of 91
admin@ecsr1n1:~> sudo -i lockdown get
unlocked
NOTE:
The -i parameter affects the working directory (/root), and sources the rest of the root environment (similar to the way the env
command works). The ECS commands do not require the -i parameter when the fully qualified path to the command is provided.
For example:
unlocked
All sudo commands are logged across the system. When syslog is set to informational, output is
sent to the customer's syslog server. Setting syslog to informational can generate substantial
amounts of syslog messages.
For example:
Once edited, you can then press Enter to intentionally execute the command.
Page 10 of 91
Multirack cabling
Where the ECS system you are installing comprises of multiple racks, or where you are extending an
ECS cluster with an extra rack, you must install the rack and connect the two racks together.
The following ports on the Dell EMC S5148F switch should be used when connecting multiple racks:
Ports To connect
39 and 40 the Dell EMC S5148F switch to Arista switches running on ECS D- or U-series (Gen 2)
hardware
41–44, and
45-48
the Dell EMC S5148F switch to another the Dell EMC S5148F switch running on the ECS
(10 Gb or EX300, EX500, or EX3000, EX5000 (EX-series) hardware. For example, 41–44 could be
25 Gb used to connect to one EX-series rack, and 45–48 could be used to connect to another EX-
uplink) series rack.
51–52, and Or
53–54
the Dell EMC S5148F switch to an aggregation switch.
(100 Gb
uplink)
Network cabling for the supported configurations is provided in the ECS EX-series EX300, EX500,
EX3000, and EX5000 Hardware Guide.
To provide a more robust interconnection, which is less susceptible to network split, you can create a
loop connection by connecting the final rack back to Rack 1. However, if you connect in this way, you
must provide an aggregation switch to enable connection to the iDRAC network.
Network connections and configurations are described in the ECS Networks Guide for EX Series
Hardware (available in SolVe Desktop).
Installation resources
Review the following items prior to arriving at the customer site.
NOTE:
See ECS Product Documentation for a complete list of ECS documentation.
Table 1. Resources for installation The table lists the Resources for installation
Page 11 of 91
SolVe An internal, confidential repository that SolVe Desktop can be installed on your local
Desktop or includes all ECS Service procedures, system. Installation files can be downloaded
Online documentation, and the following from: https://solve.emc.com/desktopbinaries/
(internal resources: SolVe Online can bee accessed at:
and https://solveonline.emc.com/solve/products/15
confidential • Latest ECS internal
) documentation, ECS software
Release Notes and README.txt
• ECS Designer
• Link to the ECS Support site
• Latest, supported switch
configuration files
• Latest version of xDoctor
• Dell EMC Power Calculator
• Detailed ESRS setup
instructions
NOTE:
On the support page, search for the latest ECS Release
Notes.
Knowledge Ensure that you have access to the The following article lists the KB articles relevant
Base (KB) Knowledge Base (KB) articles relevant to each ECS release: KB 485299
articles to the ECS version required at the
customer site.
ECS An Excel based application, Dell EMC, Information to complete and use the ECS
Designer internal resource used to streamline the Designer is available at:
design and implementation of ECS. The https://psapps.emc.com/central/solution/ECS-
ECS Designer is used for the collection Designer.
of initial ECS configuration information.
ECS Service Personnel are required to
fill out the ECS Designer prior in
installing ECS.
You must decide before arriving onsite if you will use the installation scripts generated by ECS Designer
or the manual procedures described in this guide. You cannot use both.
Page 12 of 91
An ECS deployment consists of the following components:
• One or more racks connected to the customer network for both data traffic and remote
management:
o The rack and all nodes must be powered on to ensure that there are no hardware
issues.
o Powered off all the nodes to install the operating system.
o Reserve valid IP addresses for the nodes which will be configured statically, or plan for
them to be assigned by DHCP.
o If deployment is multi-rack, ensure that the Rack ID matches the ECS Designer
specification. Refer to Configure the Rack ID for specific steps.
• BIOS updates require that a crash cart (monitor and keyboard) or a KVM to allow using a laptop
to run the installer.
• The following infrastructure must be in place at the customer site:
o Dynamic Host Configuration Protocol (DHCP) server — Required if assigning IP
addresses dynamically. The IPs must be reserved by the DHCP server when
dynamically assigned.
o Domain Name Server (DNS) (also called forwarder) — Requires a minimum of two
servers.
o Fully qualified domain name in DNS — Required to solve both forward and reverse
lookup.
o Network Time Protocol (NTP) servers — Recommended a minimum of 4 servers. Refer
to Selecting Offsite NTP Servers at:
http://support.ntp.org/bin/view/Support/SelectingOffsiteNTPServers#Section_5.3.3.
You should ensure that:
12.The public node name must match the node name that is resolved on the public.mgmt
interface if public.mgmt is separated out.
13.The uplink topology and network configuration has been validated with the customer.
14.You've reviewed the list of ports used for ECS communications. Consult the ECS Security
Configuration Guide for a list of required firewall ports to open.
• Download the latest ECS Service Console installer, and release notes
• Obtain the switch OS firmware package
• Obtain the completed ECS Designer
• Obtain the ECS license file
• Obtain the PSNT and SWID
• Identify the rack ID to assign to each rack
• Obtain the latest released ECS installer package
Page 13 of 91
• Determine the customer's branding requirement for the ECS portal
Download the latest ECS Service Console installer, and release notes
Updated versions of the Service Console are periodically released. When Installing ECS software, be
sure to have the latest version of the Service Console and Service Console release notes available for
the ECS software post-installation checks.
To obtain the latest version of the ECS Service Console installer, and Service Console Release Notes:
NOTE:
It takes maximum 48 hours for generating your scripts.
Page 14 of 91
4. Select the entitlements to activate and click Start Activation Process.
5. Select Add a Machine to specify any meaningful string for grouping licenses.
The "machine name" does not have to be a machine name at all; enter any string that will help
you keep track of the licenses.
6. Enter the quantities for each entitlement to be activated, or select Activate All. Click Next.
If you are obtaining licenses for a multi-site (geo) configuration, you should distribute the
controllers as appropriate in order to obtain individual license files for each VDC.
7. Optionally specify an addressee to receive an email summary of the activation transaction.
8. Click Finish.
9. Click Save to File to save the license file (.lic) to a folder on your computer.
The SWID is used during ECS SRS deployment and when creating ECS VDC dial-in devices, and the
device names are <SWID>-1 thru 20. The SWID is in the ECS License file, <xxx>.lic, and the field
containing the SWID is SWID=xxxx.
WARNING:
Only China and Russia use the non-encrypted package.
2. Under Browse Products in the left pane, select the latest release version.
3. Expand Full Release.
4. Click the latest version of the Fabric and Object Installation link to download the latest software
installation package.
If you use Internet Explorer (IE) to download the.tgz file, IE changes the file extension to .gz.
Page 15 of 91
5. Expand Operating Environment.
6. Click the latest version of the ECS - OS Setup - <version> link to download the OS installation
package.
Get the image files for third-party branding of the ECS Portal
Optionally, customers can choose to brand ECS with their own company logo, which would replace the
Dell EMC logo on the ECS Portal.
To re-brand the ECS Portal with a third-party logo, you will need the following three images prior to
installing the ECS Software:
• company_name.txt,: Shows the specific company name to display on the ECS Portal
Welcome page.
• loginLogo.png,: Shows the specific company logo to display on the ECS Portal Login page.
• logo.png,: Shows the specific company logo to display on the ECS Portal menu pane.
See Image and .txt file specifications for third-party branding for file requirements.
Create a directory to store the image files such as: /host/files/.
Page 16 of 91
4. Using the arrow keys, select iDRAC Settings >Network.
5. Arrow down to IPv4 setting and select Disable DHCP if it is not already disabled, and manually
enter the following information:
a. IP Address: 192.168.219.101
b. Gateway: 0.0.0.0
c. Subnet Mask: 255.255.255.0
NOTE:
These settings configure a service iDRAC network default on the master node for the install process.
Page 17 of 91
6. Click Back to return to the iDRAC Settings page, and scroll down to the User Configuration and
set the following:
d. Enable User: should be enabled.
e. User Name: root
f.LAN User Privilege: Administrator
g. Serial Port User Privilege: Administrator
h. Change Password: passwd
Page 18 of 91
7. Click Back.
8. Click Finish, and select Yes when prompted to save the changes.
9. Tab to the Finish menu, and hit the Enter key, and hit enter again, to confirm exiting the System
Setup.
The server will reboot.
10. Once the node is rebooting, power off node 1 by holding down the power button until the power
light goes out.
11. Continue to Install the latest ECS OS.
This procedure requires admin user access. ECS disables the root account to enhance security. Prefix
commands with sudo to gain administrator access.
WARNING:
Type (or copy and paste) all multiline commands as a single line of text with no breaks.
1. If you are installing the ECS operating system image on an ECS appliance that is deployed on a
third-party rack, continue to step 2.
If you are installing ECS software on an ECS appliance that has been prepackaged and
deployed on a Dell EMC rack, check the ECS operating system version running on the
Page 19 of 91
appliance. If the latest operating system image is not running on the appliance, reimage the
system with the latest operating system as described in the following steps. If the appliance is
running the latest ECS operating system image, you do not need to reimage the ECS operating
system.
2. Download the latest version of the ECS - operating system Setup files.
1. Go to: https://www.dell.com/support/home/en-us/product-support/product/ecs-appliance-
software-with-encryption/drivers.
2. Under Browse Products, select the current version of ECS.
3. Expand Operating Environment, and click the latest version of the ECS - OS Setup files.
3. Configure the service laptop as follows to be able to access iDRAC for the operating system
image:
If iDRAC IPs are configured on the private network:
1. Connect the service laptop to the service port on the private (Fox) switch; ports 34 or 36.
2. Configure your laptop with the following network parameters:
▪ IP: 192.168.219.99
▪ Netmask: 255.255.255.0
▪ No Gateway
3. Start a command prompt on your laptop and verify connectivity to iDRAC.
# ping 192.168.219.101
Page 20 of 91
i. Username: root
j. Password: passwd
6. This guide demonstrates the iDRAC operating system installation procedures on HTML5. If you
are running ECS EX3000 or EX5000, you are running iDRAC8. iDRAC8 runs on Java by default,
but can also run on HTML5. Refer to Configure the iDRAC Virtual Console to run on HTML5 to
continue to use this procedure to install the ECS operating system.
If you want to run the Java version of the iDRAC8 Virtual Console, see the iDRAC8
documentation for requirements and detailed steps. iDRAC8 documentation is available at:
http://en.community.dell.com/techcenter/systems-management/w/wiki/12334.idrac8-home.
7. From the iDRAC, Dashboard, click Launch Virtual Console.
8. From the iDRAC Virtual Console, click Connect Virtual Media.
9. From the Virtual Media page, Map CD/DVD >Image File area, select Choose File.
10. Browse to the location of the latest version of the ECS operating system Setup file, and select
the ECS operating system Setup file (ecs-os-setup<version>.iso).
11. Click Map Device.
12. Once the ECS operating system Setup file has been mapped, close out the Virtual Media page.
The status report in the Virtual Console confirms that the device has been mapped.
Page 21 of 91
NOTE:
This happens very fast. Be ready to press F11 when you see the prompt that is shown in the following steps or you can
press F11 multiple times at any point during the BIOS boot before this prompt is displayed.
15. When prompted for the password, type emcbios, and press Enter.
CAUTION:
If you do NOT see this next screen during boot up (and you have been carefully watching it), the ISO was not mounted
correctly (booted) and you will have to restart. Turn off the current node and all other nodes except for node 1. Restart
the operating system imaging of node 1—making sure that all other nodes are turned off before you begin the installing
the ECS operating system. This might happen if all the nodes were not shut down when you were imaging node 1.
16. From the Boot Manager Main Menu, select One-shot BIOS Boot Menu >Virtual Optical Drive.
NOTE:
If you missed the F11 Boot Menu selection, and the node begins to boot, press Control-Alt-Del.
17. Quickly select Install_ecs-os-setup-target.
Page 22 of 91
NOTE:
If the connection to console is lost and you receive No Signal, press Alt+F1.
18. After the operating system installs, and node 1 reboots, validate the ECS operating system
version that is shown above the login prompt, for example:
Once the ECS version is verified, use the remaining steps to update the peer nodes using PXE
boot.
19. Verify that you can log in to node 1 using the default credentials, and then exit.
k. Username: admin
l. Password: ChangeMe
The remaining operating system installation steps can run using an SSH Putty session which
enables for copy and paste of the commands.
20. Open an SSH session using Putty to 192.168.219.1:
m.Username: admin
n. Password: ChangeMe
WARNING:
o. In ECS 3.4 for EX500 and EX3000, a mandatory minimum version of NIC is required.
p. In ECS 3.5 for EX300, a mandatory minimum version of NIC is required.
Page 23 of 91
q. In ECS 3.7 for EX5000 NIC compatibility checks are applicable.
Refer to Firmware update for details.
If the operating system imaging process detects an older version of NIC, the minimum supported
version of NIC firmware is automatically applied on EX5000, EX3000, EX500, and EX300. To
apply the firmware properly, an extra power off and power on sequence may be required. Follow
the next steps to ensure that the proper power cycle sequence is achieved.
After initial master node is installed online, perform following steps.
21. Check if the boot flag is set to cold (EX500/EX3000), or warm (EX300) on the master node:
NOTE:
The following command checks the boot flag status post operating system imaging to determine whether the flag is
properly cleared after the NIC auto-update.
NOTE:
Commands are different for different EX-Series hardware.
For EX500:
NOTE:
This command issues the required racadm command sequence to ensure that firmware is applied properly. It
may take 10 minutes for the node to perform firmware reconfiguration and boot after the above command is
initiated. On EX500, cold boot flag is automatically cleared as part of powercycleneedex500cold2none
sequence.
# sudo powercycleneedex500cold2none
For EX300:
2. Before proceeding to next step, wait for the node to boot and come online, and reopen
the SSH session using Putty to 192.168.219.1.
22. Enable the rack install server for each rack to enable node 1 to PXE boot the other nodes.
Page 24 of 91
# sudo setrackinfo -p RackInstallServer yes
23. Verify that the rack install server was set to yes by running this command on each rack:
# getrackinfo -p RackInstallServer
If the result is no, for any rack repeat the previous step on the rack install server that returned
the no value.
24. Power on node 2. To image the ECS operating system using PXE boot (you always start with
node 2), wait for 30 seconds before powering on the next node so that all the commands have
been run on each node before continuing to the next node. Nodes 2–16 in the rack.
25. Wait for 30 seconds, and repeat step 24 for the next node until all the nodes are imaged.
26. Monitor the progress of ECS operating system imaging on the nodes.
# watch getrackinfo -i
Loading the operating system takes approximately 15 minutes per node. Sometimes it may take
longer for the IP column to update with the expected private IP addresses. Operating system
installation is complete once the Status is Done! as demonstrated.
27. Validate that the nodes have rebooted, and are up again.
Once the operating system has been loaded on the nodes, the nodes reboot. Rebooting the
nodes may take several minutes to complete. Use the following command to monitor the nodes,
and verify that they have been rebooted before continuing:
# getrackinfo
Sample output
Page 25 of 91
192.168.219.3 3 SA 24:6e:96:d2:2a:1e 10.249.234.57
d0:94:66:60:a3:de 10.249.234.53 orem-beige
192.168.219.4 4 SA 24:6e:96:d5:7b:7c 10.249.234.58
d0:94:66:60:8c:b1 10.249.234.54 ogden-beige
28. Create a MACHINES file that can be used to perform distributed commands. The MACHINES file
contains the list of private IP addresses for each node in a single rack.
1. Change to the /home/admin directory:
# cd /home/admin
2. Create the MACHINES file in the /home/admin directory (this directory is the home
directory for the admin user).
# getrackinfo -c MACHINES
# cat MACHINES
You can also use the pingall command which pings all nodes in the MACHINES file:
# pingall
29. Verify that the ECS operating system version above the login prompt for each node displays the
expected version as follows:
Disabling prevents the rack master node from PXE booting the other nodes in the system. This
adds more security on top of the MAC address ignore list.
31. Verify the setting:
Page 26 of 91
# sudo getrackinfo -p RackInstallServer
1. If the above command returns cold or warm for any node, SSH to each of those nodes
and run:
NOTE:
cold requires the node to fully shut down and then power on, warm requires only a
reboot.
If EX500:
NOTE:
This command issues the required racadm command sequence to ensure that firmware is applied properly. It
may take 10 minutes for node to perform firmware reconfiguration and boot after above command is initiated.
On EX500, cold boot flag is automatically cleared as part of powercycleneedex500cold2none sequence.
# sudo powercycleneedex500cold2none
If EX3000 or EX5000:
Then, power on the node using power button, iDrac console, or using ipmitool
command form another node in the rack:
If EX300:
#sudo reboot
2. Repeat step a for each node which had cold or warm flag set.
3. After all nodes have been restarted properly, check for cold or warm boot flag again to
ensure all are none.
Page 27 of 91
Validate the switch configruation
This chapter includes the following information.
See ECS EX-Series Switch OS10 Matrix guide to determine latest supported Switch firmware version
that is required to be installed in this section.
1. Review the required items to perform the firmware update:
o Firmware package with .zip filename for example: ecs-os-firmware-dell-
<version>.zip
o ECS Firmware Update Tool bundled with above firmware .zip file: fabfit
2. Obtain the latest ecs-os-firmware-dell-<version>.zip file from:
ECS product Support site: https://www.dell.com/support/home/en-in/product-
support/product/ecs-appliance-gen-3/drivers
3. Use SCP or WinSCP to copy the ecs-os-firmware-dell-<version>.zip file to the
/home/admin directory on node 1 of the rack containing the switches that require firmware
update:
# scp ecs-os-firmware-<version>.zip
[email protected]:/home/admin
4. When prompted, enter the password for the ECS system admin user.
Output similar to the following displays:
5. If you are not connected to node 1 of the rack containing the switches that require firmware
update, then open an SSH session using Putty to 192.168.219.1.
6. If you are not already in a screen session, enter into a screen session.
NOTE:
The screen session (or program) allows you to use multiple windows (virtual VT100 terminals) in UNIX.
# screen -S install
# screen -list
Page 28 of 91
There is a screen on:
178214.install (Attached)
1 Socket in /var/run/screesn/S-admin.
# unset TMOUT
If you lose the local or remote connection, the login sessions you establish through the screen
session do not go away. You can resume your screen sessions with the following command:
# screen -r
# unzip ecs-os-firmware-dell-<version>.zip
Archive: ecs-os-firmware-dell-3.4.0.0-771.42787bf.99.zip
creating: 3.4.0.0-771.42787bf.99/
inflating: 3.4.0.0-771.42787bf.99/fabfit
inflating: 3.4.0.0-771.42787bf.99/ecs-os-setup.x86_64-3.4.0.0-
771.42787bf.99.firmware.tbz
inflating: 3.4.0.0-771.42787bf.99/VERSION.firmware
inflating: 3.4.0.0-771.42787bf.99/MD5SUMS.firmware
# cd 3.4.0.0-771.42787bf.99
9. Create a target node file that includes the IP address of node 1 only:
10. Run the fabfit command to install the firmware files on node 1 in the rack:
Page 29 of 91
20190909-194713 Info logging to /opt/emc/firmware.d/3.4.0.0-
771.42787bf.99/log/20190909-194713.push-status.log
20190909-194713 Info Checking checksums locally
20190909-194716 Info Distributing files to MACHINES list of nodes
20190909-194716 Info Make /opt/emc/firmware.d/3.4.0.0-771.42787bf.99 on MACHINES
list of nodes
20190909-194717 Info pushing /home/admin/3.4.0.0-771.42787bf.99/3.4.0.0-
771.42787bf.99
20190909-194730 Info Running fabfit on MACHINES list of nodes
192.168.219.1
========================================
20190909-194730 Info logging to /opt/emc/firmware.d/3.4.0.0-
771.42787bf.99/log/20190909-194730.local-status.log
20190909-194730 Info Obtaining lock...
20190909-194730 Info Obtained fabfit lock.
20190909-194730 Info Checking checksums of pushed files
20190909-194734 Info Expanding tarball in /opt/emc/firmware.d/3.4.0.0-
771.42787bf.99
20190909-195003 Info Upgrade: emc-dell-firmware-1.3-3484.5396c74d.x86_64.rpm emc-
dell-DELLOS_10_3_xx-1.3-3484.5396c74d.x86_64.rpm emc-dell-DELLOS_10_4_xx-1.3-
3484.5396c74d.x86_64.rpm
grep: in.txt: No such file or directory
Done: running /opt/emc/bin/domulti -f /home/admin/FIRMWARE
/opt/emc/firmware.d/3.4.0.0-771.42787bf.99/fabfit install
emc-dell-firmware-1.3-3484.5396c74d.x86_64
emc-dell-DELLOS_10_3_xx-1.3-3484.5396c74d.x86_64
emc-dell-DELLOS_10_4_xx-1.3-3484.5396c74d.x86_64
# ls /usr/share/emc-dell-firmware/firmware
Page 30 of 91
NOTE:
When firmware is extracted, the previous versions of firmware and the versions that are supported for other
hardware platforms are also listed.
# ls /usr/share/emc-dell-firmware/firmware/latest/
PKGS_OS10-Enterprise-<version>-installer-x86_64.bin
NOTE:
The fabfit tool is designed to Autodefect hardware platform and place the correct firmware package in the
latest directory that is based on the switch model that is supported for the underlying hardware. See ECS EX-
Series Switch OS10 Matrix Guide to determine the latest supported Switch firmware version for target switch
models in your ECS rack, and confirm that the latest directory has the correct version.
12. Clean up the firmware installation /tmp files:
# cd /home/admin
14. Update switch firmware on all Dell switches installed in the rack using one updateDell execution.
In ECS 3.4 and earlier versions:
# getrackinfo -c /home/admin/MACHINES
# sudo updateDell <switch IP address or name> -m /home/admin/MACHINES
NOTE:
Enter multiple entries by leaving a space in between IPs or names. UpdateDell utility updates and reboots one switch at
a time when multiple switches are provided.
When you update multiple switches, the questions and output vary as compared to when one
switch at a time is changed:
o Type Y, when asked for single password for all switches.
Page 31 of 91
o Enter password when asked.
o Type Y, when asked if you would like to update the firmware on each switch.
o Type Y, when asked to restart each switch.
For example,
Page 32 of 91
Starting download of running-configuration
Download to running-configuration.fox.rack.2021-09-23.0856.txt completed.
Download to running-configuration.hare.rack.2021-09-23.0856.txt completed.
Download to running-configuration.hound.rack.2021-09-23.0856.txt completed.
Download to running-configuration.rabbit.rack.2021-09-23.0856.txt completed.
Starting download of license
Starting download of license
Starting download of license
Starting download of license
Download to license.hare.rack.2021-09-23.0856.lic completed.
Download to license.fox.rack.2021-09-23.0856.lic completed.
Download to license.hound.rack.2021-09-23.0856.lic completed.
Download to license.rabbit.rack.2021-09-23.0856.lic completed.
Transferring image to rabbit.rack.
Transferring image to hound.rack.
Transferring image to hare.rack.
Transferring image to fox.rack.
Image transfer completed on rabbit.rack.
Image transfer completed on hound.rack.
Image transfer completed on hare.rack.
Image transfer completed on fox.rack.
Image installing on rabbit.rack...
Image installing on hound.rack...
Image installing on hare.rack...
Image installing on fox.rack...
Install successfully completed on hound.rack.
Install successfully completed on fox.rack.
Install successfully completed on hare.rack.
Install successfully completed on rabbit.rack.
Upgrade completed on hare.rack. Please run updateDell -r if both switches in
pair are upgraded.
Upgrade completed on rabbit.rack. Please run updateDell -r if both switches in
pair are upgraded.
Upgrade completed on hound.rack. Please run updateDell -r if both switches in
pair are upgraded.
Upgrade completed on fox.rack. Please run updateDell -r if both switches in
pair are upgraded.
NOTE:
Page 33 of 91
Before proceeding, perform the steps from 1 to 13 described in the Update to the latest supported Dell
switch operating system version section and run the following commands against each switch pair in the
rack:
Run the following commands:
For BE switches:
Example output:
WARNING:
Service personnel MUST confirm that the customer has retained a copy of the switch configuration before leaving the site.
Be sure to review:
Page 34 of 91
Figure 1. Switch configuration for customer uplinks
The majority of uplink configurations are against vPC (multi-switch port-channel for Cisco). This
configuration will work out of the box, irrespective of the amount of uplink ports required from the
customer, and will also allow for later expansion without an ECS customer support representative. The
default configuration is not intended for network separation use cases. The configuration can however be
loaded as a base and then modified for network separation. For information about network separation
refer to the ECS Network Separation Guide, which is available in SolVe.
Determine whether to use the installation scripts created through ECS designer or install ECS manually
At this point you must decide whether you will use the information gathered for the ECS Designer to
install ECS manually, or if you will use the installation scripts created through ECS designer to automate
the installation process.
If you decided to use the installation scripts to automate the installation process, review the steps in the
email you received from ECS Designer. Refer to Obtain the ECS Automated Script for further details.
Once you have completed the procedures provided to you in the email you received from ECS Designer,
continue with the steps provided in Post Installation of this guide after the ECS software has been
installed.
Page 35 of 91
1. Verify the connection to the private network on the on the private (Fox) switch; port 34 on the
front of the switch, or 36, on the back of the switch. and that the service laptop is configured with
IP address 192.168.219.101.
2. SSH to Node 1 of the rack, 192.168.219.1. The default credentials are admin/ChangeMe.
3. Use getrackinfo to retrieve the current setting. The -a parameter ensures that all nodes in
the rack are included in the command output.
# getrackinfo -a
Locate the Node Name column in the output to view the color currently assigned to the rack.
For example:
Node Name
provo-clear
sandy-clear
orem-clear
ogden-clear
4. If the color is set to clear or to the wrong color, then set it by using setrackinfo passing in
the color by index (using the -i parameter), or the name (by passing the -n parameter). For
example, if you are installing the first rack in the data center, the index is 1 and the name is red.
You can use either command:
# sudo setrackinfo -i 1
OR
See Rack IDs and default node hostnames for a complete list.
If you are using installation scripts to automate the installation process, the static IPs are automatically
configured by the scripts, and you can skip the following steps.
Page 36 of 91
Separating networks
If you are planning to separate any one of the ECS networks (mgmt, repl, data) from the public networks:
in preparation of network separation refer to the:
Configure DNS
1. Use setrackinfo -a to configure the DNS server. Repeat the command for each DNS server
at the customer site.
2. Use setrackinfo -a to configure the DNS search. You must run the command once for each
specification as shown in the following sample format:
In addition, the order (or sequence) is very important when running this command. You must run
the command showing the more specific domain first for example: corp.customer.com) and
the less specific domain last (like customer.com) as shown in our example below.
Configure NTP
Use this procedure to add NTP servers.
NOTE:
Run getrackinfo -p NTPServer command to check if there are any existing NTP servers and add the correct server by
following the procedure that is given below.
1. Run setrackinfo -a once for each NTP server you would like to add:
# sudo getrackinfo -r
Page 37 of 91
Adjust system time after configuring NTP servers
ECS 3.4 and later honors the NTP panic threshold which has a default 1000 seconds. File modification
times are used for file cleanups and large adjustments may cause unplanned cleanups.
ECS 3.4 and later will ignore NTP servers exceeding panic threshold until the time is set on nodes to be
within panic threshold and services are restarted. Therefore, when enabling NTP, the time may have to
be manually adjusted on all nodes for nodes to accept NTP servers. To adjust the system time perform
the following steps.
1. Stop NTP services on all nodes
2. Set time on nodes to time on NTP servers and ignore panic threshold for one time only
3. Verify the time and date that were set on the nodes is correct with "date" command.
4. Start NTP services on all nodes.
NOTE:
This is applicable only to ECS 3.4 and ECS 3.5 releases.
getrackinfo -a
2. Use setrackinfo --static-ip-set to set the static IP addresses for the Public interface of
ALL nodes that are attached to the same back end (fox or hound) switch as node 1.
1. Run the command on the master node:
For example:
Page 38 of 91
# sudo setrackinfo --static-ip-set 1 192.0.2.10 255.255.255.0 192.0.2.24
2. Wait until the ifcfg file is updated with the static IP information.
Run the following command to check:
cat /etc/sysconfig/network/ifcfg-public
Example Output
BONDING_MASTER=yes
BONDING_MODULE_OPTS="miimon=100 mode=4
xmit_hash_policy=layer3+4"
BONDING_SLAVE0=slave-0
BONDING_SLAVE1=slave-1
BOOTPROTO=static
IPADDR=192.0.2.10/24
NOTE:
Ensuring that the static information is updated is essential if the customer elected to set
up the optional network separation configuration during this deployment. Extra IP
addresses that are required for the separated networks will not be allocated correctly, if
the static IP has not been set. See the ECS Network Separation document which is
available in SolVe for details. For ECS release 3.1 and later, customers can hire Dell
EMC Services to return to the site to set up network separation.
3. Run the set the static IP addresses command in step 2a above for the remaining nodes.
4. Validate static configuration settings:
getrackinfo -s
The getrackinfo command can take some time (5-10 minutes) to show the updated status
with the IP addresses.
# getrackinfo --hostname-list
Page 39 of 91
Perform system checks
This chapter includes the following information.
Firmware update
Table 1. Firmware update
• If the operating system imaging process detects an older version of NIC, the minimum supported
version of NIC firmware is automatically applied.
• For this firmware to be properly applied, you may have to perform an extra power off and power on
sequence.
• Perform all steps described in earlier sections to ensure that proper power cycle sequence is
achieved, before proceeding.
• Follow the ECS EX-Series Firmware Matrix and Firmware Upgrade Guide, to ensure that latest
recommended versions of all specified components are at recommended level, before proceeding
to next steps.
• There are 2 Firmware bundles for EX -Series - Node Firmware and Disk Firmware, follow
Firmware Matrix and Firmware Upgrade Guide for instructions. The guides can be found under
How to Procedures in Solve. The Firmware bundles can be found under Tools and Downloads.
#getrackinfo
admin@nile-318-96a8299-69:~> getrackinfo
Node private Node Public BMC
Ip Address Id Status Mac Ip Address Mac
Ip Address Node Name
=============== ====== ====== ================= =================
================= ================= =========
192.168.219.1 1 MA!1 24:6e:96:96:cd:14 0.0.0.0
d0:94:66:25:7b:b4 192.168.219.101 provo-cyan
Page 40 of 91
192.168.219.2 2 SA!1 24:6e:96:bb:ff:e8 0.0.0.0
d0:94:66:3f:42:62 192.168.219.102 sandy-cyan
192.168.219.3 3 SA!1 24:6e:96:bb:f8:a8 0.0.0.0
d0:94:66:44:fe:a2 192.168.219.103 orem-cyan
192.168.219.4 4 SA!1 24:6e:96:bb:fc:c8 0.0.0.0
d0:94:66:45:4a:87 192.168.219.104 ogden-cyan
192.168.219.5 5 SA!1 24:6e:96:bc:0a:a8 0.0.0.0
d0:94:66:45:44:7c 192.168.219.105 layton-cyan
192.168.219.6 6 SA!1 24:6e:96:bc:1e:08 0.0.0.0
d0:94:66:45:48:10 192.168.219.106 logan-cyan
4. If outputs in the status column contain warnings ! as shown in the previous output, run:
admin@provo-gen3-cyan:~> getrackinfo
Node private Node Public BMC
Ip Address Id Status Mac Ip Address Mac
Ip Address Node Name
=============== ====== ====== ================= =================
================= ================= =========
192.168.219.1 1 MA 24:6e:96:96:cd:14 10.241.207.93
d0:94:66:25:7b:b4 192.168.219.101 provo-cyan
192.168.219.2 2 SA 24:6e:96:bb:ff:e8 10.241.207.94
d0:94:66:3f:42:62 192.168.219.102 sandy-cyan
192.168.219.3 3 SA 24:6e:96:bb:f8:a8 10.241.207.95
d0:94:66:44:fe:a2 192.168.219.103 orem-cyan
192.168.219.4 4 SA 24:6e:96:bb:fc:c8 10.241.207.96
d0:94:66:45:4a:87 192.168.219.104 ogden-cyan
192.168.219.5 5 SA 24:6e:96:bc:0a:a8 10.241.207.97
d0:94:66:45:44:7c 192.168.219.105 layton-cyan
192.168.219.6 6 SA 24:6e:96:bc:1e:08 10.241.207.98
d0:94:66:45:48:10 192.168.219.106 logan-cyan
Status:
For example:
Page 41 of 91
/dev/sda [15:0:0:0] host15 0:0 ahci non-rotational 480.1 ATA DL43
SSDSCKJB480G7R PHDW7406026J480B
7. Validate that only one control node exists, and that it is accessible by all nodes:
Page 42 of 91
PING 192.168.219.254 (192.168.219.254) 56(84) bytes of data.
64 bytes from 192.168.219.254: icmp_seq=1 ttl=64 time=0.098 ms
64 bytes from 192.168.219.254: icmp_seq=2 ttl=64 time=0.115 ms
getrackinfo -r
Master Info
================
ChassisID = 00:1c:73:fa:05:5b
RackID = 1
RackName = Red
epoxy = False
10. Validate that forward and reverse lookups are set by running each of the following commands:
Page 43 of 91
The first validates forward lookup, the second the reverse lookup. Do not proceed if it is not set
properly.
To validate forward lookup:
Name: sandy-red.corp.customer.com
Address: 10.249.249.212
Name: provo-red.corp.customer.com
Address: 10.249.249.211
Name: orem-red.corp.customer.com
Address: 10.249.249.213
Name: ogden-red.corp.customer.com
Address: 10.249.249.214
Page 44 of 91
211.249.249.12.in-addr.arpa name = sandy-red.corp.customer.com
11. Validate that NTP is set, and that the value in the Offset column is within 5 s for each node.
The offset value is in milliseconds, so a value of 1000 is 1 s.
1. If the value in the Offset column is greater than 5 s, restart the service:
2. Verify that the value in the Offset column is now within the five-second limit:
12. Validate the ignore list is populated correctly and includes the correct number of nodes present
in the racks you are installing:
Page 45 of 91
13. Validate the private.4 IPs and public interfaces are up and that IP addresses are assigned to
them.
1. Verify that the private.4 IP interface is up:
14. Verify the MD5checksum of the operating system image files is consistent across all nodes.
1. Run the following command:
Page 46 of 91
Output from host : 192.168.219.3
500c0e94aec365fd007d25f61ab753ad /srv/tftpboot/boot/initrd
7720bab96a0eef99491ea4a686dc38bc /srv/tftpboot/boot/linux
2. Run the following command, and check that the MD5checksums for each image is the
same on all nodes.
15. Verify that the bonding of interfaces is correct; there should be two slave nodes and one public
interface displayed:
Page 47 of 91
16. Verify that the bonding of interfaces is correct; there should be two slave nodes and one private
interface displayed:
1. If the bonding of the interfaces is not correct, restart the interfaces by running:
2. Validate the bonding and mode is now set up properly by repeating the two previous
steps.
18. Check the /boot volume is on the same '/dev/sdX' as the LVM 'ECS' volume group for
/root.
1. Command:
Page 48 of 91
/dev/sda1 597M 87M 511M 15% /boot
The integer following the 'X' in '/dev/sdX' must be different, '1' for /boot and '2' for
the ECS VG.
2. Command:
19. With the introduction of ECS operating system 3.2.2 and gen3 there is a third system partition to
assist with firmware updates, check fwflash is present with:
Page 49 of 91
/dev/sda3 400M 8.0K 400M 1% /opt/emc/fwflash
20. Check the platform information and capture it by running each of the following commands:
21. View the System event log Summary for all nodes:
# viprexec -i 'sudo -i ipmitool sel elist | cut -d"|" -f4- | sort | uniq -c'
The output of the command is used to look for CATERR or DIMM-related failures.
22. Run svc_network check available on xDoctor during Perform system checks. Alternately, you
can create MACHINE files for corresponding interfaces and collect the IPs then run it using
pingall.
Page 50 of 91
<All Nodes> <All Nodes> PASS PASS
admin@ecs03:~>
1. Set the iDRAC IP address using the BIOS Setup Utility, and install the latest ECS OS.
2. Validate the switch configuration.
3. Configure the Rack ID and Static IP addresses.
Multirack checks
Perform the following checks to verify interrack connectivity.
1. Create a file that contains the IP addresses of all nodes in the cluster.
When you use the viprexec command to run commands on multiple nodes, the MACHINES file
is used by default. However, it is possible to specify the name of a file that contains all nodes for
the whole cluster, not just a single rack. In this example, the file is called VDCLIST.
You can create it using the getclusterinfo command:
# getclusterinfo -a VDCLIST
Alternatively, you can create it manually by adding the private.4 IP addresses of the nodes:
# vi /home/admin/VDCLIST
The private.4 IP addresses are in the form 169.254.<rack id>.<node id>, for example:
169.254.1.1
169.254.1.2
169.254.1.3
169.254.1.4
169.254.2.1
169.254.2.2
169.254.2.3
169.254.2.4
# cat VDCLIST
Page 51 of 91
3. Check that you have connectivity to all nodes using the pingall command:
# pingall -f VDCLIST
Copy the required ECS Software installation files to the installer node
Use this procedure to install the ECS Software installation package on the installer node (the first node in
the first rack at the site).
1. SSH into the installer node (or node 1).
2. If you are performing a manual installation create /tmp/installer directory:
#mkdir /tmp/installer
# cd /tmp/installer
NOTE:
When selecting the software installation package, you should ensure that you do not select the ecs-<version>-
production.tgz package for countries that do not permit encryption. Only two countries require this file: Russia and
China.
5. Extract ecs-<version>-production.tgz. Or, extract ecs-<version>-production-no-
encryption.tgz (non-D@RE deployments) depending on your installation. For example:
Example Output
Page 52 of 91
docker/
docker/
docker/images/
docker/images/fabric-syslog.txz
docker/images/fabric-registry.txz
docker/images/nile-object.txz
docker/images/caspian-fabric.txz
docker/images/fabric-agent.txz
docker/images/fabric-zookeeper.txz
docker/images/fabric-proxy.txz
docker/images/utilities.txz
manifest.txt
rpms/
rpms/emc-lsi-storelibir-2-17.01-657.1b62e78.1.RHEL.x86_64.rpm
rpms/viprhal-1.5.1.0-1704.055352c.COREOS.x86_64.rpm
rpms/fabric-agent-1.4.0.0-3250.634e3bb-SLES.rpm
rpms/fabric-cli-1.4.0.0-3250.634e3bb-x86_64.rpm
rpms/fabric-agent-1.4.0.0-3250.634e3bb-RHEL.rpm
rpms/fabric-agent-1.4.0.0-3250.634e3bb-COREOS.rpm
rpms/emc-lsi-storelibir-2-17.01-657.1b62e78.1.COREOS.x86_64.rpm
rpms/viprhal-1.5.1.0-1704.055352c.SLES.x86_64.rpm
rpms/installer-1.4.0.0-3250.634e3bb.rpm
rpms/viprhal-1.5.1.0-1704.055352c.RHEL.x86_64.rpm
rpms/checker-1.4.0.0-3250.634e3bb-x86_64.rpm
run.sh
tars/
tars/viprhal-1.5.1.0-1704.055352c.COREOS.x86_64.tgz
tars/jre-8u121-linux-x64.tgz
# sudo ./run.sh
Example Output
installer-tool/conf/log4j.properties
installer-tool/lib/
installer-tool/lib/jackson-mapper-asl-1.9.13.jar
installer-tool/lib/jetty-http-9.3.8.v20160314.jar
...
Copy tars to installer ...
Copy tars done
Copy manifest to installer ...
Copy manifest done
All done. Please use installer /opt/emc/caspian/installer to start the installation
when ready
The ECS software installation files are available for the install.
Page 53 of 91
Run the Compatibility Checker (also known as pre-check tool)
A successful run of the Compatibility Checker tool is a prerequisite for installing ECS software, and must
run the same day as the ECS system deployment. Services personnel cannot proceed to create a
topology configuration file, or install the software until the checks in this section run without errors. The
workflow involves completing the following tasks:
1. Create a machines.txt file
2. Run the Compatibility Checker COLLECT operation
3. Run the PRECHECK and REPORT operation
See Compatibility Checker Guide, which is available in SolVe, for additional details.
# cd /opt/emc/caspian/checker
2. Create a machines.txt file that contains the IP addresses of all the nodes in the cluster:
NOTE:
If there is a multirack VDC, this command adds private.4 IP addresses from all nodes and racks in the VDC.
The following example shows the comments that are embedded in the template file:
cd /opt/emc/caspian/checker
sudo vi conf/machines.txt
cat conf/machines.txt
# This file describes the format of the machines file which is to
# be used by the checker. The machines file contains information
# on the IP v4 addresses of the cluster nodes.
#
# Each line of machines file may contain IP v4 address or range
# of IP v4 addresses.
#
# For example:
# 192.168.1.1
# 192.168.100.1-192.168.100.10
#
Page 54 of 91
# Lines beginning with a hash '#' are considered comments.
#
169.254.1.1
169.254.1.2
169.254.1.3
169.254.1.4
Run the Compatibility Checker with the COLLECT operation and your machines.txt file.
#cd /opt/emc/caspian/checker
#sudo bin/checker --operation COLLECT --machines conf/machines.txt
The operation creates a logfile and HAL inventory file for every node in the
/opt/emc/caspian/checker/log/machines.txt file.
Output example:
+-
+-Execute pre-check script
|-[169.254.1.1] ExecutePrecheck PASS
|-[169.254.1.2] ExecutePrecheck PASS
|-[169.254.1.3] ExecutePrecheck PASS
+-[169.254.1.4] ExecutePrecheck PASS
Page 55 of 91
# sudo bin/checker --operation REPORT --machines conf/machines.txt
Test Results:
Number of nodes with status PASSED: 1.
Number of nodes with status WARNING: 0.
Number of nodes with status FAILED: 4.
HTML report put into /opt/emc/caspian/checker/results/report_2018-08-06_17-33-27
Summary TXT report put into /opt/emc/caspian/checker/results/report_2018-08-06_1
Full TXT report put into /opt/emc/caspian/checker/results/report_2018-08-06_17-3
When an issue is detected, you can go to the Failed tests to determine the reason for the failure.
For detailed steps, see the Compatibility Checker Guide, which is available in SolVe.
NOTE:
You must type (or copy and paste) all multi-line commands as a single line of text with no breaks.
1. On Node 1 of the first rack, copy the topology template file located at
/opt/emc/caspian/installer/conf/topology.txt to a local directory. For example:
# mkdir /var/tmp/installer
# getrackinfo -g nan -o ip
Example Output:
Page 56 of 91
getrackinfo -g nan -o ip
169.254.1.1
169.254.1.2
169.254.1.3
169.254.1.4
NOTE:
If installing more than one rack make sure to run step 2. on all racks to collect the private.4 IPs.
3. Open the topology file copy for editing. For example:
# sudo vi /var/tmp/installer/topology
4. Modify the topology file entries to represent what you are installing, and ensure the file contains
the correct addresses in step 2.
There is one entry for each node you are installing. Each node is described on its own line. for
example:
# ID,HOSTNAME,ADDRESS,LOCATION,RACK,PSNT,SHELF
Table 1. Options and Description The table lists the option and its description details
Option Description
ID The rack number/node number (r#n#) that the line describes. For example, r1n1 identifies
the first node in the first rack; r2n3 identifies the third node in the second rack.
NOTE:
You cannot insert the node's short name.
RACK ID The RACK ID is very important because it identifies racks during SKU distribution. Each
rack MUST list a unique RACK ID in the topology.txt file. Additionally, the RACK ID
MUST always correspond to the rack value in the ID. For example:
17.If ID is r1nX, RACK ID should be rack1
18.If ID is r4nX, RACK ID should be rack4
PSNT The PSNT of the rack (r1, r2, r3...) in which the nodes. If multiple racks, there will be a
unique PSNT for each rack (each node in the rack will have same PSNT). It is important to
Page 57 of 91
enter the correct PSNT value in the topology file prior to installing the ECS software. The
PSNT value entered here is used by ESRS to identify the system. If no value, or an
incorrect value is entered in the topology file then you will not be able to use ESRS to
identify the issues with the system.
SHELF A string that identifies the shelf. This value is not currently used by ECS. For simplicity, you
can use the value specified in the template.
The following example shows a topology configuration file for a four-node cluster.
# ID,HOSTNAME,ADDRESS,LOCATION,RACK,SHELF
r1n1,layton-red.sea.corp.company.com,169.254.1.1,Cambridge,rack1,psnt1,shelf1
r1n2,logan-red.sea.corp.company.com,169.254.1.2,Cambridge,rack1,psnt1,shelf1
r1n3,lehi-red.sea.corp.company.com,169.254.1.3,Cambridge,rack1,psnt1,shelf1
r1n4,murray-red.sea.corp.company.com,169.254.1.4,Cambridge,rack1,psnt1,shelf1
If the cluster comprises two racks, the nodes of the additional racks must also be added to the
topology file. In this case, second rack containing 4 nodes is added.
# ID,HOSTNAME,ADDRESS,LOCATION,RACK,SHELF
r1n1,layton-red.sea.corp.company.com,169.254.1.1,Cambridge,rack1,psnt1,shelf1
r1n2,logan-red.sea.corp.company.com,169.254.1.2,Cambridge,rack1,psnt1,shelf1
r1n3,lehi-red.sea.corp.company.com,169.254.1.3,Cambridge,rack1,psnt1,shelf1
r1n4,murray-red.sea.corp.company.com,169.254.1.4,Cambridge,rack1,psnt1,shelf1
r2n1,layton-green.sea.corp.company.com,169.254.2.1,Cambridge,rack2,psnt2,shelf1
r2n2,logan-green.sea.corp.company.com,169.254.2.2,Cambridge,rack2,psnt2,shelf1
r2n3,lehi-green.sea.corp.company.com,169.254.2.3,Cambridge,rack2,psnt2,shelf1
r2n4,murray-green.sea.corp.company.com,169.254.2.4,Cambridge,rack2,psnt2,shelf1
• The auto-discovery commands fail if the topology file is not populated with the customer-specific
node and rack details.
• The auto-generated SKU displays during the install process (see output below).
19.If you are adding a custom brand to ECS you must complete the instructions provided in Install
ECS with a custom brand before continuing with the following steps.
20.If not already in a screen session, enter into one.
NOTE:
The screen session (or program) allows you to use multiple windows (virtual VT100 terminals) in Unix.
# screen -S install
Page 58 of 91
To verify if you are in a screen session:
screen -list
If you lose the local or remote connection, the login sessions you establish through the screen
session don't go away. You can resume your screen sessions with the following command:
# screen -r
# cd /opt/emc/caspian/installer
Example Output
Initializing...
######################################################################################
######################################################################################
#####################################
# Automatically generated SKU is shown below.
[__AUTO_SKU__]
fabric_installer = [r1n1]
fabric_registry = [r1n1]
fabric_lifecycle = [r1n1,r1n2,r1n3]
fabric_zookeeper = [r1n1:1,r1n2:2,r1n3:3]
object_main = [r1n1,r1n2,r1n3,r1n4]
######################################################################################
######################################################################################
#####################################
|-Install HAL
| |-Check and Install JRE and Installer Tool
| | |-Check Java Runtime environment
| | | |-[169.254.92.8] JavaRuntimeHealth PASS (1/0)
| | | |-[169.254.92.6] JavaRuntimeHealth PASS (2/0)
| | | |-[169.254.92.5] JavaRuntimeHealth PASS (3/0)
Page 59 of 91
| | | +-[169.254.92.7] JavaRuntimeHealth PASS (4/0, 1 sec)
| | +-Check Installer tool
| | |-[169.254.92.5] InstallerToolHealth PASS (5/0)
| | |-[169.254.92.8] InstallerToolHealth PASS (6/0, 1 sec)
| | |-[169.254.92.7] InstallerToolHealth PASS (7/0, 2 sec)
| | +-[169.254.92.6] InstallerToolHealth PASS (8/0, 2 sec)
| |-Check HAL Rpm dependencies
| | |-[169.254.92.8] CheckDependencies PASS (9/0, 1 sec)
| | |-[169.254.92.5] CheckDependencies PASS (10/0, 1 sec)
| | |-[169.254.92.6] CheckDependencies PASS (11/0, 1 sec)
| | +-[169.254.92.7] CheckDependencies PASS (12/0, 1 sec)
| |-Install HAL Rpm
| | |-[169.254.92.8] Install PASS (13/0, 15
sec)
| | |-[169.254.92.7] Install PASS (14/0, 15
sec)
| | |-[169.254.92.6] Install PASS (15/0, 15
sec)
| | +-[169.254.92.5] Install PASS (16/0, 15
sec)
| +-Configure HAL
| |-[169.254.92.7] ConfigureHAL PASS (17/0, 2 sec)
| |-[169.254.92.6] ConfigureHAL PASS (18/0, 2 sec)
| |-[169.254.92.5] ConfigureHAL PASS (19/0, 2 sec)
| +-[169.254.92.8] ConfigureHAL PASS (20/0, 3 sec)
|-Install Fabric
| |-Install Validation Phase
| | |-Validators
| | | |-[0.0.0.0] ValidateTopology PASS (21/0)
| | | |-[0.0.0.0] ValidateRegisteries PASS (22/0)
| | | |-[0.0.0.0] ValidateOddNodes PASS (23/0)
| | | |-[0.0.0.0] ValidateOddNodes PASS (24/0)
| | | +-[0.0.0.0] ValidateExtendedNodes PASS (25/0)
| | |-Check docker health
| | | |-[169.254.92.7] DockerHealth PASS (26/0)
| | | |-[169.254.92.6] DockerHealth PASS (27/0)
| | | |-[169.254.92.5] DockerHealth PASS (28/0)
| | | +-[169.254.92.8] DockerHealth PASS (29/0)
| | +-Check custom registries
| |-Install Agent
| | |-Check and Install JRE and Installer Tool
| | | |-Check Java Runtime environment
| | | | |-[169.254.92.6] JavaRuntimeHealth PASS (30/0)
| | | | |-[169.254.92.5] JavaRuntimeHealth PASS (31/0)
| | | | |-[169.254.92.7] JavaRuntimeHealth PASS (32/0)
| | | | +-[169.254.92.8] JavaRuntimeHealth PASS (33/0, 1 sec)
| | | +-Check Installer tool
| | | |-[169.254.92.8] InstallerToolHealth PASS (34/0)
| | | |-[169.254.92.6] InstallerToolHealth PASS (35/0, 1 sec)
| | | |-[169.254.92.5] InstallerToolHealth PASS (36/0, 1 sec)
| | | +-[169.254.92.7] InstallerToolHealth PASS (37/0, 2 sec)
| | |-Uninstall Fabric
Page 60 of 91
| | | |-Uninstall Agent Service
| | | | |-[169.254.92.7] Uninstall PASS (38/0, 1 sec)
| | | | |-[169.254.92.8] Uninstall PASS (39/0, 1 sec)
| | | | |-[169.254.92.6] Uninstall PASS (40/0, 1 sec)
| | | | +-[169.254.92.5] Uninstall PASS (41/0, 1 sec)
| | | |-Remove Fabric Services
| | | | |-Remove Registry
| | | | | +-[169.254.92.5] NukeService PASS (42/0)
| | | | |-Remove LM
| | | | | |-[169.254.92.7] NukeService PASS (43/0)
| | | | | |-[169.254.92.5] NukeService PASS (44/0)
| | | | | +-[169.254.92.6] NukeService PASS (45/0)
| | | | |-Remove ZooKeeper
| | | | | |-[169.254.92.7] NukeService PASS (46/0)
| | | | | |-[169.254.92.5] NukeService PASS (47/0)
| | | | | +-[169.254.92.6] NukeService PASS (48/0)
| | | | +-Remove Syslog
| | | | |-[169.254.92.5] NukeService PASS (49/0)
| | | | |-[169.254.92.7] NukeService PASS (50/0)
| | | | |-[169.254.92.8] NukeService PASS (51/0)
| | | | +-[169.254.92.6] NukeService PASS (52/0)
| | | |-Cleanup FabricImages
| | | | |-Remove Registry Image
| | | | | |-[169.254.92.7] RemoveImage 'registry' PASS (53/0)
| | | | | |-[169.254.92.6] RemoveImage 'registry' PASS (54/0)
| | | | | |-[169.254.92.5] RemoveImage 'registry' PASS (55/0)
| | | | | +-[169.254.92.8] RemoveImage 'registry' PASS (56/0)
| | | | |-Remove LM Image
| | | | | |-[169.254.92.7] RemoveImage 'fabric' PASS (57/0)
| | | | | |-[169.254.92.8] RemoveImage 'fabric' PASS (58/0)
| | | | | |-[169.254.92.5] RemoveImage 'fabric' PASS (59/0)
| | | | | +-[169.254.92.6] RemoveImage 'fabric' PASS (60/0)
| | | | |-Remove ZooKeeper Image
| | | | | |-[169.254.92.8] RemoveImage 'zookeeper' PASS (61/0)
| | | | | |-[169.254.92.6] RemoveImage 'zookeeper' PASS (62/0)
| | | | | |-[169.254.92.7] RemoveImage 'zookeeper' PASS (63/0)
| | | | | +-[169.254.92.5] RemoveImage 'zookeeper' PASS (64/0)
| | | | +-Remove Syslog Image
| | | | |-[169.254.92.7] RemoveImage 'syslog' PASS (65/0)
| | | | |-[169.254.92.5] RemoveImage 'syslog' PASS (66/0)
| | | | |-[169.254.92.6] RemoveImage 'syslog' PASS (67/0)
| | | | +-[169.254.92.8] RemoveImage 'syslog' PASS (68/0)
| | | |-Cleanup Fabric Services Data
| | | | |-Remove Registry Data
| | | | | +-[169.254.92.5] RemoveServiceData PASS (69/0)
| | | | |-Remove LM Data
| | | | | |-[169.254.92.5] RemoveServiceData PASS (70/0)
| | | | | |-[169.254.92.7] RemoveServiceData PASS (71/0)
| | | | | +-[169.254.92.6] RemoveServiceData PASS (72/0)
| | | | +-Remove ZooKeeper Data
| | | | |-[169.254.92.5] RemoveServiceData PASS (73/0)
| | | | |-[169.254.92.7] RemoveServiceData PASS (74/0)
Page 61 of 91
| | | | +-[169.254.92.6] RemoveServiceData PASS (75/0)
| | | |-Uninstall FCLI Rpm
| | | | |-[169.254.92.8] Uninstall PASS (76/0)
| | | | |-[169.254.92.7] Uninstall PASS (77/0)
| | | | |-[169.254.92.5] Uninstall PASS (78/0)
| | | | +-[169.254.92.6] Uninstall PASS (79/0)
| | | |-Cleanup FCLI Data
| | | | |-[169.254.92.6] RemoveServiceData PASS (80/0)
| | | | |-[169.254.92.8] RemoveServiceData PASS (81/0)
| | | | |-[169.254.92.7] RemoveServiceData PASS (82/0)
| | | | +-[169.254.92.5] RemoveServiceData PASS (83/0)
| | | +-Cleanup Agent Data
| | | |-[169.254.92.8] RemoveServiceData PASS (84/0)
| | | |-[169.254.92.7] RemoveServiceData PASS (85/0)
| | | |-[169.254.92.5] RemoveServiceData PASS (86/0)
| | | +-[169.254.92.6] RemoveServiceData PASS (87/0)
| | |-Check Agent Rpm Dependencies
| | | |-[169.254.92.8] CheckDependencies PASS (88/0, 1 sec)
| | | |-[169.254.92.6] CheckDependencies PASS (89/0, 1 sec)
| | | |-[169.254.92.7] CheckDependencies PASS (90/0, 1 sec)
| | | +-[169.254.92.5] CheckDependencies PASS (91/0, 1 sec)
| | |-Check Fabric CLI Rpm Dependencies
| | | |-[169.254.92.7] CheckDependencies PASS (92/0, 1 sec)
| | | |-[169.254.92.8] CheckDependencies PASS (93/0, 1 sec)
| | | |-[169.254.92.5] CheckDependencies PASS (94/0, 1 sec)
| | | +-[169.254.92.6] CheckDependencies PASS (95/0, 1 sec)
| | |-Configure Agent
| | | |-[169.254.92.5] PrepareAgentConfiguration PASS (96/0, 14
sec)
| | | |-[169.254.92.6] PrepareAgentConfiguration PASS (97/0)
| | | |-[169.254.92.7] PrepareAgentConfiguration PASS (98/0)
| | | +-[169.254.92.8] PrepareAgentConfiguration PASS (99/0)
| | |-Install Agent Service
| | | |-[169.254.92.8] Install PASS (100/0, 25
sec)
| | | |-[169.254.92.5] Install PASS (101/0, 25
sec)
| | | |-[169.254.92.7] Install PASS (102/0, 25
sec)
| | | +-[169.254.92.6] Install PASS (103/0, 25
sec)
| | |-Install Fabric CLI Rpm
| | | |-[169.254.92.6] Install PASS (104/0, 14
sec)
| | | |-[169.254.92.7] Install PASS (105/0, 14
sec)
| | | |-[169.254.92.5] Install PASS (106/0, 14
sec)
| | | +-[169.254.92.8] Install PASS (107/0, 14
sec)
| | |-Setup Installation SSL
Page 62 of 91
| | | +-[169.254.92.5] SetupInstallationSsl PASS (108/0, 48
sec)
| | |-Verify Agent Health
| | | |-[169.254.92.5] AgentHealth PASS (109/0, 1
sec)
| | | |-[169.254.92.7] AgentHealth PASS (110/0, 1
sec)
| | | |-[169.254.92.6] AgentHealth PASS (111/0, 1
sec)
| | | +-[169.254.92.8] AgentHealth PASS (112/0, 1
sec)
| | |-Validate Disks
INFO: [169.254.92.8] Number of found disks is 12
INFO: [169.254.92.6] Number of found disks is 12
| | | |-[169.254.92.8] StorageHealth PASS (113/0)
| | | |-[169.254.92.6] StorageHealth PASS (114/0)
INFO: [169.254.92.5] Number of found disks is 12
| | | |-[169.254.92.5] StorageHealth PASS (115/0)
INFO: [169.254.92.7] Number of found disks is 12
| | | +-[169.254.92.7] StorageHealth PASS (116/0, 1
sec)
| | +-Emit Topology Info
| | |-[169.254.92.7] EmitTopologyInfo PASS (117/0)
| | |-[169.254.92.6] EmitTopologyInfo PASS (118/0)
| | |-[169.254.92.5] EmitTopologyInfo PASS (119/0)
| | +-[169.254.92.8] EmitTopologyInfo PASS (120/0)
| |-Install Registries
| | |-Install Head Registry
| | | |-[169.254.92.5] LoadImage 'registry' PASS (121/0, 8
sec)
| | | +-[169.254.92.5] InstallUpgradeService PASS (122/0, 11
sec)
| | |-Push Fabric Images to Head Registry
| | | |-[169.254.92.5] PushImage 'registry' PASS (123/0, 15
sec)
| | | |-[169.254.92.5] LoadImage 'zookeeper' PASS (124/0, 18
sec)
| | | |-[169.254.92.5] PushImage 'zookeeper' PASS (125/0, 16
sec)
| | | |-[169.254.92.5] LoadImage 'fabric' PASS (126/0, 18
sec)
| | | |-[169.254.92.5] PushImage 'fabric' PASS (127/0, 2
sec)
| | | |-[169.254.92.5] LoadImage 'syslog' PASS (128/0, 15
sec)
| | | +-[169.254.92.5] PushImage 'syslog' PASS (129/0, 1
sec)
| | |-Pull Registry Image
| | |-Install Remote Registries
| | +-Push Fabric Images to Remote Registries
| |-Push Images to Custom Registries
| | |-Load Images
Page 63 of 91
| | +-Push Images
| |-Install Zookeeper
| | |-Pull Zookeeper Image
| | | |-[169.254.92.5] PullImage 'zookeeper' PASS (130/0)
| | | |-[169.254.92.7] PullImage 'zookeeper' PASS (131/0, 8
sec)
| | | +-[169.254.92.6] PullImage 'zookeeper' PASS (132/0, 8
sec)
| | +-Install Zookeeper Container
| | |-[169.254.92.5] InstallUpgradeService PASS (133/0, 9
sec)
| | |-[169.254.92.7] InstallUpgradeService PASS (134/0, 9
sec)
| | +-[169.254.92.6] InstallUpgradeService PASS (135/0, 9
sec)
| |-Install Fabric Container
| | |-[0.0.0.0] GenerateClusterPassword PASS (136/0, 2
sec)
| | |-Pull Fabric Image
| | | |-[169.254.92.5] PullImage 'fabric' PASS (137/0)
| | | |-[169.254.92.7] PullImage 'fabric' PASS (138/0, 2
sec)
| | | +-[169.254.92.6] PullImage 'fabric' PASS (139/0, 2
sec)
| | |-Install Fabric Container
| | | |-[169.254.92.7] InstallUpgradeService PASS (140/0, 9
sec)
| | | |-[169.254.92.6] InstallUpgradeService PASS (141/0, 9
sec)
| | | +-[169.254.92.5] InstallUpgradeService PASS (142/0, 9
sec)
| | |-[0.0.0.0] CreateClusterRequest PASS (143/0, 30
sec)
| | |-[0.0.0.0] JoinClusterRequest PASS (144/0, 2
sec)
| | |-Verify Lifecycle Health
| | | |-[169.254.92.5] LifecycleHealth PASS (145/0)
| | | |-[169.254.92.6] LifecycleHealth PASS (146/0)
| | | +-[169.254.92.7] LifecycleHealth PASS (147/0)
| | |-[0.0.0.0] PutRootCertIntoLM PASS (148/0)
| | |-[0.0.0.0] AddAgentRequest PASS (149/0, 4
sec)
| | +-[0.0.0.0] CustomerSyrInfo PASS (150/0)
| |-Install Syslog
| | |-Pull Syslog Image
| | | |-[169.254.92.5] PullImage 'syslog' PASS (151/0)
| | | |-[169.254.92.6] PullImage 'syslog' PASS (152/0, 1
sec)
| | | |-[169.254.92.7] PullImage 'syslog' PASS (153/0, 1
sec)
| | | +-[169.254.92.8] PullImage 'syslog' PASS (154/0, 8
sec)
Page 64 of 91
| | +-Install Syslog Container
| | |-[169.254.92.5] InstallUpgradeService PASS (155/0)
| | |-[169.254.92.7] InstallUpgradeService PASS (156/0)
| | |-[169.254.92.6] InstallUpgradeService PASS (157/0)
| | +-[169.254.92.8] InstallUpgradeService PASS (158/0)
| |-Install Fabric Application Definition
| | |-[0.0.0.0] InstallApplication PASS (159/0)
| | |-[0.0.0.0] InstallApplication PASS (160/0)
| | |-[0.0.0.0] InstallApplication PASS (161/0)
| | +-[0.0.0.0] InstallApplication PASS (162/0)
| +-Update Fabric Ownership To Lifecycle Manager
| |-Update Ownership For Registry
| | +-[169.254.92.5] UpdateOwnership PASS (163/0)
| |-Update Ownership For Zookeeper
| | |-[169.254.92.5] UpdateOwnership PASS (164/0)
| | |-[169.254.92.7] UpdateOwnership PASS (165/0)
| | +-[169.254.92.6] UpdateOwnership PASS (166/0)
| |-Update Ownership For Lifecycle Manager
| | |-[169.254.92.5] UpdateOwnership PASS (167/0)
| | |-[169.254.92.7] UpdateOwnership PASS (168/0)
| | +-[169.254.92.6] UpdateOwnership PASS (169/0)
| +-Update Ownership For Syslog
| |-[169.254.92.7] UpdateOwnership PASS (170/0)
| |-[169.254.92.6] UpdateOwnership PASS (171/0)
| |-[169.254.92.5] UpdateOwnership PASS (172/0)
| +-[169.254.92.8] UpdateOwnership PASS (173/0)
+-Install Services
|-Push Services Images to Head Registry
| |-[169.254.92.5] LoadImage 'object' PASS (174/0, 1 min
27 sec)
| |-[169.254.92.5] PushImage 'object' PASS (175/0, 45
sec)
|-Push Services Images to Custom Registries
|-Pull Services Images from Head Registry
|-Push Services Images to Remote Registries
|-Install Services
| |-[0.0.0.0] InstallApplication PASS (176/0, 7
sec)
|-Set Services Operational Mode To Normal
| |-[0.0.0.0] SetApplicationRoleOperationalMode PASS (177/0)
+-Post-check Services Health
|-Validate Object Service
| |-[169.254.92.5] ServiceHealth PASS (178/0, 21
sec)
| |-[169.254.92.8] ServiceHealth PASS (179/0, 1 min
4 sec)
| |-[169.254.92.7] ServiceHealth PASS (180/0, 1 min
4 sec)
| +-[169.254.92.6] ServiceHealth PASS (181/0, 1 min
5 sec)
Elapsed time is 9 min .
Page 65 of 91
INSTALL_ALL completed successfully
Information has been written to the
"/opt/emc/caspian/installer/discover/INSTALL_ALL.2018-07-16T22:48:25/provisioning.txt"
Information has been written to the
"/opt/emc/caspian/installer/discover/INSTALL_ALL.2018-07-16T22:48:25/topology.txt"
Post-installation
This part includes the following chapters.
# cat /opt/emc/config/local/cluster.ini
NOTE:
The Health_Check command is case sensitive.
The following shows an example of a Service Console health check with no errors:
ECS
==============================================================================
ECS.Service Procedures
==============================================================================
20161230 22:36:34.267: Service Procedures Setup
20161230 22:36:38.832: | PASS
Fabric Health Check :: Check of an ECS Fabric health. Failure of t...
20161230 22:36:38.834: Fabric Health Check
20161230 22:36:38.835: | Run Keywords Independently
Page 66 of 91
20161230 22:36:38.836: | | Persist installer configuration information on all
nodes in the system
20161230 22:36:40.725: | | | PASS
20161230 22:36:40.726: | | Get additional cluster information and validate fabric-
agent status
20161230 22:38:52.369: | | | PASS
20161230 22:38:52.370: | | Validate that OS version is equal between nodes
...
ECS | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Debug: /opt/emc/caspian/service-
console/log/run_Health_Check_2017_03_23_11_23_20/dbg_robot.log
Output: /opt/emc/caspian/service-
console/log/run_Health_Check_2017_03_23_11_23_20/output.xml
Log: /opt/emc/caspian/service-
console/log/run_Health_Check_2017_03_23_11_23_20/log.html
Report: /opt/emc/caspian/service-
console/log/run_Health_Check_2017_03_23_11_23_20/report.html
6. Alternatively, if errors display in the output, contact Technical Support regarding next steps.
If you are running the Service Console health check prior to creating a VDC, you will receive the
following warning messages in your results. You can ignore the VDC error messages if you have
not created a VDC.
[WARN] Failed to get list of vdcs. Please make sure that the system is provisioned
20180326 21:06:34.519: | | | | | PASS
20180326 21:06:34.521: | | | | Check Object version across the cluster
[WARN] Failed to get list of vdcs. Please make sure that the system is provisioned
20180326 21:06:36.954: | | | | | PASS (2 sec)
20180326 21:06:36.956: | | | | Validate that all nodes aren't in Maintenance Mode
(SSM)
[WARN] Failed to get list of vdcs. Please make sure that the system is provisioned
Page 67 of 91
Run xDoctor checks
Run system-level diagnostic checks using xDoctor.
1. Validate that all nodes are running the same version of xDoctor:
NOTE:
You must upgrade if the output shows that the installed xDoctor version is not the same on all nodes (or not the latest
version). See ECS xDoctor User's Guide available in ECS SolVe Desktop for complete details.
2. Log into the current Rack master node:
# ssh master.rack
3. Launch xDoctor. The default (Full Diagnosis Suite) tests run on the entire system:
# sudo -i xdoctor
4. Locate the Session Report from the xdoctor output for example:
5. Run the session report to display only "Warning, Error, and Critical" events:
Output sample
Page 68 of 91
Extra = Threshold=[5 days, 0:00:00], Node:Value=[169.254.4.2:3 days, 17:50:33]
Timestamp = 2018-09-21_145914
Category = DNS
Source = Config
Severity = WARNING
Message = The following NameServers are not accessible
Extra = 10.241.172.60
Timestamp = 2018-09-21_145914
Category = health
Source = fcli
Severity = WARNING
Message = Fabric indicates a non-healthy disk
Extra =
169.254.4.1,169.254.4.1,169.254.4.3,169.254.4.3,169.254.4.3,169.254.4.3,169.254.4.2,16
9.254.4.2,169.254.4.2,169.254.4.2,169.254.4.2
Dell EMC Secure Remote Services (SRS), RAP069, error messages are returned until SRS has
been configured. Once you have configured SRS, you must restart xDoctor as described in:
Restart xDoctor after configuring SRS.
WARNING:
If the report returns any Warning, Errors, or Critical issues, other than SRS issues, you must resolve those issues before
continuing.
6. Log out from the master node:
# logout
SNMP is configured from the ECS Portal. For steps to enable SNMP query support refer to Enable
SNMP query support. For additional information on how to configure SNMP traps refer to the ECS
Administration Guide.
Page 69 of 91
The Compatibility Checker removes all older utility images automatically when you deploy a new utility
image.
Run the Compatibility Checker with the CLEANUP operation. Specify your machines.txt file.
#cd /opt/emc/caspian/checker
# sudo bin/checker --operation CLEANUP --machines conf/machines.txt
Initialize...
+-
+-Cleanup utilities docker image
|-[169.254.127.1] CleanupDockerArtifacts PASS
|-[169.254.127.2] CleanupDockerArtifacts PASS
|-[169.254.127.3] CleanupDockerArtifacts PASS
+-[169.254.127.4] CleanupDockerArtifacts PASS
NOTE:
The first time you log in to the ECS Portal use the default user, root, which is the default System Administrator role.
1. Type the public IP address or the fully qualified domain name (FQDN) of any node in the system,
or the address of the load balancer that is configured as the front end, in the address bar of the
browser: https://<node1_public_ip>.
2. Log in with the default root credentials:
o User Name: root
o Password: ChangeMe
NOTE:
The first system administrator to login to the ECS Portal is prompted to acknowledge the End User License Agreement.
Once acknowledged, the user is prompted to change the password.
3. After you change the password at first login, click Save.
You are logged out and the ECS login screen is displayed.
Page 70 of 91
4. Type the User Name and Password.
The ECS portal launches with the Getting Started Checklist opened.
5. If you are configuring an ECS instance that is not being added to a geo-configuration, or you are
configuring the first VDC of a geo configuration, you can use the Getting Started Checklist to
provision the initial resources in the ECS.
If you are configuring an ECS instance that is being added to an existing VDC in a geo-
configuration, click No thanks, I'll get started on my own, and perform the postinstallation steps
manually as described in the following sections.
NOTE:
If you continue to provision the ECS instance, you cannot add the ECS instance to the VDC without reinstalling the
system. Refer to Provision ECS resources for details.
6. To log out of the ECS Portal, in the upper-right menu bar, click the arrow beside your user name,
and then click logout.
NOTE:
If you are not adding this ECS instance to VDC in a geo-configuration, you can opt for automatic guidance through the licensing,
and initial provisioning steps using the Getting Started Checklist in the ECS portal. See the ECS Administrator's Guide for
information to access the Getting Started Checklist. If you are adding the ECS instance to a VDC in a geo-configuration, use the
following steps to upload the ECS license file.
Page 71 of 91
The license is updated.
NOTE:
The supported features vary depending on the license file.
ECS geo
An ECS geo is created when multiple ECS appliances, which may be in different geographical locations,
are connected, and can then be managed through a single ECS portal. It is possible to add ECS
appliances to an existing geo-configuration at a later time, but if you know ahead of time you will be
configuring an ECS appliance for a geo-configuration, complete installation on all the ECS appliances
prior to provisioning the ECS appliances. If you installed ECS on a system which will be added to a VDC,
do not perform the initial ECS setup steps through the Getting Started Checklist .
The ECS Portal is launched with the Getting Started Checklist opened by default. If you intend to add
this instance of ECS to an existing VDC you must manually perform the steps as follows:
1. Login to the ECS Portal.
2. When the Getting Started Checklist opens, click No thanks, I'll get started on my own.
3. Upload the ECS license file.
4. Create storage pools. For specific steps, refer to the ECS Administrator's Guide.
5. Add the instance of ECS to the VDC initially configured for the geo environment. For specific
steps, refer to the ECS Administrator's Guide.
6. Continue final provisioning steps from the VDC initially configured, as described below.
NOTE:
After adding the ECS instance to the VDC (step 5 above), you must continue provisioning the ECS instance from the existing VDC
in the geo environment. If you do not, you will not be able to add the ECS instance to the VDC without reinstalling the system.
NOTE:
If you are adding the ECS instance to an existing VDC for a geo configuration, perform the following steps for the ECS
instance from the existing VDC to which you are adding this instance of ECS.
30.Creating virtual data centers (VDC).
Page 72 of 91
Optionally, you can run the post-install Service Console health check again, after you have
created the VDC.
31.Creating replication groups.
32.Creating Namespaces, and users.
Provisioning operations can be performed manually through the ECS Portal, ECS REST API, or the ECS
CLI. Optionally, you can allow ECS to guide you through initial provisioning steps using the Getting
Started Checklist which is built-in to the ECS portal only when you are not adding the ECS instance to a
VDC as part of a geo configuration. Refer to the ECS Administrator's Guide for specific steps to manually
provision ECS resources, or information to access the Getting Started Checklist.
NOTE:
Since the Dell acquisition of EMC, the name of the EMC Secure Remote Services has changed to Secure Remote Services (SRS).
Any references to ESRS, or SRS in this guide, or in any one of the ECS interfaces, refers to the Secure Remote Services feature.
To connect ECS to SRS, you must add the SRS Gateway to ECS through the ECS Portal, as described
in Add the SRS (ESRS) gateway to ECS.
Before you add an Secure Remote Service server to ECS using the ECS Portal, you must:
The Secure Remote Services Gateway server facilitates communication between ECS and the
back-end Secure Remote Service server at Dell EMC. For more information on configuring
Secure Remote Service Gateway servers, see the Secure Remote Services VE documentation.
Page 73 of 91
• Verify that the Activated Site ID defined in the ECS license is supported on the Secure Remote
Services Gateway server that will be used by ECS.
The Activated Site ID is the license site number for the physical site where ECS is installed. You
can obtain the Activated Site ID in the Activated Site column in the ECS Portal on the Settings
>Licensing page. To verify the Activated Site ID is supported on the Secure Remote Services
Gateway server:
1. Log in to the Secure Remote Services Gateway web UI.
2. In the top menu bar, click Devices >Manage Device.
A list of ECS managed devices, such as VDCs and racks, displays. At the top of the list
there is a row with the heading Site ID: followed by a comma-separated list of Site ID
numbers. For example: Site ID: 67520, 89645, 111489
3. If the Activated Site ID you obtained from the ECS Portal is:
▪ listed in the Site ID row at the top of the Managed Device list in the Secure
Remote Services Gateway web UI, it is supported on the Secure Remote
Services Gateway server.
▪ not listed in the Site ID row at the top of the Managed Device list in the Secure
Remote Services Gateway web UI, you must add it by clicking the Add SiteID
button at the bottom of the page.
• Verify that you have full access (administrator) rights to the Activated Site ID and that the VDC
Serial Number is associated with the Activated Site ID.
You can obtain the VDC Serial Number on the Settings >Licensing in the ECS Portal.
1. Go to https://support.emc.com/myservice360/.
2. Log in and validate that you have full access rights.
If you can access this page, then you can use your user account credentials to configure
ESRS.
3. To verify that the VDC Serial number is associated with the Activated Site ID, click the
Install Base link near the center of the page.
If you have access to multiple sites, select the appropriate site in the My Sites drop-
down list.
4. In the search box, type the VDC Serial Number.
The VDC Serial Number is verified when it displays in the Product ID column in the table
below the search box.
• Optionally, validate that all DELL EMC Secure Remote Services backend systems have been
updated with the SiteID and SWID information by opening a service request as described here:
https://www.dell.com/support/kbdoc/en-us/000132371/how-to-create-a-support-request-online-
for-dell-emc?lang=en
Page 74 of 91
33.your support.emc.com account email
34.VDC Serial Number
35.Activated Site ID
• In an ECS geo-federated system, you must add an ESRS server for each VDC in the system.
• Review Secure Remote Services prerequisites before performing this procedure in the ECS
Portal.
• This operation requires the System Administrator role in ECS.
The following steps describe how to add a Secure Remote Services gateway server after initial
installation of ECS. Refer to the ECS Adminstration Guide for steps to manage connectivity to Secure
Remote Services after the initial connection of ECS and SRS.
36.In the ECS Portal, select Settings >ESRS >New Server.
37.On the New ESRS Server page:
a.In the FQDN/IP field, type the Secure Remote Services Gateway FQDN or IP address.
b.In the PORT field, type the Secure Remote Services Gateway port (9443 by default).
c. In the Username field, you can use your Dell EMC login and FOB, or the login user name
used by the customer to interface with ECS support. This is the same login user name
used by the customer to log in to https://support.emc.com/myservice360/.
d.In the Password field, use your FOB password or type the password set up with the
customer's login user name.
38.Click Save.
Server connectivity may take a few minutes to complete. To monitor the process, click the
refresh button. Possible states of transition are Processing, Connected, Disconnected, or Failed.
If the status is Failed, mouse over the status result to determine the cause of the failure.
NOTE:
If you receive an INVALID_CREDENTIALS error message, email [email protected] with a description of the issue,
your account email, VDC serial number, and Activated Site ID.
For more information regarding Secure Remote Services error messages, refer to the Secure
Remote Services documentation which is available from: https://www.dell.com/support/home/en-
us/.
39.If Secure Remote Services is configured with more than one gateway for high availability, repeat
steps 1 to 3 to add additional Secure Remote Services gateway servers.
40.Once connectivity to the Secure Remote Services gateway server is complete, you can verify
the connection through the ECS Portal.
Page 75 of 91
Verify that Secure Remote Services call home works
You can test the Secure Remote Services call home functionality by generating a test call home alert,
and then verifying that the alert is received.
Restart xDoctor, and verify ECS is registered with the xDoctor portal
If you have configured SRS after running xDoctor, restart xDoctor to ensure SRS is configured to send
notification and alerts. After restarting xDoctor verify that it was restarted successfully, and that ECS has
been registered with the xDoctor portal.
1. Run the following commands to restart xDoctor.
2. Verify xDoctor is fully functional after restart by running the following command to generate an
xDoctor status report.
3. Find the Session Report line in the --rap=RAP069 command output and run a report to
verify the session is running correctly, for example:
If the report does not display any issues, xDoctor is configured and working correctly.
Page 76 of 91
If xDoctor has failed, the message will display the issue. There are a number of issues that could
be causing the issues. KB 520782 lists solutions to all the known issues with xDoctor. The
following is an example of an issue you may experience after restarting xDoctor.
xDoctor configuration.
Displaying xDoctor Report (2018-07-27_203854) Filter:[] ...
Timestamp = 2018-07-27_203854
Category = connecthome
Source = os
Severity = ERROR
Message = xDoctor needs to be Kickstarted
Extra = No Connected ESRS
RAP = RAP069
Solution = 520782
4. Verify that ECS is registered with the xDoctor portal as described in KB525506.
NOTE:
For 2.2 and later, the SSH credentials have changed from root/ChangeMe to admin/ChangeMe.
2. Check the Operating System (OS) version on each node using the following command:
Example output:
example: ecs-os-base-<current_version>.noarch
Page 77 of 91
Before you can install the ECS software, you must set the iDRAC IP address, and then install the latest
ECS operating system onto the ECS nodes through the iDRAC Virtual Console. You will be accessing
different versions of iDRAC, depending on which version of ECS hardware you are running.
The iDRAC8 Virtual Console runs on Java, or HTML5. This guide demonstrates the procedures using
HTML5. If you are running iDRAC8 on Java, see the iDRAC 8 documentation for requirements and
detailed steps. iDRAC documentation is available at: http://en.community.dell.com/techcenter/systems-
management/w/wiki/12334.idrac8-home. If you want to change your iDRAC Virtual Machine settings to
run on HTML 5:
41.Connect to the iDRAC address you set up in Set the iDRAC IP address using the BIOS Setup
Utility.
42.Use a browser to connect to the iDRAC IP address of node 1 (also known as provo), and enter
the login credentials, and click Log in.
a. Username: root
b. Password: passwd
43.From the iDRAC 8 dashboard, select System Summary.
44.From the Virtual Console Preview select, Settings.
45.Next to Plug-in Type select HTML5, and click Apply.
# sudo vi /etc/snmp/snmpd.conf
Page 78 of 91
# sec.name source community
com2sec local localhost secret42
com2sec cust1_sec 192.168.219.0/24 public
com2sec cust1_sec 192.168.219.0/24 public
com2sec snmpv3_user 192.168.219.0/24 public
com2sec snmpv3_user 192.168.219.0/24 public
The sudo command is required with the viprscp command in this case as reading the
snmpd.conf file requires root permissions.
6. Create a user:
1. Stop the snmp service:
# sudo vi /var/lib/net-snmp/snmpd.conf
Page 79 of 91
7. For SNMP v2c, test the SNMP agent using the following command. Compare the output with the
sample output below:
Example Output:
8. For SNMP v3, the SNMP username is snmp3_user. Test the UCD-SNMP-MIB using this
snmpwalk command:
Example Output:
NOTE:
If your SSH connection continues to drop, you may need to disable your Cisco VPN client.
3. Start a command prompt on your laptop and verify connectivity to iDRAC.
# ping 192.168.219.101
2. Use a browser to connect to the iDRAC IP address of node 1 (also known as provo), and enter
the login credentials, and click Log in.
Page 80 of 91
a. Username: root
b. Password: passwd
company_name.txt
The company_name.txt file must contain the name of the company (case sensitive) as it will appear in
the ECS Portal (see example below).
loginLogo.png
The loginLogo.png must measure 350 x 80 pixels to replace the logo on the login page highlighted
below.
Page 81 of 91
Figure 2. loginLogo.png location in the ECS Portal
logo.png
The logo.png must measure 250 x 45 pixels to replace the logo on the top navigation section
highlighted below.
• You must have the topology file populated with the customer-specific node and rack details, or
the Auto-SKU commands will fail.
Page 82 of 91
• The auto-generated SKU displays during the install process.
46.Obtain the third-party rebranding files which should have been put into the /host/files/
directory as described in: Get the image files for third-parting branding of the ECS Portal.
a. company_name.txt,: Shows the specific company name to display on the ECS Portal
Welcome page.
b. loginLogo.png,: Shows the specific company logo to display on the ECS Portal
Login page.
c. logo.png,: Shows the specific company logo to display on the ECS Portal menu pane.
See Image and .txt file specifications for third-party branding for file requirements.
WARNING:
You must type (or copy and paste) all multi-line commands as a single line of text with no breaks.
47.Ensure you are in a screen session.
48.Change to the installer directory:
# cd /opt/emc/caspian/installer
49.Run the installer to install the ECS software so that the ECS Portal will include the third-party
branding files provided by the customer.
a. company_name.txt
b. loginLogo.png
c. logo.png
Remember to replace the example parameters company_name.txt,loginLogo.png,,
logo.png, with the names of the files provided by customers.
NOTE:
List disks do not run if steps to install HAL failed.
Page 83 of 91
# viprexec cs_hal list disks
Example Output
total: 10
NOTE:
o After the new install or extend, if the Maintenance Page displays disks that must be
replaced in the Data Disks column, or if any disks are listed in the Unassigned Disks
column, contact ECS Remote Services.
o If the disks are not automatically added to the object, contact the support team.
2. List volume details to ensure that the volumes are configured successfully:
• You have connected to the rack as described in: Connect the service laptop to the ECS
appliance.
50.Uninstall fabric and object components.
a.Change to the installer directory.
Page 84 of 91
cd /opt/emc/caspian/installer
For example:
WARNING:
Performing these steps destroy all data on the disks and makes ECS unusable. It is an irreversible process.
NOTE:
The system disks on which the operating system is installed do not need to be freshened as they will be re-imaged with ECS
operating system.
1. If you have not already done so, create a MACHINES file that can be used to perform distributed
commands. The MACHINES file contains the list of private IP addresses for each node in a single
rack.
1. Change to the /home/admin directory:
# cd /home/admin
2. Create the MACHINES file in the /home/admin/ directory ( this is the home directory for
the admin user):
# getrackinfo -c MACHINES
Page 85 of 91
# cat MACHINES
You can also use the pingall which will ping all nodes in the MACHINES file:
# pingall
# viprexec "blkid"
Example output of a system without partition or filesystem UUIDs showing after successful
factory freshen of the storage volumes.
Note the remaining block IDs are related to the system and include only LVM, ROOT, BOOT,
swap, and docker, and possibly a second partition on a block device with no related filesystem
(hybrid on the C70 only).
Page 86 of 91
/dev/mapper/docker-254:1-658505856-pool: UUID="193a8f15-b7ee-4cac-a2fd-d8d765057fef"
TYPE="xfs"
Table 1. Rack ID 1 to 50 The default rack IDs and color names are assigned in installation order as
shown in the following table.
Page 87 of 91
8 violet 25 amaranth 42 navy
17 silver 34 eggplant - -
Nodes are assigned node names based on their order within the server chassis and within the rack itself.
Table 2. Default node names The following table lists the default node names.
Nodes that are positioned in the same slot in different racks at a site have the same node name. For
example, node 4 is called ogden, assuming that you use the default node names.
The getrackinfo command identifies nodes by a unique combination of node name and rack name.
For example, node 4 in rack 4 and node 4 in rack 5 are identified as:
ogden-green
ogden-blue
and can be pinged using their NAN resolvable (through mDNS) name:
Page 88 of 91
ogden-green.nan.local
ogden-blue.nan.local
NOTE:
The cloud DVR feature is available from ECS 3.6, and it is applicable only to the EX500 and EX3000 systems. By default, the cloud
DVR feature is disabled. See Enabling Cloud DVR to enable it.
For the first time, you must enable the cloud DVR feature using Service Console. After enabling cloud
DVR, by default, for all the new nodes cloud DVR is enabled.
Run the below command in Service Console to enable the cloud DVR feature:
Sample Output
Page 89 of 91
================================================================================
Status: PASS
Time Elapsed: 10 min 32 sec
Sample output
================================================================================
Disable_CloudDVR
20200609 16:38:58.609: | Validate that data recovery is enabled for all nodes
Page 90 of 91
20200609 16:39:05.281: | | PASS (6 sec)
================================================================================
Status: PASS
Document feedback
To provide any feedback or suggestions on the document, go to Content Feedback Router portal. For
more information, see Content Feedback Router - Support.
Page 91 of 91