NetApp Study Notes
NetApp Study Notes
• Volumes are logical entities that hold user data. They are accessible from one
or more access protocols supported by Data ONTAP
(NFS,CIFS,HTTP,WebDAV, DAFS, FCP and iSCSI)
• Aggregates provide the underlying physical storage for traditional and flexible
volumes
• Traditional volume is contained by a single, dedicated aggregate.The only way
to change the size of a traditonal volume
is to add disks
• Flexible volume is contained within a single aggregate. You can increase and
decrease the size of flexible volumes
almost instantaneously
• Volume name
• Volume size
• Security style: determines whether a volume can contan fiels that use UNIX
security or Windows NTFS file security
• Option to designate the volume as one that uses CIFS oplocks (opportunistic
locks)
• Suppoted language
• Level of space guranteed (FlexVol only)
• Disk space and file limits (quotas)
• Snapshot schedule (optional)
• Option to designate the volume as a SnapLock volume
• Option to designate the volume as a root volume. The root volume in Data
ONTAP is named vol0 by default.
CIFS Administration
# To add a new local user
useradmin user add user_name -g group_name
# To delete a group
useradmin group delete group_name
# To delete an ACL (share-level access control list) entry for a user on a share
# Note: the -g option specifies that the user is the name of a UNIX group.
cifs access -delete <share> [-g] [user]
cifs access -delete datatree1 everyone
## Multiprotocol options:
# A CIFS user can access the file without disrupting UNIX permissions
# When enabled, UNIX qtree's appear as NFS volumes. (ONTAP 7.2 or later)
options cifs.preserve_unix_security on
## Reconfiguring CIFS
# Disconnect users and stop the CIFS server
cifs terminate
# Reconfigure the CIFS service
cifs setup
• Use "sysstat –x 1" to determine how many CIFS ops/s and how much CPU is
being utilized
• Check /etc/messages for any abnormal messages, especially for oplock break
timeouts
• Use "perfstat" to gather data and analyze (note information from "ifstat",
"statit", "cifs stat", and "smb_hist", messages, general cifs info)
• "pktt" may be necessary to determine what is being sent/received over the
network
• "sio" should / could be used to determine how fast data can be written/read
from the filer
• Client troubleshooting may include review of event logs, ping of filer, test using
a different filer or Windows server
• If it is a network issue, check "ifstat –a", "netstat –in" for any I/O errors or
collisions
• If it is a gigabit issue check to see if the flow control is set to FULL on the filer
and the switch
• On the filer if it is one volume having an issue, do "df" to see if the volume is
full
• Do "df –i" to see if the filer is running out of inodes
• From "statit" output, if it is one volume that is having an issue check for disk
fragmentation
• Try the "netdiag –dv" command to test filer side duplex mismatch. It is
important to find out what the benchmark is and if it’s a reasonable one
• If the problem is poor performance, try a simple file copy using Explorer and
compare it with the application's performance. If they both are same, the issue
probably is not the application. Rule out client problems and make sure it is
tested on multiple clients. If it is an application performance issue, get all the
details about:
◦ The version of the application
◦ What specifics of the application are slow, if any
◦ How the application works
◦ Is this equally slow while using another Windows server over the
network?
◦ The recipe for reproducing the problem in a NetApp lab
• If the slowness only happens at certain times of the day, check if the times
coincide with other heavy activity like SnapMirror, SnapShots, dump, etc. on
the filer. If normal file reads/writes are slow:
◦ Check duplex mismatch (both client side and filer side)
◦ Check if oplocks are used (assuming they are turned off)
◦ Check if there is an Anti-Virus application running on the client. This
can cause performance issues especially when copying multiple small
files
◦ Check "cifs stat" to see if the Max Multiplex value is near the
cifs.max_mpx option value. Common situations where this may need
to be increased are when the filer is being used by a Windows Terminal
Server or any other kind of server that might have many users opening
new connections to the filer. What is CIFS Max Multiplex?
◦ Check the value of OpLkBkNoBreakAck in "cifs stat". Non-zero numbers
indicate oplock break timeouts, which cause performance problem
NFS Administration
# Exportable resources
Volume
Directory/Qtree
File
#
# Target examples from /etc/exports
#
# Access restrictions that specify what operations a target can perform on a resource
• Default is read-write (rw) and UNIX Auth_SYS (sys)
• "ro" option provides read-ony access to all hosts
• "ro=" option provides read-only access to specified hosts
• "rw=" option provides read-write access to specified hosts
• "root=" option specifies that root on the target has root permissions
# Displays the UNIX user mappings for the specified Windows account
wcc -s ntname
# local admin
wcc -s administrator
# domain admin
wcc -s development\administrator
# WCC rules
• A Windows-to-UNIX user mapping is not stored in the WCC
• The WCC contains the cached user mappings for the UNIX user identities (UID/
GID) to Windows identities (SID's)
• The wcc command useful for troubleshooting user mappings issues
• the cifs.trace._login option must be enabled.
•
# Displays number and type of NFS v2,v3 requests received by all FlexCache volumes
nfsstat -C
# To enable mountd tracing of denied mount requests against the storage system.
option nfs.mountd.trace on
# Display Top NFS clients currently most active for the storage system
netapp-top.pl -i 30 filerX
# Captures all needed performance information from the storage system and hosts
(clients).
perfstat -f filerX -h host1 -t 5 -i 12 >perfstat.out
Solaris:
rw,bg,hard,nointr,rsize=32768,wsize=32768,proto=tcp,forcedirectio,noac,vers=3
AIX:
cio,rw,bg,hard,intr,proto=tcp,vers=3,rsize=32768,wsize=32768
#
# Recommended test to collect NFS statistics
#
# Check the nfsstat output against retransmissions, timeouts and bad calls
• timeout>5%. Requests timing out before the server can answer them
• badxid~timeout. Server slow. Check nfsstat -m.
• badxid~0 and timeouts > 3%. Packets lost in the network; check netstat. If
this number is the same as bad calls, the network is congested.
• retrans. May indicate network of routing problem if retransmit >5%.
• null>0. Automounter timing out. Increase the timeout parameter on the
Automounter configuration.
# In the output of the "nfsstat -m`' command, the following parameters are critical
• srtt. Smoothed round-trip time.
• dev. Estimated deviation.
# NFS troubleshooting
Resolution Tips
Resolution Tips
• Use ping to contact the hostname of the storage system (server) from client
• Use ping to contact the client from the storage system
• Check ifconfig from the storage system
• Check that the correct NFS version is enabled
• Check all nfs options on the storage system
• Check /etc/rc file for nfs options
• Check nfs license
Resolution Tips
Resolution Tips
Resolution Tips
• snaprestore
snaprestore enables rapid revert (restore) of single files or volumes so
operations can resume quickly.
• snapmirror
The are two types of snapmirror solutions:
◦ Asynchronous snapmirror
This is an automated file system or qtree replication for disaster
recovery of data distribution.
Updates of new and changed data from the source to the destination
occur on a scheduled defined by the storage administrator.
◦ Synchronous snapmirror
Replicates writes from the source volume to the partner destination
volume at the same time it is written to the source volume.
Updates are performed in real time intervals.
• snapvault
Is a low-overhead, disk-based online backup of homogeneous storage systems
for last and simple restores.
• Open Systems SnalVault (OSSV)
Is a heterogeneous disk-based data protection feature of Data ONTAP that
enables data stored on multiple Open Systems platforms (Windows/Unix based
clients)
to be backed up to and restored from a central storage system.
snaplock technology is a software feature that allows companies to implement the data
perfromance functionality of traditional WORM
(write once, read many) storage in an easier-to-manage, faster access, lower cost
magnetic disk-based solution.
There are two types:
• snaplock
◦ snaplock Compliance
Is designed for comprehensive archival solution that meets UD
Securities and Exchange Commission regulations for
data retention. snaplock volumes of this type cannot be altered
or deleted before the expiration of retention period.
◦ snaplock Enterprise
This solution is designed for organizations whith self-regulated
and best-practice requirements for protecting digital assets with
WORM-like storage devices. Data written on a snaplock
Enterprise volume can be deleted by an administrator.
• Data Fabric Manager (DFM)
Data Fabric Manager provides centralized management of distributed NetApp,
NetCache, storage, and NearStore appliances.
snapshot technology
Creates a read-only copy of a storage appliance's file system, readily accessible via
special subdirectories (i.e. .snaphot), taken automatically on a schedule
manually. Creating snaphot copies is very quicly because if it an index to the file
system.
Disk Sanitization
Disk sanitization is the process of physically removing data from a disk by overwriting
patterns on the disk in a manner that precludes the
recovery of that data by any known recovery methods.
2. SnapRestore
SnapshotRestore Considerations
• You can use SnapRestore to revert a volume or file to a snapshot at any time
• NOTE: Reverting an Aggregate snaphot will revert ALL volumes in the
Aggregate.
• Prerequisites
◦ SnapRestore licensed
◦ Snapshots must exist of the appliance so that you can select a
snapshot for the reversion.
◦ The volume to be reverted must be online.
◦ The volume to be reverted must not be a mirror used for data
replication.
◦ Enough free space must be available for recovery of a single file.
Cautions
3. Enter the name of the snapshot to be used for reverting the volume
NOTES:
3. Enter the name of the snapshot to be used for reverting the file
NOTES:
• A file can only be restored to an existing directory. The SnapRestore default is
to restore the file to its
original directory path. The "-r" option can be used to specify a different
(existing) directory.
• NFS users who try to access a reverted file without first reopening it might get
a the "Stale File Handle" error message after the reversion.
3. SnapMirror
snapmirror overview
SnapMirror provides a fast and flexible enterprise solution for replicating data over
local area,
wide area and Fibre Channel networks. SnapMirror addresses mutiple applications
areas such as
mission critical data protection, and business continuanance in case of a disaster.
• Data migration
• Disaster Recovery
• Remote access to data and load sharing
• Remote tape archival
SnapMirror Modes
SnapMirror Terminology
SnapMirror Components
• Source volumes and qtrees: SnapMirror source volumes and qtrees are
writable data objects.
• Destination volumes and qtrees: the SnapMirror destination volumes and
qtrees are read-only objects, usually on a separate storage system.
The destination volumes and qtrees are normally accessed by users ony when
a distaser takes down the source system
and the administrator uses SnapMirror commands to make the replicated data
at the destination accessible and writable.
• Space guarantee
◦ volume-disable automatically on the destinal volume
◦ As a result, it is possible to overcommiting the aggregate
◦ When the relationship is broken, space mode is identical on source and
destination
• Overcommiting the aggregate
◦ More efficient disk space utilization on the destination
◦ When the relationship is broken, turn off vol options
fs_sized_fixed and use vol size to re-size the destnation volume.
◦ To overcommit an aggregate volume, create the destination flexvol
with quarantee set to none of file
Arguments
kbs=kbs
Maximum transfer speed, in kilobytes per second, that Data ONTAP can use to transfer
data.
• Never: Transfers are always restarted from the beginning of a transfer and
never from where they
were before an interruption
• Always: Transfers are always restarted if possible from where they were
before an interruption
• Default: Transfers are restarted if they do not conflict with a scheduled
transfer.
Schedule
For asynchronous SnapMirror, a schedule must be set per relationship and consists of:
minute hour day_of_month day_of_week
Where:
• minute can be a value from 0-59
• hour can be 0 (midnight) to 23 (11 pm)
• day_of_month can be 1-31
• day_of_week can be 0 (sunday) to 6 (saturday)
• all possible values can applied with an "*"
• A "-" means "never" and prevents this schedule entry from executing
Options Snapmirror
Caution
With VSM, if you upgrade your systems to a later version of Data ONTAP, upgrade the
SnapMirror destination before you upgrade
the SnapMirror source system.
Snapmirror Snapshot copies are distinguished from the system Snapshot copies by a
more elaborate naming convention
and the label snapmirror in parentheses with the "snap list" command.
Dest_system(sysid)_name-src.number|dst.number
NOTE: If you use SnapMirror for data migration, you can copy the /etc/quotas
entries from the source to the /etc/quotas file of the destination before
you convert the mirror to a regular volume or qtree. However, if you use SnapMirror
for Disaster Recovery, you must keep a copy on the destination
storage system of all /etc/quotas entries used by the source.
Example:
When the relationship is broken, subsequent updates will fail. To resume incremental
updates, you have first to re-establish
the relationship. The snapmirror resync (on the source or destination) command
enables you to do this without executing a new initial base transfer.
You can release a mirror volume or qtree when you want to permanently remove it
from a mirrored relationship.
Releasing the mirror deletes Snapshot copies from the volume.
SnapMirror to Tape
1. Initial large-sized vol baseline snapshot replicated to tape from source filer.
2. Tape physically transported to tape drive on destination site
3. Load the tape and start replication to the destination filer.
4. Incremental SnapMirror updates are made via low bandwidth connection
Example:
The command starts the initial baseline transfer from the source volume to the
tape.
2. Remove the base snapshot on the source when the backup to tape is
completed:
NOTE: If you retrieve a backup tape into a file system that does not match the disk
geometry of the
source storage system used when writing the data onto tape, the retrieve can be
extremely slow.
4. SnapVault
SnapVault is a disk-based storage backup feature of ONTAP that enables data stored
on multiple storage systems
to be backuped up to a central, secondary NetApp storage system as read-only
Snapshot copies.
Both primary and secondary systems use SnapVault for backup and restore operations,
based on the
same logical replication engine as qtree SnapMirror.
SnapVault Basic Deployment
Restoration on Request
Users can perform a restore of their own data without intervention of system
administrator.
To do a restore, issue the snapvault restore command from the primary system
whos qtree needs to be restored.
Ater succesful restore of data, you use the snapvault start -r command to restart
the SnapVault relationship
between primary and secondary qtree.
Note: when you use snapvault restore command to restore a primary qtree,
SnapVault places a residual SnapVault
Snapshot copy on the volume of the restored primary qtree. This Snapshot copy is not
automatically deleted.
You cannot use the snapvault restore command to restore a single file. For single
file restores, you must use the ndmpcopy command.
Port 10566 must be open in both directions for SnapVault backup and restore
operations.
If NDMP is used for control management, then Port 10000 must be open on primary
and secondary.
• On Primary
◦ Add the license sv_ontap_pri
◦ Enable SnapVault service and configure SnapVault options
• On Secondary
◦ Add the license sv_ontap_sec
◦ Enable SnapVault service and configure SnapVault options
◦ Initialize the first baseline transfer
• On Primary and Secondary
◦ Schedule SnapVault snapshots creation
◦ Monitor transfers progress, status and snapshots
Example:
Command Syntax:
Snapshot schedule results: This schedule keeps the two most recent weekly
Snapshot copies, the six most recent nightly Snapshot copies,
and the eight most recent hourly Snapshot copies, created at 8 a.m., noon, 4
p.m., and 8 p.m. every day.
Whenever the Snapshot schedule creates a new Snapshot copy of a particular
type, it deletes the oldest one and renames the existing ones.
On the hour, for example, the system deletes hourly.7, renames hourly.0 to
hourly.1, and so on.
Example:
SnapVault commands
• Perform the initialization baseline transfer from the primary qtree to the
secondary qtree
• Resume the SnapVault relationship between the restored qtree and its backup
qtree on the secondary
• On primary, lists all the known destinations for SnapVault primary qtrees.
• VSM copies all snaphots from a read/write source into a read-only destination
• Qtree SnapMirror is to be used in an environment requiring an immediate
failover capability
• SnapVault is to be used with applications that can afford to lose some data and
not require immediate failover
• Qtree SnapMirror allows replication in both directions (source and destination
can run on same storage system)
• Qtree SnapMirror does not allow snapshot creation or deletion on the read-only
destination.
• SnapVault replicates in one direction (source and destination cannot run on
same storage system)
• SnapVault adds snapshot scheduling, retention, and expiration, providing
versions (backups) on secondary
• SnapMirror provides up to per minute updates
• SnapVault provides up to per hour updates
In case of a disater, when the prinmary becomes unavailable, you might want to
convert the read-only qtrees replica
to a writeable file system to redirect CIFS and NFS clients access to the secondary.
SnapVault does not currently have the ability to create a writable destination on the
secondary. You can use
SnapMirror/SnapVault bundle to convert the SnapVault destination qtree to a
SnapMirror destination qtree,
making it a typical SnapMirror destinations qtree that can be quiesced and broken.
Note: if any changes made while in the broken state need to be copied back to
the primary, you also need a SnapMirror license on primary
• Involve NetApp Support (when entering the priv set diag mode)
• Convert snapvaulted qtree to a snapmirrored qtree
• Quiesce the snapmirror qtree
• Break the mirror, making it writeable
• Restablishing the SnapVault relationship
◦ Preserve the changes made on the secondary
◦ Or discard all changes made on the secondary
Example:
2. Quiesce the destination qtree and break the releationship (makes qtree writable)
Sec*> snapmirror on
Sec*> snapmirror quiesce /vol/sec_vol/sec_qtree
Sec*> snapmirror break /vol/sec_vol/sec_qtree
Sec> snapmirror on
Sec> snapvault on
Scenario 2: Discard all the changes made on the secondary during the DR
period.
Sec> snapmirror on
Sec> snapvault on
Symptoms are:
• Initialization or transfer updates lagging, the lag is above the expectation, and
the transfer duration does not meet the SLA
• The transfer duration meets the SLA, but the throughput is low.
◦ Check /etc/snapmirror.conf or snapvault snap sched, define what
is the expected lag (exptected time between two scheduled updates)
◦ Then explore the snapmirror status -l or snapvault status -l
outputs to get a view of the mirror implementation:
▪ How many systems are involved?
▪ How many mirror/backup services are active?
▪ Which systems are a source and a destination at the same
time?
▪ How many relationships are set per source and destination
systems?
◦ Note the transfer lag and define the data/time the last transfer
succeeded
◦ Analyze the SnapMirror logs /etc/log/snapmirror and syslog
messages /etc/messages to trace what happened before and after the
last succesful transfer has completed:
when was the request sent, started and ended. Are there any errors?
6. NDMP Fundamentals
7. Snaplock
SAN Administration
Initiator/Target Relationship
• Storage systems and hosts have HBA's (Host Bus Adapters) so they can be
connected directly to each other or to FC switches
• Each FCP node is identified by a World Wide Node Name (WWNN) and a World
Wide Port Name (WWPN)
• WWPN's are used to create igroups, which controls host access to specific
LUN's
FC SAN Topologies
IP SAN topogies
There a three steps required on the storage system and two additional steps performed
on the FCP or iSCSI accessed host.
Persistant binding permanently bind a particular target ID on the host to the storage
system WWPN. On some system you must create persistent
binding between the storage system (target) and the host (initiator) to guarantee that
the storage system is always available at the correct SCSI target ID
on the host.
Use the command fcp show adapters to display the WWPN for each HBA on the
storage system.
On the Solaris host, use one of the following methods to specify the adapter on the
storage system HBA.
• create_binding.pl
• /usr/sbin/lpfc/lputil
• /kernel/drv/lpfc.conf
• HBAnywhere (Emulex adapter)
• SANsurfer (Qlogic adapter)
To determine the WWPN of the HBA installed on the AIX or HPUX host.
To find the WWPN for the WWPN installed on the Linux host.
LUNs created on storage system that will be accessed via FCP must be configured on
the SUN Solaris host.
• Edit /kernel/drv/sd.conf file with the appropriate target and LUN IDs. The
/kernel/drv/lpfc.conf file will help determine
what should be in the sd.conf file.
• Run devfsadm on the host to allow discovery of the new LUN's or use the
reboot command reboot -- -r
• Use the sanlun command to verify that the new LUN's are now visable
• Use the format command to label the new LUN's as Solaris disks
• Create a UNIX file system on the disk, or use it as a raw device
• Run cfgmgr command to discover the new LUN's. Allows the host to log into
the fabric, check for new devices and create new device entries.
• Run the sanlun lun show command to verify that the host has discovered the
new LUN's
• Run smit vg command to create a volume group.
• Rim smit to access storage on a volume group.
• Run smit fs to create a file system
• Run lsvg newvg command to verify the information on the new volume group.
LUN cloning