RAC12c Theory Guide v8.0

Oracle 12c: RAC
Theory Guide
www.ucliditschool.com
Uclid IT School RAC Theory Guide
Copyright © 2018, Uclid IT School. All rights reserved.
Disclaimer
This document is proprietary of Uclid IT School. You may use this document solely for your own use. The
document may not be modified or altered in any way. Except where your use constitutes "fair use" under
copyright law, you may not use, share, download, upload, copy, print, display, perform, reproduce,
publish, license, post, transmit, or distribute this document in whole or in part without the express
authorization of Uclid IT School.
The information contained in this document is subject to change without notice. If you find any errors in
this document, please report them in writing to [email protected] . This document is not warranted
to be error-free.
Oracle, Java are registered trademarks of Oracle Corporation and/or its affiliates. Other names may be
trademarks of their respective owners.
.
V8.0 www.ucliditschool.com Page 2 of 132

Contents
1. INTRODUCTION TO ORACLE RAC ........................................................................................4
2. ORACLE CLUSTERWARE CONCEPTS ................................................................................ 12
3. ASM CONCEPTS ....................................................................................................................... 21
4. NEW FEATURES IN RAC 12C ............................................................................................... 32
5. PREPARING FOR GRID INFRASTRUCTURE INSTALLATION ..................................... 38
6. ADMINISTERING ORACLE CLUSTERWARE .................................................................... 51
7. ADMINISTERING ASM INSTANCES .................................................................................... 67
8. ADMINISTERING ASM DISK GROUPS ............................................................................... 79
9. ASM COMMAND LINE UTILITY ........................................................................................... 91
10.ORACLE RAC ARCHITECTURE ............................................................................................ 97
11.ADMINISTERING RAC INSTANCES & DATABASE ...................................................... 102
12.WORKLOAD MANAGEMENT WITH SERVICES ............................................................ 115
13.PATCHING RAC ENVIRONMENTS ................................................................................... 122
14.MIGRATING NON-ASM DATABASE TO ASM ................................................................ 127
15.CONVERTING SINGLE INSTANCE DATABASE TO RAC DATABASE ...................... 130

Introduction to Oracle RAC

Introduction to Clusters
A cluster consists of a group of independent but interconnected computers whose combined
resources can be applied to a processing task. A common cluster feature is that it should
appear to an application as though it were a single server. Most cluster architectures use a
dedicated network (cluster interconnect) for communication and coordination between cluster
nodes.
A common cluster architecture for data-intensive transactions and computations is built around
shared disk storage. Shared-nothing clusters use an alternative architecture where storage is
not shared and data must be either replicated or segmented across the cluster. Shared-nothing
clusters are commonly used for workloads that can be easily and predictably divided into small
units that can be spread across the cluster in parallel. Shared disk clusters can perform these
tasks but also offer increased flexibility for varying workloads.
Load balancing clusters allow a single application to balance its workload across the cluster.
Alternatively, in a failover cluster, some nodes can be designated as the primary host for an
application, whereas others act as the primary host for different applications. In a failover
cluster, the failure of a node requires that the applications it supports be moved to a surviving
node.
Load balancing clusters can provide failover capabilities but they can also run a single
application across multiple nodes providing greater flexibility for different workload
requirements. Oracle supports a shared disk cluster architecture providing load balancing and
failover capabilities. In an Oracle cluster, all nodes must share the same processor architecture
and run the same operating system.
The following figure shows a typical cluster.

m
co
l.
ai
gm
About Clusterware
7@
Clusterware is a term used to describe software that provides interfaces and services that
enable and support a cluster. Different cluster architectures require clusterware that delivers
71
different services.
n2
wa
For example, in a simple failover cluster the clusterware may monitor the availability of
applications and perform a failover operation if a cluster node becomes unavailable. In a load
ha
balancing cluster, different services are required to support workload concurrency and
id
coordination.
it
Typically, clusterware includes capabilities that:

Pr
 Allow the cluster to be managed as a single entity (not including OS requirements), if

desired
 Protect the integrity of the cluster so that data is protected and the cluster continues to
function even if communication with a cluster node is severed
 Maintain a registry of resources so that their location is known across the cluster and so that
dependencies between resources is maintained
 Deal with changes to the cluster such as node additions, removals, or failures
 Provide a common view of resources such as network addresses and files in a file system

What is RAC?
Real Application Clusters (RAC) is a software that enables you to use clustered hardware by
running multiple Instances against the same database. The database files are stored on disks
that are either physically or logically connected to each node, so that every active instance can
read from or write to them.
RAC software manages data access, so that changes are coordinated between the instances
and each instance sees a consistent image of the database. The cluster interconnect enables
instances to pass coordination information and data images between each other.
The following block diagram shows the RAC model.
Oracle Real Application Clusters (Oracle RAC) enables an Oracle database to run across a
cluster of servers, providing fault tolerance, performance, and scalability with no application
changes necessary. Oracle RAC provides high availability for applications by removing the
single point of failure with a single server.
Oracle RAC enables you to combine smaller commodity servers into a cluster to create scalable
environments that support mission critical business applications. Applications that you deploy on
Oracle RAC databases can operate without code changes. In an Oracle RAC database, multiple
instances access a single set of database files (i.e. single database) . Having multiple database
instances accessing a single set of data files prevents the server from single point of failure. If a
node in the cluster fails, then the Oracle database continues running on the remaining nodes.
Individual nodes can be shut down for maintenance while application users continue to work. An
Oracle RAC database can have up to 100 instances, all of which access one database.

Oracle RAC extends Oracle Database so that you can store, update, and efficiently retrieve
data using multiple database instances on different servers at the same time. Oracle RAC
provides the software that manages multiple servers and instances as a single group. The data
files that comprise the database must reside on shared storage that is accessible from all
servers that are part of the cluster. Each server in the cluster runs the Oracle RAC software.
History of Oracle RAC

A cluster is a group of two or more interconnected computers or servers that appear as if they
are one server to end users and applications and generally share the same set of physical
disks. The key benefit of clustering is to provide a highly available framework where the failure
of one node (for example a database server running an instance of Oracle) does not bring down
an entire application. In the case of failure with one of the servers, the other surviving server (or
servers) can take over the workload from the failed server and the application continues to
function normally as if nothing has happened.
The concept of clustering computers actually started several decades ago. The first successful
cluster product was developed by DataPoint in 1977 named ARCnet. The ARCnet product
enjoyed much success by academia types in research labs, but didn't really take off in the
commercial market. It wasn't until the 1980's when Digital Equipment Corporation (DEC)
released its VAX cluster product for the VAX/VMS operating system.
With the release of Oracle 6 for the Digital VAX cluster product, Oracle was the first commercial
database to support clustering at the database level. It wasn't long, however, before Oracle
realized the need for a more efficient and scalable distributed lock manager (DLM) as the one
included with the VAX/VMS cluster product was not well suited for database applications.
Oracle decided to design and write their own DLM for the VAX/VMS cluster product which
provided the fine-grain block level locking required by the database. Oracle's own DLM was
included in Oracle 6.2 which gave birth to Oracle Parallel Server (OPS) - the first database to
run the parallel server.
By Oracle 7, OPS was extended to include support for not only the VAX/VMS cluster product
but also with most flavours of UNIX. This framework required vendor-supplied clusterware which
worked well, but made for a complex environment to setup and manage given the multiple
layers involved. By Oracle8, Oracle introduced a generic lock manager that was integrated into
the Oracle kernel. In later releases of Oracle, this became known as the Integrated Distributed
Lock Manager (IDLM) and relied on an additional layer known as the Operating System
Dependant (OSD) layer. This new model paved the way for Oracle to not only have their own
DLM, but to also create their own clusterware product in future releases.
Oracle Real Application Clusters (RAC), introduced with Oracle9i, is the successor to Oracle
Parallel Server. Using the same IDLM, Oracle 9i could still rely on external clusterware but was
the first release to include their own clusterware product named Cluster Ready Services (CRS).
With Oracle 9i, CRS was only available for Windows and Linux. By Oracle 10g release 1,
Oracle's clusterware product was available for all operating systems and was the required
cluster technology for Oracle RAC. With the release of Oracle Database 10g Release 2 (10.2),
Cluster Ready Services was renamed to Oracle Clusterware. When using Oracle 10g or higher,
Oracle Clusterware is the only clusterware that you need for most platforms on which Oracle
RAC operates (except for Tru cluster, in which case you need vendor clusterware). You can still
use clusterware from other vendors if the clusterware is certified, but keep in mind that Oracle
RAC still requires Oracle Clusterware as it is fully integrated with the database software. This
guide uses Oracle Clusterware which as of 11g Release 2 (11.2), is now a component of Oracle
grid infrastructure.

A big difference between Oracle RAC and OPS is the addition of Cache Fusion. With OPS a
request for data from one instance to another required the data to be written to disk first, then
the requesting instance can read that data (after acquiring the required locks). This process was
called disk pinging. With cache fusion, data is passed along a high-speed interconnect using a
sophisticated locking algorithm.
Not all database clustering solutions use shared storage. Some vendors use an approach
known as a Federated Cluster, in which data is spread across several machines rather than
shared by all. With Oracle RAC, however, multiple instances use the same set of disks for
storing data. Oracle's approach to clustering leverages the collective processing power of all the
nodes in the cluster and at the same time provides failover security.
Components of Oracle RAC

The major components of Oracle RAC system include:
 Shared disk system
 Oracle Clusterware
 Cluster Interconnects
 Oracle Kernel Components
Shared Disk system

For all installations, you must choose the storage option to use for Oracle Clusterware files and
Oracle Database files. The Oracle Grid Infrastructure for a cluster and Oracle RAC software is
installed on disks local to each node, not on a shared file system.
Oracle Clusterware supports NFS, iSCSI, Direct Attached Storage (DAS), Storage Area
Network (SAN) storage, and Network Attached Storage (NAS).
All data files (including an undo tablespace for each instance) and redo log files (at least two for
each instance) for an Oracle RAC database must reside on shared storage. Oracle
recommends that you use Oracle ASM to store these files in an Oracle ASM disk group.
Once you have the storage attached to the servers, you have 2 choices on how to setup the
disks
 Cluster File System (CFS) - used to hold all the Oracle data files. Can be used by
windows and Linux, its not used widely
 Automatic Storage Management (ASM) - Oracle choice of storage management, its a
portable, dedicated and optimized cluster file system
Oracle strongly recommends using Automatic Storage Management for Oracle RAC.
Oracle Clusterware
A cluster comprises of multiple interconnected computers or servers that appear as if they are
one server to end users and applications. The Oracle RAC option with Oracle Database enables
you to cluster Oracle databases. Oracle RAC uses Oracle Clusterware for the infrastructure to
bind multiple servers so they operate as a single system. Oracle Clusterware is a portable
cluster management solution that is integrated with Oracle Database. Oracle Clusterware is a
required component for using Oracle RAC that provides the infrastructure necessary to run
Oracle RAC. Oracle Clusterware provides all of the features required to manage the cluster
database including node membership, group services, global resource management, and high

availability functions. Oracle Clusterware also manages resources, such as Virtual Internet
Protocol (VIP) addresses, databases, listeners, services, and so on. In addition, Oracle
Clusterware enables both non-cluster Oracle databases and Oracle RAC databases to use the
Oracle high-availability infrastructure. Oracle Clusterware along with Oracle Automatic
Storage Management (Oracle ASM) (the two together comprise the Oracle Grid
Infrastructure) enables you to create a clustered pool of storage to be used by any
combination of non-cluster and Oracle RAC databases. Oracle Clusterware is the only
Clusterware that you need for most platforms on which Oracle RAC operates.
When you create an Oracle RAC database using any of the management tools, the database is
registered with and managed by Oracle Clusterware, along with the other required components
such as the VIP address, the Single Client Access Name (SCAN) (which includes the SCAN
VIPs and the SCAN listener), Oracle Notification Service, and the Oracle Net listeners. These
resources are automatically started when the node starts and automatically restart if they fail.
The Oracle Clusterware daemons run on each node.
Anything that Oracle Clusterware manages is known as a CRS resource. A CRS resource can
be a database, an instance, a service, a listener, a VIP address, or an application process.
Oracle Clusterware manages CRS resources based on the resource's configuration information
that is stored in the Oracle Cluster Registry (OCR). You can use SRVCTL commands to
administer any Oracle-defined CRS resources. Oracle Clusterware provides the framework that
enables you to create CRS resources to manage any process running on servers in the cluster
which are not predefined by Oracle. Oracle Clusterware stores the information that describes
the configuration of these components in OCR.
In Oracle 11g Release 2, Oracle Clusterware and ASM are installed into a single home directory
called Oracle Grid Infrastructure. Oracle Clusterware supports up to 100 nodes in a cluster on
configurations running Oracle Database 10g release 2 (10.2) and later releases.
Further, in Oracle 12c Release 1, Oracle has introduced Flex cluster concept. A flex cluster
uses hub and spoke topology. Flex Cluster is designed to scale up to 64 hub nodes and many
more Leaf nodes.
Cluster Interconnect
The cluster interconnect is used to synchronize the resources of the RAC cluster, and also used
to transfer some data from one instance to another. This interconnect should be private, highly
available and fast with low latency. Ideally they should be on a minimum 1GB private network.
Oracle Kernel Components

The kernel components relate to the background processes, buffer cache and shared pool and
managing the resources without conflicts and corruptions requires special handling.
In RAC, as more than one instance is accessing the resource, the instances require better
coordination at the resource management level. Each node will have its own set of buffers but
will be able to request and receive data blocks currently held in another instance's cache. The
management of data sharing and exchange is done by the Global Cache Services (GCS).
All the resources in the cluster group form a central repository called the Global Resource
Directory (GRD), which is distributed. Each instance masters some set of resources and
together all instances form the GRD. The resources are equally distributed among the nodes

based on their weight. The GRD is managed by two services called Global Caches Services
(GCS) and Global Enqueue Services (GES), together they form and manage the GRD. When a
node leaves the cluster, the GRD portion of that instance needs to be redistributed to the
surviving nodes, a similar action is performed when a new node joins.
Benefits of RAC
 RAC architecture enables users and applications to benefit from the processing power of
multiple machines.
 RAC harnesses the power of multiple low-cost computers to serve as single large
computer for database processing and provides the only viable alternative to large scale
symmetric multiprocessing (SMP) for all types of applications. If your application scales
transparently on SMP machines, it is realistic to expect it to scale well on RAC, without
having to make any changes to the application code.
 RAC, which is based on shared-disk architecture, can grow or shrink on demand
 RAC allows you to perform dynamic provisioning of nodes, storage, CPUs, and memory
to maintain service levels more easily and efficiently.
 RAC architecture also achieves redundancy in the case of system failure or
unavailability. The application can still access the database on surviving instances.
 RAC eliminates the database instance and/or the node , as a single point of failure and
ensures database integrity in case of such failures
 RAC offers automatic workload management for services. Services in RAC enable
continuous, uninterrupted database operations
 The combined processing power of the multiple servers can provide greater throughput
and scalability than is available from a single server.
Tool for Managing RAC

The following is a description of the tools used for installing, configuring, and managing the
Oracle Clusterware, Oracle Automatic Storage Management (Oracle ASM), and Oracle Real
Application Clusters (Oracle RAC):
 Oracle Universal Installer (OUI): OUI installs the Oracle grid infrastructure software (which
consists of Oracle Clusterware and Oracle ASM) and the Oracle Database software with
Oracle RAC. After you configure the nodes that you want to use in your cluster, OUI installs
the Oracle software on the specified nodes using a network connection
 Cluster Verification Utility (CVU): CVU is a command-line tool that you can use to verify a
range of cluster and Oracle RAC components such as shared storage devices, networking
configurations, system requirements, and Oracle Clusterware, as well as operating system
groups and users. You can use CVU for preinstallation as well as post installation checks of
your cluster environment. CVU is especially useful during preinstallation and during
installation of Oracle Clusterware and Oracle RAC components. OUI runs CVU after the
Oracle Clusterware installation to verify your environment.
 Oracle Enterprise Manager: Oracle Enterprise Manager Cloud Control GUI interface for
managing both noncluster database and Oracle RAC database environments. Oracle
recommends that you use Oracle Enterprise Manager to perform administrative tasks
whenever feasible. You can use Oracle Enterprise Manager Cloud Control to also manage
Oracle RAC One Node databases.

 SQL*Plus: SQL*Plus commands operate on the current instance. The current instance can
be either the local default instance on which you initiated your SQL*Plus session, or it can
be a remote instance to which you connect with Oracle Net Services.
 Server Control Utility (SRVCTL): SRVCTL is a command-line interface that you can use to
manage an Oracle RAC database from a single point. You can use SRVCTL to start and
stop the database and instances and to delete or move instances and services. You can
also use SRVCTL to manage configuration information, Oracle Real Application Clusters
One Node (Oracle RAC One Node), Oracle Clusterware, and Oracle ASM.
 Cluster Ready Services Control (CRSCTL): CRSCTL is a command-line tool that you can
use to manage Oracle Clusterware daemons. These daemons include Cluster
Synchronization Services (CSS), Cluster-Ready Services (CRS), and Event Manager
(EVM). You can use CRSCTL to start and stop Oracle Clusterware and to determine the
current status of your Oracle Clusterware installation.
 DBCA: The recommended utility for creating and initially configuring Oracle RAC and Oracle
noncluster databases.
 Oracle Automatic Storage Management Configuration Assistant (ASMCA): ASMCA is a

utility that supports installing and configuring Oracle ASM instances, disk groups, volumes,
and Oracle Automatic Storage Management Cluster File System (Oracle ACFS). ASMCA
provides both a GUI and a non-GUI interface.
 NETCA: Configures the network for your Oracle RAC environment.

Oracle Clusterware Concepts

Oracle Clusterware provides a complete set of cluster services to support the shared disk, load
balancing cluster architecture of the Oracle Real Application Cluster (RAC) database. Oracle
Clusterware can also be used to provide failover clustering services for single instance Oracle
databases and other applications.
The services provided by Oracle Clusterware include:

 Cluster Management, which allows cluster services and application resources to be
monitored and managed from any node in the cluster
 Node Monitoring, which provides real-time information regarding which nodes are currently
available and the resources they support. Cluster integrity is also protected by evicting or
fencing unresponsive nodes.
 Event Services, which publishes cluster events so that applications are aware of changes
in the cluster
 Time Synchronization, which synchronizes the time on all nodes of the cluster
 Network Management, which provisions and manages Virtual IP (VIP) addresses that are
associated with cluster nodes or application resources to provide a consistent network
identity regardless of which nodes are available. In addition, Grid Naming Service (GNS)
manages network naming within the cluster.
 High Availability, which services, monitors, and restarts all other resources as required
About OCR & Voting Disks

Oracle Clusterware includes two important components that manage configuration and node
membership: Oracle Cluster Registry (OCR), which also includes the local component Oracle
Local Registry (OLR), and Voting disks.
 OCR manages Oracle Clusterware and Oracle RAC database configuration information
 OLR resides on every node in the cluster and manages Oracle Clusterware configuration
information for each particular node
 Voting disks manage information about node membership. Each voting disk must be
accessible by all nodes in the cluster for nodes to be members of the cluster
You can store OCR and voting disks on Oracle Automatic Storage Management (Oracle ASM),
or a certified cluster file system
Voting Disks
The voting disk (or quorum disk) is a file, where heartbeat information of all nodes will be stored
by CSSD continuously. Oracle Clusterware uses voting disk files to determine which nodes are
members of a cluster. You can configure voting disks on Oracle ASM, or you can configure
voting disks on shared storage. If you configure voting disks on Oracle ASM, then you do not
need to manually configure the voting disks. Depending on the redundancy of your disk group,
an appropriate number of voting disks are created.
When voting disks are placed into an ASM diskgroup, the number of voting disks that are
created depend on the redundancy level of the diskgroup. For external redundancy, 1 voting
disk is created. When running ASM redundancy, voting disks have a higher requirement for the
number of disks. Normal redundancy creates 3 voting disks, and high redundancy creates 5.
Oracle recommends at least 400MB per voting disk file, and 400MB for each copy of the OCR.

If you do not configure voting disks on Oracle ASM, then for high availability, Oracle
recommends that you have a minimum of three voting disks on physically separate storage.
This avoids having a single point of failure. If you configure a single voting disk, then you must
use external mirroring to provide redundancy. You should have at least three voting disks,
unless you have a storage device, such as a disk array that provides external redundancy.
Oracle recommends that you do not use more than five voting disks.
As far as voting disks are concerned, a node must be able to access strictly more than half of
the voting disks at any time. So if you want to be able to tolerate a failure of “n” voting disks, you
must have at least 2n+1 configured. (n=1 means 3 voting disks). You can configure minimum of
1 and maximum 15 copies of voting disks.
Storing Voting Disks on ASM

Oracle ASM manages voting disks differently from other files that it stores. If you choose to
store your voting disks in Oracle ASM, then Oracle ASM stores all the voting disks for the
cluster in the disk group you choose.
Once you configure voting disks on Oracle ASM, you can only make changes to the voting
disks' configuration using the crsctl replace votedisk command. This is true even in
cases where there are no working voting disks. Despite the fact that crsctl query css
votedisk reports zero vote disks in use, Oracle Clusterware remembers the fact that Oracle
ASM was in use and the replace verb is required. Only after you use the replace verb to
move voting disks back to non-Oracle ASM storage are the verbs add css votedisk and
delete css votedisk again usable.
The number of voting files you can store in a particular Oracle ASM disk group depends upon
the redundancy of the disk group.
 External redundancy: A disk group with external redundancy can store only 1 voting disk
 Normal redundancy: A disk group with normal redundancy stores 3 voting disks
 High redundancy: A disk group with high redundancy stores 5 voting disks
Starting from Oracle 11g release 2 (11.2), if voting disks are stored on Oracle ASM with normal
or high redundancy, and the storage hardware in one failure group suffers a failure, then if there
is another disk available in a disk group in an unaffected failure group, Oracle ASM
automatically recovers the voting disk in the unaffected failure group. In previous versions, this
automation did not exist. If a voting disk went offline, the DBA had to manually create a new
voting disk.
Keep in mind that this feature will only take effect if there must be another failgroup available to
place the voting disk in. If you only have 3 failgroups for your normal redundancy (or 5 for your
high redundancy) OCR/voting diskgroup, this automatic recreation will not occur.
Using the crsctl replace votedisk command, you can move a given set of voting
disks from one Oracle ASM disk group into another, or onto a certified file system. If you move
voting disks from one Oracle ASM disk group to another, then you can change the number of
voting disks by placing them in a disk group of a different redundancy level as the former disk
group.

Note:
 You cannot directly influence the number of voting disks in ASM disk group.
 You cannot use the crsctl add | delete votedisk commands on voting
disks stored in Oracle ASM disk groups because Oracle ASM manages the number of
voting disks according to the redundancy level of the disk group.
Oracle Cluster Registry (OCR)

Oracle Clusterware uses the Oracle Cluster Registry (OCR) to store and manage information
about the components that Oracle Clusterware controls, such as Oracle RAC databases,
listeners, virtual IP addresses (VIPs), and services and any applications. OCR stores
configuration information in a series of key-value pairs in a tree structure.
What is stored in OCR?

The following information is stored in OCR:
 node membership information i.e. which nodes are part of the cluster
 software active version
 location of voting disks
 server pools
 status for the cluster resources such as RAC databases, listeners, instances, and services
− server up/down
− network up/down
− database up/down
− instance up/down
− listener up/down …
 configuration for the cluster resources such as RAC databases, listeners, instances, and
services.
− dependencies
− management policy (automatic/manual)
− callout scripts
− retries
 cluster database instance to node mapping

 ASM instance, disk groups etc.
 CRS application resource profiles such as VIP addresses, services etc.
 database services’ characteristics
− preferred/available nodes
− TAF policy
− load balancing goal
 information about clusterware processes
 information about interaction and management of third party applications controlled by CRS
 details of the network interfaces held by the cluster network
 communication settings where the Clusterware daemons or background processes listen
 information about OCR backups
Who updates OCR?

OCR, which contains information about the high-availability components of the RAC cluster, is
maintained and updated by several client applications including the following:

m
co
l.
ai

gm
CSSd during cluster setup - to update the status of servers
 CSS during node addition/deletion – to add/delete node names
7@
 CRSd about status of nodes during failure/reconfiguration

71
Oracle Universal Installer (OUI)
n2
 SRVCTL (used to manage clusters and RAC databases/instance)
 Cluster control utility – CRSCTL (to manage cluster/local resources)
wa
 Enterprise Manager (EM)

ha
 Database Configuration assistant (DBCA)

id
 Database Upgrade Assistant (DBUA)

it
 Network Configuration Assistant (NETCA)

 ASM Configuration Assistant (ASMCA).
Pr
Each node in the cluster maintains a copy of the OCR in memory for better performance and
each node is responsible for updating the OCR as required. The CRSd process is responsible
for reading and writing to the OCR files as well as refreshing the local OCR cache and the
caches on the other nodes in the cluster. Oracle uses a distributed shared cache architecture
during cluster management to optimize queries against the cluster repository. Each node
maintains a copy of the OCR in memory. Oracle Clusterware uses a background process to
access the OCR cache. Only one CRSd process (designated as the master) in the cluster
performs any disk read/write activity. Once any new information is read by the master CRSd
process, it performs a refresh of the local OCR cache and the OCR cache on other nodes in the
cluster. Since the OCR cache is distributed across all nodes in the cluster, OCR clients (srvctl,
crsctl etc.) communicate directly with the local OCR process on the node to obtain required
information. Clients communicate via the local CRSd process for any updates on the physical
OCR binary file.
The following block diagram shows the architecture of Oracle Cluster Registry.
How is the info stored in OCR?

The OCR uses a file-based repository to store configuration information in a series of key-value
pairs, using a directory tree-like structure. It contains information pertaining to all tiers of the
clustered database. Various parameters are stored as name-value pairs used and maintained at
different levels of the architecture. Each tier is managed and administrated by daemon
processes with appropriate privileges to manage them. For example,

 All SYSTEM level resource or application definitions would require root, or super user,
privileges to start, stop, and execute resources defined at this level.
 those defined at the DATABASE level will require dba privileges to execute
Considering the criticality of the OCR contents to the cluster functionality, Oracle strongly
recommends you to multiplex the OCR file. From Oracle 11g Release 2, you can have up to 5
OCR copies.
NOTE: When using ASM Diskgroup for Grid Infrastructure, the following files are generally
stored in ASM Diskgroup:
 Oracle Cluster Registry file (OCR)

 Voting Files
 Shared SPFILE for the ASM Instances
Oracle Local Registry

In Oracle Clusterware 11g Release 2 an additional component related to the OCR called the
Oracle Local Registry (OLR) is installed on each node in the cluster. The OLR is a local registry
for node specific resources. The OLR is not shared by other nodes in the cluster. It is installed
and configured when Oracle clusterware is installed
Multiple processes on each node have simultaneous read and write access to the OLR
particular to the node on which they reside, regardless of whether Oracle Clusterware is running
or fully functional.
OLR is the very first file that is accessed to startup clusterware when OCR is stored on ASM.
OCR should be accessible to find out the resources which need to be started on a node. If OCR
is on ASM, it can’t be read until ASM (which itself is a resource for the node and this information
is stored in OCR) is up. To resolve this problem, information about the resources which need to
be started on a node is stored in an operating system file which is called Oracle Local Registry
or OLR. Since OLR is an operating system file, it can be accessed by various processes on the
node for read/write irrespective of the status of the clusterware (up/down). Hence, when a node
joins the cluster, OLR on that node is read, various resources, including ASM are started on the
node. Once ASM is up, OCR is accessible and is used henceforth to manage all the clusterware
resources. If OLR is missing or corrupted, clusterware can’t be started on that node.
The OLR stores:

- version of clusterware
- clusterware configuration
- configuration of various resources which need to be started on the node etc…
By default, OLR is located at Grid_home/cdata/host_name.olr on each node. The

location of OLR is stored in /etc/oracle/olr.loc. and used by OHASD.
You can manage OLR using the OCRCHECK, OCRDUMP, and OCRCONFIG utilities as root
with the -local option.

GPnP Profile
The GPnP profile is a small XML file located in GRID_HOME/gpnp/<hostname>/profiles/peer
under the name profile.xml. It is used to establish the correct global personality of a node.
Each node maintains a local copy of the GPnP Profile and is maintained by the GPnP Daemon
(GPnPD).
GPnP Profile is used to store necessary information required for the startup of Oracle
Clusterware like SPFILE location, ASM Disk String etc. It contains various attributes defining
node personality like:
 Cluster name
 Network classifications (Public/Private)
 Storage to be used for CSS
 Storage to be used for ASM : SPFILE location, ASM Disk String etc
 Digital signature information
Profile is updated whenever changes are made to a cluster with configuration tools like
− oifcfg (Change network)
− crsctl (change location of voting disk)
− asmcmd (change ASM_DISKSTRING, SPfile location) etc.
GPnPd daemon replicates changes to the profile during installation, system boot and when
updated.
How is GPnP profile used by clusterware?

To start clusterware, voting disk needs to be accessed. If voting disk is on ASM, this information
(that voting disk is on ASM) is read from GPnP profile. The voting disk is read using kfed utility
even if ASM is not up. Next, the clusterware checks if all the nodes have the updated GPnP
profile and the node joins the cluster based on the GPnP configuration. Whenever a node is
started/added to the cluster, the clusterware software on the starting node starts a GPnP agent.
Next CRSD needs to read OCR to startup various resources on the node. Since OCR is also on
ASM, location of ASM SPfile should be known.
The order of searching the ASM SPfile is:

- GPnP profile
- ORACLE_HOME/dbs/spfile<sid.ora>
- ORACLE_HOME/dbs/init<sid.ora>
In cluster environment, the location of SPfile for ASM is read from GPnP profile. Using the
parameters in SPfile, ASM is started. Once ASM is up, OCR is read by CRSD and various
resources on the node are started. Each node reads network information in GPnP profile and
using GNS, negotiates appropriate network identity for itself. Hence, nodes can be dynamically
added/deleted.
About Cluster Interconnect

A cluster consists of one or more servers. The hardware in a server in a cluster (or cluster
member or node) is similar to a standalone server. However, a server that is part of a cluster,
otherwise known as a node or a cluster member, requires a second network. This second
network is referred to as the interconnect. For this reason, cluster member nodes require at

least two network interface cards: one for a public network and one for a private network. The
interconnect network is a private network using a
switch (or multiple switches) that only the nodes in the cluster can access.
You can obtain the details of cluster interconnect by using the following query.
SQL> select * from GV$CLUSTER_INTERCONNECTS;
About Virtual IP (VIP)

In a typical RAC environment the database is served by multiple listeners running on multiple
nodes. For Failover configuration the listener configuration includes the IP addresses of all
nodes on which listeners are running. Pre Oracle 10g (i.e until Oracle 9i) releases used Physical
IPs for this purpose. When a user process tries to get connection through a listener which is
down then Oracle RAC automatically fails over this request to the next available listener in the
configuration. However the connection need to wait till TCP/IP connection timeout before the
failover to next listener occurs. This timeout could be as high as 10 minutes. Without using
VIPs, clients connected to a node that died will often wait a 10-minute TCP timeout period
before getting an error. As a result, you do not really have a good High Availability solution
without using VIPs. When you use VIPs in listener configuration (instead of Physical IP) the
connection need not wait till TCP/IP timeout. Once ONS finds that a particular listener is down, it
diverts the connection request immediately to another listener running on surviving nodes
without waiting for TCP/IP timeout.
Oracle 10g introduced the concept of Virtual IP. With Oracle 10g RAC, each node requires a
VIP. Starting with Oracle RAC 11g Release 2, three (3) additional SCAN VIPs are required for
the cluster. When a node fails, the VIP associated with it is automatically failed over to some
other node.
Virtual IP (VIP) addresses are all about availability of applications when an entire node fails.
When a node fails, the VIP address associated with it automatically fails over to some other
node in the cluster. When this occurs:
 The new node indicates the new Media Access Control (MAC) address for the VIP. For
directly connected clients, this usually causes them to see errors on their connections to
the old address.
 Subsequent packets sent to the VIP address go to the new node, which will send error
reset (RST) packets back to the clients. This results in the clients getting errors
immediately
The VIPs should be registered in the DNS. The VIP addresses must be on the same subnet as
the public host network addresses.
Single Client Access Name (SCAN)

Oracle RAC 11g release 2 (11.2) introduced the Single Client Access Name (SCAN). The single
client access name is the address used by clients connecting to the cluster. The SCAN is a fully
qualified host name (host name + domain) registered to three IP addresses.
If you use GNS, and you have DHCP support, then the GNS will assign addresses dynamically
to the SCAN. If you do not use GNS, the SCAN should be defined in the DNS to resolve to the

three addresses assigned to that name. This should be done before you install Oracle Grid
Infrastructure.
The SCAN and its associated IP addresses provide a stable name for clients to use for
connections, independent of the nodes that make up the cluster.
SCANs function like a cluster alias. However, SCANs are resolved on any node in the cluster,
so unlike a VIP address for a node, clients connecting to the SCAN no longer require updated
VIP addresses as nodes are added to or removed from the cluster. Because the SCAN
addresses resolve to the cluster, rather than to a node address in the cluster, nodes can be
added to or removed from the cluster without affecting the SCAN address configuration.
During installation, listeners are created on each node for the SCAN IP addresses. Oracle
Clusterware routes application requests to the cluster SCAN to the least loaded instance
providing the service.
SCAN listeners can run on any node in the cluster. SCANs provide location independence for
the databases so that client configuration does not have to depend on which nodes run a
particular database.
Oracle Database 11g Release 2 and later instances register with SCAN listeners only as remote
listeners. Upgraded databases register with SCAN listeners as remote listeners, and also
continue to register with all other listeners. If you specify a GNS domain during installation, the
SCAN defaults to clustername-scan.GNS_domain. If a GNS domain is not specified at
installation, the SCAN defaults to clustername-scan.current_domain.
SCAN and Local Listeners

When a client submits a connection request, the SCAN listener listening on a SCAN IP address
and the SCAN port is contracted on the client’s behalf. Because all services on the cluster are
registered with the SCAN listener, the SCAN listener replies with the address of the local
listener on the least-loaded node where the service is currently being offered. Finally, the client
establishes connection to the service through the listener on the node where service is offered.
All of these actions take place transparently to the client without any explicit configuration
required in the client.
During installation, listeners are created on nodes for the SCAN IP addresses. Oracle Net
Services routes application requests to the least loaded instance providing the service. Because
the SCAN addresses resolve to the cluster, rather than to a node address in the cluster, nodes
can be added to or removed from the cluster without affecting the SCAN address configuration
The following diagram shows how SCAN and Node listeners (local listeners) are used in RAC.

The following diagram shows Oracle RAC cluster and various components of RAC cluster

ASM Concepts
This chapter describes Oracle Automatic Storage Management (Oracle ASM) concepts and
provides an overview of Oracle ASM features.
About Oracle ASM

Oracle ASM is a volume manager and a file system for Oracle database files that supports
single-instance Oracle Database and Oracle Real Application Clusters (Oracle RAC)
configurations. Oracle ASM is Oracle's recommended storage management solution that
provides an alternative to conventional volume managers, file systems, and raw devices.
Oracle ASM uses disk groups to store data files; an Oracle ASM disk group is a collection of
disks that Oracle ASM manages as a unit. Within a disk group, Oracle ASM exposes a file
system interface for Oracle database files. The content of files that are stored in a disk group is
evenly distributed to eliminate hot spots and to provide uniform performance across the disks.
The performance is comparable to the
performance of raw devices.
You can add or remove disks from a disk group while a database continues to access files from
the disk group. When you add or remove disks from a disk group, Oracle ASM automatically
redistributes the file contents and eliminates the need for downtime when redistributing the
content.
The Oracle ASM volume manager functionality provides flexible server-based mirroring options.
The Oracle ASM normal and high redundancy disk groups enable two-way and three-way
mirroring respectively. You can use external redundancy to enable a Redundant Array of
Independent Disks (RAID) storage subsystem to perform the mirroring protection function.
Oracle ASM also uses the Oracle Managed Files (OMF) feature to simplify database file
management. OMF automatically creates files in designated locations. OMF also names files
and removes them while relinquishing space when tablespaces or files are deleted.
Oracle ASM reduces the administrative overhead for managing database storage by
consolidating data storage into a small number of disk groups. The smaller number of disk
groups consolidates the storage for multiple databases and provides for improved I/O
performance.
Oracle ASM files can coexist with other storage management options such as raw disks and
third-party file systems. This capability simplifies the integration of Oracle ASM into pre-existing
environments.
Oracle Enterprise Manager includes a wizard that enables you to migrate non-Oracle ASM
database files to Oracle ASM. Oracle ASM also has easy to use management interfaces such
as SQL*Plus, the Oracle ASM Command Line Utility (ASMCMD) command-line interface,
Oracle ASM Configuration Assistant, and Oracle Enterprise Manager.

About Oracle ASM Instances

An Oracle ASM instance is built on the same technology as an Oracle Database instance. An
Oracle ASM instance has a System Global Area (SGA) and Background Processes that are
similar to those of Oracle Database. However, because Oracle ASM performs fewer tasks than
a database, an Oracle ASM SGA is much smaller than a database SGA. In addition, Oracle
ASM has a minimal performance effect on a server. Oracle ASM instances mount disk groups to
make Oracle ASM files available to database instances; Oracle ASM instances do not mount
databases.
Oracle ASM is installed in the Oracle Grid Infrastructure home before Oracle Database is
installed in a separate Oracle home. Oracle ASM and database instances require shared
access to the disks in a disk group. Oracle ASM instances manage the metadata of the disk
group and provide file layout information to the database instances.
Oracle ASM metadata is the information that Oracle ASM uses to control a disk group and the
metadata resides within the disk group. Oracle ASM metadata includes the following
information:
 The disks that belong to a disk group
 The amount of space that is available in a disk group
 The filenames of the files in a disk group
 The location of disk group data file extents
 A redo log that records information about atomically changing metadata blocks
 Oracle ADVM volume information
Oracle ASM instances can be clustered using Oracle Clusterware; there is one Oracle ASM
instance for each cluster node. If there are several database instances for different databases
on the same node, then the database instances share the same single Oracle ASM instance on
that node.
If the Oracle ASM instance on a node fails, then all of the database instances on that node also
fail. Unlike a file system driver failure, an Oracle ASM instance failure does not require restarting
the operating system. In an Oracle RAC environment, the Oracle ASM and database instances
on the surviving nodes automatically recover from an Oracle ASM instance failure on a node.
The following figure shows a single node configuration with one Oracle ASM instance and
multiple database instances. The Oracle ASM instance manages the metadata and provides
space allocation for the Oracle ASM files. When a database instance creates or opens an
Oracle ASM file, it communicates those requests to the Oracle ASM instance. In response, the
Oracle ASM instance provides file extent map information to the database instance.

The following figure shows an Oracle ASM cluster in an Oracle RAC environment where Oracle
ASM provides a clustered pool of storage. There is one Oracle ASM instance for each node
serving multiple Oracle RAC or single-instance databases in the cluster. All of the databases
are consolidated and share the same two Oracle ASM disk groups.
To share a disk group among multiple nodes, you must install Oracle Clusterware on all of the
nodes, regardless of whether you install Oracle RAC on the nodes. Oracle ASM instances that

are on separate nodes do not need to be part of an Oracle ASM cluster. However, if the Oracle
ASM instances are not part of an Oracle ASM cluster, they cannot communicate with each
other. Multiple nodes that are not part of an Oracle ASM cluster cannot share a disk group.
About Oracle ASM Disk Groups

A disk group consists of multiple disks and is the fundamental object that Oracle ASM manages.
Each disk group contains the metadata that is required for the management of space in the disk
group. Disk group components include disks, files, and allocation units.
Files are allocated from disk groups. Any Oracle ASM file is completely contained within a single
disk group. However, a disk group might contain files belonging to several databases and a
single database can use files from multiple disk groups. For most installations you need only a
small number of disk groups, usually two, and rarely more than three.
You can store the following file types in ASM disk groups:
 Control files
 Datafiles, temporary datafiles, and datafile copies
 SPFILEs
 Online redo logs, archive logs, and Flashback logs
 RMAN backups
 Disaster recovery configurations
 Change tracking bitmaps
 Data Pump dumpsets
Note: Oracle executables and ASCII files, such as alert logs and trace files, cannot be
stored in ASM disk groups.
About Mirroring and Failure Groups

Mirroring protects data integrity by storing copies of data on multiple disks. When you create a
disk group, you specify an Oracle ASM disk group type based on one of the following three
redundancy levels:
 Normal for 2-way mirroring

 High for 3-way mirroring
 External to not use Oracle ASM mirroring, such as when you configure hardware RAID
The redundancy level controls how many disk failures are tolerated without dismounting the disk
group or losing data. The disk group type determines the mirroring levels with which Oracle
creates files in a disk group.
Oracle ASM mirroring is more flexible than traditional RAID mirroring. For a disk group specified
as NORMAL redundancy, you can specify the redundancy level for each file. For example, two
files can share the same disk group with one file being mirrored while the other is not.
A Failure Group is a subset of the disks in a disk group. Failure groups define disks that share
components, such that if one fails then other disks sharing the component might also fail. An
example of what you might define as a failure group would be a set of SCSI disks sharing the
same SCSI controller. Failure groups are used to determine which Oracle ASM disks to use for
storing redundant data. For example, if two-way mirroring is specified for a file, then redundant

m
co
l.
ai
gm
copies of file extents must be stored in separate failure groups. By default, Oracle ASM puts
each voting disk in its own failure group within the disk group.
7@
71
A normal redundancy disk group must contain at least two failure groups but if you are storing
n2
your voting disks on Oracle ASM, then a normal redundancy disk group must contain at least
three failure groups. A high redundancy disk group must contain at least three failure groups.
wa
However, Oracle recommends using several failure groups. A small number of failure groups, or
ha
failure groups of uneven capacity, can create allocation problems that prevent full use of all of
id
the available storage.

it
When Oracle ASM allocates an extent for a mirrored file, Oracle ASM allocates a primary copy
Pr
and a mirror copy. Oracle ASM chooses the disk on which to store the mirror copy in a different
failure group than the primary copy. Failure groups are used to place mirrored copies of data so
that each copy is on a disk in a different failure group. The simultaneous failure of all disks in a
failure group does not result in data loss.
You define the failure groups for a disk group when you create an Oracle ASM disk group. After
a disk group is created, you cannot alter the redundancy level of the disk group. If you omit the
failure group specification, then Oracle ASM automatically places each disk into its own failure
group. Normal redundancy disk groups require at least two failure groups. High redundancy disk
groups require at least three failure groups. Disk groups with external redundancy do not use
failure groups.
How Oracle ASM Manages Disk Failures

Depending on the redundancy level of a disk group and how you define failure groups, the
failure of one or more disks could result in either of the following:
 The disks are first taken offline and then automatically dropped. In this case, the disk
group remains mounted and serviceable. In addition, because of mirroring, all of the disk
group data remains accessible. After the disk drop operation, Oracle ASM performs a
rebalance to restore full redundancy for the data on the failed disks.
 The entire disk group is automatically dismounted, which means loss of data
accessibility
Guidelines for Using Failure Groups

The following are guidelines for using failure groups:
 Each disk in a disk group can belong to only one failure group.
 Failure groups should all be of the same size. Failure groups of different sizes may lead
to reduced availability.
 Oracle ASM requires at least two failure groups to create a normal redundancy disk
group and at least three failure groups to create a high redundancy disk group.
About Oracle ASM Disks

Oracle ASM disks are the storage devices that are provisioned to Oracle ASM disk groups.
Examples of Oracle ASM disks include:
 A disk or partition from a storage array

 An entire disk or the partitions of a disk

 Logical volumes
 Network-attached files (NFS)
When you add a disk to a disk group, you can assign an Oracle ASM disk name or Oracle ASM
assigns the Oracle ASM disk name automatically. This name is different from the path name
used by the operating system. In a cluster, a disk may be assigned different operating system
device names on different nodes, but the disk has the same Oracle ASM disk name on all of the
nodes. In a cluster, an Oracle ASM disk must be accessible from all of the instances that share
the disk group.
Oracle ASM spreads the files proportionally across all of the disks in the disk group. This
allocation pattern maintains every disk at the same capacity level and ensures that all of the
disks in a disk group have the same I/O load. Because Oracle ASM load balances among all of
the disks in a disk group, different Oracle ASM disks should not share the same physical drive.
Allocation Units
Every Oracle ASM disk is divided into allocation units (AU). An allocation unit is the fundamental
unit of allocation within a disk group. A file extent consists of one or more allocation units. An
Oracle ASM file consists of one or more file extents.
When you create a disk group, you can set the Oracle ASM allocation unit size with the
AU_SIZE disk group attribute. The values can be 1, 2, 4, 8, 16, 32, or 64 MB, depending on the
specific disk group compatibility level. Larger AU sizes typically provide performance
advantages for data warehouse applications that use large sequential reads.
About Oracle ASM Files

Files that are stored in Oracle ASM disk groups are called Oracle ASM files. Each Oracle ASM
file is contained within a single Oracle ASM disk group. Oracle Database communicates with
Oracle ASM in terms of files. This is similar to the way Oracle Database uses files on any file
system. You can store the various file types in Oracle ASM disk groups, including:
 Control files
 Data files, temporary data files, and data file copies
 SPFILEs
 Online redo logs, archive logs, and Flashback logs
 RMAN backups
 Disaster recovery configurations
 Change tracking bitmaps
 Data Pump dumpsets
Oracle ASM automatically generates Oracle ASM file names as part of file creation and
tablespace creation. Oracle ASM file names begin with a plus sign (+) followed by a disk group
name. You can specify user-friendly aliases for Oracle ASM files and create a hierarchical
directory structure for the aliases.
The following sections describe the Oracle ASM file components:

 Extents
 Oracle ASM Striping

Extents
The contents of Oracle ASM files are stored in a disk group as a set, or collection, of extents
that are stored on individual disks within disk groups. Each extent resides on an individual disk.
Extents consist of one or more allocation units (AU). To accommodate increasingly larger files,
Oracle ASM uses variable size extents.
Variable size extents enable support for larger Oracle ASM data files, reduce SGA memory
requirements for very large databases, and improve performance for file create and open
operations. The initial extent size equals the disk group allocation unit size and it increases by a
factor of 4 or 16 at predefined thresholds. This feature is automatic for newly created and
resized data files when specific disk group compatibility attributes are set to 11.1 or higher.
The extent size of a file varies as follows:

 Extent size always equals the disk group AU size for the first 20000 extent sets (0 -
19999).
 Extent size equals 4*AU size for the next 20000 extent sets (20000 - 39999).
 Extent size equals 16*AU size for the next 20000 and higher extent sets (40000+).
Oracle ASM Striping

Oracle ASM striping has two primary purposes:
 To balance loads across all of the disks in a disk group
 To reduce I/O latency
Coarse-grained striping provides load balancing for disk groups while fine-grained striping
reduces latency for certain file types by spreading the load more widely. To stripe data, Oracle
ASM separates files into stripes and spreads data evenly across all of the disks in a disk group.
The fine-grained stripe size always equals 128 KB in any configuration; this provides lower I/O
latency for small I/O operations. The coarse-grained stripe size is always equal to the AU size
(not the data extent size).
Oracle ASM Disk Group Concepts

This section describes the concepts related to Oracle ASM disk group administration.
About Discovering Disks

The disk discovery process locates the operating system names for disks that Oracle ASM can
access. Disk discovery finds all of the disks that comprise a disk group to be mounted. The set
of discovered disks also includes disks that could be added to a disk group.
An Oracle ASM instance requires an ASM_DISKSTRING initialization parameter value to

specify its discovery strings. Only path names that the Oracle ASM instance has permission to
open are discovered. The path names that an operating system accepts are always usable as
discovery strings.
About Mounting and Dismounting Disk Groups

A disk group must be mounted by a local Oracle ASM instance before database instances can
access the files in the disk group. Mounting the disk group requires discovering all of the disks
and locating the files in the disk group that is being mounted.
You can explicitly dismount a disk group. Oracle reports an error if you attempt to dismount a
disk group without the force option when any of the disk group files are open. It is possible to

have disks fail in excess of the Oracle ASM redundancy setting. If this happens, then the disk
group is forcibly dismounted. If the disk group is forcibly dismounted, a database cannot access
files in the disk group.
About Adding and Dropping Disks

You can add a disk to an existing disk group to add space and to improve throughput. The
specified discovery string identifies the disk or disks that you could add. The disks that you add
must be discovered by every Oracle ASM instance using its ASM_DISKSTRING initialization
parameter. After you add a disk, Oracle ASM rebalancing operations move data onto the new
disk. To minimize the rebalancing I/O, it is more efficient to add multiple disks at the same time.
You can drop a disk from a disk group if it fails or to re-purpose capacity. Use the Oracle ASM
disk name to drop a disk, not the discovery string device name. If an error occurs while writing
to a disk, then Oracle ASM drops the disk automatically.
About Online Storage Reconfigurations and Dynamic Rebalancing

Any storage configuration change will trigger a rebalance. The main objective of the rebalance
operation is to provide an even distribution of file extents and space usage across all disks in
the diskgroup.
Rebalancing is performed on all database files on a per file basis. The ASM background
process, RBAL, responsible for rebalancing activity.
The Rebalance process examines each file extent map, and the new extents are re-plotted
on to the new configuration. For example, consider an 8-disk diskgroup, with a datafile with 40
extents (each disk will house 5 extents). When 2 new drives (of same size) are added, that
datafile is rebalanced and spread across 10 drives, with each drive containing 4 extents. Only 8
extents need to move to complete the rebalance.
Rebalancing a disk group moves data between disks to ensure that every file is evenly spread
across all of the disks in a disk group. When all of the files are evenly dispersed, all of the disks
are evenly filled to the same percentage; this ensures load balancing. Rebalancing does not
relocate data based on I/O statistics nor is rebalancing started based on I/O statistics. Oracle
ASM rebalancing operations are controlled by the size of the disks in a disk group.
The following is a typical process flow for ASM rebalancing:

1. On the ASM instance, DBA adds (or drops) a disk to (from) a diskgroup.
2. This invokes the RBAL process to create the rebalance plan and then begin coordination of
the redistribution
3. RBAL will calculate estimation time and work required to perform the task and then message
the ARB0, ARB1, etc to actually process the request. The number of ARBx processes invoked
is directly determined by the asm_power_limit initialization parameter.
4. The Continuing Operations Directory (metadata) will be updated to reflect a rebalance
activity.
5. Each extent to be relocated is assigned to an ARBx process.
6. ARBx performs rebalance on these extents. Each extent is locked, relocated, and unlocked.
This is shown as Operation REBAL.
Rebalancing involves physical movement of file extents. This impact is generally low because
the rebalance is done one extent/AU at a time; therefore, there will be only be one outstanding

I/O at any given time. This should not adversely affect online database activity. However, it is
generally advisable to schedule the rebalance operation during off-peak hours.
The init.ora parameter, asm_power_limit, influences the throughput and speed of the
rebalance operation. The range of values for asm_power_limit are 1 to 11; where a value of 11
is full throttle and a value of 1 is low speed. However, the power value can also be set for a
specific rebalance activity using the alter diskgroup command. This value is only effective for
the specific rebalance task. A power value of 0 indicates that no rebalance should occur
for this rebalance. This setting is particularly important to set when adding or removing
storage (that has redundancy), and then deferring the rebalance to a later time.
Oracle ASM automatically initiates a rebalance after storage configuration changes, such as
when you add, drop, or resize disks. You can manually start a rebalance to change the power
setting of a running rebalance. A rebalance is automatically restarted if the instance on which
the rebalancing is running stops. Databases can remain operational during rebalancing
operations.
OCR and Voting Files in Oracle ASM Disk Groups

The voting files and OCR are two important components of Oracle Clusterware. You can store
Oracle Cluster Registry (OCR) and voting files in Oracle ASM disk groups.
Voting files manage information about node membership. OCR is a system that manages
cluster and Oracle Real Application Clusters (Oracle RAC) database configuration information.
For Oracle Clusterware files a minimum of three disk devices or three failure groups is required
with a normal redundancy disk group.
Interaction between ASM instance RDBMS instance

Oracle ASM instances mount disk groups to make Oracle ASM files available to database
instances; Oracle ASM instances do not mount databases. ASM instance manage the metadata
needed to make ASM files available to all RDBMS instances running on the node. Both ASM
Instance and Database instances require shared access to the disks in a disk group.
When a database instance creates, opens, or deletes an Oracle ASM file, it communicates
those requests to the Oracle ASM instance. In response, the Oracle ASM instance provides file
extent map information to the database instance. Database instances access contents of ASM
files directly, communicating with an ASM instance only to get information about the layout of
these files. Oracle Database communicates with Oracle ASM in terms of files. ASM and
Database Instances also interact if the storage configuration changes, i.e. when disks are
added, dropped or fail.
The file creation process provides an illustration of the interactions that take place between
Database Instances and ASM. The file creation process occurs as follows:
1. The database requests file creation.
2. An ASM foreground process creates a Continuing Operation Directory (COD) entry and
allocates space for the new file across the disk group.
3. A background database process (ASMB) receives an extent map for the new file.
4. The file is now open and the database process initializes the file directly.

5. After initialization, the database process requests that the file creation be committed.
This causes the ASM foreground process to clear the COD entry and mark the file as
created.
6. Acknowledgement of the file commit implicitly closes the file. The database instance will
need to reopen the file for future I/O.
The above sequence of steps is depicted in the following diagram:
There are two important points to consider from this example:

 The database instance and ASM instance work together in a coordinated fashion. A
database instance must interact with ASM to map database files to ASM extents. It also
receives a stream of messages relating to ASM operations (such as disk group rebalancing)
that may lock or move ASM extents.
 Database I/O is not channeled through the ASM instance. In fact, the database conducts
I/O operations directly against ASM files as mentioned in the below diagram.

All ASM management commands are directed through the ASM instance, no ASM commands
are issued to any regular Oracle database instance that is using ASM files.

New Features in RAC 12c

This chapter explains the new features introduced in Oracle 12c RAC.
Oracle Flex Cluster

Oracle Flex Clusters is a cluster that contains two types of nodes arranged in a hub and spoke
architecture: Hub Nodes and Leaf Nodes.
The number of Hub Nodes in an Oracle Flex Cluster can be as many as 64. The number of Leaf
Nodes can be many more. Hub Nodes and Leaf Nodes can host different types of applications.
Hub Nodes are similar to Oracle Grid Infrastructure nodes in an Oracle Clusterware standard
Cluster configuration: they are tightly connected among themselves via private network and
have direct access to shared storage just like in previous versions of Oracle. These nodes are
the ones that access the Oracle Cluster Registry (OCR) and Voting Disk (VD) directly.
Leaf Nodes are different from standard Oracle Grid Infrastructure nodes, in that they do not
require direct access to shared storage, but instead request data through Hub Nodes.
These nodes are lighter and are not connected among them; neither accesses the shared
storage like the Hub Nodes. Each Leaf Node communicates with the Hub Node that is attached
to, and its connected to the cluster via the Hub Node that is linked to.
Hub Nodes can run in an Oracle Flex Cluster configuration without having any Leaf Nodes as
cluster member nodes, but Leaf Nodes must be members of a cluster that includes at least one
Hub Node.
This topology allows loosely coupled application servers to form a cluster with tightly coupled
database servers. Tightly coupled servers are Hub Servers that share storage for database,
OCR and Voting devices as well as peer-to-peer communication with other Hub Servers in the
cluster. A loosely coupled server is a Leaf Server that has a loose communication association
with a single Hub Server in the cluster and does not require shared storage nor peer-to-peer
communication with other Hub or Leaf Servers in the cluster, except to communicate with the
Hub to which it is associated.
Overall, Flex Clusters enhance the ability of Grid Infrastructure to manage the complete server
environment: application servers as well as database servers.
The following figure shows a typical Oracle Flex Cluster with four Leaf nodes and two Hub
nodes.

Oracle Flex ASM

Prior to Oracle 12c, for a database instance to use ASM it is expected that the ASM instance
must be up and running on all nodes before the database instance is brought up. Failure of
ASM instance to come-up means that database instance using ASM at the storage level cannot
be brought up.
The following figure shows the ASM and Database instance association in Oracle versions prior
to Oracle 12c.
With the launch of Oracle 12c the above constraint has been addressed with the feature called
Oracle Flex ASM which primarily has a feature to fail over to another node in the cluster.

Essentially a Hub and Leaf architecture, the connection of a failed node is seamlessly
transferred to another participating node by way of a replacement ASM instance by Oracle
Clusterware. The number of ASM instances running in a given cluster is called ASM cardinality
with a default value of 3.
The Oracle Flex ASM can be implemented in two ways:

 Pure 12c Flex ASM
Both Grid Infrastructure (GI) and Database running on Oracle 12c
 Pre Oracle 12c Mixed (Standard ASM)
As normal ASM instance will run on each node with Flex configuration to support
pre 12c database. The compatibility parameter of ASM disk group is used to
manage the compatibility of between and among database instances. Advantage
of this approach is that if an Oracle 12c database instance loses connectivity with
an ASM instance, then the database connections will failover to another ASM
instance on a different server. This failover is achieved by setting the cardinality
to all.
The following figure shows Standard Oracle Flex ASM configuration:

m
co
l.
ai
gm
The following figure shows ASM Instance Failure on Oracle Flex ASM configuration:
7@
71
n2
wa
ha
id
it
Pr
Oracle Flex ASM Cluster Network & ASM Listener

In previous releases (before 12c), the Oracle clusterware required a public network for client
access and a private network for inter-node communication including ASM inter-node
communication in the cluster.
Starting with Oracle Grid Infrastructure 12c Release 1 (12.1), as part of an Oracle Flex Cluster
installation, Oracle ASM is configured within Oracle Grid Infrastructure to provide storage
services. Each Oracle Flex ASM cluster has its own name that is globally unique within the
enterprise.
In Flex ASM, ASM network provides the capability to isolate ASM’s internal network traffic to its
own dedicated private network. The OUI presents the DBA with the choice as to whether a
dedicated network is to be used for ASM. The ASM network is the communication path in which
all the traffic between database instances and ASM instances commence. This traffic is mostly
the metadata such as a particular file’s extent map. If the customer chooses, the ASM private
network can be dedicated for ASM traffic or shared with CSS and a dedicated network is not
required.
There will be a local ASM listener that runs on each node where the flex ASM instance is
running. ASM clients connect to Flex ASM over the ASM network by using highly available
connection endpoints supplied by up to three ASM listeners. Connections are load balanced
across the ASM cluster.
Oracle Flex ASM can use either the same private networks as Oracle Clusterware, or use its
own dedicated private networks. Each network can be classified PUBLIC, PRIVATE, ASM or
ASM & PRIVATE.

Grid Infrastructure Management Repository (-MGMTDB)

The Oracle Grid Infrastructure Management Repository:
 Is an Oracle database that stores real-time operating system metrics collected by
Cluster Health Monitor (CHM)
 Runs on one node in the cluster (this must be a Hub Node in an Oracle Flex Cluster
configuration), and must support failover to another node in case of node or storage
failure.
 Data files are located in the same disk group as the OCR and voting file.
Oracle Grid Infrastructure Management Repository is automatically configured during

installation of or upgrade to Oracle Clusterware 12c (12.1.0.2.0) on a cluster.
IPV6 Support for Public Networks

Oracle Clusterware 12c Release 1 (12.1) supports IPv6-based public IP and VIP addresses.
IPv6-based IP addresses have become the latest standard for the information technology
infrastructure in today's data centers. With this release, Oracle RAC and Oracle Grid
Infrastructure support this standard. You can configure cluster nodes during installation with
either IPv4 or IPv6 addresses on the same network.
Database clients can connect to either IPv4 or IPv6 addresses. The Single Client Access Name
(SCAN) listener automatically redirects client connection requests to the appropriate database
listener for the IP protocol of the client request.
IPv6 addresses are not supported for the Private Network.
Shared Grid Naming Service (GNS)

Oracle Clusterware enables a dynamic Grid Infrastructure through the self-management of the
network requirements for the cluster. Oracle Clusterware 11g release 2 (11.2) onwards supports
the use of dynamic host configuration protocol (DHCP) for all private interconnect addresses, as
well as for most of the VIP addresses. DHCP provides dynamic configuration of the host's IP
address, but it does not provide an optimal method of producing names that are useful to
external clients.
When you are using Oracle RAC, all of the clients must be able to reach the database. This
means that the VIP addresses must be resolved by the clients. This problem is solved by the
addition of the Oracle Grid Naming Service (GNS) to the cluster. GNS is linked to the corporate
domain name service (DNS) so that clients can easily connect to the cluster and the databases
running there. Activating GNS in a cluster requires a DHCP service on the public network.
To implement GNS, you must collaborate with your network administrator to obtain an IP
address on the public network for the GNS VIP. DNS uses the GNS VIP to forward requests for
access to the cluster to GNS. The network administrator must delegate a subdomain in the
network to the cluster. The subdomain forwards all requests for addresses in the subdomain to
the GNS VIP. GNS and the GNS VIP run on one node in the cluster. The GNS daemon listens
on the GNS VIP using port 53 for DNS requests. Oracle Clusterware manages the GNS and
the GNS VIP to ensure that they are always available. If the server on which GNS is running
fails, then Oracle Clusterware fails GNS over, along with the GNS VIP, to another node in the
cluster.
With DHCP on the network, Oracle Clusterware obtains an IP address from the server along
with other network information, such as what gateway to use, what DNS servers to use, what

domain to use, and what NTP server to use. Oracle Clusterware initially obtains the necessary
IP addresses during cluster configuration and it updates the Oracle Clusterware resources with
the correct information obtained from the DHCP server.
For small clusters, you can use a static configuration of IP addresses. For large clusters,
manually maintaining the large number of required IP addresses becomes too cumbersome.
The Oracle Grid Naming Service is used with large clusters to ease administration costs.
In previous releases (up to release 11gR2) , the Grid Naming Service (GNS) was dedicated to
one Oracle Grid Infrastructure-based cluster, providing name resolution only for its own cluster
member nodes.
With 12c release, one Oracle GNS can now manage just the cluster member nodes in its own
cluster, or GNS can provide naming resolution for all nodes across all clusters in the data center
that are delegated to Oracle GNS for resolution.
Using only one Oracle GNS for all nodes that are part of an Oracle Grid Infrastructure cluster in
the data center not only streamlines the naming convention, but also enables a data center
cloud, minimizing day-to-day administration efforts.
Oracle Cluster Registry backup in Oracle ASM Diskgroups

The Oracle Cluster Registry (OCR) backup mechanism enables storing the OCR backup in an
Oracle ASM disk group. Storing the OCR backup in an Oracle ASM disk group simplifies OCR
management by permitting access to the OCR backup from any node in the cluster should an
OCR recovery become necessary.
Shared Oracle ASM Password File in a Disk group

This feature implements the infrastructure needed to address the bootstrapping issues of Oracle
ASM shared password file in an Oracle ASM disk group. This feature solves the bootstrapping
problem for storing shared Oracle ASM password files in a disk group.
An individual password file for Oracle Database or Oracle ASM can reside on a designated
Oracle ASM disk group. Having the password files reside on a single location accessible across
the cluster reduces maintenance costs and situations where passwords become out of sync.
You can use a password file located on a disk group for authentication only if the Oracle ASM
instance is running and the designated disk group is mounted. Otherwise, operating system
authentication must be used to bootstrap the startup of the Oracle ASM instance and stack.
The COMPATIBLE.ASM disk group attribute must be set to at least 12.1 for the disk group
where the password file is to be located. The SYSASM privilege is required to manage the
Oracle ASM password file.
The shared password file in a disk group is managed by ASMCMD commands, the ORAPWD
tool, and SRVCTL commands. ORAPWD supports the creation of password files on an Oracle
ASM disk group. All other password file manipulation is performed with ASMCMD or SRVCTL
commands.
You must specify the disk group location and database unique name when using orapwd to
create a database password file on a disk group.

Preparing for Grid Infrastructure Installation

This chapter contains the information about how to configure a two node cluster for Oracle Grid
Infrastructure (GI) & Oracle RAC installation.
This chapter has the steps to install Oracle 12c Release 1 (12.1.0.2.0) GI & RAC software on
Oracle Enterprise Linux 6.x (OEL 6.x) 64-bit platform. This guide assumes that you are using
Oracle ASM for shared storage.
In general you need to perform the following tasks before installing Oracle Grid Infrastructure
and Oracle Software for RAC.
1. General System Requirements

2. Configure Groups, Users and Resource Limits
3. Configure SSH
4. Create Oracle Base and Oracle Home directories
5. Configure Disk Device Persistence Using ASMLIB
1. General System Requirements

Package requirements
The following packages (or later versions) must be installed
binutils-2.20.51.0.2-5.11.el6 (x86_64)
compat-libcap1-1.10-1 (x86_64)
compat-libstdc++-33-3.2.3-69.el6 (x86_64)
compat-libstdc++-33-3.2.3-69.el6.i686
gcc-4.4.4-13.el6 (x86_64)
gcc-c++-4.4.4-13.el6 (x86_64)
glibc-2.12-1.7.el6 (i686)
glibc-2.12-1.7.el6 (x86_64)
glibc-devel-2.12-1.7.el6 (x86_64)
glibc-devel-2.12-1.7.el6.i686
ksh
libgcc-4.4.4-13.el6 (i686)
libgcc-4.4.4-13.el6 (x86_64)
libstdc++-4.4.4-13.el6 (x86_64)
libstdc++-4.4.4-13.el6.i686
libstdc++-devel-4.4.4-13.el6 (x86_64)
libstdc++-devel-4.4.4-13.el6.i686
libaio-0.3.107-10.el6 (x86_64)
libaio-0.3.107-10.el6.i686
libaio-devel-0.3.107-10.el6 (x86_64)
libaio-devel-0.3.107-10.el6.i686
libXext-1.1 (x86_64)
libXext-1.1 (i686)
libXtst-1.0.99.2 (x86_64)
libXtst-1.0.99.2 (i686)
libX11-1.3 (x86_64)
libX11-1.3 (i686)
libXau-1.0.5 (x86_64)
libXau-1.0.5 (i686)
libxcb-1.5 (x86_64)
libxcb-1.5 (i686)
libXi-1.3 (x86_64)
libXi-1.3 (i686)

make-3.81-19.el6
sysstat-9.0.4-11.el6 (x86_64)
Kernel Parameters and Resource Limits

Verify that the kernel parameters shown in the following table are set to values greater than or
equal to the minimum value shown.
Parameter Minimum Value

Semmsl 250
Semmns 32000
Semopm 100
Semmni 128
Shmall Half the size of physical memory in pages
Shmmax Half the size of physical memory in bytes
Shmmni 4096
file-max 6815744
ip_local_port_range Minimum: 9000 maximum: 65500
rmem_default 262144
rmem_max 4194304
wmem_default 262144
wmem_max 1048576
aio-max-nr 1048576
If the current value of any parameter is higher than the value listed in this table, then do not
change the value of that parameter.
To view the current value specified for these kernel parameters, enter commands similar to the
following
# /sbin/sysctl –a | grep parameter
For example:
# /sbin/sysctl –a | grep rmem_max
If the value of any kernel parameter is different from the recommended value, then using vi
editor, create or edit the /etc/sysctl.conf file, and add or edit lines similar to the following:
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 536870912
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586

After updating the values of kernel parameters in the /etc/sysctl.conf file, run the command
sysctl -p ( or restart the computer ) to make the changes in the /etc/sysctl.conf file available in
the active kernel memory.
# /sbin/sysctl -p
Setting Network Time Protocol for Cluster Time Synchronization

Oracle Clusterware requires the same time zone environment variable setting on all cluster
nodes.
You have two options for time synchronization:

 An operating system configured network time protocol (NTP)
 Oracle Cluster Time Synchronization Service
Oracle Cluster Time Synchronization Service is designed for organizations whose cluster
servers are unable to access NTP services. If you use NTP, then the Oracle Cluster Time
Synchronization daemon (ctssd) starts up in observer mode. If you do not have NTP daemons,
then ctssd starts up in active mode and synchronizes time among cluster members without
contacting an external time server
Note: Before starting the installation of Oracle Grid Infrastructure, Oracle recommends that you
ensure the clocks on all nodes are set tothe same time.
If you have NTP daemons on your server but you cannot configure them to synchronize time
with a time server, and you want to use Cluster Time Synchronization Service to provide
synchronization service in the cluster, then deactivate and deinstall the NTP.
To deactivate the NTP service, you must stop the existing ntpd service, disable it from
the initialization sequences and remove the ntp.conf file. To complete these step on
Oracle Linux, and Asianux systems, run the following commands as the root user
# /sbin/service ntpd stop

# chkconfig ntpd off
# mv /etc/ntp.conf /etc/ntp.conf.org
Also remove the following file:

/var/run/ntpd.pid
This file maintains the pid for the NTP daemon.
When the installer finds that the NTP protocol is not active, the Cluster Time Synchronization
Service is installed in active mode and synchronizes the time across the nodes. If NTP is found
configured, then the Cluster Time Synchronization Service is started in observer mode, and no
active time synchronization is performed by Oracle Clusterware within the cluster.
To confirm that ctssd is active after installation, enter the following command as the Grid
installation owner:
$ crsctl check ctss

Memory & Storage Requirements

Each node in the cluster must satisfy the following minimum requirements.
 At least 4 GB of RAM for Oracle Grid Infrastructure for a Cluster installations, including
installations where you plan to install Oracle RAC
 An amount of swap space that is at least equal to the amount of RAM
 Temporary space (at least 1 GB) available in /tmp
 Adequate local disk space to install Oracle software:
− At least 8 GB of available disk space for the Grid home directory, which includes
both the binary files for Oracle Clusterware and Oracle Automatic Storage
Management
− Additional disk space (100 GB is recommended) for the Grid home directory for
associated log files and patches.
− At least 3.5 GB of space for the Oracle base of the Oracle Grid Infrastructure
installation owner (Grid user). The Oracle base includes Oracle Clusterware and
Oracle ASM log files.
− At least 5.8 GB of disk space for the Oracle home (the location for the Oracle
Database software binaries).
Shared Storage Requirements

An Oracle RAC database is a shared everything database. All data files, control files, redo log
files, and the server parameter file (SPFILE) used by the Oracle RAC database must reside on
shared storage that is accessible by all the Oracle RAC database instances. The Oracle RAC
installation explained in this guide uses Oracle ASM for the shared storage for Oracle
Clusterware and Oracle Database files.
These Oracle Clusterware components require the following disk space on a shared file system:
 Three Oracle Clusterware Registry (OCR) files, a least 400 MB for each, or 1.2 GB total disk
space
 Three voting disk files, 300 MB each, or 900 MB total disk space
Oracle Cluster Registry (OCR)–Maintains cluster configuration information and configuration

information about any cluster database within the cluster. The OCR contains information such
as which database instances run on which nodes and which services run on which databases.
The OCR also stores information about processes that Oracle Clusterware controls. The OCR
resides on shared storage that is accessible by all the nodes in the cluster. Oracle Clusterware
can multiplex, or maintain multiple copies of, the OCR and Oracle recommends that you use
this feature to ensure high availability.
Voting disk–Manages cluster membership and arbitrates cluster ownership between the nodes
in case of network failures. The voting disk is a file that resides on shared storage. For high
availability, Oracle recommends that you have multiple voting disks, and that you have an odd
number of voting disks. If you define a single voting disk, then use mirroring at the file system
level for redundancy.
Hardware Requirements
 A processor type (CPU) that is certified with the release of the Oracle software being
installed
 A minimum of 1024 x 786 display resolution, so that Oracle Universal Installer (OUI)
displays correctly

 All servers in the cluster have the same chip architecture, for example, all 64-bit processors
To determine if the system architecture can run the software, enter the following command:
# uname -m
For example, you should see the following for a x86-64 bit system:
x86_64
Network Requirements
 Each node must have at least two network adapters or network interface cards (NICs): one
for the public network interface, and one for the private network interface (the interconnect).
 Interface for Oracle Flex ASM Network
Oracle Flex ASM can use either the same private networks as Oracle Clusterware,
or use its own dedicated private networks. Each network can be classified PUBLIC,
ASM & PRIVATE, PRIVATE, or ASM.
 A public IP address for each node
 A virtual IP address for each node
 A private IP address for each node
 A Single Client Access Name (SCAN) for the cluster with three static IP addresses
configured on the domain name server (DNS) or one static IP address configured in
/etc/hosts file.
 You must use Grid Naming Service (GNS) with an Oracle Flex Cluster Deployment
 You must configure the GNS VIP as a static IP address for Hub Nodes.
Public IP addresses, VIP addresses, and SCAN addresses should be on the same subnet as all
other public IP addresses. Private IP addresses should be configured on a separate, private
network, with its own subnet, that is not resolvable except by other cluster member nodes.
When performing an advanced installation of Oracle Grid Infrastructure for a cluster, you can
choose to use Grid Naming Service (GNS) and Dynamic Host Configuration Protocol
(DHCP) for virtual IPs (VIPs). GNS uses multicast Domain Name Server (mDNS) to enable
the cluster to assign host names and IP addresses dynamically as nodes are added and
removed from the cluster, without requiring additional network address configuration in the
domain name server (DNS).
In this course you will perform a typical installation, which does not use GNS.
2. Configuring Users, Groups and Resource Limits

This section describes the users, groups and environment settings to complete before you
install Oracle Grid Infrastructure for a Cluster and Oracle Real Application Clusters.
Oracle Inventory Group

When you install Oracle software on the system for the first time, OUI creates the oraInst.loc
file. This file identifies the name of the Oracle Inventory group (by default, oinstall), and the path
of the Oracle central inventory directory. An oraInst.loc file has contents similar to the following:
inventory_loc=central_inventory_location
inst_group=group

For Oracle Grid Infrastructure installations, the central inventory must be on local storage on the
node. If you have an existing Oracle central inventory, then ensure that you use the same
Oracle Inventory for all Oracle software installations, and ensure that all Oracle software users
you intend to use for installation have permissions to write to this directory.
If the oraInst.loc file does not exist, then create the Oracle Inventory group by entering a
command similar to the following:
# /usr/sbin/groupadd -g 54321 oinstall
Note: Group and user IDs must be identical on all nodes in the cluster. Check to make sure
that the group and user IDs you want to use are available on each cluster member node, and
confirm that the primary group for each Oracle Grid Infrastructure for a cluster installation owner
has the same name and group ID.
Users and Groups

Oracle recommends that you create one software owner to own each Oracle software product.
In this course, you will be installing Oracle Grid Infrastructure under the user grid and Oracle
RAC Database under the user oracle.
Oracle software owners (grid & oracle) must have the Oracle Inventory group (oinstall) as their
primary group, so that each Oracle software installation owner can write to the central inventory
(oraInventory), and so that OCR and Oracle Clusterware resource permissions are set correctly.
The database software owner must also have the OSDBA group (dba) as secondary group.
About Job Role Separation

A job role separation configuration of Oracle Database and Oracle ASM is a configuration with
groups and users to provide separate groups for operating system authentication.
With Oracle Database job role separation, each Oracle Database installation has separate
operating system groups to provide authentication for system privileges on that Oracle
Database, so multiple databases can be installed on the cluster without sharing operating
system authentication for system privileges.
In addition, each Oracle software installation is owned by a separate installation owner, to

provide operating system user authentication for modifications to Oracle Database binaries.
Job role separation configuration enables database security, and does not restrict user roles in
starting and stopping various Clusterware resources.
With Oracle Grid Infrastructure job role separation, Oracle ASM has separate operating system
groups that provides operating system authentication for Oracle ASM system privileges for
storage tier administration. This operating system authentication is separated from Oracle
Database operating system authentication. In addition, the Oracle Grid Infrastructure installation
owner provides operating system user authentication for modifications to Oracle Grid
Infrastructure binaries.
During the Oracle Database installation, Oracle Universal Installer (OUI) prompts you to specify
the name of the OSDBA, OSOPER, OSBACKUPDBA, OSDGDBA and OSKMDBA groups.
Members of these groups are granted operating system authentication for the set of database
system privileges each group authorizes

Note: Job Role Separation configuration is optional, to restrict user access to Oracle software
by responsibility areas for different administrator users. You can choose to create one
administrative user and one group for operating system authentication for all system privileges
on the storage and database tiers.
If you do not want to use role allocation groups, then Oracle strongly recommends that you use
at least two groups:
 A system privileges group whose members are granted administrative system privileges,
including OSDBA, OSASM, and other system privileges groups.
 An installation owner group (the oraInventory group) whose members are granted
Oracle installation owner system privileges (the OINSTALL system privilege).
Oracle ASM Groups for Job Role Separation

Members of the OSDBA group formerly were granted SYSASM privilege on Oracle ASM
instances, including mounting and dismounting disk groups.
Starting with Oracle Automatic Storage Management 11g release 2(11.2), members of the
database OSDBA group are not granted SYSASM privileges, unless the operating system
group designated as the OSASM group is the same group designated as the OSDBA group.
This privileges grant is removed with Oracle Grid Infrastructure 11g Release 2 (11.2), if different
operating system groups are designated as the OSDBA and OSASM groups.
OSASM Group for Oracle ASM Administration (typically asmadmin):

Members of this group are granted the SYSASM system privileges to administer Oracle ASM.
Members of the OSASM group can use SQL to connect to an Oracle ASM instance as
SYSASM using operating system authentication. The SYSASM privileges permit mounting and
dismounting disk groups, and other storage administration tasks. SYSASM privileges provide no
access privileges on an RDBMS instance.
OSDBA for ASM Database Administrator group for ASM (typically asmdba):
Members of the ASM Database Administrator group (OSDBA for ASM) are granted read and
write access to files managed by Oracle ASM. The Oracle Grid Infrastructure installation owner
and all Oracle Database software owners must be a member of this group, and all users with
OSDBA membership on databases that have access to the files managed by Oracle ASM must
be members of the OSDBA group for ASM.
OSOPER for ASM Group for ASM Operators (typically asmoper):

This is an optional group. Create this group if you want a separate group of operating system
users to have a limited set of Oracle ASM instance administrative privileges (the SYSOPER for
ASM privilege), including starting up and stopping the Oracle ASM instance. By default,
members of the OSASM group also have all privileges granted by the SYSOPER for ASM
privilege.
The following section, shows the commands to create various groups and users for Oracle Grid
Infrastructure and Oracle RAC.
Creating Groups:
# groupadd –g 1100 oinstall

m
co
l.
ai
# groupadd –g 1101 dba
gm
# groupadd –g 1102 asmdba
7@
# groupadd –g 1103 asmoper
71
# groupadd –g 1104 asmadmin
n2
Creating Separate Owner for Grid Infrastructure and Oracle RAC

wa
ha
# useradd –u 1200 –g oinstall –G dba, asmdba oracle

# useradd –u 1201 –g oinstall –G asmdba, asmadmin, asmoper grid
id
it
Create the above mentioned groups and users on each node of the cluster.
Pr
Resource Limits
Also, before you start the installation, for each installation software owner, check the resource
limits for installation, using the following recommended ranges
Resource Shell Limit Resource Soft Limit Hard Limit

Open File Descriptors nofile At least 1024 At least 65536
No of processes available nproc At least 2047 At least 16384
to a single user
Size of the stack segment stack At least 10240 KB At least 10240 KB
of the process and at most 32768
KB
To check resource limits:
1. Log in as an installation owner.
2. Check the soft and hard limits for the file descriptor setting. Ensure that the result is in the
recommended range. For example:
$ ulimit -Sn
4096
$ ulimit -Hn
65536
3. Check the soft and hard limits for the number of processes available to a user. Ensure
that the result is in the recommended range. For example:
$ ulimit -Su
2047
$ ulimit -Hu
16384
4. Check the soft limit for the stack setting. Ensure that the result is in the recommended
range. For example:
$ ulimit -Ss
10240

$ ulimit -Hs
32768
If necessary, update the resource limits in the /etc/security/limits.conf configuration file for the
installation owner. For example
# vi /etc/security/limits.conf (make sure the following entries in the file)
oracle soft nproc 2047

oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft stack 10240
oracle hard stack 32768
3. Configuring SSH
SSH is required for Oracle Grid Infrastructure installation. OpenSSH should be included in the
Linux distribution minimal installation.
To install Oracle software, Secure Shell (SSH) connectivity must be set up between all cluster
member nodes. OUI uses the ssh and scp commands during installation to run remote
commands on and copy files to the other cluster nodes. You must configure SSH so that these
commands do not prompt for a password.
You can configure SSH from the Oracle Universal Installer (OUI) interface during installation for
the user account running the installation. The automatic configuration creates passwordless
SSH connectivity between all cluster member nodes. Oracle recommends that you use the
automatic procedure if possible.
If automatic ssh configuration is not possible, you can manually configure user equivalence as
described below. The following steps show how to configure SSH between 2 nodes named
node1 and node2.
Note: If you are using separate operating system accounts for installing Grid Infrastructure and
Oracle RAC software, then you need to repeat the following steps for each linux account.
Node1:
1. Login as the oracle software owner (as grid or oracle)

2. Change to home directory
3. $ssh-keygen –t dsa
4. $cat id_dsa.pub >> authorized_keys
Node2:
1. Login as the oracle software owner (as grid or oracle)

2. Change to home directory
3. $ssh-keygen –t dsa
4. $cat id_dsa.pub >> authorized_keys_node2
5. $scp ~/.ssh/authorized_keys_node2 node1:~/.ssh/

Node1:
$cat authorized_keys_node2 >> authorized_keys

$scp authroized_keys node2:~/.ssh
Test Connectivity
check all ssh combinations as mentioned below
on node1:
$ssh node1
$ssh node2
on node2:
$ssh node1
$ssh node2
Make sure the above tests do not prompt for password. If any of the above tests prompt
for password, then the SSH is not properly configured. In that case you need to fix the
issue and make sure SSH is working correctly.
4. Create Oracle Home and Oracle Base Directories

The Oracle home for Oracle Grid Infrastructure software (Grid home) should be located in a
path that is different from the Oracle home directory paths for any other Oracle software.
For Oracle Grid Infrastructure for a cluster installations, note the following restrictions for the
Oracle Grid Infrastructure binary home (Grid home directory for Oracle Grid Infrastructure):
 It must not be placed under one of the Oracle Base directories, including the Oracle
base directory of the Oracle Grid Infrastructure installation owner.
 It must not be placed in the home directory of an installation owner.
 Placing Oracle Grid Infrastructure for a cluster binaries on a cluster file system is not
supported.
Note: On Linux and UNIX systems, you must ensure the Grid home directory is not a
subdirectory of the Oracle base directory. Installing Oracle Clusterware in an Oracle base
directory causes installation
errors.
Oracle recommends that you create Oracle Grid Infrastructure Grid home and Oracle base
homes manually, particularly if you have separate Oracle Grid Infrastructure for a cluster and
Oracle Database software owners, so that you can separate log files for the Oracle Grid
Infrastructure installation owner in a separate Oracle base, and prevent accidental placement of
the Grid home under an Oracle base path.
Use commands similar to the following to create Oracle Home and Oracle Base directories for
Oracle Grid Infrastructure and Oracle Software for RAC.
# mkdir -p /u01/app/12.1.0/grid (This will be ORACLE_HOME for grid)

# mkdir -p /u01/app/grid (This will be the ORACLE_BASE for grid)
# mkdir -p /u01/app/oracle (This will be the ORACLE_BASE for oracle)

#mkdir –p /u01/app/oracle/product/12.1.0/dbhome_1 (This will be the ORACLE_HOME for

oracle)
5. Configuring Disk Device Persistence Using ASMLIB

By default, the Linux 2.6 kernel device file naming scheme udev dynamically creates device file
names when the server is started, and assigns ownership of them to root. If udev applies default
settings, then it changes device file names and owners for voting disks or Oracle Cluster
Registry partitions, corrupting them when the server is restarted.
For example, a voting disk on a device named /dev/sdd owned by the user grid may be on a
device named /dev/sdf owned by root after restarting the server.
If you use ASMLIB, then you do not have to ensure permissions and device path persistency in
udev. If you do not use ASMLIB, then you must create a custom rules file for the shared disks
mounted on each node.
If you use ASMLIB to configure the shared disks, then when you restart the node:
 The disk device names do not change
 The ownership and group membership for these disk devices remains the same
The following sections describe how to install and configure ASMLIB, and how to use ASMLIB
to configure your shared disk devices:
Installing ASMLIB
The ASMLIB software for various Linux platforms is available from the Oracle Technology
Network. Select the link for your platform on the ASMLIB download page at:
http://www.oracle.com/technetwork/topics/linux/asmlib/index-101839.html
You should see four to six packages for your Linux platform. The oracleasmlib package
provides the actual Oracle ASM library. The oracleasm-support package provides the utilities
used to get the Oracle ASM driver up and running. Both of these packages must be installed.
The remaining packages provide the kernel driver for the Oracle ASM library. Each package
provides the driver for a different kernel. You must install the appropriate package for the kernel
you run.
Download and install the ASMLIB packages on each node in your cluster.
For Example:
# rpm -Uvh oracleasm-support-2.1.3-1.el4.x86_64.rpm

# rpm -Uvh oracleasmlib-2.0.4-1.el4.x86_64.rpm
# rpm -Uvh oracleasm-2.6.9-55.0.12.ELsmp-2.0.3-1.x86_64.rpm

Configuring ASMLIB
After the ASMLIB software is installed, a few steps have to be taken by you to make the Oracle
ASM driver available. The Oracle ASM driver must be loaded, and the driver file system must be
mounted. This is taken care of by the initialization script, /usr/sbin/oracleasm.
To configure the ASMLIB software after installation:
1. As the root user, run the following command:
# /usr/sbin/oracleasm configure -i
The script prompts you for the default user and group to own the Oracle ASM driver access
point. Specify the Oracle GI software owner (grid) and the OSDBA group (oinstall).
The script also prompts you to specify whether you want to start the ASMLIB driver when the
node is started and whether you want to scan for presence of any Oracle Automatic Storage
Management disks when the node is started. Answer yes for both of these questions.
2. Repeat step 1 on each node in your cluster
Using ASMLIB to Create Oracle ASM Disks

Every disk that is used in an Oracle ASM disk group must be accessible on each node. After
you make the physical disk available to each node, you can then mark the disk device as an
Oracle ASM disk. The /usr/sbin/oracleasm script is used for this task.
If the target disk device supports partitioning, then you must first create a single partition that
encompasses the entire disk.
If the target disk device does not support partitioning, then you do not have to create a partition
on the disk.
To create Oracle ASM disks using ASMLIB:

1. As the root user, use oracleasm to create Oracle ASM disks using the following syntax:
# /usr/sbin/oracleasm createdisk disk_name device_partition_name
In this command, disk_name is the name you choose for the Oracle ASM disk. The name you
choose must contain only ASCII capital letters, numbers, or underscores, and the disk name
must start with a letter, for example, DISK1 or VOL1, or RAC_FILE1. The name of the disk
partition to mark as an Oracle ASM disk is the device_partition_name. For example:
# /usr/sbin/oracleasm createdisk DISK1 /dev/sdb1
If you must unmark a disk that was used in a createdisk command, then you can use the
following syntax:
# /usr/sbin/oracleasm deletedisk disk_name
2. Repeat step 1 for each disk that Oracle ASM uses.
3. After you have created all the Oracle ASM disks for your cluster, use the listdisks command
to verify their availability:

# /usr/sbin/oracleasm listdisks
DISK1
DISK2
DISK3
4. On all the other nodes in the cluster, use the scandisks command to view the newly created
Oracle ASM disks. You do not have to create the Oracle ASM disks on each node, only on
one node in the cluster.
# /usr/sbin/oracleasm scandisks
Scanning system for ASM disks [ OK ]
5. After scanning for Oracle ASM disks, display the available Oracle ASM disks on each node
to verify their availability:
# /usr/sbin/oracleasm listdisks
DISK1
DISK2
DISK3
Note: At this point, you should restart each node on which you are installing Oracle Grid
Infrastructure for a cluster.

Administering Oracle Clusterware

This chapter describes how to administer the installed Oracle Clusterware environment.
Oracle Real Application Clusters (Oracle RAC) uses Oracle Clusterware as the infrastructure
that binds together multiple nodes that then operate as a single server. In an Oracle RAC
environment, Oracle Clusterware monitors all Oracle components (such as instances and
listeners). If a failure occurs, Oracle Clusterware automatically attempts to restart the failed
component and also redirects operations to a surviving component.
Oracle Clusterware includes a high availability framework for managing any application that
runs on your cluster. Oracle Clusterware manages applications to ensure they start when the
system starts. Oracle Clusterware also monitors the applications to make sure that they are
always available. For example, if an application process fails, then Oracle Clusterware attempts
to restart the process based on scripts that you customize. If a node in the cluster fails, then you
can program application processes that typically run on the failed node to restart on another
node in the cluster.
Oracle Clusterware is managed using CRSCTL and SRVCTL command-line utilities installed
under the Oracle Grid Infrastructure home directory. However, most CRSCTL commands are
available only to Clusterware administrators, whereas SRVCTL commands are available to
other groups such as database administrators.
CRSCTL is used to manage Clusterware related operations like Starting and Stopping Oracle
Clusterware, Enabling and disabling Oracle Clusterware daemons, Registering Cluster
resources.
SRVCTL is used to manage Oracle resource related operations like Starting and Stopping
database instances and services.
Apart from CRSCTL & SRVCTL command-line utilities Oracle Enterprise Manager also provides
browser based graphical interface to manage Oracle Clusterware.
Oracle Clusterware Processes

Oracle Clusterware comprises several processes that facilitate cluster operations. The Cluster
Ready Service (CRS), Cluster Synchronization Service (CSS), Event Management (EVM), and
Oracle Notification Service (ONS) components communicate with other cluster component
layers in the same cluster database environment. These components are also the main
communication links between Oracle Database, applications, and the Oracle Clusterware high
availability components. In addition, these background processes monitor and manage
database operations. The following list describes some major Oracle Clusterware background
processes.
Oracle Clusterware consists of two separate stacks: an upper stack anchored by the Cluster
Ready Services (CRS) daemon (crsd) and a lower stack anchored by the Oracle High
Availability Services daemon (ohasd). These two stacks have several processes that facilitate
cluster operations.

 The Cluster Ready Services Stack

 The Oracle High Availability Services Stack
The following sections describe these stacks in more detail:
The Cluster Ready Services Stack

The list in this section describes the processes that comprise CRS. The list includes
components that are processes on Linux and UNIX operating systems, or services on
Windows.
 Cluster Ready Services (CRS): The primary program for managing high availability
operations in a cluster.
The CRS daemon (crsd) manages cluster resources based on the configuration information that
is stored in OCR for each resource. This includes start, stop, monitor, and failover operations.
The crsd process generates events when the status of a resource changes. When you have
Oracle RAC installed, the crsd process monitors the Oracle database instance, listener, and so
on, and automatically restarts these components when a failure occurs.
The CRS process manages two types of CRS resources:
 Cluster resources: A cluster resource is an Oracle Clusterware resource. Cluster

resources are viewed, added, modified, or removed using the crsctl command.
 Local resources: A local resource runs on every node in the cluster (no failover) and
can be, for example, a listener, ASM, a disk group, or Oracle Notification Service (ONS).
 Cluster Synchronization Services (CSS): Manages the cluster configuration by controlling

which nodes are members of the cluster and by notifying members when a node joins or
leaves the cluster.
CSS has three separate processes:

− CSS Daemon (ocssd)
− CSS Agent (cssdagent)
− CSS Monitor (cssdmonitor)
The cssdagent process monitors the cluster and provides I/O fencing. This service formerly was
provided by Oracle Process Monitor Daemon (oprocd), also known as OraFenceService on
Windows. A cssdagent failure may result in Oracle Clusterware restarting the node.
 Event Management (EVM): A background process that publishes events that Oracle
Clusterware creates.
 Oracle Notification Service (ONS): A publish and subscribe service for communicating
Fast Application Notification (FAN) events.
 Oracle ASM: Provides disk management for Oracle Clusterware and Oracle Database.
 Cluster Time Synchronization Service (CTSS): Provides time management in a cluster for
Oracle Clusterware.

 Oracle Agent (oraagent): Extends clusterware to support Oracle-specific requirements and

complex resources. This process runs server callout scripts when FAN events occur. This
process was known as RACG in Oracle Clusterware 11g release 1 (11.1).
 Oracle Root Agent (orarootagent): A specialized oraagent process that helps crsd
manage resources owned by root, such as the network, and the Grid virtual IP address.
The Cluster Synchronization Service (CSS), Event Management (EVM), and Oracle Notification
Services (ONS) components communicate with other cluster component layers on other nodes
in the same cluster database environment. These components are also the main
communication links between Oracle database, applications, and the Oracle clusterware high
availability components. In addition, these background processes monitor and manage
database operations.
The Oracle High Availability Services Stack

This section describes the processes that comprise the Oracle High Availability Services stack.
The list includes components that are processes on Linux and UNIX operating systems, or
services on Windows.
 Cluster Logger Service (ologgerd): Receives information from all the nodes in the cluster
and persists in a CHM repository-based database. This service runs on only one node in a
cluster and an additional service will be started for every 32 nodes.
 System Monitor Service (osysmond): The monitoring and operating system metric
collection service that sends the data to the cluster logger service. This service runs on
every node in a cluster.
 Grid Plug and Play (GPNPD): Provides access to the Grid Plug and Play profile, and
coordinates updates to the profile among the nodes of the cluster to ensure that all of the
nodes have the most recent profile.
 Grid Inter Process Communication (GIPC): A support daemon that enables Redundant
Interconnect Usage.
 Multicast Domain Name Service (mDNS): Used by Grid Plug and Play to locate profiles in
the cluster, as well as by GNS to perform name resolution. The mDNS process is a
background process on Linux and UNIX and on Windows.
 Oracle Grid Naming Service (GNS): Handles requests sent by external DNS servers,
performing name resolution for names defined by the cluster.
 Cluster Kill Daemon (oclskd): handles instance/node eviction requests that have been
escalated to CSS.
 Disk Monitory Daemon (diskmon): Monitors and performs input/output fencing for Oracle
Exadata Storage server. As Exadata storage can be added to any RAC node at any point in
time, the diskmon daemon is always started when OCSSD is started
Clusterware processes are organized into several component groups as mentioned in below
table:

Component Processes Owner

Cluster Ready Service (CRS) crsd root
Cluster Synchronization Service (CSS) ocssd, cssdmonitor, cssdagent grid owner, root, root
Event Manager (EVM) evmd, evmlogger grid owner
Cluster Time Synchronization Service (CTSS) octssd root
Oracle Notification Service (ONS) ons, eons grid owner
Oracle Agent oraagent grid owner
Oracle Root Agent orarootagent root
Grid Naming Service (GNS) gnsd root
Grid Plug and Play (GPnP) gpnpd grid owner
Multicast domain name service (mDNS) mdnsd grid owner
Cluster Time Synchronization Services (CTSS) octssd Root
Oracle Clusterware Startup

During the installation of Oracle Clusterware, the init.ohasd startup script is copied to /etc/init.d.
The wrapper script is responsible for setting up environment variables and then starting the
Oracle Clusterware daemons and processes.
The Oracle High Availability Services daemon (ohasd) is responsible for starting in proper order,
monitoring, and restarting other local Oracle daemons, up through the crsd daemon, which
manages cluster wide resources. When init starts ohasd on Clusterware startup, ohasd starts
orarootagent, oraagent and cssdagent. These processes then carry out the following tasks:
• orarootagent starts crsd.
− crsd starts another orarootagent process responsible for root-owned CRS resources
including the SCAN VIPS.
• cssdagent starts cssd (ocssd).
• oraagent starts mdnsd, evmd, ASM, ctssd, gpnpd, gsd, ONS, and the listeners.
Some of the high availability daemons will be running under the root user with real-time priority,
and others will be running under the Clusterware owner with user-mode priorities after they are
started. When a command is used to stop Oracle Clusterware, the daemons will be stopped, but
the ohasd process will remain running.

m
co
l.
ai
gm
The following diagram depicts the Cluster startup process
7@
71
n2
wa
ha
id
it
Pr
Controlling Oracle Clusterware

When a node that contains Oracle Clusterware is started, the Oracle Clusterware wrapper script
is automatically started by /etc/init.d/ohasd startup script.
You may have to manually control the Oracle Clusterware stack using the following mentioned
commands while applying patches or during planned outages.
To start or stop Oracle Clusterware on a specific node (need root access)
#crsctl start crs

#crsctl stop crs
To enable or disable Oracle Clusterware on a specific node (need root access)
#crsctl enable crs

#crsctl disable crs

To determine the overall health of the cluster on a specific node:
$crsctl check crs
To check the viability of Cluster Synchronization Services (CSS) across nodes:
$crsctl check cluster
To check the status of the individual daemons like crsd, evmd, and cssd
$crsctl check <daemon>
For example: $crsctl check evmd
To check the cluster status across all nodes:
$crsctl check cluster -all
OCR & VOTING Disks

Oracle Clusterware uses two primary configuration files: the voting disk and the OCR. The
voting disk is a file that manages information about node membership. The OCR is a file that
contains information about the cluster node list, instance-to-node mapping information, and
information about Oracle Clusterware resource profiles for resources that you have customized.
Each node in a cluster also has its own local OCR, called an Oracle Local Registry (OLR),that is
created when Oracle Clusterware is installed. Multiple processes on each node have
simultaneous read and write access to the OLR particular to the node on which they reside,
whether or not Oracle Clusterware is fully functional. By default, OLR is located at
GRID_HOME/cdata/$HOSTNAME.olr
The Oracle Clusterware installation process creates the voting disk and the OCR on shared
storage. If you select the option for normal redundant copies during the installation process,
then Oracle Clusterware automatically maintains redundant copies of these files to prevent the
files from becoming single points of failure. When you use normal redundancy, Oracle
Clusterware automatically maintains two copies of the OCR file and three copies of the voting
disk file.
Tools to manage OCR

 OCRCONFIG – command line tool to manage backup, restore, import, export, repair and
replace OCR file
 OCRCHECK – checks integrity and displays the version of the OCR’s block format, total
space available, used space and the OCR locations that you have configured
 OCRDUMP – view the OCR contents by writing OCR content to a file or stdout in a
reliable format
To determine the location of Voting disk from any node
#crsctl query css votedisk

To determine the location of OCR
$cat /etc/oracle/ocr.loc
To check the integrity of Voting disk
To check the integrity of Voting disks, examine ocssd.log. Errors with voting disks appear in the
log.
$grep voting <grid_home>/log/<hostname>/cssd/ocssd.log
To check the integrity of OCR file
$cluvfy comp ocr –n all -verbose
or
$ocrcheck
Adding and Removing Voting Disks

You can dynamically add and remove voting disks after installing Oracle RAC. Do this using the
following commands where path is the fully qualified path for the additional voting disk.
1.To add one or more voting disks to non-ASM storage, run the following command as root:
#crsctl add css votedisk path_to_new_voting_disk
2. To add a voting disk to ASM
#crsctl replace votedisk +asm_disk_group
3.To replace voting disk A with voting disk B on non-ASM storage, first add voting disk B and
then delete voting disk A
#crsctl add css votedisk path_to_voting_diskB
#crsctl delete css votedisk path_to_voting_diskA
4.To replace voting disk on ASM.
#crsctl replace votedisk +asm_disk_group
You do not have to delete any voting disks from ASM using this command
Backing Up Voting Disks

The voting disk records node membership information. A node must be able to access more
than half of the voting disks at any time. To avoid simultaneous loss of multiple voting disks,
each voting disk should be on a storage device that does not share any components (controller,
interconnect, and so on) with the storage devices used for the other voting disks.

For example, if you have five voting disks configured, then a node must be able to access at
least three of the voting disks at any time. If a node cannot access the minimum required
number of voting disks it is evicted, or removed, from the cluster. After the cause of the failure
has been corrected and access to the voting disks has been restored, you can instruct Oracle
Clusterware to recover the failed node and
restore it to the cluster.
If you lose all copies of the voting disk and do not have a backup, the only safe way to re-create
the voting disk is to reinstall Oracle Clusterware. Having a backup of the voting disk can
drastically simplify the recovery of your system.
Backing up Voting Disks

Before Oracle 11g Release 2, backing up voting disks using dd command (unix & linux) was a
required post installation task.
Starting with Oracle Clusterware 11g Release 2, backing up and restoring a voting disk using
dd command may result in the loss of the voting disk, so this procedure is not supported.
Backing up voting disks manually is no longer required because voting disk data is backed up
automatically in the OCR as part of any configuration change and voting disk data is
automatically restored to any added voting disks.
The voting disk files are backed up automatically by Oracle Clusterware if the contents of the
files have changed in the following ways:
 Configuration parameters, for example misscount, have been added or modified

 After performing voting disk add or delete operations
Backing up OCR
The information stored in OCR is much more dynamic in nature than the voting disk. Oracle
Clusterware automatically performs routine backups of the OCR file. These are Physical
backups. Only one node has the responsibility to perform these backups, but that responsibility
can transfer to any other node in the cluster when outages occur. The default target location of
each automatically generated OCR backup file is <Grid_Home>/cdata/<clustername>
directory.
Oracle Clusterware automatically creates OCR backups (auto backups) every 4 hours (called 4
hourly backups), daily at the end of the day (daily backups), every week (weekly backup). At
any one time, Oracle Clusterware retains the latest 3 copies of 4 hourly backup, last 2 copies of
daily and weekly backups.
You cannot customize the backup frequencies or the number of files that Oracle Clusterware
retains. You can use any backup software to copy the automatically generated backup files at
least once daily to a different device from where the primary OCR file resides.
The backup file names cannot be changed and are named as follows:
backup00.ocr, backup01.ocr, backup02.ocr, day.ocr, day_.ocr, week.ocr and week_.ocr
Viewing OCR Backups

You can view the backups generated automatically by Oracle Clusterware using the following
command.

$ocrconfig –showbackup
Changing OCR Backup Location

Because the automatic backup is performed only by the master node to the local files system by
default, it is recommended that you change the OCR automatic backup location to one that is
shared by all nodes in the cluster by using the following command.
#ocrconfig –backuploc <path to shared CFS or NFS>
The new backup location will be used for both automatic and manual backups.
Manually Backing Up the OCR

Use the ocrconfig utility to force Oracle Clusterware to perform a backup of OCR at any time,
rather than wait for the automatic backup that occurs at 4-hour intervals.This option is especially
useful when you want to obtain a binary backup on demand, such as before you make changes
to OCR. When significant changes to the configuration have occurred, a manual, on-demand
backup is suggested.
To perform a manual Physical backup:
#ocrconfig –manualbackup
Note: you are not allowed to specify the name used for the manual backup. A file will be created
with the name backup_<date>_<time>.ocr and placed into the default backup location.
To display a list of manual backups
$ocrconfig -showbackup manual
To perform a Logical backup of OCR
#ocrconfig –export /home/oracle/ocr.backup
Note: the export will create a binary file containing a logical backup of the OCR keys and values.
Changing the Oracle Cluster Registry Configuration

This section describes how to administer the OCR. The OCR contains information about the
cluster node list, which instances are running on which nodes, and information about Oracle
Clusterware resource profiles for applications that have been modified to be managed by Oracle
Clusterware.
Note: The operations in this section affect the OCR for the entire cluster. However, the
ocrconfig command cannot modify OCR configuration information for nodes that are shut down
or for nodes on which Oracle Clusterware is not running. Avoid shutting down nodes while
modifying the OCR using the ocrconfig command.
Adding an OCR Location

Oracle Clusterware supports up to 5 OCR copies. You can add an OCR location after an
upgrade or after completing the Oracle RAC installation. Additional OCR copies provide greater
fault tolerance.

To add an OCR file:
As the root user, enter the following command to add a new OCR file:
# ocrconfig -add new_ocr_file_name
This command updates the OCR configuration on all the nodes on which Oracle Clusterware is
running.
For Example:
#ocrconfig –add +DATA2
Replacing an OCR
If you need to change the location of an existing OCR, or change the location of a failed OCR to
the location of a working one, you can use the following procedure as long as one OCR file
remains online.
1. Use the OCRCHECK utility to verify that a copy of the OCR other than the one you are going
to replace is online, using the following command:
$ocrcheck
Note: The OCR that you are replacing can be either online or offline.
2. Use the following command to verify that Oracle Clusterware is running on the node on which
the you are going to perform the replace operation:
$crsctl check cluster –all
3. As the root user, enter the following command to designate a new location for the specified
OCR file:
# ocrconfig -replace source_ocr_file -replacement destination_ocr_file
This command updates the OCR configuration on all the nodes on which Oracle Clusterware is
running.
For example:
#ocrconfig –replace /dev/sde4 –replacement +DATA2
4. Use the OCRCHECK utility to verify that OCR replacement file is online:
$ocrcheck
Removing an OCR
To remove an OCR file, at least one copy of the OCR must be online. You can remove an OCR
location to reduce OCR-related overhead or to stop mirroring your OCR because you moved the
OCR to a redundant storage system, such as a redundant array of independent disks (RAID).

1. Use the OCRCHECK utility to ensure that at least one OCR other than the OCR that you are
removing is online.
$ocrcheck
2. As the root user, run the following command on any node in the cluster to remove a specific
OCR file:
# ocrconfig -delete ocr_file_name
For example:
#ocrconfig –delete +DATA2
Troubleshooting Oracle Cluster Registry

The OCRCHECK utility displays the data block format version used by the OCR, the available
space and used space in the OCR, the ID used for the OCR, and the locations you have
configured for the OCR. The OCRCHECK utility calculates a checksum for all the data blocks in
all the OCRs that you have configured to verify the integrity of each block. It also returns an
individual status for each OCR file as well as a result for the overall OCR integrity check. The
following is a sample of the OCRCHECK output:
Status of Oracle Cluster Registry is as follows :

Version : 3
Total space (kbytes) : 262144
Used space (kbytes) : 16256
Available space (kbytes) : 245888
ID : 570929253
Device/File Name : +CRS_DATA
Device/File integrity check succeeded
...
Decive/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
The OCRCHECK utility creates a log file in Grid_home/log/hostname/client directory, where

Grid_home is the location of the Oracle grid infrastructure installation, and hostname is the
name of the local node.
The log files have names of the form ocrcheck_nnnnn.log, where nnnnn is the process ID of
the operating session that issued the ocrcheck command.
Oracle Local Registry (OLR)

Starting with Oracle Clusterware 11g Release 2, each node in a cluster has a local registry for
node-specific resources, called an Oracle Local Registry (OLR),that is installed and configured
when Oracle Clusterware installs OCR. Multiple processes on each node have simultaneous
read and write access to the OLR particular to the node on which they reside, regardless of
whether Oracle Clusterware is running or fully functional.
The OLR provides various Oracle Clusterware processes with access to key configuration
information even when Oracle Clusterware is not running on the node. One of its functions is to
facilitate the Oracle Clusterware startup process in situations where the ASM stores the OCR

and Voting disks. During the startup process, the OLR is referenced to determine the exact
location of the voting disks. This enables to node to join the cluster. After this initial phase, ASM
is started. After ASM is started, processes that require the full OCR can start and the
Clusterware startup process completes.
By default, OLR is located at Grid_Home/cdata/hostname.olr.

The OLR is backed up at the end of an installation or an upgrade. After that, you can only
manually back up the OLR. Automatic backups are not supported for the OLR.
You can manage the OLR using ocrcheck, ocrdump and ocrconfig utilities with the –local
option.
 You can check the status of OLR on the local node using the OCRCHECK utility:
#ocrcheck –local
 You can display the content of OLR on the local node to the text terminal that initiated the
program using the OCRDUMP utility:
#ocrdump –local –stdout
 You can perform administrative tasks on OLR on the local node using the OCRCONFIG
utility.
 To export OLR to a file (logical backup):

#ocrconfig –local –export file_name
 To manually back up OLR (physical backup):

Automatic backups are not supported for OLR.
#ocrconfig –local –manualbakcup
 To change the OLR backup location:
#ocrconfig –local –backuploc new_olr_backup_path
 To view the contents of the OLR backup file
#ocrdump –local –backupfile olr_backup_file_name
Monitoring Oracle Clusterware

You can use various tools to monitor Oracle Clusterware. While Oracle recommends that you
use Oracle Enterprise Manager to monitor the everyday operations of Oracle Clusterware,
Cluster Health Monitor (CHM) monitors the complete technology stack, including the operating
system, for the purpose of ensuring smooth cluster operations. Both tools are enabled, by
default, for any Oracle cluster, and Oracle strongly recommends that you use them.
This section discusses Cluster Health Monitor (CHM).

Cluster Health Monitor

The Cluster Health Monitor (CHM) detects and analyzes operating system and cluster resource-
related degradation and failures. CHM stores real-time operating system metrics in the Oracle
Grid Infrastructure Management Repository that you can use for later triage with the help of My
Oracle Support should you have cluster issues.
This section includes the following CHM topics:

 CHM Services
 Collecting CHM Data
CHM Services
CHM consists of the following services:
 System Monitor Service
 Cluster Logger Service
 Oracle Grid Infrastructure Management Repository
System Monitor Service

There is one system monitor service on every node. The system monitor service (osysmond) is
a real-time, monitoring and operating system metric collection service that sends the data to the
cluster logger service. The cluster logger service receives the information from all the nodes and
persists in an Oracle Grid Infrastructure Management Repository database.
Cluster Logger Service

There is one cluster logger service (OLOGGERD) for every 32 nodes in a cluster. Another
OLOGGERD is spawned for every additional 32 nodes (which can be a sum of Hub and Leaf
Nodes). If the cluster logger service fails (because the service is not able come up after a fixed
number of retries or the node where it was running is down), then Oracle Clusterware starts
OLOGGERD on a different node. The cluster logger service manages the operating system
metric database in the Oracle Grid Infrastructure Management Repository.
Oracle Grid Infrastructure Management Repository

The Oracle Grid Infrastructure Management Repository:
 Is an Oracle database that stores real-time operating system metrics collected by CHM. You
configure the Oracle Grid Infrastructure Management Repository during an installation of or
upgrade to Oracle Clusterware 12c on a cluster.
 Runs on one node in the cluster (this must be a Hub Node in an Oracle Flex Cluster
configuration), and must support failover to another node in case of node or storage failure.
You can locate the Oracle Grid Infrastructure Management Repository on the same node as
the OLOGGERD to improve performance and decrease private network traffic.
 Communicates with any cluster clients (such as OLOGGERD and OCLUMON) through the
private network. Oracle Grid Infrastructure Management Repository communicates with
external clients over the public network, only.
 Data files are located in the same disk group as the OCR and voting file.
NOTE: Oracle increased the Oracle Clusterware shared storage requirement to
accommodate the Oracle Grid Infrastructure Management Repository.
 Size and retention is managed with OCLUMON

Collecting CHM Data

You can collect CHM data from any node in the cluster by running the
Grid_home/bin/diagcollection.pl script on the node. You must run this script as privileged
user.
Syntax
Use the diagcollection.pl script with the following syntax:
diagcollection.pl {--collect [--crs | --acfs | -all] [--chmos [--
incidenttime time [--incidentduration time]]]
[--adr location [--aftertime time [--beforetime time]]]
[--crshome path | --clean | --coreanalyze}]
For example, run the following command to collect CHM data

# Grid_home/bin/diagcollection.pl –collect --chmos
To limit the amount of data collected, use the following command syntax:
# Grid_home/bin/diagcollection.pl --collect --chmos
--incidenttime time --incidentduration duration
In the preceding command, the format for the --incidenttime argument is

MM/DD/YYYY24HH:MM:SS and the format for the --incidentduration argument is HH:MM.
For example:
# Grid_home/bin/diagcollection.pl --collect --crshome Grid_home
--chmos --incidenttime 08/23/2015 01:00:00 --incidentduration 00:30
OCLUMON
The OCLUMON command-line tool is included with CHM and you can use it to query the CHM
repository to display node-specific metrics for a specified time period. You can also use
OCLUMON to perform miscellaneous administrative tasks, such as changing the debug levels,
querying the version of CHM, and changing the metrics database size.
The following examples show commands and sample output:

$oclumon version
Cluster Health Monitor (OS), Version 12.1.0.0.2 - Production Copyright 2013

Oracle. All rights reserved.
$ oclumon manage -get MASTER

Master = node1
$ oclumon manage -get alllogger -details

Logger = node1
Nodes = node1,node2
$ oclumon manage -repos changeretentiontime 86400

$ oclumon manage -repos changerepossize 6000

m
co
l.
ai
gm
Oracle Clusterware Diagnostic and Alert Log files
7@
Oracle Clusterware uses Oracle Database fault diagnosability infrastructure to manage
diagnostic data and its alert log. As a result, most diagnostic data resides in the
71
Automatic Diagnostic Repository (ADR), a collection of directories and files located under a
n2
base directory that you specify during installation. This section describes clusterware-specific
wa
aspects of how Oracle Clusterware uses ADR.

ha
This section includes the following topics:

id
 ADR Directory Structure

 Files in the Trace Directory
it
 The Oracle Clusterware Alert Log

Pr
 Incident Trace Files

 Other Diagnostic Data
ADR Directory Structure

Oracle Clusterware ADR data is written under a root directory known as the ADR base. You
specify the file system path to use as the base during Oracle Grid Infrastructure installation and
can only be changed if you reinstall the Oracle Grid Infrastructure.
ADR files reside in an ADR home directory. The ADR home for Oracle Clusterware running on a
given host always has this structure:
ORACLE_BASE/diag/crs/host_name/crs
Under the ADR home are various directories for specific types of ADR data. The directories of
greatest interest are trace and incident. The trace directory contains all normal (non-incident)
trace files written by Oracle Clusterware daemons and command-line programs as well as the
simple text version of the Oracle Clusterware alert log. This organization differs significantly
from versions prior to Oracle Clusterware 12c release 1 (12.1.0.2), where diagnostic log files
were written under distinct directories per daemon.
Files in the Trace Directory

Starting with Oracle Clusterware 12c release 1 (12.1.0.2), diagnostic data files written by Oracle
Clusterware programs are known as trace files and have a .trc file extension, and appear
together in the trace subdirectory of the ADR home.
The Oracle Clusterware Alert Log

Besides trace files, the trace subdirectory in the Oracle Clusterware ADR home contains the
simple text Oracle Clusterware alert log. It always has the name alert.log. The alert log is also
written as an XML file in the alert subdirectory of the ADR home, but the text alert log is most
easily read.
The alert log is the first place to look when a problem or issue arises with Oracle Clusterware.
Oracle Clusterware alert messages are written for most significant events or errors that occur.
Incident Trace Files

Certain errors occur in Oracle Clusterware programs that will raise an ADR incident. In most
cases, these errors should be reported to My Oracle Support for diagnosis. The occurrence of
an incident normally produces one or more descriptive messages in the Oracle Clusterware
alert log.

In addition to alert messages, incidents also cause the affected program to produce a special,
separate trace file containing diagnostic data related to the error. These incident-specific trace
files are collected in the incident subdirectory of the ADR home rather than the trace
subdirectory. Both the normal trace files and incident trace files are collected and submitted to
Oracle when reporting the error.
Other Diagnostic Data

Besides ADR data, Oracle Clusterware collects or uses other data related to problem diagnosis.
Starting with Oracle Clusterware 12c release 1 (12.1.0.2), this data resides under the same
base path used by ADR, but in a separate directory structure with this form:
ORACLE_BASE/crsdata/host_name.
In this example, ORACLE_BASE is the Oracle base path you specified when you installed the
Grid Infrastructure and host_name is the name of the host.
In this directory, on a given host, are several subdirectories. The two subdirectories of greatest
interest if a problem occurs are named core and output.
The core directory is where Oracle Clusterware daemon core files are written when the normal
ADR location used for core files is not available (for example, before ADR services are
initialized in a program).
The output directory is where Oracle Clusterware daemons redirect their C standard output and
standard error files. These files generally use a name structure consisting of the executable
name with the characters OUT appended to a .trc file extension (like trace files). For example,
the redirected standard output from the Cluster Time Synchronization Service daemon is named
octssdOUT.trc. Typically, daemons write very little to these files, but in certain failure scenarios
important data may be written there.
Diagnostics Collection Script

When an Oracle Clusterware error occurs, run the diagcollection.pl diagnostics collection
script to collect diagnostic information from Oracle Clusterware into trace files. The diagnostics
provide additional information so My Oracle Support can resolve problems. Run this script as
root from the Grid_home/bin directory.
Syntax
Use the diagcollection.pl script with the following syntax:
diagcollection.pl {--collect [--crs | --acfs | -all] [--chmos [--
incidenttime time [--incidentduration time]]]
[--adr location [--aftertime time [--beforetime time]]]
[--crshome path | --clean | --coreanalyze}]

Administering ASM Instances

This chapter describes how to administer Automatic Storage Management (Oracle ASM)
instances. It explains how to configure Oracle ASM instance parameters and how to set Oracle
Database parameters for use with Oracle ASM.
Administering an Oracle ASM instance is similar to administering an Oracle Database instance,

but the process requires fewer procedures. You can use Oracle Enterprise Manager and
SQL*Plus to perform Oracle ASM instance administration tasks.
Oracle ASM is installed in the Oracle Grid Infrastructure home separate from the Oracle
Database home. Only one Oracle ASM instance is supported on a server. When managing an
Oracle ASM instance, the administration activity must be performed in the Oracle Grid
Infrastructure home.
Oracle ASM Instances

An Oracle ASM instance is built on the same technology as an Oracle Database instance. An
Oracle ASM instance has a System Global Area (SGA) and background processes that are
similar to those of Oracle Database. However, because Oracle ASM performs fewer tasks than
a database, an Oracle ASM SGA is much smaller than a database SGA.
Also, unlike standard database instance, the ASM instance does not contain any physical files
such as Control files, Redo log file and Data files and only require a few init.ora parameters for
startup.
Upon startup, an ASM instance will spawn all the basic background processes, plus some new
one background processes that are specific to the operation of ASM. All the ASM instance
background processes names begin with asm, as opposed to the database instance, whose
process names begin with ora.
Oracle ASM instances mount disk groups to make Oracle ASM files available to database
instances.
Oracle ASM and database instances require shared access to the disks in a disk group. Oracle
ASM instances manage the metadata of the disk group and provide file layout information to the
database instances.
Oracle ASM instances can be clustered using Oracle Clusterware; there is one Oracle ASM
instance for each cluster node. If there are several database instances for different databases
on the same node, then the database instances share the same single Oracle ASM instance on
that node.
If the Oracle ASM instance on a node fails, then all of the database instances on that node also
fail. Unlike a file system driver failure, an Oracle ASM instance failure does not require restarting
the operating system. In an Oracle RAC environment, the Oracle ASM and database instances
on the surviving nodes automatically recover from an Oracle ASM instance failure on a node.

ASM Instance Background Processes

An ASM instance also has most of the same background processes as ordinary database
instance (SMON, PMON, LGWR….). However ASM instance has two specific background
processes i.e. RBAL and ARB. The ASM instance’s RBAL background process coordinates
the rebalance activity for disk groups and the ARB background process performs the actual
rebalance activity. At any given time there can be many ARB processes started (ARB0, ARB1
etc...).
Each database instance using ASM has two new background processes called ASMB and
RBAL. RABL performs global opens of the disks in the disk groups. A global open means that
more than one database instance can be accessing the ASM disks at a time. At database
instance startup, ASMB connects as a foreground process into the ASM instance. All
communication between the database and ASM instances is performed via this bridge. This
includes physical file changes such as data file creation and deletion. Over this connection,
periodic messages are exchanged to update statistics and to verify that both instances are
healthy
ASM Instance
RBAL coordinates the rebalancing when a new disk is add or removed
(rebalance master)
ARB[1-9] actually does the work requested by the RBAL process (upto 9 of these)
(rebalance)
RDBMS Instance
RBAL performs global opens of all the disks in the disk groups
ASMB The ASMB process contacts Cluster Synchronization Service (CSS)

using the diskgroup name, and acquires the associated ASM connect
string. This connect string is then used to connect into ASM instance as
a foreground process. Using this persistent connection, periodic
messages are exchanged to update statistics and provide a heartbeat
mechanism. During operations that require ASM intervention, such as
a file creation by a database foreground, the database foreground
connects directly to the ASM instance to perform the operation. Also,
database file extent maps are sent by ASM to ASMB via this pipe.
Additionally, ASMB also sends database IO statistics to ASM instance.
ASM registers its name and disks with the RDBMS via the cluster synchronization service
(CSS). This is why the oracle cluster services must be running, even if the node and instance is
not clustered.
Initialization Parameter Files for an Oracle ASM Instance

Like a database instance, ASM instances also require an initialization file. You can store spfile
in cluster file system or ASM disk group. Oracle recommends storing spfile in ASM disk group
When installing Oracle ASM in a clustered Oracle ASM environment where the Oracle Grid
Infrastructure home is shared among all of the nodes, OUI creates a single SPFILE for Oracle
ASM in a disk group. In a clustered environment without a shared Oracle Grid Infrastructure
home, OUI creates an SPFILE in a disk group for the Oracle ASM instance on each node.
You can use an SPFILE or a text-based initialization parameter file (PFILE) as the Oracle ASM
instance parameter file. If you use an SPFILE in a clustered Oracle ASM environment, then you

must place the SPFILE in a disk group, on a shared raw device, or on a cluster file system.
Oracle recommends that the Oracle ASM SPFILE is placed in a disk group.
If you do not use a shared Oracle Grid Infrastructure home, then the Oracle ASM instance can
use a PFILE. The same rules for file name, default location, and search order that apply to
database initialization parameter files also apply to Oracle ASM initialization parameter files.
You can administer Oracle ASM initialization parameter files with SQL*Plus, Oracle Enterprise
Manager, ASMCA, and ASMCMD commands.
During startup, Oracle searches for the spfile in the order below:
1. In a RAC environment the path is retrieved from the Grid Plug and Play profile (GPnP). GPnP
is an XML file that stores the configuration information related to Grid Infrastructure like virtual
ips, public ips etc. It also stores the spfile path and disk discovery path, which are used during
startup. GPnP is identical in all nodes of a cluster and it is stored under
"<Grid_Home>/gpnp/<Node_Name>/profiles/peer/profile.xml"
If it is a single instance configuration, the path of spfile is retrieved from the OLR (Oracle Local
Registry). OLR is a binary file. It contains similar configuration data as GPnP profile. The path of
OLR is kept in "/etc/oracle/olr.loc" file.
2. If spfile path is not set in GPnP profile or OLR, then it is searched under GI home. The path of
spfile should be "$ORACLE_HOME/dbs/spfile+ASM.ora"
3. If no spfile exists then pfile is searched under GI home. The path of pfile should be
"$ORACLE_HOME/dbs/init+ASM.ora"
Backing Up, Copying, and Moving an Oracle ASM Initialization Parameter File
You can back up, copy, or move an Oracle ASM SPFILE with the ASMCMD spbackup,
spcopy or spmove commands. You can also use the SQL CREATE SPFILE to create an
Oracle ASM SPFILE when connected to the Oracle ASM instance. You can copy and move an
Oracle ASM PFILE with the commands available on the specific platform, such as cp for Linux.
If the COMPATIBLE.ASM disk group attribute is set to 11.2 or greater for a disk group, you can
create, copy, or move an Oracle ASM SPFILE into the disk group.
For example, create an Oracle ASM SPFILE from the existing PFILE.
SQL> CREATE SPFILE = '+DATA/asmspfile.ora'

FROM PFILE = '$ORACLE_HOME/dbs/asmpfile.ora';
The CREATE SPFILE statement also updates the Grid Plug and Play (GPnP) profile. You can
check the location of the Oracle ASM SPFILE in the GPnP profile with the ASMCMD spget
command.
Initialization Parameters for Oracle ASM Instance

There are several initialization parameters that you must set for an Oracle ASM instance. You
can set these parameters with Oracle ASM Configuration Assistant (ASMCA). You can also set

some parameters after database creation using Oracle Enterprise Manager or SQL ALTER
SYSTEM or ALTER SESSION statements.
The INSTANCE_TYPE initialization parameter is the only required parameter in the Oracle ASM
instance parameter file. The Oracle ASM* parameters use suitable defaults for most
environments. You cannot use parameters with names that are prefixed with Oracle ASM* in
database instance parameter files
Some database initialization parameters are also valid for an Oracle ASM instance initialization
file. In general, Oracle ASM selects the appropriate defaults for database parameters that are
relevant to an Oracle ASM instance
The following section contains information about some important parameters for Oracle ASM.
INSTANCE_TYPE
The INSTANCE_TYPE initialization parameter must be set to Oracle ASM for an Oracle ASM
instance. This parameter is optional for an Oracle ASM instance in an Oracle Grid Infrastructure
home.
The following is an example of the INSTANCE_TYPE parameter in the initialization file:
INSTANCE_TYPE = ASM
ASM_DISKGROUPS
The ASM_DISKGROUPS initialization parameter specifies a list of the names of disk groups
that an Oracle ASM instance mounts at startup.
ASM_DISKSTRING
The ASM_DISKSTRING initialization parameter specifies a comma-delimited list of strings that
limits the set of disks that an Oracle ASM instance discovers. The discovery strings can include
wildcard characters. Only disks that match one of the strings are discovered.
ASM_POWER_LIMIT
The ASM_POWER_LIMIT initialization parameter specifies the default power for disk
rebalancing in a disk group. The range of values is 0 to 1024. The default value is 1. A value of
0 disables rebalancing. Higher numeric values enable the rebalancing operation to complete
more quickly, but might result in higher I/O overhead and more rebalancing processes.
 For disk groups that have the disk group ASM compatibility set to 11.2.0.2 or greater (for
example, COMPATIBLE.ASM = 11.2.0.2), the operational range of values is 0 to 1024
for the rebalance power.
 For disk groups that have the disk group ASM compatibility set to less than 11.2.0.2, the
operational range of values is 0 to 11 inclusive. If the value for ASM_POWER_LIMIT is
larger than 11, a value of 11 is used for these disk groups
MEMORY_TARGET
This parameter enables Automatic Memory Management for Oracle ASM Instance. Oracle
strongly recommends that you use automatic memory management for Oracle ASM. Automatic
memory management is enabled by default on an Oracle ASM instance, even when the
MEMORY_TARGET parameter is not explicitly set. The default value used for
MEMORY_TARGET is acceptable for most environments

The minimum MEMORY_TARGET for Oracle ASM is 256 MB. If you set MEMORY_TARGET to
100 MB, then Oracle increases the value for MEMORY_TARGET to 256 MB automatically.
DB_CACHE_SIZE
This buffer cache is used to store metadata blocks .You do not have to set a value for the
DB_CACHE_SIZE initialization parameter if you use automatic memory management. The
setting for the DB_CACHE_SIZE parameter determines the size of the buffer cache. The default
value for this parameter is suitable for most environments.
SHARED_POOL_SIZE
Used for standard memory usage (like control structures) to manage the instance. Also used to
store extent maps. You do not have to set a value for the SHARED_POOL_SIZE initialization
parameter if you use automatic memory management. The setting for the
SHARED_POOL_SIZE parameter determines the amount of memory required to manage the
instance. The default value for this parameter is suitable for most environments.
LARGE_POOL_SIZE
Used for large allocations. You do not have to set a value for the LARGE_POOL_SIZE
initialization parameter if you use automatic memory management. The default value for this
parameter is suitable for most environments.
DIAGNOSTIC_DEST
The DIAGNOSTIC_DEST initialization parameter specifies the directory where diagnostics for
an instance are located. The default value for an Oracle ASM instance is the $ORACLE_BASE
directory for the Oracle Grid Infrastructure installation.
PROCESSES
The PROCESSES initialization parameter affects Oracle ASM, but the default value is usually
suitable. However, if multiple database instances are connected to an Oracle ASM instance,
you can use the following formula:
PROCESSES = 50 + 50*n
where n is the number database instances connecting to the Oracle ASM instance
REMOTE_LOGIN_PASSWORDFILE
The REMOTE_LOGIN_PASSWORDFILE initialization parameter specifies whether the Oracle
ASM instance checks for a password file. This parameter operates the same for Oracle ASM
and database instances.
You can use ALTER SYSTEM command to adjust ASM instance parameter settings in an
SPFILE. You can add an optional SID clause to specify that the setting is instance specific or
use the default SID= ‘*’ to explicitly state that the parameter applies to all instances.
For example:
To adjust your spfile so that the +ASM1 instance uses a specific power-limit setting
SQL> ALTER SYSTEM SET ASM_POWER_LIMIT=5 SCOPE=SPFILE SID=’+ASM1’;
To adjust your spfile so that all the ASM instances use the same setting

SQL> ALTER SYSTEM SET ASM_POWER_LIMIT=5 SCOPE=SPFILE SID=’*’ ;
Authentication for accessing ASM Instance

An Oracle ASM instance does not have a data dictionary, so the only way to authenticate to
Oracle ASM instance is by using Operating system authentication and/or Oracle Password file
based authentication.
About Privileges for Oracle ASM

During Oracle ASM installation, you can use one operating system group for all users or divide
system privileges so that database administrators, storage administrators, and database
operators each have distinct operating system privilege groups.
Whether you create separate operating system privilege groups or use one group to provide
operating system authentication for all system privileges, you should use SYSASM to
administer an Oracle ASM instance. The SYSDBA privilege cannot be used to administer an
Oracle ASM instance. If you use the SYSDBA privilege to run administrative commands on an
Oracle ASM instance, the operation results in an error. The SYSDBA privilege is intended to be
used by the database to access disk groups.
Oracle also recommends the use of a less privileged user, such as ASMSNMP with SYSDBA
privileges that is created during installation, for monitoring the Oracle ASM instance.
Operating system authentication using membership in the group or groups designated as

OSDBA, OSOPER, and OSASM is valid on all Oracle platforms. Connecting to an Oracle ASM
instance as SYSASM grants you full access to all of the available Oracle ASM disk groups and
management functions.
Using One Operating System Group for Oracle ASM Users

If you do not want to divide the privileges for system access into separate operating system
groups, then you can designate one operating system group as the group whose members are
granted access as OSDBA, OSOPER, and OSASM for Oracle ASM privileges. The default
operating system group name for all of these is usually dba and that group is typically chosen
for the default configuration.
Using Separate Operating System Groups for Oracle ASM Users

You can designate separate operating system groups as the operating system authentication
groups for privileges on Oracle ASM. The following list describes the separate operating system
authentication groups for Oracle ASM and the privileges that their members are granted.
 OSASM group
This group is granted the SYSASM privilege, which provides full administrative privileges for the
Oracle ASM instance. For example, the group could be asmadmin.
 OSDBA for Oracle ASM group

This group is granted the SYSDBA privilege on the Oracle ASM instance, which grants access
to data stored on Oracle ASM. This group has a subset of the privileges of the OSASM group.
When you implement separate administrator privileges, choose an OSDBA group for the Oracle
ASM instance that is different than the group that you select for the database instance, such as
dba. For example, the group could be asmdba.

You can connect as SYSDBA to use SQL*Plus or ASMCMD commands to manage Oracle ASM
components associated with the database. When running SQL or ASMCMD operations with the
SYSDBA privilege, connect to the database instance rather than the Oracle ASM instance.
Connecting as SYSDBA to the database instance has a limited set of Oracle ASM privileges.
For example, you cannot create a disk group when connected with the SYSDBA privilege.
When connected as SYSDBA to the database instance, the Oracle ASM operations are
limited to:
− Create and delete files, aliases, directories, and templates

− Examine various Oracle ASM instance views
− Operate on files that were created by this user or only access files to which another user
had explicitly granted access
− Granting Oracle ASM File Access Control to other users
 OSOPER for Oracle ASM group

This group is granted the SYSOPER privilege on the Oracle ASM instance, which provides
operations such as startup, shutdown, mount, dismount, and check disk group. This group has a
subset of the privileges of the OSASM group. For example, the group could be asmoper.
The following commands are available to SYSOPER users:

Startup/shutdown
Alter diskgroup mount/dismount
Alter diskgroup online/offline disk
Alter diskgroup rebalance
Alter diskgroup check
Access to all V$ASM_* views
All other commands, such as CREATE DISKGROUP, ADD/DROP/RESIZE DISK and so on

require the SYSASM privilege and are not allowed with SYSOPER privilege.
When you implement separate Oracle ASM and database administrator duties, this
configuration requires different group and different software owners. Implicitly this
implementation requires that the OSASM and OSDBA are different groups. For this
configuration, you must create an OSDBA for Oracle ASM group and a database instance must
be a member of that group to access the Oracle ASM instance.
In an installation that has been configured as Oracle Grid Infrastructure, the Oracle ASM user,
such as grid, does not have to be a member of the Oracle Database OSDBA group, such as
dba1 or dba2, because the Oracle Clusterware database agent runs as the database owner and
can use SYSDBA to connect to the database. Additionally, the owner of the operating system
disk devices should be the same as the owner of the Oracle ASM software.
There are three modes of connecting to Oracle ASM instances:
1. Local Connection Using OS Authentication

Members of OSASM (Operating System ASM) group are granted SYSASM privilege. In an
installation on Linux environment, the "asmadmin" group is OSASM. Shortly, if any OS user is a
member of “asmadmin” group, he/she can connect to the ASM instance with SYSASM
privileges.

Oracle also allows you to separate OSASM group from OSDBA group. Members of OSASM
group will have SYSASM privilege while members of OSDBA group will have SYSDBA
privilege.
2. Local Connection Using a Password File

Like a database instance, ASM instances can have a password file and it allows you to add new
users to the password file. If you use ASMCA for creating disk groups, ASMCA creates a
password file implicitly. As a default, user SYS is added to password file.
3. Remote Connection Using Oracle NET and Password File

It is possible to connect to an ASM instance remotely using Oracle NET. There must be at least
one listener running on host server and ASM instance should be registered to that listener.
$ sqlplus sys/<sys_password>@ASM_MYDB as sysasm
Creating Users with the SYSASM Privilege

When you are logged in to an Oracle ASM instance as SYSASM, you can use the combination
of CREATE USER and GRANT SQL statements to create a user who has the SYSASM
privilege. You also can revoke the SYSASM privilege from a user using the REVOKE command,
and you can drop a user from the password file using the DROP USER command.
Note: These commands update the password file for the local Oracle ASM instance only.
For example:
SQL> CREATE USER new_user IDENTIFIED by new_user_passwd;

SQL> GRANT SYSASM TO new_user;
SQL> REVOKE SYSASM FROM new_user;

SQL> DROP USER new_user;
Starting & Stopping ASM Instances

You start an Oracle ASM instance similarly to the way in which you start an Oracle database
instance with some minor differences.
When starting an Oracle ASM instance, note the following:
 To connect to a local Oracle ASM instance with SQL*Plus, set the ORACLE_SID
environment variable to the Oracle ASM system identifier (SID).
The default Oracle ASM SID for a single-instance database is +ASM, and the default SID for
Oracle ASM for an Oracle RAC node is +ASMnode_number where node_number is the
number of the node. The ORACLE_HOME environment variable must be set to the Grid
Infrastructure home where Oracle ASM was installed.
Note: Oracle recommends that you do not change the default Oracle ASM SID name.
 The initialization parameter file must contain the following entry:

INSTANCE_TYPE = ASM

m
co
l.
ai
gm
This parameter indicates that an Oracle ASM instance, not a database instance, is starting.
7@
 71
When you run the STARTUP command, rather than trying to mount and open a database,
n2
this command attempts to mount Oracle ASM disk groups.
 The associated Oracle database instance does not have to be running when you start the
wa
associated Oracle ASM instance.

ha
id
You can start & stop ASM instances either using SRVCTL or SQL*PLUS.
it
The following commands shows how to start ASM instances using SRVCTL
Pr
To start/stop ASM Instance on a particular node

$srvctl start asm –n nodename
$srvctl stop asm –n nodename
To start/stop ASM Instances on all nodes

$srvctl start asm
$srvctl stop asm
To view the status of ASM instance on a particular node

$srvctl status asm –n nodename
To view the status of ASM instance on all nodes

$srvctl status asm
Starting ASM instances using SQLPLUS

$export ORACLE_SID=+ASM1
$export ORACLE_HOME=/u01/app/12.1.0/grid
$sqlplus / as sysasm
SQL> startup <startup parameter>

For example:
SQL> STARTUP MOUNT
The following list describes how Oracle ASM interprets SQL*Plus STARTUP command
parameters.
 FORCE Parameter
Issues a SHUTDOWN ABORT to the Oracle ASM instance before restarting it.
 MOUNT or OPEN Parameter
Mounts the disk groups specified in the ASM_DISKGROUPS initialization parameter. This is
the default if no command parameter is specified.
 NOMOUNT Parameter
Starts up the Oracle ASM instance without mounting any disk groups.

 RESTRICT Parameter
Starts up an instance in restricted mode that enables access only to users with both the
CREATE SESSION and RESTRICTED SESSION system privileges. You can use the
RESTRICT clause in combination with the MOUNT, NOMOUNT, and OPEN clauses.
In restricted mode, database instances cannot use the disk groups. In other words, databases
cannot open files that are in that disk group. Also, the disk group cannot be mounted by any
other instance in the cluster. Mounting the disk group in restricted mode enables only one
Oracle ASM instance to mount the disk group. This mode is useful to mount the disk group for
repairing configuration issues.
At startup, the Oracle ASM instance attempts to mount the following disk groups:
 Disk groups specified in the ASM_DISKGROUPS initialization parameter
 Disk group used by Cluster Synchronization Services (CSS) for voting files
 Disk groups used by Oracle Clusterware for Oracle Cluster Registry (OCR)
 Disk group used by the Oracle ASM instance to store the ASM server parameter file
(SPFILE)
If no disk groups are found in the previous list, then the Oracle ASM instance does not mount
any disk groups at startup. After the Oracle ASM instance has started, you can mount disk
groups with the ALTER DISKGROUP...MOUNT command
To shut down an Oracle ASM instance using SQL*PLUS, perform the following steps:
$sqlplus / as sysasm
SQL> SHUTDOWN NORMAL
The following list describes the SHUTDOWN modes and the behavior of the Oracle ASM
instance in each mode.
 NORMAL Clause
Oracle ASM waits for any in-progress SQL to complete before performing an orderly
dismount of all of the disk groups and shutting down the Oracle ASM instance. Before the
instance is shut down, Oracle ASM waits for all of the currently connected users to
disconnect from the instance. If any database instances are connected to the Oracle ASM
instance, then the SHUTDOWN command returns an error and leaves the Oracle ASM
instance running. NORMAL is the default shutdown mode.
 IMMEDIATE or TRANSACTIONAL Clause
Oracle ASM waits for any in-progress SQL to complete before performing an orderly
dismount of all of the disk groups and shutting down the Oracle ASM instance. Oracle ASM
does not wait for users currently connected to the instance to disconnect. If any database
instances are connected to the Oracle ASM instance, then the SHUTDOWN command
returns an error and leaves the Oracle ASM instance running. Because the Oracle ASM
instance does not contain any transactions, the TRANSACTIONAL mode behaves the same
as IMMEDIATE mode.

 ABORT Clause
The Oracle ASM instance immediately shuts down without the orderly dismount of disk
groups. This causes recovery to occur upon the next Oracle ASM startup. If any database
instance is connected to the Oracle ASM instance, then the database instance aborts.
Oracle strongly recommends that you shut down all database instances that use the Oracle
ASM instance before attempting to shut down the Oracle ASM instance. If Oracle Cluster
Registry (OCR) or voting files are stored in a disk group, the disk group can only be dismounted
by shutting down the Oracle ASM instance as part of shutting down the clusterware on a node.
Starting and Stopping Node Listener

A standard clustered ASM installation configures an Oracle Network listener under the Grid
home directory. This listener can be manually started and stopped using the lsnrctl utility
installed as part of the ASM installation as mentioned below.
$lsnrctl start listener
$lsnrctl stop listener
You can alternatively use the Server Control Utility (SRVCTL) to start and stop the ASM listener
as below:
$srvctl start listener –n <nodename>
$srvctl stop listener –n <nodename>
The lsnrctl and srvctl utilities exist in both the grid_home and RDBMS home directories.
Which one you use depends on where the listener configuration files reside. The Grid
Infrastructure installation will start a listener with the configuration files in the grid_home. By
default, the database installation will use that listener. In this case, set the ORACLE_HOME and
PATH environment variables to use the grid_home and run the utilities.
If you create a new listener with configuration files in the RDBMS home, set the
ORACLE_HOME and PATH environment variables to use the RDBMS home and run the
utilities.

Creating Oracle ASM Instances with Oracle ASM Configuration Assistant

Oracle ASM Configuration Assistant enables you to create an Oracle ASM instance if an Oracle
ASM instance is not found on the host, as shown in the following figure.
On the Create ASM page, you enter the SYS password for the administrator of the Oracle ASM
instance and the password for the ASMSNMP user that can monitor this Oracle ASM instance.
Note that ASMSNMP is a less privileged user that is primarily used by Oracle Enterprise
Manager to monitor Oracle ASM instances. You can select the listener with which to register
this Oracle ASM instance. Click ASM Parameters to update the initialization parameters for the
Oracle ASM instance.

Administering ASM Disk Groups

This chapter describes how to administer Oracle Automatic Storage Management (Oracle ASM)
disk groups. This information includes how to create, alter, drop, mount, and dismount Oracle
ASM disk groups. The database instances that use Oracle ASM can continue operating while
you administer disk groups.
You can administer ASM Disk groups both with SQL commands and Oracle ASM Configuration
Assistant (ASMCA) graphical tool.
Administering ASM Diskgroups with SQL commands

To use SQL commands, the SQL*Plus should run from the Oracle Grid home where Oracle
ASM is installed and the Oracle environmental variables are to be set to this home.
About Disk Group Attributes

Disk Group attributes, introduced in ASM version 11.1, can be used to fine tune the disk
group capabilities.Disk group attributes are parameters that are bound to a disk group, rather
than an Oracle ASM instance. Some attributes can be set only at the time the disk group is
created, some only after the disk group is created, and some attributes can be set at any
time, by altering the disk group. Some of the attributes are Exadata specific and some are
available in certain versions only.
The following section describes some of the important disk group attributes.
 AU_SIZE
This attribute can only be set when creating a disk group.
 COMPATIBLE.ASM
The value for the disk group COMPATIBLE.ASM attribute determines the minimum software
version for an Oracle ASM instance that can use the disk group.
For Oracle ASM in Oracle Database 11g, 10.1 is the default setting for the COMPATIBLE.ASM
attribute when using the SQL CREATE DISKGROUP statement. When creating a disk group
with ASMCA, the default setting is 11.2.
 COMPATIBLE.RDBMS
The value for the disk group COMPATIBLE.RDBMS attribute determines the minimum
COMPATIBLE database initialization parameter setting for any database instance that is
allowed to use the disk group.
Before advancing the COMPATIBLE.RDBMS attribute, ensure that the values for the
COMPATIBLE initialization parameter for all of the databases that access the disk group are set
to at least the value of the new setting for COMPATIBLE.RDBMS.
For example, if the COMPATIBLE initialization parameters of the databases are set to either
11.1 or 11.2, then COMPATIBLE.RDBMS can be set to any value between 10.1 and 11.1
inclusively.

For Oracle ASM in Oracle Database 11g, 10.1 is the default setting for the
COMPATIBLE.RDBMS attribute when using the SQL CREATE DISKGROUP statement as well
as ASMCA
Note: The database initialization parameter COMPATIBLE enables you to use a new release of
Oracle Database, while at the same time guaranteeing backward compatibility with an earlier
release.
 DISK_REPAIR_TIME
Specifies the amount of time ASM should wait before an OFFLINE disk is dropped and start
disk rebalance operation. By default, Oracle ASM drops a disk in 3.6 hours after it is taken
offline.
The time can be specified in units of minutes (m or M) or hours (h or H). If you omit the unit,
then the default unit is hours. The REPAIR_TIMER column of V$ASM_DISK shows the amount
of time left (in seconds) before an offline disk is dropped.
This attribute can only be set when altering a disk group.
You can display disk group attributes with the V$ASM_ATTRIBUTE view
 FAILGROUP_REPAIR_TIME
This attribute specifies a default repair time for the failure groups in the disk group. The failure
group repair time is used if Oracle ASM determines that an entire failure group has failed. The
default value is 24 hours (24h). If there is a repair time specified for a disk, such as with the
DROP AFTER clause of the SQL ALTER DISKGROUP OFFLINE DISK statement, that disk
repair time overrides the failure group repair time.
This attribute can only be set when altering a disk group and is only applicable to normal and
high redundancy disk groups.
Creating Disk Groups

You can use the CREATE DISKGROUP SQL statement to create a disk group.
When creating a disk group, you:
 Assign a unique name to the disk group.
Disk group name is not case sensitive and is always converted to uppercase when stored
internally.
 Specify the redundancy level (normal, high, external) of the disk group.
After a disk group is created, you cannot alter the redundancy level of the disk group. To
change the redundancy level, you must create another disk group with the appropriate
redundancy and then move the files to the new disk group.
 Specify the disks that are to be formatted as Oracle ASM disks belonging to the
disk group.

The disks can be specified using operating system dependent wildcard characters in search
strings that Oracle ASM then uses to find the disks. You can specify names for the disks with
the NAME clause or use the system-generated names. System-generated names are in
the form diskgroup_nnnn, where nnnn is the disk number for the disk in the disk group.
For ASMLIB disks, the disk name defaults to the ASMLIB name that is the user label of the disk;
for example, mydisk is the default Oracle ASM disk name for ORCL:mydisk.
 Optionally specify the disks as belonging to specific failure groups.
Oracle recommends that you create failure groups of equal size to maintain space balance and
even distribution of mirror data.
 Optionally specify disk group attributes, such as software compatibility or allocation unit size.
 Optionally specify the type of failure group (quorum or regular)

Oracle ASM programmatically determines the size of each disk. If for some reason this is not
possible, or to restrict the amount of space used on a disk, you can specify a SIZE clause for
each disk. Oracle ASM creates operating system–independent names for the disks in a disk
group that you can use to reference the disks in other SQL statements. Optionally, you can
provide your own name for a disk using the NAME clause. Disk names are available in the
V$ASM_DISK view.
The Oracle ASM instance ensures that any disk in a newly created disk group is addressable
and is not currently a member of another disk group. You must use FORCE only when adding a
disk that was dropped with FORCE. If a disk is dropped with NOFORCE, then you can add it
with NOFORCE.
For example, a disk might have failed and was dropped from its disk group. After the disk is
repaired, it is no longer part of any disk group, but Oracle ASM still recognizes that the disk had
been a member of a disk group. You must use the FORCE flag to include the disk in a new disk
group. In addition, the disk must be addressable, and the original disk group must not be
mounted. Otherwise, the operation fails.
Note: Use caution when using the FORCE option to add a previously used disk to a disk group;
you might cause another disk group to become unusable.
Disks with a valid header status of candidate, former, or provisioned are the only ones
allowed to be diskgroup members. This check and validation prevents ASM from destroying any
in-use data devices.
The CREATE DISKGROUP statement mounts the disk group for the first time, and adds the disk
group name to the ASM_DISKGROUPS initialization parameter if a server parameter file is being
used. If a text initialization parameter file is being used and you want the disk group to be
automatically mounted at instance startup, then you must remember to add the disk group name
to the ASM_DISKGROUPS initialization parameter before you shut down and restart the Oracle
ASM instance.
The following SQL statement creates disk group named fra with the default disk group
attribute values.

CREATE DISKGROUP fra NORMAL REDUNDANCY

DISK '/devices/diskc*';
The following SQL statement creates a disk group named data with normal redundancy
consisting of two failure groups controller1 or controller2 with four disks in each failure
group.
CREATE DISKGROUP data NORMAL REDUNDANCY

FAILGROUP controller1 DISK
'/devices/diska1' NAME diska1,
'/devices/diska4' NAME diska4
FAILGROUP controller2 DISK
'/devices/diskb1' NAME diskb1,
'/devices/diskb4' NAME diskb4
ATTRIBUTE 'au_size'='4M',
'compatible.asm' = '11.2',
'compatible.rdbms' = '11.2';
Altering Disk Groups

You can use the ALTER DISKGROUP SQL statement to alter a disk group configuration. You
can add, resize, or drop disks while the database remains online. Whenever possible, multiple
operations in a single ALTER DISKGROUP statement are recommended. Grouping operations
in a single ALTER DISKGROUP statement can reduce rebalancing operations.
Oracle ASM automatically rebalances when the configuration of a disk group changes. By
default, the ALTER DISKGROUP statement does not wait until the operation is complete before
returning. Query the V$ASM_OPERATION view to monitor the status of this operation.
Use the REBALANCE WAIT clause to cause the ALTER DISKGROUP statement processing to
wait until the rebalance operation is complete before returning. This is especially useful in
scripts.
You can interrupt a rebalance running in wait mode by typing CTRL-C on most platforms. This
causes the statement to return immediately with the message ORA-01013: user requested
cancel of current operation, and then to continue the operation asynchronously.
Typing CTRL-C does not cancel the rebalance operation or any disk add, drop, or resize
operations.
To control the speed and resource consumption of the rebalance operation, you can include the
REBALANCE POWER clause in statements that add, drop, or resize disks.
Adding Disks to a Disk Group

You can use the ADD clause of the ALTER DISKGROUP statement to add a disk or a failure
group to a disk group. After you add new disks, the new disks gradually begin to accommodate
their share of the workload as rebalancing progresses.

The following statement successfully adds disks /devices/diska5 through

/devices/diska8 to data1. Because no FAILGROUP clauses are included in the
ALTER DISKGROUP statement, each disk is assigned to its own failure group.
ALTER DISKGROUP data1 ADD DISK

'/devices/diska8' NAME diska8;
The following statement runs with a rebalance power of 5, and does not return until the
rebalance operation is complete
'/devices/diskd*'
REBALANCE POWER 5 WAIT;
If a disk was previously member of a disk group that no longer exists, then you could use the
FORCE option to add the disk as a member of another disk group. For example, the following
example uses the FORCE clause to enable /devices/diskc3 to be added to data2, even
though it is a current member of data3. For this statement to succeed, data3 cannot be
mounted.

'/devices/diskc3' FORCE;
Dropping Disks from Disk Groups

To drop disks from a disk group, use the DROP DISK clause of the ALTER DISKGROUP
statement. You can also drop all of the disks in specified failure groups using the DROP DISKS
IN FAILGROUP clause.
When a disk is dropped, the disk group is rebalanced by moving all of the file extents from the
dropped disk to other disks in the disk group. A drop disk operation might fail if not enough
space is available on the other disks. The best approach is to perform both the add and drop
operation with the same ALTER DISKGROUP statement. This has the benefit of rebalancing
data extents only one time and ensuring that there is enough space for the rebalance operation
to succeed.
Caution: The ALTER DISKGROUP...DROP DISK SQL statement returns to SQL prompt
before the drop and rebalance operations are complete. Do not reuse, remove, or disconnect
the dropped disk until the HEADER_STATUS column for this disk in the V$ASM_DISK view
changes to FORMER. You can query the V$ASM_OPERATION view to determine the amount of
time remaining for the drop/rebalance operation to complete.
If you specify the FORCE clause for the drop operation, the disk is dropped even if Oracle ASM
cannot read or write to the disk. You cannot use the FORCE flag when dropping a disk from an
external redundancy disk group.
The following example drops diska5 from disk group data1.

ALTER DISKGROUP data1 DROP DISK diska5;
The below example drops diska5 from disk group data1, and also illustrates how multiple
actions are possible with one ALTER DISKGROUP statement.
ALTER DISKGROUP data1 DROP DISK diska5

ADD FAILGROUP failgrp1 DISK '/devices/diska9' NAME diska9;
The following example advances the Oracle ASM compatibility for disk group data3 to 11.2.
ALTER DISKGROUP data3 SET ATTRIBUTE 'compatible.asm' = '11.2';
Undropping Disks in Disk Groups

The UNDROP DISKS clause of the ALTER DISKGROUP statement enables you to cancel
all pending drops of disks within disk groups. If a drop disk operation has completed, then this
statement cannot be used to restore it. This statement cannot be used to restore disks that are
being dropped as the result of a DROP DISKGROUP statement, or for disks that are being
dropped using the FORCE clause.
The following example cancels the dropping of disks from disk group data1:
ALTER DISKGROUP data1 UNDROP DISKS;
Mounting and Dismounting Disk Groups

Disk groups that are specified in the ASM_DISKGROUPS initialization parameter are mounted
automatically at Oracle ASM instance startup. This makes them available to all database
instances running on the same node as Oracle ASM. The disk groups are dismounted at Oracle
ASM instance shutdown. Oracle ASM also automatically mounts a disk group when you initially
create it, and dismounts a disk group if you drop it.
When a disk group is mounted, a disk group number is chosen. This number may change
across disk group mounts. A disk group number is not recorded in any persistent structure, but
the current value can be viewed in the GROUP_NUMBER column of the V$ASM views.
When you want to mount or dismount disk groups manually, use the ALTER DISKGROUP
...MOUNT or ALTER DISKGROUP...DISMOUNT statement. You can mount or dismount disk
groups by name, or specify ALL.
If you try to dismount a disk group that contains open files, the statement fails, unless you also
specify the FORCE clause.
In a clustered Oracle ASM environment in RESTRICTED mode, a disk group is mounted in

single-instance exclusive mode. No other Oracle ASM instance in that cluster can mount that
disk group. In this mode the disk group is not usable by any Oracle ASM client. Use this mode
to perform a fast rebalance.
The following SQL statement dismounts all disk groups that are currently mounted to the Oracle
ASM instance:
ALTER DISKGROUP ALL DISMOUNT;

m
co
l.
ai
The following SQL statement mounts disk group data1:
gm
7@
ALTER DISKGROUP data1 MOUNT;
71
n2
Mounting Disk Groups Using the FORCE Option
Oracle ASM provides MOUNT FORCE option with ALTER DISKGROUP to enable Oracle ASM
wa
disk groups to be mounted in normal or high redundancy modes even though some Oracle ASM
ha
disks may be unavailable to the disk group at mount time. The default behavior without the
id
FORCE option is to fail to mount a disk group that has damaged or missing disks.
it
The MOUNT FORCE option is useful in situations where a disk is temporarily unavailable and
Pr
you want to mount the disk group with reduced redundancy while you correct the situation that
caused the outage.
To successfully mount with the MOUNT FORCE option, Oracle ASM must be able to find at least
one copy of the extents for all of the files in the disk group. In this case, Oracle ASM can
successfully mount the disk group, but with potentially reduced redundancy.
The disks that Oracle ASM cannot access are placed in an offline mode. Oracle ASM then
begins timing the period that these disks are in an offline mode. If the disk offline time period
exceeds the timer threshold set by DISK_REPAIR_TIME disk group attribute, then those disks
are permanently dropped from the disk group.
The following example shows how to use the FORCE option to force the mount of the data1
disk group:
ALTER DISKGROUP data1 MOUNT FORCE;
Checking the Internal Consistency of Disk Group Metadata

You can check the internal consistency of disk group metadata using the ALTER DISKGROUP
statement with the CHECK keyword. You can use this statement to check specific files in a disk
group, specific disks or all disks in a disk group, or specific failure groups within a disk group.
The disk group must be mounted to perform these checks.
By default, the CHECK DISK GROUP clause verifies all of the metadata directories. Oracle ASM
displays summary errors and writes the details about the errors in an alert log. The CHECK
keyword performs the following operations:
 Verifies the consistency of the disk

 Cross checks all of the file extent maps and allocation tables for consistency
 Checks that the alias metadata directory and file directory are linked correctly
 Verifies that the alias directory tree is linked correctly
 Checks that Oracle ASM metadata directories do not have unreachable allocated
blocks
The REPAIR | NOREPAIR clause specifies whether Oracle ASM should attempt to repair
errors that are found during the check. The default is NOREPAIR. Use the NOREPAIR clause to
receive alerts about inconsistencies and to suppress Oracle ASM from resolving the errors
automatically.

The following example statement checks for consistency in the metadata for all disks in the
data1 disk group:
ALTER DISKGROUP data1 CHECK ALL;
Dropping Disk Groups

The DROP DISKGROUP statement enables you to delete an Oracle ASM disk group and
optionally, all of its files. You can specify the INCLUDING CONTENTS clause if you also want to
delete all files that are contained in the disk group. The default is EXCLUDING CONTENTS,
which provides syntactic consistency and prevents you from dropping the disk group if it has
any contents.
The Oracle ASM instance must be started and the disk group must be mounted with none of the
disk group files open, in order for the DROP DISKGROUP statement to succeed. The statement
does not return until the disk group has been dropped.
When you drop a disk group, Oracle ASM dismounts the disk group and removes the disk group
name from the ASM_DISKGROUPS initialization parameter if a server parameter file is being
used. If a text initialization parameter file is being used, and the disk group is mentioned in the
ASM_DISKGROUPS initialization parameter, then you must remove the disk group name from
the ASM_DISKGROUPS initialization parameter before the next time that you shut down and
restart the Oracle ASM instance.
The following statement deletes data1:
DROP DISKGROUP data1 INCLUDING CONTENTS;
After ensuring that none of the files contained in data1 are open, Oracle ASM rewrites the
header of each disk in the disk group to remove Oracle ASM formatting information.
If you cannot mount a disk group but must drop it, you can use the FORCE option of the DROP
DISKGROUP statement. This command enables you to remove the headers on disks that
belong to a disk group that cannot be mounted by any Oracle ASM instances as shown in the
following example:
DROP DISKGROUP data1 FORCE;
The disk group on which you perform this operation should not be mounted anywhere in the
cluster. When you use the FORCE option, the Oracle ASM instance does not attempt to verify
that a disk group is being used by another Oracle ASM instance in the same storage
subsystem.
Administering ASM Diskgroups with ASMCA

Oracle ASM Configuration Assistant provides a GUI interface for installing and configuring
Oracle ASM instances, disk groups, volumes, and file systems.
Oracle ASM Configuration Assistant enables you to configure or create Oracle ASM disk groups
with the Configure ASM Disk Groups tab, as shown in the below figure. The disk group tab
displays selected details about the disk group, such as name, size, free space, usable space,
redundancy level, and state of the disk group.

You can right click a selected disk group in the screen to display the configuration options
menu. The menu options enable you to:
 Add disks to the disk group
 Edit the disk group attributes
 Manage templates for the disk group
 Create an Oracle ACFS file system on the disk group for a database home
 Dismount and mount the disk group
 Drop the disk group
There are additional options for creating a disk group, mounting all disk groups, and
dismounting all disk groups. Oracle ASM Configuration Assistant enables you to create an
Oracle ASM disk group by clicking Create on the Oracle ASM Configuration Assistant disk
group page. The page for creating a disk group is shown in the below figure.

On this page, you can enter a name for the disk group, select the redundancy level, change the
disk discovery path to locate disks in a different directory, identify the disks to be in the disk
group.
You can choose to show advanced options when creating a disk group, as shown in the below
figure. The advanced options page enables you to set additional options, such as disk group
compatibility settings and allocation unit (AU) size. When you are finished with the create disk
group specifications, click OK.

Oracle ASM Storage Limits

Oracle ASM provides near unlimited capacity for future growth, but does have some storage
limits. For example, Oracle ASM has the following limits on the number of disk groups, disks,
and files in 12c Release 1.
 511 disk groups in a storage system for Oracle Database 12c Release 1 or later
 10,000 Oracle ASM disks in a storage system
 1 million files for each disk group
Without any Oracle Exadata Storage, Oracle ASM has the following storage limits if the
COMPATIBLE.ASM or COMPATIBLE.RDBMS disk group attribute is set to less than 12.1:
 2 terabytes (TB) maximum storage for each Oracle ASM disk
 20 petabytes (PB) maximum for the storage system

Without any Oracle Exadata Storage, Oracle ASM has the following storage limits if
the COMPATIBLE.ASM and COMPATIBLE.RDBMS disk group attributes are set to 12.1
or greater:
 4 PB maximum storage for each Oracle ASM disk with the allocation unit (AU) size
equal to 1 MB
 8 PB maximum storage for each Oracle ASM disk with the AU size equal to 2 MB
 320 exabytes (EB) maximum for the storage system
With all Oracle Exadata Storage, Oracle ASM has the following storage limits:
 320 EB maximum for the storage system
The maximum size limit of a disk group equals the maximum disk size multiplied by the
maximum number of disks in a disk group (10,000). The maximum number of disks across all
disk groups is 10,000. The 10,000 disks can be in one disk group or distributed across a
maximum of 511 disk groups.
File size limits are dependent on the value of the disk group compatibility attributes. Oracle ASM
supports file sizes greater than 128 TB in any redundancy mode when the
COMPATIBLE.RDBMS disk group attribute is set greater than10.1.
If COMPATIBLE.RDBMS is set to 10.1, the file size limits are less. For example, with
COMPATIBLE.RDBMS equal to 10.1 and the AU size equal to 1 MB, Oracle ASM filesize limits
are:
 External redundancy: 16 TB
 Normal redundancy: 5.8 TB
 High redundancy: 3.9 TB
Note: Oracle Database supports data file sizes up to 128 TB depending on the file system. In
addition, Oracle Database has a file size limit that is dependent on the DB_BLOCK_SIZE
initialization parameter.

ASM Command Line Utility

About ASMCMD
ASMCMD is a command-line utility that you can use to manage Oracle ASM instances, disk
groups, file access control for disk groups, files and directories within disk groups, templates for
disk groups, and volumes.
Types of ASMCMD Command

The following table lists various types of ASMCMD commands that you can use.
Category Commands
ASMCMD Instance Management Commands dsget, dsset, lsct, lsop, lspwusr, orapwusr,
shutdown, spbackup, spcopy, spget, spmove,
spset, startup
ASMCMD File Management Commands cd, cp, du, find, ls, lsof, mkalias, pwd, rm, rmalias
ASMCMD Disk Group Management Commands chdg, chkdg, dropdg, iostat, lsattr, lsdg, lsdsk, lsod,
md_
backup, md_restore, mkdg, mount, offline, online,
rebal,
remap, setattr, umount
ASMCMD Template Management Commands chtmpl, lstmpl, mktmpl, rmtmpl
ASMCMD File Access Control Commands chgrp, chmod, chown, groups, grpmod, lsgrp, lsusr,
mkgrp, mkusr, passwd, rmgrp, rmusr
ASMCMD Volume Management Commands volcreate, voldelete, voldisable, volenable, volinfo,
volresize, volset, volstat
About Oracle ASM Files, Filenames, Directories, and Aliases

ASMCMD works with Oracle ASM files, directories, and aliases. Before using ASMCMD, you
should understand how these common computing concepts apply to the Oracle ASM
environment.
System-Generated Filename or Fully Qualified Filename

Every file created in Oracle ASM gets a system-generated filename, otherwise known as a fully
qualified filename. This is similar to a complete path name in a local file system.
Oracle ASM generates filenames according to the following scheme:

+diskGroupName/databaseName/fileType/fileTypeTag.fileNumber.incarnation
An example of a fully qualified filename is the following:

+data/orcl/CONTROLFILE/Current.256.541956473
In the previous fully qualified filename, data is the disk group name, orcl is the database name,
CONTROLFILE is the file type, and so on.
Only the slash (/) is supported by ASMCMD. Filenames are not case sensitive, but are
case retentive. If you type a path name as lowercase, ASMCMD retains the lowercase.

Directory
As in other file systems, an Oracle ASM directory is a container for files, and an Oracle ASM
directory can be part of a tree structure of other directories. The fully qualified filename
represents a hierarchy of directories in which the plus sign (+) represents the root directory. In
each disk group, Oracle ASM automatically creates a directory hierarchy that corresponds to the
structure of the fully qualified filenames in the disk group. The directories in this hierarchy are
known as system-generated directories.
ASMCMD enables you to move up and down in this directory hierarchy with the cd (change
directory) command. The ASMCMD ls (list directory) command lists the contents of the current
directory, while the pwd command prints the name of the current directory.
When you start ASMCMD, the current directory is set to root (+). For an Oracle ASM instance
with two disk groups, for example, data and fra, entering an ls command with the root
directory as the current directory produces the following output:
ASMCMD> ls
data/
fra/
The following example demonstrates navigating the Oracle ASM directory tree (refer to the fully
qualified filename shown previously):
ASMCMD> cd +data/orcl/CONTROLFILE
ASMCMD> ls
Current.256.541956473
Current.257.541956475
You can create your own directories as subdirectories of the system-generated directories using
the ASMCMD mkdir command. The directories that you create can have subdirectories, and
you can navigate the hierarchy of both system-generated directories and user-created
directories with the cd command.
The following example creates the directory mydir under orcl in the disk group data:
ASMCMD> mkdir +data/orcl/mydir
If you start ASMCMD with the -p flag, then ASMCMD shows the current directory as part of its
prompt
ASMCMD [+] > cd data/orcl

ASMCMD [+data/orcl] >
ASMCMD retains the case of the directory that you entered.
Alias
Aliases are filenames that are references or pointers to system-generated filenames. However,
aliases are user-friendly names. Aliases are similar to symbolic links in UNIX or Linux
computers. You can create aliases to simplify Oracle ASM filename administration.

You can create aliases with the mkalias ASMCMD command or a SQL ALTER DISKGROUP
command. An alias has at a minimum the disk group name as part of its complete path. You can
create aliases at the disk group level or in any system-generated or user-created subdirectory.
The following are examples of aliases:

+data/ctl1.f
+data/orcl/ctl1.f
+data/mydir/ctl1.f
If you run the ASMCMD ls (list directory) with the -l flag, each alias is listed with the system-
generated file to which the alias refers.
ctl1.f => +data/orcl/CONTROLFILE/Current.256.541956473
Absolute Path and Relative Path

When you run an ASMCMD command that accepts a filename or directory name as an
argument, you can use the name as either an absolute path or a relative path.
An absolute path refers to the full path of a file or directory. An absolute path begins with a plus
sign (+) followed by a disk group name, followed by subsequent directories in the directory tree.
The absolute path includes directories until the file or directory of interest is reached. A
complete system-generated filename, otherwise known as the fully qualified filename, is an
example of an absolute path to a file.
Using an absolute path enables the command to access the file or directory regardless of where
the current directory is set. The following rm command uses an absolute path for the filename:
ASMCMD [+] > rm +data/orcl/datafile/users.259.555341963
A relative path includes only the part of the filename or directory name that is not part of the
current directory. That is, the path to the file or directory is relative to the current directory.
In the following example, the rm command operates on the file undotbs1.272.557429239,

which in this case is a relative path
ASMCMD [+] > cd +data

ASMCMD [+data] > cd orcl/DATAFILE
ASMCMD [+data/orcl/DATAFILE] > ls
EXAMPLE.269.555342243
SYSAUX.257.555341961
SYSTEM.256.555341961
UNDOTBS1.258.555341963
UNDOTBS1.272.557429239
USERS.259.555341963
ASMCMD [+data/orcl/DATAFILE] > rm undotbs1.272.557429239
In addition, you can use the pseudo-directories "." and ".." rather than a directory name. The "."
pseudo-directory is the current directory. The ".." pseudo-directory is the parent directory of the
current directory.
The following example demonstrates how to use relative directory paths and pseudo-directories:

ASMCMD [+data/orcl] > cd DATAFILE

ASMCMD [+data/orcl/DATAFILE] >cd ..
ASMCMD [+data/orcl] >
Wildcard Characters
The wildcard characters * and % match zero or more characters anywhere within an absolute or
relative path, which saves typing of the full directory or file name. The two wildcard characters
behave identically. There are various ASMCMD commands that accept wildcards. These
include cd, du, find, ls, lsattr, lsdg, lsdsk, lsgrp, lsusr, and rm.
ASMCMD [+] > cd +data/orcl/C*

ASMCMD [+data/orcl/CONTROLFILE] >
Preparing to Run ASMCMD

Before running ASMCMD, review the items in the following list.
• Log in to the host which contains the Oracle ASM instance that you plan to administer.
You must log in as a user that has SYSASM or SYSDBA privileges through operating
system authentication. The SYSASM privilege is the required connection to administer the
Oracle ASM instance.
• To connect to the Oracle ASM instance, run ASMCMD that is located in the bin
subdirectory of the Oracle Grid Infrastructure home (Oracle ASM home).
Connect as SYSASM, the default connection, to administer an Oracle ASM instance.
Ensure that the ORACLE_HOME and ORACLE_SID environment variables to refer to the Oracle
ASM instance.
The default value of the Oracle ASM SID for a single-instance database is +ASM. In Oracle Real
Application Clusters environments, the default value of the Oracle ASM SID on any node is
+ASMnode#.
• You can connect to the database instance as SYSDBA by running ASMCMD that is located
in the bin directory of the Oracle Database home to manage Oracle ASM components
associated with the database.
Ensure that the ORACLE_HOME and ORACLE_SID environment variables to refer to the
database instance. With this connection, there is a limited set of operations that can be run. For
example, you cannot create a disk group when connected with the SYSDBA privilege.
When connected as SYSDBA to the database instance, the Oracle ASM operations are limited
to:
• Create and delete files, aliases, directories, and templates
• Examine various Oracle ASM instance views
• Operate on files that were created by this user or only access files to which another user
had explicitly granted access
• Granting Oracle ASM File Access Control to other users

m
co
l.
ai
You can specify the -a option to choose the type of connection, either SYSASM or SYSDBA.
gm
The default value is SYSASM and is used when administering an Oracle ASM instance. Connect
7@
as SYSDBA when connecting to the database instance.
71
n2
For example:
$ asmcmd -a sysasm
wa
ha
You can specify the -p option with the asmcmd command to include the current directory in the
id
ASMCMD prompt as shown in the following example:

it
$ asmcmd -p
Pr
ASMCMD [+] > cd data

ASMCMD [+data] >
You can specify the -V option when starting asmcmd to display the asmcmd version number.
After displaying the version number, asmcmd immediately exits.
For example:
$ asmcmd -V
asmcmd version 12.1.0.2.0
Getting Help
Type help at the ASMCMD prompt to view general information about ASMCMD and a list of
available ASMCMD commands.
You can type help command to display help text for a specific command, including usage
information about how to run the command with its options.
The following is an example of the use of the help command.
ASMCMD [+] > help startup
startup [--nomount] [--restrict] [--pfile <pfile.ora>]

Start the ASM instance.
[--nomount] specifies the nomount option.
[--restrict] start the instance in restricted mode.
[--pfile <pfile.ora>] specifies the location of the pfile.
ASMCMD command examples

The following section provides some frequently used ASMCMD commands:
The following example uses dsget to retrieve the current discovery diskstring value from the
GPnP profile and the ASM_DISKSTRING parameter.
ASMCMD [+] > dsget

profile: /devices/disk*
parameter: /devices/disk*
To shutdown ASM instances

ASMCMD [+] > shutdown --normal

ASMCMD [+] > shutdown --immediate
ASMCMD [+] > shutdown –abort
To backup spfile of ASM instance

ASMCMD> spbackup +DATA/asm/asmparameterfile/registry.253.721810181
+DATA/spfileBackASM.bak
To display location of ASM spfile
ASMCMD [+] > spget

+DATA/asm/asmparameterfile/registry.253.691575633

Oracle RAC Architecture

Instance in RAC
You may see few additional background processes associated with a RAC instance than you
would with single-instance database. These additional processes are primarily used to maintain
database coherency among each instance. They manage what is called as Global Resources.
About Global Resources

A resource is an identifiable entity. It basically has a name or a reference. It can be an area in
memory, a disk file or an abstract entity. A resource can be owned or locked in various states
(exclusive or shared). Any shared resource is lockable and if it is not shared no access conflict
will occur.
A resource can be local or global, if it is local it can be acted upon without consulting other
instances, if it is global it cannot be acted upon without consulting or informing remote
instances. GCS is used as a messaging agent to coordinate manipulation of a global resource.
A global resource is a resource that is visible to all the nodes within the cluster. Data buffer
cache blocks are the most obvious and most heavily used global resource, transaction
enqueues and database data structures are other examples.
A local resource is a resource that is accessible to only one instance within the cluster.
In single-instance environments, locking coordinates access to a common resource such as a

row in a table. Locking prevents two processes from changing the same resource (or row) at the
same time.
In RAC environments, internode synchronization is critical because it maintains proper

coordination between processes on different nodes, preventing them from changing the same
resource at the same time. Internode synchronization guarantees that each instance sees the
most recent version of a block in its buffer cache. RAC uses Global Resource Directory (GRD)
to record information about how resources are used within a cluster database. Global Cache
Services (GCS) and Global Enqueue Services (GES) manage the information in the GRD. GRD
is a dedicated area within the shared pool.
Each instance maintains a part of the GRD in its System Global Area. GCS and GES nominate
one instance to manage all information about a particular resource. This instance is called the
resource master. Also, each instance knows which instance masters which resource.
Maintaining cache coherency is an important part of RAC activity. Cache Coherency is the
technique of keeping multiple copies of a block consistent between different oracle instances.
GCS implements cache coherency by using Cache Fusion algorithm. The GCS processes
manage the routing of inter-instance traffic among Oracle RAC instances.
The sequence of operation would go as below

1. When instance A needs a block of data to modify, it reads the bock from disk, before
reading it must inform the GCS (DLM). GCS keeps track of the lock status of the data
block by keeping an exclusive lock on it on behalf of instance A
2. Now instance B wants to modify that same data block, it too must inform GCS, GCS will
then request instance A to release the lock, thus GCS ensures that instance B gets the

latest version of the data block (including instance A modifications) and then exclusively
locks it on instance B behalf.
3. At any one point in time, only one instance has the current copy of the block, thus
keeping the integrity of the block.
GCS maintains data coherency and coordination by keeping track of all lock status of each
block that can be read/written to by any nodes in the RAC. GCS is an in memory database that
contains information about current locks on blocks and instances waiting to acquire locks. This
is known as Parallel Cache Management (PCM).
GES manages all non-cache fusion inter instance resource operations and tracks the status of
all Oracle enqueuing mechanisms. The primary resources that GES controls are dictionary
cache locks and library cache locks. GES also performs deadlock detection to all deadlock-
sensitive enqueues and resources.
GCS handle data buffer cache blocks and GES handle all the non-data block resources.
The following diagram shows the instance in RAC.
Cache Fusion in RAC

Traditional shared disk database systems use disk I/O for synchronizing data access across
multiple nodes. Typically, when two or more nodes contend for the same data block, the node
that has a lock on the data block writes it to disk before the other nodes can access the same
data block. This requires disk I/O that involves moving components; it is inherently slow. It also
requires synchronization in the form of messages that nodes need to exchange to communicate
about lock status. While inter-node message communication can be performed more efficiently
with a good design and optimal implementation strategy, the costs of disk I/O for
synchronization pose significant challenges for scaling non-partitioned workloads or high
contention workloads on traditional shared disk database systems.
The Cache Fusion architecture overcomes this fundamental weakness in traditional shared disk
systems by utilizing the collective caches of all the nodes in the cluster to satisfy database

requests. The Global Cache Service manages the status and transfer of data blocks across the
buffer caches of the instances.
The Global Cache Service is tightly integrated with the buffer cache manager to enable fast
lookup of resource information in the Global Resource Directory. This directory is distributed
across all instances and maintains the status information about resources including any data
blocks that require global coordination. Query requests can now be satisfied by the local cache
or any of the other caches. This reduces disk I/O. Update operations do not require disk I/O for
synchronization since the local node can obtain the needed block directly from any of the cluster
database node caches. Expensive disk I/Os are only performed when none of the collective
caches contain the necessary data and when an update transaction performs a COMMIT
operation that requires disk write guarantees. This implementation effectively expands the
working set of the database cache and reduces disk I/O to dramatically speed up database
operation.
Let’s look, however, at the specific types of cross-node contention for data and how the Cache
Fusion architecture addresses each of them. There are four situations that warrant
consideration when multiple Oracle instances access the same data. For reasons of simplicity,
the examples refer to a two-node cluster with “node 1” and “node 2.”
read/read — User on node 1 wants to read a block that user on node 2 has recently read.
read/write — User on node 1 wants to read a block that user on node 2 has recently updated.
write/read — User on node 1 wants to update a block that user on node 2 has recently read.
write/write — User on node 1 wants to update a block that user on node 2 has recently
updated.
The read/read case typically requires little or no coordination, depending the on specific
database implementation. In a traditional shared disk implementation the request by the user on
node 1 will be satisfied either via local cache access or by way of disk read operations. In the
Cache Fusion implementation, the read request may be served by any of the caches in the
cluster database where the order of access preference is local cache, remote cache, and finally
disk I/O. If the query request is served by a remote cache, the block is transferred across the
high speed cluster interconnect from one node’s cache to another and expensive disk I/O is
avoided.
Both the read/write and write/write cases, in which a user on node 2 updates the block,
coordination between the instances becomes necessary so that the block being read is a read
consistent image (for read/write) and the block being updated preserves data integrity (for
write/write). In both cases, the node that holds the initially updated data block ships the block to
the requesting node across the high speed cluster interconnect and avoids expensive disk I/O
for read. It does this with full recovery capabilities.
In the case of the write/read scenario, node 1 wants to update a block that’s already read and
cached by node 2. An update operation typically involves reading the relevant block into
memory and then writing the updated block back to disk. In this scenario disk I/O for read is
avoided and performance is increased as the block is shipped from the cache of node 2 into the
cache of node 1.
This “fusing of the caches,” to provide users an extended database cache for queries and
updates, makes the Cache Fusion architecture a promising one.

Background Processes in RAC

Each node has its own background processes and memory structures. In RAC, there are
additional background processes to manage the shared resource. These additional processes
maintain cache coherency across the nodes.
The following section describes the RAC specific background processes.
Lock Manager Server Process (LMSn)

Also called as Global Cache Services Process. This is the cache fusion part (GCS) and the
most active process. It maintains records of the data file statuses and each cached block by
recording information in GRD. The LMS process also controls the flow of messages to remote
instances and manages global data block access and transmits block images between the
buffer caches of different instances. This processing is part of Cache Fusion algorithm. It also
rolls back any uncommitted transactions. There can be up to 10 LMS processes running and
can be started dynamically if demand requires it. It also handles global deadlock detection and
monitors for lock conversion timeouts.
Lock Monitor Process (LMON)
Global Enqueue Service Monitor. This process manages the GES, it maintains consistency of
GCS memory structure in case of process death. It is also responsible for cluster reconfiguration
and locks reconfiguration (node joining or leaving), it checks for instance deaths and listens for
local messaging.
Lock Manager Daemon (LMD)
This process manages the GES . It manages the enqueue manager service requests for the
GCS. It also handles deadlock detention and remote resource requests from other instances.
Lock Process(LCK0)
Instance Enqueue Processes. This process manages the GES. It manages instance resource
requests and cross-instance call operations for shared resources. It builds a list of invalid lock
elements and validates lock elements during recovery. It manages non-cache fusion resource
requests such as library and row cache requests.
Atomic Control file to Memory Service (ACMS)

In a RAC environment, the ACMS per-instance process is an agent that contributes to ensuring
a distributed SGA memory update is either globally committed on success or globally aborted if
a failure occurs.
Global Transaction Process (GTX[0-j]):
The GTX[0-j] processes provide transparent support for XA global transactions in a RAC
environment. The database automatically tunes the number of these processes based on the
workload of XA global transactions.
Oracle RAC Management Processes (RMSn)

The RMSn processes perform manageability tasks for Oracle RAC. Tasks accomplished by an
RMSn process include creation of resources related to Oracle RAC when new instances are
added to the clusters.
Remote Slave Monitor (RSMN)
The RSMN process manages background slave process creation and communication on
remote instances. These background slave processes perform tasks on behalf of a coordinating
process running in another instance

Database in RAC
The primary difference between RAC storage and storage for non-RAC (single instance)
database is that all data files in RAC must reside on shared devices to be shared by all
instances that access the same database. You must also create at least two redo log groups for
each instance, and all the redo log groups must also be stored on shared devices for instance
or crash recovery purpose. Each instance’s online redo log groups are called an instance’s
thread of online redo. You must also create one undo tablespace for each instance. Each
instance’s undo tablespace must be shared by all other instances for recovery purposes.
You can place Archive log files on a local file system or shared storage that accessible by all
nodes of the cluster. If you store archive log files on local file system of each node, then you are
always forced to make archive logs available to the other cluster members at the time of
recovery. For this reason Oracle recommends to place archive logs on shared storage. If you
are using the recommended Fast Recovery Area (FRA) feature, then it must be created on
shared storage so that all instances can access it.
The following diagram shows the database architecture in RAC.
Global Dynamic Performance Views

Global Dynamic Performance views retrieve information about all started instances accessing
one RAC database. In contrast, standard Dynamic performance views retrieve information
about the local instance only.
For each of the V$ views available, there is a corresponding GV$ view, with a few exceptions.
You can query GV$ views from any started instance.

Administering RAC Instances & Database

This chapter describes how to administer Oracle Real Application Clusters (Oracle RAC)
databases and database instances.
Most administration tasks are the same for Oracle single-instance and Oracle RAC databases.
This chapter provides additional instructions for database administration tasks specific to Oracle
RAC, and recommendations for managing Oracle RAC databases.
Overview of Oracle RAC Database Administration

Oracle RAC databases support two different management styles and deployment models:
 Administrator-managed deployment is based on the Oracle RAC deployment types

that requires you to statically configure each database instance to run on a specific node
in the cluster, and that you configure database services to run on specific instances
belonging to a certain database using the preferred and available designation.
 Policy-managed deployment, introduced in Oracle Database 11g release 2 (11.2), is

based on server pools, where database services run within a server pool as singleton
or uniform across all of the servers in the server pool. Databases are deployed in one or
more server pools and the size of the server pools determine the number of database
instances in the deployment.
You can manage databases with either the administrator-managed or policy-managed

deployment model using the same commands or methods (such as DBCA or Oracle Enterprise
Manager). All commands and utilities maintain backward compatibility to support the
management of Oracle databases that only support administrator-based management (Oracle
databases before Oracle Database 11g release 2).
In general, a database is defined as a resource in Oracle Clusterware. The database resource

is automatically created when you create your database with DBCA, or you can manually create
the database resource by adding your database with SRVCTL. The database resource contains
the Oracle home, the SPFILE, one or more server pools, and one or more Oracle ASM disk
groups required for the database. The database resource also has a weak start dependency on
the listener type, which means that the resource tries to start all listeners for the node when the
database instance starts. Oracle Clusterware tries to start listeners on the node where the
database instance starts. Starting the listeners in turn starts the VIP for the node.
This course covers Administrator-managed databases.
Tools for Administering Oracle RAC

The following sections introduce Oracle RAC administration using the three tools that you
commonly use to manage Oracle RAC databases and instances:
 SRVCTL utility
 Oracle Enterprise Manager
 SQL*Plus.

Administering Oracle RAC with SRVCTL

The Server Control Utility (SRVCTL) is a command-line interface that you can use to manage
Oracle Databases in a centralized manner.You can use SRVCTL to start and stop the database
and instances, and to delete or move instances and services. You can also use SRVCTL to add
services and manage configuration information, in addition to other resources in the cluster.
When you use SRVCTL to perform configuration operations on your cluster, SRVCTL stores
configuration data in the Oracle Cluster Registry (OCR) in a cluster. SRVCTL performs other
operations, such as starting and stopping instances, by configuring and managing Oracle
Clusterware resources, which define agents that perform database startup and shutdown
operations using Oracle Call Interface APIs.
Administering Oracle RAC with Oracle Enterprise Manager

Oracle Enterprise Manager provides a central point of control for the Oracle RAC environment,
allowing you to perform administrative tasks simultaneously on multiple cluster databases.
Based on the Oracle Enterprise Manager Cloud Control (Grid Control in Oracle Enterprise
Manager 11g) graphical user interface (GUI), you can manage both non-clustered and Oracle
RAC environments.
In Oracle Enterprise Manager, Oracle RAC-specific administrative tasks generally focus on two
levels: tasks that affect an entire cluster database and tasks that affect specific instances. For
example, you can use Oracle Enterprise Manager to start, stop, and monitor databases, cluster
database instances, and their listeners, and to schedule jobs or set up alert thresholds for
metrics. Or you can perform instance-specific commands such as setting parameters or creating
resource plans. You can also use Oracle Enterprise Manager to manage schemas, security,
and cluster database storage features.
Administering Oracle RAC with SQL*Plus

Unlike SRVCTL or Oracle Enterprise Manager, SQL*Plus is an instance-oriented management
tool. This means that SQL*Plus commands operate on the current instance. The current
instance can be either the local default instance on which you initiated your SQL*Plus session,
or it can be a remote instance to which you connect with Oracle Net Services.
Starting a SQL*Plus session and connecting to the database without specifying an instance
directs all SQL*Plus commands to the local instance. In this case, the default instance is also
the current instance. To connect to a different instance in SQL*Plus, issue a new CONNECT
command and specify a remote instance net service name, as shown in the following example:
CONNECT user_name@net_service_name
Enter password:
Connecting as SYSOPER or SYSDBA enables you to perform privileged operations, such as

instance startup and shutdown. Multiple SQL*Plus sessions can connect to the same instance
at the same time. SQL*Plus automatically disconnects you from the first instance whenever you
connect to another one.
Starting and Stopping RAC Instances and Databases

You can start and stop instances with Oracle Enterprise Manager, SQL*Plus or SRVCTL as
described in the following sections. Both Oracle Enterprise Manager and SRVCTL provide
options to start and stop all of the instances in an Oracle RAC database with a single step.

To start an Oracle RAC database instance on a node in the cluster, you must first start the
Oracle Grid Infrastructure stack on the node. An Oracle RAC database instance will not start on
a server on which the Oracle Grid Infrastructure stack is not running.
The procedure for shutting down Oracle RAC instances is identical to shutting down instances
in noncluster Oracle databases, with the following exceptions:
 In Oracle RAC, shutting down one instance does not interfere with the operation of other
running instances.
 To shut down an Oracle RAC database completely, shut down every instance that has
the database open or mounted
 After a NORMAL or IMMEDIATE shutdown, instance recovery is not required.

Recovery is required, however, after you issue the SHUTDOWN ABORT command or
after an instance terminates abnormally. An instance that is still running performs
instance recovery for the instance that shut down. If no other instances are running, the
next instance to open the database performs instance recovery for any instances
needing it.
 Using the SHUTDOWN TRANSACTIONAL command with the LOCAL option is useful
to shut down a particular Oracle RAC database instance. Transactions on other
instances do not block this operation. If you omit the LOCAL option, then this operation
waits until transactions on all other instances that started before you ran the
SHUTDOWN command either commit or rollback, which is a valid approach, if you
intend to shut down all instances of an Oracle RAC database.
Note: SHUTDOWN TRANSACTIONAL and SHUTDOWN TRANSACTIONAL LOCAL both

perform the same action on a nonclustered database but the two commands are different on an
Oracle RAC database
The following section shows various SRVCTL commands to start/stop Oracle RAC instances
and Databases.
 To start or stop your entire cluster database, that is, all of the instances and its enabled
services, enter the following SRVCTL commands:
$ srvctl start database -db db_unique_name [-startoption start_options]

$ srvctl stop database -db db_unique_name [-o stop_options]
 The following SRVCTL command, for example, mounts all of the non-running instances of
an Oracle RAC database:
$ srvctl start database -db orcl -startoption mount
 To start administrator-managed databases, enter a comma-delimited list of instance names:
$ srvctl start instance -db db_unique_name -instance instance_name_list [-startoption

start_options]
 To start policy-managed databases, enter a single node name:

m
co
l.
ai
gm
$ srvctl start instance -db db_unique_name -node node_name [-startoption start_ options]
7@
71
Note that this command also starts all enabled and non-running services that have
AUTOMATIC management policy, and for which the database role matches one of the
n2
service's roles.
wa
ha
 To stop one or more instances, enter the following SRVCTL syntax from the command line:
id
$ srvctl stop instance -db db_unique_name [ -instance "instance_name_list" | -node

node_name ] [ -stopoption stop_options ]
it
Pr
You can enter either a comma-delimited list of instance names to stop several instances or
you can enter a node name to stop one instance.
This command also stops the services related to the terminated instances on the nodes
where the instances were running.
As an example, the following command shuts down the two instances, orcl3 and orcl4,
on the orcl database using the immediate stop option:
$ srvctl stop instance -db orcl -instance "orcl3,orcl4" -stopoption immediate
Starting and Stopping Individual Instances Using SQL*Plus

If you want to start or stop just one instance and you are connected to your local node, you
should first ensure that your current environment includes the SID for the local instance.
To start or shutdown your local instance, initiate a SQL*Plus session and connect with the
SYSDBA or SYSOPER privilege and then issue the required command. For example:
SQL> CONNECT / AS SYSDBA

SQL> STARTUP
You can start multiple instances from a single SQL*Plus session on one node using Oracle
Net Services. Connect to each instance in turn by using a Net Services connection string,
typically an instance-specific alias from your tnsnames.ora file.
For example, you can use a SQL*Plus session on a local node to perform a transactional
shutdown for two instances on remote nodes by connecting to each in turn using the
instance's individual alias name.
SQL> CONNECT sys/xxx@db1 AS SYSDBA

SQL> SHUTDOWN TRANSACTIONAL
SQL> CONNECT SYS/XXXX@db2 AS SYSDBA

SQL> SHUTDOWN TRANSACTIONAL
It is not possible to start or stop multiple instances, simultaneously, with SQL*Plus, so you
cannot start or stop all of the instances for a cluster database with a single SQL*Plus
command.

NOTE: Oracle recommends that you do not use SQL*Plus to manage Oracle ASM
instances in an Oracle RAC environment. Oracle Clusterware automatically manages Oracle
ASM instances, as required. If manual intervention is necessary, then use respective
SRVCTL commands.
Verify that Instances are Running

Using SRVCTL:
You can use SRVCTL to verify that instances are running on a particular database.
$ srvctl status database -db yahoo
This command returns output similar to the following:

Instance yahoo1 is running on node Uclid-rac1
Instance yahoo2 is running on node Uclid-rac2
Using SQL*Plus
1. On any node, from a SQL*Plus prompt, connect to a database instance by using a Net
Services connection string, typically an instance-specific alias from your tnsnames.ora
file.
CONNECT SYS/XXX@db1 as SYSDBA
2. Query the V$ACTIVE_INSTANCES view, using the following statment:
CONNECT SYS as SYSDBA

Enter password: password
SELECT * FROM V$ACTIVE_INSTANCES;
This query returns output similar to the following:
INST_NUMBER INST_NAME
--------------------- -----------------
1 uclid-rac:yahoo1
2 uclid-rac:yahoo2
3 uclid-rac:yahoo3
Terminating Sessions
You can use the ALTER SYSTEM KILL SESSION statement to terminate a session on a
specific instance. When a session is terminated, any active transactions of the session are
rolled back, and resources held by the session (such as locks and memory areas) are
immediately released and available to other sessions.
To terminate sessions, follow these steps:
1. Query the value of the INST_ID column in the GV$SESSION dynamic performance view to
identify which session to terminate.

2. Issue the ALTER SYSTEM KILL SESSION and specify the session index number (SID)
and serial number of a session that you identified with the GV$SESSION dynamic
performance view.
ALTER SYSTEM KILL SESSION 'integer1, integer2[, @integer3]' ;
 For integer1, specify the value of the SID column.

 For integer2, specify the value of the SERIAL# column.
 For the optional integer3, specify the ID of the instance where the session to be killed
exists. You can find the instance ID by querying the GV$ tables.
To use this statement, your instance must have the database open, and your session and
the session to be terminated must be on the same instance unless you specify integer3.
If the session is performing some activity that must be completed, such as waiting for a reply
from a remote database or rolling back a transaction, then Oracle Database waits for this
activity to complete, marks the session as terminated, and then returns control to you. If the
waiting lasts a minute, then Oracle Database marks the session to be terminated and
returns control to you with a message that the session is marked to be terminated. The
PMON background process then marks the session as terminated when the activity is
complete.
For example, the following procedure kills SCOTT’s session.
SQL> SELECT SID, SERIAL#, INST_ID FROM GV$SESSION WHERE USERNAME='SCOTT';
SID SERIAL# INST_ID

---------- ---------- ----------
80 6 2
SQL> ALTER SYSTEM KILL SESSION '80, 6, @2';
System altered.
Initialization Parameter Files in Oracle RAC

When you create the database, Oracle Database creates an SPFILE in the file location that
you specify. This location can be either an Oracle ASM disk group or a cluster file system.
Oracle RAC uses a traditional PFILE only if an SPFILE does not exist or if you specify
PFILE in your STARTUP command. Oracle recommends that you use an SPFILE to
simplify administration, to maintain parameter setting consistency, and to guarantee
parameter setting persistence across database shutdown and startup events. In addition,
you can configure RMAN to back up your SPFILE.
All instances in the cluster database use the same SPFILE at startup.
The value before the period (.) in an SPFILE entry identifies the instance to which the
particular parameter value belongs. When an asterisk (*) precedes the period, the value is
applied to all instances that do not have a subsequent, individual value listed in the SPFILE.

For example, assume that you start an instance with an SPFILE containing the following
entries:
*.OPEN_CURSORS=500
prod1.OPEN_CURSORS=1000
For the instance with the Oracle system identifier (SID) prod1, the OPEN_CURSORS
parameter is set to 1000 even though it has a database-wide setting of 500. Parameter file
entries that have the asterisk (*) wildcard character only affect the instances without an
instance-specific entry. These two types of settings can appear in any order in the
parameter file.
If another DBA runs the following statement, then Oracle Database updates the setting on
all instances except the instance with SID prod1:
ALTER SYSTEM SET OPEN_CURSORS=1500 sid='*' SCOPE=SPFILE;
The SPFILE now has the following entries for OPEN_CURSORS:
*.OPEN_CURSORS=1500
prod1.OPEN_CURSORS=1000
Parameter File Search Order in Oracle RAC

Oracle recommends that you do not use the default SPFILE names because all instances
must use the same file and they all have different SIDs. Instead, store the SPFILE on
Oracle ASM. If you store the SPFILE on a cluster file system, then use the following naming
convention for the SPFILE:
$ORACLE_HOME/dbs/spfiledb_unique_name.ora.
Create a PFILE named $ORACLE_ HOME/dbs/initsid.ora that contains the name

SPFILE=ORACLE_HOME/dbs/spfiledb_unique_name.ora.
Initialization Parameters in Oracle RAC

The following are the parameters used specifically for Oracle RAC databases.
ASM_PREFERRED_READ_ FAILURE_GROUPS
CLUSTER_DATABASE
CLUSTER_DATABASE_ INSTANCES
CLUSTER_INTERCONNECTS
DB_NAME
DISPATCHERS
GCS_SERVER_PROCESSES
INSTANCE_NAME
RESULT_CACHE_MAX_SIZE
SERVICE_NAMES
SPFILE
THREAD
The following parameters must have identical settings on all the instances:
COMPATIBLE
CLUSTER_DATABASE

CONTROL_FILES
DB_BLOCK_SIZE
DB_DOMAIN
DB_FILES
DB_NAME
DB_RECOVERY_FILE_DEST
DB_RECOVERY_FILE_DEST_SIZE
DB_UNIQUE_NAME
INSTANCE_TYPE (RDBMS or ASM)
PARALLEL_EXECUTION_MESSAGE_SIZE
REMOTE_LOGIN_PASSWORDFILE
UNDO_MANAGEMENT
The following parameters must be identical on every instance only if the parameter value is
set to zero:
DML_LOCKS
RESULT_CACHE_MAX_SIZE
Oracle recommends that you set the values for the following parameters to the same value
on all instances. Although you can have different settings for these parameters on different
instances, setting each parameter to the same value on all instances simplifies
administration.
ARCHIVE_LAG_TARGET
CLUSTER_DATABASE_ INSTANCES
LICENSE_MAX_USERS
LOG_ARCHIVE_FORMAT
SPFILE
TRACE_ENABLED
UNDO_RETENTION
Data Dictionary in RAC

For each of the V$ views available, there is a corresponding GV$ view, with a few exceptions.
You can query GV$ views from any started instance. Global Dynamic Performance views (GV$
views) retrieve information about all started instances accessing one RAC database. In contrast,
standard Dynamic performance views retrieve information about the local instance only.
Administering Storage in RAC

Most administration tasks for managing storage are the same for Oracle single-instance and
Oracle RAC databases. This section provides additional information for managing some storage
structures of an Oracle RAC database.
About Automatic Undo Management in Oracle RAC

Oracle RAC automatically manages undo segments within a specific undo tablespace that is
assigned to an instance. You assign undo tablespaces in your Oracle RAC database by
specifying a different value for the UNDO_TABLESPACE parameter for each instance in your
SPFILE or individual PFILEs.
Only the instance assigned to the undo tablespace can modify the contents of that tablespace.
However, each instance can read the undo data blocks created by any instance. Also, when

performing transaction recovery, any instance can update any undo tablespace, if that undo
tablespace is not currently being used by another instance for undo generation or transaction
recovery. Also, all instances of an Oracle RAC database must operate in the same undo mode
(AUTO or MANUAL).
Oracle ASM in RAC

Oracle Automatic Storage Management (Oracle ASM) automatically optimizes storage to
maximize performance by managing the storage configuration across the disks that Oracle ASM
manages. Oracle ASM does this by evenly distributing the storage load across all the available
storage within your cluster database environment. Oracle ASM partitions your total disk space
requirements into uniformly sized units across all the disks in a disk group. Oracle ASM can also
automatically mirror data to prevent data
loss. Because of these features, Oracle ASM also significantly reduces your administrative
overhead. As in single-instance Oracle databases, using Oracle ASM in Oracle RAC does not
require I/O tuning.
When you configure Oracle ASM after installing Oracle Grid Infrastructure for a cluster, one
Oracle ASM instance is created on each node in your Oracle RAC environment if one does not
exist. Each Oracle ASM instance has either an SPFILE or PFILE type parameter file.
In a cluster, each Oracle ASM instance manages the metadata updates to the disk groups for
the node on which it is running. In addition, each Oracle ASM instance coordinates disk group
metadata with other nodes in the cluster.
You can use Enterprise Manager, Oracle ASM configuration assistant (ASMCA), Oracle ASM
command-line utility (ASMCMD), SQL*Plus, and SRVCTL to administer disk groups for Oracle
ASM in an Oracle RAC environment.
Redo Logs in Oracle RAC

Managing redo log files in Oracle RAC environments is similar to managing redo log files in
single-instance Oracle Database environments. The following sections provide an overview of
additional concepts and procedures for configuring redo log files in Oracle RAC environments.
Redo logs contain a record of changes that have been made to data files. In a single-instance
Oracle database, redo logs are stored in two or more redo log file groups. Each of these groups
contains a redo log file and possibly one or more mirrored copies of that file. In an Oracle RAC
database, each instance requires its own set of redo log groups, which is known as a redo
thread. Mirrored copies of the redo
log files provide your system with extra protection against data loss that is due to hardware
failures or data corruption. If a redo log file is unreadable, then Oracle Database attempts to
access its mirrored copy. The redo log file mirrors should be located on different disk devices
from the primary redo log files.
Each instance's redo thread must contain at least two redo log groups. Each redo log group
should contain at least two members: a redo log and its mirrored copy. If you create your Oracle
RAC database using DBCA, then your Oracle RAC database automatically implements a
configuration that meets the Oracle recommendations.
You should create redo log groups only if you are using administrator-managed databases. For
policy-managed databases, if an instance starts due to a change in server pool cardinality, then
Oracle Database automatically creates redo log files, enables a redo thread for the instance if

there is not a redo thread allocated to that instance, and creates the undo tablespace if there is
not an undo tablespace allocated to that instance. The database must be using Oracle
Managed Files and Oracle ASM in this situation.
In an Oracle RAC database, all the redo log files reside on shared storage. In addition, each
instance must have access to the redo log files of all the other instances in the cluster. If your
Oracle RAC database uses Oracle ASM, then Oracle ASM manages the shared storage for the
redo log files and the access to those files.
In an Oracle RAC database, each instance writes and archives the redo log groups in its redo
thread in the same manner that single-instance Oracle databases do. However, in recovery
mode, the instance performing the recovery can read and process all the redo threads for the
database, regardless of which instance generated the redo thread. Being able to read all the
redo threads enables a running instance to recover the work completed by one or more failed
instances.
In case of instance failure, a surviving instance can read the redo logs of the failed instance.
Users can continue to access and update the database without waiting for the failed instance to
be restarted. For example, assume that you have an Oracle RAC database with two instances,
instance A and instance B. If instance A is down, then instance B can read the redo log files for
both instance A and B to ensure a successful recovery.
Archiving in RAC environments

During backup and recovery operations involving archived log files, the Oracle server
determines the file destinations and names from the control file. If you use RMAN, the archived
log file path names can also be stored in the optional recovery catalog. However, the archived
log file path names do not include the node name, so RMAN expects to find the files it needs on
the nodes where the channels are allocated. If you use a supported shared storage scheme,
your instances can all write to the same archive log destination. Backup and recovery of the
archive logs are easy because all logs are located in the same directory.
The archived redo log naming scheme that you use is important because when a node writes to
a log with a specific file name on its file system, the file must be readable by any node that must
access this archived redo log. For example, if node1 archives a log to
/oracle/arc_dest/log_1_100_23452345.arc, then node2 can back up this archived redo log only
if it can read /oracle/arc_dest/log_1_100_23452345.arc on its own file system.
The backup and recovery strategy that you choose depends on how you configure the archiving
destinations for each node. Whether only one node or all nodes perform archived redo log
backups, you must ensure that all archived redo logs are backed up. If you use RMAN
parallelism during recovery, then the node that performs recovery must have read access to all
archived redo logs in your cluster. Multiple nodes can restore archived logs in parallel. However,
during recovery, only one node applies the archived logs. Therefore, the node that is performing
the recovery must be able to access all of the archived logs that are needed for the recovery
operation. By default, the database determines the optimum number of parallel threads to use
during the recovery operation. You can use the PARALLEL clause in the RECOVER command
to change the number of parallel threads.
The primary consideration for Archived Redo Logs in RAC is to ensure that all archived redo
logs can be read from every node during recovery, and if possible during backups.

Also, for any archived redo log configuration, uniquely identify the archived redo logs with the
LOG_ARCHIVE_FORMAT parameter. If you do not specify a log format, then the default is
operating system-specific and includes %t, %s, and %r where t is thread number, s is log
sequence number and r is resetlogs identifier.
You can use either ASM or CFS (Cluster File System) for recovery area though Oracle
recommends to use ASM. When you use ASM storage, Oracle ASM uses an OMF based
naming format for archived redo log files. When you use CFS, you need to set
LOG_ARCHIVE_FORMAT for naming archived redo log files.
In case if you are not using either ASM or CFS, you can use local file system archiving scheme
in which each node archives to a uniquely named local directory. If recovery is required, then
you can configure the recovery node so that it can access directories on the other nodes
remotely. For example, use NFS on Linux and UNIX computers, or mapped drives on Windows
systems. Therefore, each node writes only to a local destination, but each node can also read
archived redo log files in remote directories on the other nodes.
Backups in RAC
Although RAC provides you with methods to avoid or to reduce down time due to a failure of
one or more (but not all) of your instances, you must still protect the database itself, which is
shared by all the instances. This means that you need to consider backup and recovery
strategies for your cluster database just as you would for a nonclustered database.
RMAN enables you to back up, restore, and recover data files, control files, server parameter
files (SPFILEs) and archived redo log files. RMAN is included with Oracle Database and does
not require separate installation.
Before you start using RMAN to backup RAC database you must configure various settings as
discussed in the below sections.
Configuring Channels
While performing backups in parallel, RMAN channels can connect to a different instance in the
cluster. Channel connections to the instances are determined using the connect string defined
by channel configurations. For example, in the following configuration, three channels are
allocated using dbauser/pwd@service_name. If you configure the SQL Net service name with
load balancing turned on, then the channels are allocated at a node as decided by the load
balancing algorithm.
CONFIGURE DEVICE TYPE sbt PARALLELISM 3;

CONFIGURE DEFAULT DEVICE TYPE TO sbt;
CONFIGURE CHANNEL DEVICE TYPE SBT CONNECT 'dbauser/pwd@service_name'
However, if the service name used in the connect string is not for load balancing, then you can
control at which instance the channels are allocated using separate connect strings for each
channel configuration, as follows:
CONFIGURE DEVICE TYPE sbt PARALLELISM 3;

CONFIGURE CHANNEL 1.. CONNECT 'dbauser/pwd@mydb_1';

In the previous example, it is assumed that node1, node2 and node3 are SQL Net service
names that connect to pre-defined nodes in your Oracle RAC environment.
Alternatively, you can also use manually allocated channels to backup your database files. For
example, the following command backs up the SPFILE, controlfile, data files and archived redo
logs:
RUN
{
ALLOCATE CHANNEL CH1 CONNECT 'dbauser/pwd@mydb_1';
BACKUP DATABASE PLUS ARCHIVED LOG;
}
During backup operation, if at least one channel allocated has access to the archived log, then
RMAN automatically schedules the backup of the specific log on that channel. Because the
control file, SPFILE, and data files are accessible by any channel, the backup operation of these
files is distributed across the allocated channels. You can configure channels to use automatic
load balancing or you can specify specific channels for specific instances.
During backup operation, the instances to which the channels connect must be either all
mounted or all open. For example, if the instance on node1 has the database mounted while the
instances on node2 and node3 have the database open, then the backup fails.
Instance Recovery in Oracle RAC

Instance failure occurs when software or hardware problems disable an instance. After instance
failure, Oracle Database automatically uses the online redo logs to perform recovery. In RAC
environments, depending on the fault, a single node may fail or multiple nodes may fail.
Single Node Failure in Oracle RAC

Oracle Clusterware restarts the instance automatically. When one instance performs recovery
for another instance, the surviving instance reads online redo logs generated by the failed
instance and uses that information to ensure that committed transactions are recorded in the
database. Thus, data from committed transactions is not lost. The instance performing recovery
rolls back transactions that were active at the time of the failure and releases resources used by
those transactions.
Multiple-Node Failures in Oracle RAC
When multiple node failures occur, if one instance survives, then Oracle RAC performs instance
recovery for any other instances that fail. If all instances of an Oracle RAC database fail, then
Oracle Database automatically recovers the instances the next time one instance opens the
database. The instance performing recovery can mount the database in either cluster database
or exclusive mode from any node of an Oracle RAC database. This recovery procedure is the
same for Oracle Database running in shared mode as it is for Oracle Database running in
exclusive mode, except that one instance performs instance recovery for all of the failed
instances.
Media Recovery in RAC

Media recovery must be user initiated through a client application, whereas instance recovery is
automatically performed by the database. In these situations, use RMAN to restore backups of
the data files and then recover the database. The procedures for RMAN media recovery in

Oracle RAC environments do not differ substantially from the media recovery procedures for
single-instance environments.
The node that performs the recovery must be able to restore all of the required data files. That
node must also be able to either read all the required archived redo logs on disk or be able to
restore them from backups. Each instance generates its own archive logs that are copies of its
dedicated redo log group threads. It is recommended that Automatic Storage Management
(ASM) or a cluster file system be used to consolidate these files
Parallel Recovery in RAC

Oracle Database automatically selects the optimum degree of parallelism for instance, crash
and media recovery. Oracle Database applies archived redo logs using an optimal number of
parallel processes based on the availability of CPUs. With RMAN’s RESTORE and RECOVER
commands, Oracle Database automatically uses parallelism for the following three stages of
recovery:
• Restoring Data Files: When restoring data files, the number of channels you allocate in the
RMAN recover script effectively sets the parallelism that RMAN uses. For example, if you
allocate five channels, you can have up to five parallel streams restoring data files.
• Applying Incremental Backups: Similarly, when you are applying incremental backups,
the number of channels you allocate determines the potential parallelism.
• Applying Archived Redo Logs with RMAN: Oracle Database automatically selects the
optimum degree of parallelism based on available CPU resources.
To disable parallel instance and crash recovery on a system with multiple CPUs, set the
RECOVERY_PARALLELISM parameter to 0.
Use the NOPARALLEL clause of the RMAN RECOVER command or ALTER DATABASE
RECOVER statement to force the RAC database to use nonparallel media recovery.

m
co
l.
ai
Workload Management with Services
gm
7@
This chapter describes how to manage workloads in Oracle Real Application Clusters (Oracle
71
RAC) to provide high availability and scalability for your applications.
n2
Services are a feature for workload management that organizes the universe of work execution
wa
within the database to make that work more manageable, measurable, tunable, and
ha
recoverable.
id
The concept of a service was first introduced in Oracle8i as a means for the listener to perform
it
connection load balancing between nodes and instances of a cluster. However, the concept,
Pr
definition, and implementation of services have been dramatically expanded.
A service is a grouping of related tasks within the database with common functionality, quality
expectations, and priority relative to other services. A service provides a single-system image
for managing competing applications running within a single instance and across multiple
instances and databases.Services divide workloads into mutually disjoint groupings. Each
service represents a workload with common attributes, service-level thresholds, and priorities.
A single service can represent an application, multiple applications or a subset of a single

application. For example, the Oracle E-Business suite defines a service for each responsibility,
such as general ledger, accounts receivable, order entry, and so on. A single service can be
associated with one or more instances of an Oracle RAC database, and a single instance can
support multiple services.
Services provide the following benefits:

 Provide a single entity for managing applications that compete for the same resources
 Enable each workload to be managed as a unit
 Hide the complexity of the cluster from the client
To manage workloads, you can define services that you assign to a particular application or to a
subset of an application's operations. You can also use services to manage the workload for
different types of work. For example, online users can use one service while batch processing
can use a different service and reporting can use yet another service type.
Using standard interfaces, such as Enterprise Manager, and SRVCTL, services can be
configured, administered, enabled, disabled, and measured as a single entity.
Services are dynamic in that the number of instances a service runs on can be augmented
when load increases, and reduced when load declines. This dynamic resource allocation
enables a cost-effective solution for meeting demands as they occur.
To implement the workload balancing and continuous availability features of services, Oracle
Clusterware stores the HA configuration for each service in the Oracle Cluster Registry (OCR).
The HA configuration defines a set of preferred and available instances that support the service.
A preferred instance set defines the number of instances (cardinality) that support the
corresponding service. It also identifies every instance in the cluster that the service will run on
when the system first starts up.

An available instance does not initially support a service. However, it begins accepting
connections for the service when a preferred instance cannot support the service. If a preferred
instance fails, then the service is transparently restored to an available instance defined for the
service.
Oracle Database supports two broad types of services: Application services and Internal
services. Application services are mainly functional maps to workloads. Sessions doing work
for a common business function are grouped together. For Oracle E-Business suite, AP, AR,
GL, MFG, WIP, BOM, and so on create a functional division of work within the database and
can thus be categorized as services.
There is a limitation of 115 application services per database that you can create.
In addition to application services, the RDBMS also supports two internal services.
SYS$BACKGROUND is used by the background processes only. SYS$USERS is the default
service for user sessions that are not associated with any application service. Both internal
services support all the workload management features and neither one can be stopped or
disabled.
A special Oracle database service is created by default for the Oracle RAC database. This
default service is always available on all instances in an Oracle RAC environment.
The features discussed in this chapter do not work with the default database service. You must
create cluster managed services to take advantage of these features. You can only manage the
services that you create. Any service created automatically by the database server is managed
by the database server.
Administering Services using SRVCTL

You can create and administer services with Oracle Enterprise Manager and the SRVCTL utility.
This class covers administering services with SRVCTL utility.
In addition to creating services, you can:
 Delete a service. You can delete services that you created. However, you cannot delete or
modify the properties of the default database service that Oracle Database created.
 Check the status of a service. A service can be assigned different roles among the
available instances. In a complex database with many services, you may not remember the
details of every service. Therefore, you may have to check the status on an instance or
service basis. For example, you may have to know the status of a service for a particular
instance before you make modifications to that instance or to the Oracle home from which it
runs.
 Start or stop a service for a database or an instance. A service must be started before it
can be used for client connections to that instance. If you shut down your database, for
example, by running the SRVCTL command srvctl stop database -d db_unique_name
where db_unique_name is the name of the database you want to stop, then Oracle
Database stops all services to that database. Depending on the service management policy,
you may have to manually restart the services when you start the database.

 Enable or disable a service for a database or an instance. By default, Oracle

Clusterware attempts to restart a service automatically after failures. You can prevent this
behaviour by disabling a service. Disabling a service is useful when you must perform
database or instance maintenance, for example, if you are performing an upgrade and you
want to prevent connection requests.
 Relocate a service to a different instance. You can move a service from one instance to
another instance to re-balance workloads, for example, after adding or deleting cluster
nodes.
Notes: When you use services, do not set a value for the SERVICE_NAMES parameter; Oracle
Database controls the setting for this parameter for the services that you create and for the
default database service.
Creating Services
To create a service with SRVCTL, enter a command from the command line using the following
syntax:
srvctl add service -database db_unique_name -service service_name [-eval]
-preferred preferred_list -available available_list] [-netnum network_number]
[-tafpolicy {BASIC | NONE | PRECONNECT}] [-edition edition_name]
[-role "[PRIMARY][,PHYSICAL_STANDBY][,LOGICAL_STANDBY][,SNAPSHOT_STANDBY]"
[-policy {AUTOMATIC | MANUAL}] [-notification {TRUE | FALSE}]
[-clbgoal {SHORT | LONG}] [-failovertype {NONE|SESSION|SELECT|TRANSACTION}]
[-rlbgoal {NONE | SERVICE_TIME | THROUGHPUT}] [-dtp {TRUE | FALSE}]
[-failovermethod {NONE | BASIC}] [-failoverretry failover_retries]
[-failoverdelay failover_delay] [-pdb pluggable_database]
[-sql_translation_profile sql_translation_profile]
[-global {TRUE | FALSE}] [-maxlag max_lag_time] [-commit_outcome {TRUE|FALSE}]
[-retention retention_time] [-replay_init_time replay_initiation_time]
[-session_state {STATIC|DYNAMIC}] [-pqservice pq_service]
[-pqpool pq_pool_list] [-force] [-verbose]
For example:
To create a service called GL with preferred instance RAC02 and an available instance
RAC01:
$ srvctl add service –database PROD1 –service GL -preferred RAC02 -available RAC01
Starting and Stopping Services

For applications to connect using a server, the service must be started. If you stop a service,
then it is temporarily unavailable, but is still subject to automatic restart and failover.
Enter the following SRVCTL syntax from the command line to stop or start a service:
$srvctl start service -d database_unique_name [-s service_name_list] [-i inst_name]

[-o start_options]
$srvctl stop service -d database_unique_name -s service_name_list [-i inst_name]

[-o start_options]

For example:
Start a named service on all configured instances

$ srvctl start service –database orcl –service AP
Stop a service:
$ srvctl stop service –database orcl –service AP
Enabling and Disabling Services

If you disable a service, then Oracle Clusterware does not consider the service for automatic
startup, failover, or restart. You might disable a service when performing application
maintenance, to ensure the service is not accidentally restarted by Oracle Clusterware until your
maintenance operations are complete. To make the service available for normal operation
again, you enable the service.
Use the following SRVCTL syntax from the command line to enable and disable services:
$srvctl enable service -d database_unique_name -s service_name_list [-i inst_name]

$srvctl disable service -d database_unique_name -s service_name_list [-i inst_name]
For example:
Disable a service at a named instance

$ srvctl disable service –d orcl –s AP –i orcl4
Relocating Services
Run the srvctl relocate service command from the command line to relocate a service. You
might use this command when a service has failed over to an available instance, but you want
to move it back to the preferred instance after that instance is restarted.
The following command relocates the crm service from instance apps1 to instance apps3:
$srvctl relocate service -d apps -s crm -i apps1 -t apps3
For example:
Relocate a service from one instance to another:

$ srvctl relocate service –d orcl –s AP -i orcl5 –t orcl4
Obtaining the Statuses of Services

Run the srvctl status service command from the command line to obtain the status of a service.
For example, the following command returns the status of the services that are running on the
apps database:
$srvctl status service -d apps
Service erp is running on nodes: apps02,apps03

Service hr is running on nodes: apps02,apps03
Service sales is running on nodes: apps01,apps04

Obtaining the Configuration of Services

Run the srvctl config service command from the command line to obtain the high availability
configuration of a service. For example, the following command returns the configuration of the
erp service that is running on the apps database:
$srvctl config service -d apps -s erp
Service name: erp

Service is enabled
Server pool: sp1
Cardinality: UNIFORM
Disconnect: false
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notificaitons: false
Failover type: NONE
Failover method: NONE
TAF failover retries: 0
TAF failover delay: 0
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: SERVICE_TIME
TAF policy specification: NONE
Edition:
Service is enabled on nodes:
Service is disabled on nodes:
Transparent Application Failover (TAF)

RAC provides near-continuous availability by hiding failures from end-user clients and
application server clients. This provides continuous, uninterrupted data access.
Transparent Application Failover (TAF) or Run time connection failover (RTCF) transparently re-
routs application (query) clients to an available database node in the cluster when the
connected node fails. Application clients do not see error messages describing loss of service.
Furthermore, you can configure database clients to pre-connect, or to have redundant idle
connections. These redundant connections with another database node avoid delays if
thousands of users must migrate servers during a node failure. These rich capabilities provide
great benefits to e-business applications that cannot afford downtime.
When Oracle Net Services establishes a connection to an instance, the connection remains
open until the client closes the connection, the instance is shutdown, or a failure occurs. If you
configure Transparent Application Failover (TAF) for the connection, then Oracle Database
moves the session to a surviving instance when an outage occurs.
TAF can restart a query after failover has completed but for other types of transactions, such as
INSERT, UPDATE, or DELETE, the application must rollback the failed transaction and
resubmit the transaction. You must also re-execute any session customizations, in other words,
ALTER SESSION statements, after failover has occurred. However, with TAF, a connection is
not moved during normal processing, even if the workload changes over time.
tnsnames parameter FAILOVER_MODE enables TAF. When you enable TAF you will define
the following attributes of TAF in tnsnames.ora file.

TYPE: Failover Type can be either SESSION or SELECT.

When TYPE= SESSION is specified, session will failover to the available
instance in the cluster and doesn’t continue with any in-progress SELECT
commands. OCI will start the select process again.
When TYPE=SELECT is specified, session will failover to the available instance

in the cluster and the failover process continue with any in-progress SELECT
commands without any interruption.
When TYPE=NONE disables failover.
METHOD: Failover Method can be either BASIC or PRECONNECT.
When METHOD=BASIC is specified, the second connection to the available instance is

made only if and when the primary instance is down.
When METHOD=PRECONNECT is specified, the second connection to available

instance is made at the same time as the connection to the primary instance. Having the
backup connection already in place can reduce the time needed for a failover in the
event that one needs to take place.
For example, the following tnsnames.ora entry on the client side shows the TAF settings:
ERP=
(DESCRIPTION =(FAILOVER=ON)(LOAD_BALANCE=ON)
(ADDRESS=(PROTOCOL=TCP)(HOST=cluster01-scan)(PORT=1521))
(CONNECT_DATA =
(SERVICE_NAME = ERP)
(FAILOVER_MODE =
(TYPE=SELECT)
(METHOD=BASIC)
(RETRIES=180)
(DELAY=5))))
Services & TAF

Services simplify the deployment of TAF. You can define a TAF policy for a service, and all
connections using this service will automatically have TAF enabled. This does not require any
client-side changes. The TAF setting on a service overrides any TAF setting in the client
connection definition.
The TAF setting on a service can be NONE, BASIC, or PRECONNECT and overrides any TAF
setting in the client connection definition. You can also specify a single TAF policy for all users
of a service by defining the FAILOVER_METHOD (-m option), FAILOVER_TYPE (-e option).
The TAF policy can also define the number of times that a failed session attempts to reconnect
to the service (-z option) and how long it should wait between reconnection attempts (-w option).

To define a TAF policy for a service, use SRVCTL as in the following example, where the
service name is tafconn.example.com and the database name is CRM:
$ srvctl modify service -d crm -s tafconn.example.com -P BASIC -e SELECT -z 5 -w 120

Patching RAC Environments

The following are the different types of patches that are available for Oracle Grid Infrastructure
and/or Oracle RDBMS.
 Patchset
 Bundle Patch (BP)
 One-Off Patch or Interim Patch
 Patchset Update (PSU)
 Critical Patch Update (CPU)
Patchset
Patchset is a set of patches provided on top of a base release (for example, 10.2.0.1.0 or
11.2.0.1.0). A patchset increments the fourth digit of the release number. A patchset will include
updates for Oracle Grid Infrastructure and updates for the Oracle RDBMS software. You must
update Oracle Grid Infrastructure before updating the Oracle RDBMS product. Though the
reverse is not true, Oracle Grid Infrastructure may be patched without patching the RDBMS.
Patchsets are released less frequently than other types of patches. They are cumulative and
may contain hundreds of fixes.
A patchset must be installed using Oracle Universal Installer (OUI) and is generally considered
as an "upgrade".
Prior to 11gR2, a base release had to be installed before a patchset could be applied. For
example, to install 10.2.0.5 on Linux, the 10.2.0.1 base release has to be installed first and then
upgraded to 10.2.0.5. Starting with 11gR2, patchset releases are now full releases and no
longer require a "base" release, e.g. 11.2.0.2 can be installed directly without having to install
11.2.0.1 first.
Prior to 11gR2, the same patchset download is used to patch the Clusterware, ASM and
Database homes. For example, Patch 8202632 is the 10.2.0.5 patchset, this same patch (Patch
8202632) will be used to patch the 10.2.0.x Clusterware, 10.2.0.x ASM and 10.2.0.x Database
to 10.2.0.5.
Prior to 11gR2 - even though the CRS and RDBMS base releases were provided on separate
media (downloadable zip file or separate DVD/CD) - the patchsets for both products were
delivered as one i.e. the same patchset could be applied to the CRS as well as the RDBMS
home.
Starting with 11gR2, the patchsets for Grid Infrastructure and RDBMS are delivered separately
(as they are full releases).
Grid Infrastructure patchsets can be applied in a rolling fashion, while database patchsets
cannot. For example, you can rolling upgrade Grid Infrastructure to 11.2.0.2, but you have to
shutdown database on all nodes to upgrade to database 11.2.0.2.

Bundle Patch
The second common type of patch available for Oracle Grid Infrastructure is known as Bundle
Patch (BP). A Bundle Patch is a small grouping of individual fixes, usually around 20 to 30,that
are fully regression tested and released more frequently than patchsets. Each bundle patch on
the same patchset level is cumulative for that patchset level only. The naming convention is
11.1.0.6.0 BP#1, 11.1.0.6.0 BP#2, and so on. When the patchset level is incremented, the
bundle patch numbering will restart. Bundle patches are always applied with the opatch utility
instead of the OUI utility. BPs are binaries and do not require relinking.
One-Off Patch
Another type of patch that may be applied to Oracle Grid Infrastructure is known as “One Off”
or “Interim” patch. These are installed with the opatch utility and are usually priority 1 fixes that
are not contained in a bundle yet. An interim patch contains fixes for one or in some cases
several bugs (merge patch).
Grid Infrastructure interim patches are rare, they usually are built on top of the latest PSU (at the
time) and include the entire PSU they were built on. The same does not hold true for database
interim patches, they usually do not include a PSU.
Patch Set Updates (PSU)

Patch Set Update (PSU) is patches containing important updates and fixes accumulated from
the last PSU released. The purpose of these types of patches is to provide important updates
and fixes in a single, well tested package on a regular schedule. The PSUs generally supersede
older bundle patches and establish a new baseline version for easier tracking.
 Clusterware PSUs (pre-11.2) are referred to as CRS PSUs. Grid Infrastructure PSUs
(11.2 onwards) are referred to as GI PSUs.
 Grid Infrastructure (GI) PSUs contains both GI PSU and Database PSU for a particular
quarter.
For example, 11.2.0.2.2 GI PSU contains both the 11.2.0.2.2 GI PSU and the 11.2.0.2.2
Database PSU. You are able to take note of this when you extract a GI PSU, you will
see 2 directories (named with the Patch number) one for the GI PSU and one for the
RDBMS PSU. Once the GI PSU is applied, "opatch lsinventory" will show that both GI
PSU and DB PSU are applied.
GI PSUs are always cumulative meaning that you can apply higher version GI PSU directly
without having to apply a lower version one first. For example, the 11.2.0.2.2 GI PSU can be
applied to a 11.2.0.2 home without having to apply GI PSU 11.2.0.2.1 first.
Database PSUs can be subject to overlay PSU packaging. In these cases, the PSUs are still
cumulative, but a higher PSU may require a lower PSU to be applied first; for example, to apply
database PSU 10.2.0.4.7, you must apply database PSU 10.2.0.4.4 first. If a previous PSU is a
prerequisite to a later PSU the requirement will be clearly documented in the PSU readme.

CPU Patches
Critical Patch Updates are collection of high-priority fixes for security related issues and are only
applicable to the Database Home. CPUs are released quarterly on the same cycle as PSUs and
are cumulative with respect to prior security fixes but may contain other fixes in order to address
patch conflicts with non-security patches. PSUs always contain the CPU for that respective
quarter.
Rolling Patches
A rolling patch allows one node to be patched to the latest version, while other nodes continue
to use the older version and provide business service.
Rolling patches are enabled by using a locally accessible, non-shared file system to store the
software files, known as Grid Home. Each node maintains its own copy. With this technique on
a 5-node cluster, there would be 5 copies of the Oracle Grid Infrastructure software—that is, 5
copies requiring disk space and 5 copies to update. This technique enables rolling patches and
rolling upgrades by allowing two versions in the cluster at the same time.
Rolling patches cannot be done when the Oracle Grid Infrastructure software files are stored in
a shared cluster file system in which a single copy of the software is shared among all nodes. A
single copy requires much less disk space and a single copy to patch or upgrade. However, to
patch or upgrade, the software must be stopped. Stopping the Oracle Grid Infrastructure
software also requires all databases, applications, and services that depend on Oracle Grid
Infrastructure to be stopped. This technique requires a complete outage with down time to patch
or upgrade.
Note: A patchset that can be rolled for the Grid Infrastructure may not be able to be rolled for
the RDBMS.
By default, Oracle supports rolling upgrade for patches for Oracle Grid Infrastructure. This is
available for patchsets, bundle patches, and most individual patches.
Rolling upgrades work for Oracle Clusterware and Oracle Automatic Storage Management
(Oracle ASM) (collectively known as Oracle Grid Infrastructure), but not for the Oracle Database
software.
You cannot upgrade to a new major release, maintenance release, or patch release of Oracle
Database using the rolling upgrade process in an Oracle Real Application Clusters (Oracle
RAC) environment.
You can, however, apply most database patches (sometimes referred to as one-off patches)
and all database patch set updates and critical patch updates (PSUs and CPUs) using a rolling
patch install in an Oracle RAC environment.
How to determine whether or not a One-Off patch is "rolling upgradeable"?

After you have downloaded a patch, you can go into the directory where you unpacked the
patch:
> pwd
/ora/install/4933522

m
co
l.
ai
gm
Then use the following OPatch command:
> opatch query is_rolling_patch
7@
...
71
Query ...
n2
Please enter the patch location:
/ora/install/4933522
wa
---------- Query starts ------------------

ha
Patch ID: 4933522

id
....
Rolling Patch: True.
it
---------- Query ends -------------------

Pr
Out-of-Place Oracle Grid Infrastructure Upgrade

Oracle Grid Infrastructure 11g Release 2 supports only out-of-place upgrades. Out-of-place
upgrades mean the new software is placed in a different Oracle Grid Infrastructure home
directory. In past versions, in-place upgrades were performed. In-place upgrades mean that the
software being installed is placed in the same home directory as the existing or active software.
With the out-of-place upgrade, both versions of the software are present on the nodes at the
same time, in different Grid homes; but only one is active.
Installing Oracle Grid Infrastructure in a separate home before the upgrade reduces the planned
outage time required for cluster upgrades, which assists in meeting availability service-level
agreements. After the software is installed, you can upgrade the cluster by stopping the
previous version of the software and starting the new version node by node (known as a rolling
upgrade).
Each node maintains a “software version” that indicates the version of the Oracle Grid
Infrastructure that is active on the machine. The software version is stored in the OCR. There is
also a clusterwide “active version” for the entire cluster that reflects the version of the
communication protocols and shared disk data structures that are currently in use on the
cluster.
Checking Software Versions

When a rolling patch or upgrade is being performed, two versions of the software will
temporarily coexist in the cluster. The software version is the latest version that is installed on
an individual node. You can check the software version registered in the OCR with the following
command:
$ crsctl query crs softwareversion

Oracle Clusterware version on node [host01] is [11.2.0.1.0]
The active version is the lowest version anywhere in the cluster. It applies to the cluster and not
an individual node. The active version is not updated until the last node has been updated to the
newest version. You can check the active version with the following command:
$ crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [11.1.0.6.0]

Permanently operating Oracle Grid Infrastructure at different versions is not supported. This is
allowed only for a short duration—that is, the time it takes to apply the patchset or patch to the
cluster.
Note: The version of Oracle Grid Infrastructure must be greater than the version running other
Oracle products such as the Real Application Clusters (RAC) database.
About OPatch
OPatch is a Java-based utility that allows the application and rolling back of interim patches to
an Oracle product. It is not used to apply patchsets. OPatch is included with the Oracle Grid
Infrastructure 11g installation. When patching an Oracle Grid Infrastructure installation using
the rolling-patch method, the user is prompted for which nodes to patch during the patch
installation process. A rolling patch is identified inside the patch and cannot be enabled when
invoking the patch tool.
OPatch also supports a minimum down-time mode. An inventory of all patches that have been
installed is maintained by OPatch.
Patching Oracle Grid Infrastructure has special considerations: Part of the software is owned by
root and part is owned by the account used to install the product (typically grid). To patch the
root-owned components, the Grid Infrastructure software will need to be unlocked by running
special rootcrs.pl . After patching, the clusterware software will need to be returned to its
previous secure settings by running rootcrs.pl script.

Migrating Non-ASM Database to ASM

This chapter describes how to migrate a database using Filesystem storage to Oracle Automatic
Storage Management (Oracle ASM) storage.
Purpose of Oracle ASM Data Migration

Alternatives to Oracle ASM storage include file systems, raw disks, and SAN configurations.
Oracle ASM includes numerous benefits over these storage alternatives, including performance
optimization, redundancy protection, and load balancing. You do not need a third-party Logical
Volume Manager because Oracle ASM manages disks for you. Oracle Real Application
Clusters (Oracle RAC) databases benefit from Oracle ASM because it provides ready-made
shared storage.
If a database currently uses a storage system other than Oracle ASM, then you can migrate all
or part of the database into Oracle ASM, thereby simplifying database administration. You can
also migrate a fast recovery area to Oracle ASM. Native operating system commands such as
Linux cp or Windows COPY cannot write or read files in Oracle ASM storage. Because RMAN
can read and write Oracle ASM files, you can use RMAN to copy data files into and out of
Oracle ASM storage or between Oracle ASM disk groups. This technique is useful if you must
store backups on user-managed disks.
Basic Concepts of Oracle ASM Data Migration

You can migrate data to Oracle ASM with RMAN even if you are not using RMAN as your
primary backup tool. The migration requires one RMAN database backup.
After you set the location of the new recovery area, existing backups remain in the old recovery
area and count against the total disk quota of the recovery area. The backups are deleted from
the old recovery area when space is needed. These backups are usable by RMAN. It is not
necessary to move legacy backups to the new Oracle ASM recovery area unless you need disk
space. To free space consumed by files in the old recovery area, you can back them up to tape
or migrate them to the Oracle ASM recovery area.
Basics Steps of Data Migration to Oracle ASM Using RMAN

To migrate the entire database and fast recovery area from alternative storage to Oracle ASM,
perform the following steps:
1. Back up the database and server parameter file, and disable Oracle Flashback Database.
1. Restore files to Oracle ASM, recover the database, and optionally migrate the fast recovery
area to Oracle ASM.
Migrating the Database to Oracle ASM using RMAN

This section explains how to migrate the database from Filesystem to Oracle ASM. This section
makes the following assumptions:
a) The database to be migrated to Oracle ASM storage is named yahoo

b) The database to be moved to ASM is in ARCHIVELOG mode and uses SPFILE
c) Two ASM disk groups named DATADG, FRADG will be used to move the database to
ASM

1. With the target database open, edit the initialization parameter control_files and
db_create_file_dest to point to the ASM disk group +DATADG. Also configure
db_recovery_file_dest to point to the ASM disk group +FRADG:
SQL> ALTER SYSTEM SET control_files='+DATADG' SCOPE=spfile;

SQL> ALTER SYSTEM SET db_create_file_dest='+DATADG' SCOPE=spfile;
SQL> ALTER SYSTEM SET db_recovery_file_dest='+FRADG' SCOPE=spfile;
2. Startup the target database in NOMOUNT mode:
SQL> SHUTDOWN IMMEDIATE

SQL> STARTUP NOMOUNT
3. From an RMAN session, copy one of your control files from the local file system to its new
location in ASM. The new control file will be copied to the value specified in the initialization
parameter control_files:
RMAN> RESTORE CONTROLFILE FROM '/u01/…../yahoo/control01.ctl';

RMAN> ALTER DATABASE MOUNT;
4. From an RMAN session, copy the database files from the local file system to ASM, switch
the database to copy, perform recovery and open the database.
RMAN> BACKUP AS COPY DATABASE FORMAT '+DATADG';
RMAN> SWITH DATABASE TO COPY;

RMAN> RECOVER DATABASE;
Move Temporary Files
RMAN> run
{
Set newname for tempfile 1 to ‘+DATADG’;
Switch tempfile all;
}
Open the database
RMAN> ALTER DATABASE OPEN NORESETLOGS;
5. Drop and recreate on ASM any online redo logfiles that are still currently on the local file
system
SQL> alter database drop logfile group 1;

SQL> alter database add logfile group 1 size 50m;
Repeat this step until all logs are rebuilt in ASM

6. Relocate the SPFILE from the local file system to an ASM disk group
RMAN> BACKUP SPFILE;

RMAN> RESTORE SPFILE TO ‘+DATADG/spfileSID.ora’;
RMAN> SHUTDOWN IMMEDIATE;
Delete spfilesid.ora and initsid.ora from the ORACLE_HOME/dbs directory and create a new
initsid.ora with the following line of content:
SPFILE='+DATADG/spfilesid.ora‘
7. Start the database using new SPFILE
SQL> STARTUP;
At this point, the target database is open with all of its datafiles, controlfiles, online redo logfiles,
tempfiles, and SPFILE stored in ASM. You may optionally remove the old files related to
database that were stored on local Filesystem.

Converting Single Instance Database to RAC Database

You can use any of the following tools to convert a single instance database to RAC
 rconfig
 dbca
 Oracle Enterprise Manager
In this chapter you will use rconfig utility to convert a single instance database to Oracle RAC.
rconfig is a command line tool introduced in Oracle Database 10g R2 to convert Single-Instance
10g R2 Database to RAC(Real Application Cluster). rconfig is located at $ORACLE_HOME/bin/.
rconfig takes in a xml input file and converts the Single Instance database whose information is
provided in the xml. A sample xml can be found
$ORACLE_HOME/assistants/rconfig/sampleXMLs/ ConvertToRAC_AdminManaged.xml
rconfig performs following steps :

 Migrate the database to ASM storage (Only if ASM is specified as storage option in the
configuration xml file)
 Create Database Instances on all nodes in the cluster
 Configure Listener and Net Service entries
 Configure and register CRS resources
 Start the instances on all nodes in the cluster.
Prerequisites to use rconfig

Before you convert a single-instance database to a RAC database using rconfig, ensure that the
following conditions are met for each cluster node that you intend to make a RAC database
node:
 Oracle Clusterware is installed, configured, and running.
 Oracle Real Application Clusters software is installed.
 The Oracle binary is enabled for RAC.
 Shared storage, either Oracle Cluster File System or ASM, is available and accessible
from all nodes.
 User equivalence exists for the oracle user.
 Default listener should be running from Grid Infrastructure home and preferably on port
1521.
How to test rconfig before converting to RAC?

Oracle has provided a sample rconfig input XML file called ConvertToRAC_AdminManaged.xml
under the directory $ORACLE_HOME/assistants/rconfig/sampleXMLs. Make a copy of the
sample XML file & make your own customised copy of input giving your Instance specific
details. The Sample XML file includes comments to make understand how to edit each variable.
For testing purpose you need to modify the "Convert verify = "ONLY" option in the xml file.
Then run the tool :
$ ./rconfig ConvertToRAC_AdminManaged.xml
The Convert verify option in the ConverttoRAC_AdminManaged.xml file has three options:

 Convert verify="YES": rconfig performs checks to ensure that the prerequisites for
single-instance to RAC conversion have been met before it starts conversion
 Convert verify="NO": rconfig does not perform prerequisite checks, and starts
conversion
 Convert verify="ONLY" : rconfig only performs prerequisite checks; it does not start
conversion after completing prerequisite checks
rconfig log files

rconfig log files are located at $ORACLE_HOME/cfgtoollogs/rconfig. Also one important thing to
note is that rconfig re-writes the log file everytime you run the rconfig tool. So make sure have a
copy in case you want to refer to the results of earlier run.
Restarting Rconfig
After running RCONFIG in Convert verify="YES" Mode, if you get any fatal error(e.g. Disk space
not available, Some issue with 10g Parameters, Shared Storage Issues) which exits the rconfig
tool & stops the conversion.
You can restart the rconfig tool again by running the command
$ ./rconfig ConvertToRAC_AdminManaged.xml
This will perform the clean-up operation on converted Instance or will delete the files created by
earlier run. It again starts the process of conversion from start i.e. taking rman backup of Single
Instance. After the conversion you can check the filesystem to verify if there is anything left from
the previous failed run.
Performing the conversion to RAC

Backup $ORACLE_HOME/assistants/rconfig/sampleXMLs/ConvertToRAC_AdminManaged.xml
and modify it:
<?xml version="1.0" encoding="UTF-8"?>

< n:RConfig xmlns:n="http://www.oracle.com/rconfig"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.oracle.com/rconfig rconfig.xsd">
< n:ConvertToRAC>
< n:Convert verify="YES">
< n:SourceDBHome>/u02/app/oracle/database/12.1</n:SourceDBHome>
< n:TargetDBHome>/u02/app/oracle/database/12.1</n:TargetDBHome>
< !--Specify SID of non-rac database and credential. User with sysdba role is
required to perform conversion -->
< n:SourceDBInfo SID="test">
< n:Credentials>
< n:User>sys</n:User>
< n:Password>oracle</n:Password>
< n:Role>sysdba</n:Role>
< /n:Credentials>
< /n:SourceDBInfo>
< n:NodeList>
< n:Node name="vmrac1"/>
< n:Node name="vmrac2"/>
< /n:NodeList>
< n:InstancePrefix>test</n:InstancePrefix>
< n:SharedStorage type="ASM">

< n:TargetDatabaseArea>+DATADG</n:TargetDatabaseArea>
< !--Specify Fast Recovery Area to be configured for rac database. -->
< n:TargetFlashRecoveryArea>+FRADG</n:TargetFlashRecoveryArea>
< /n:SharedStorage>
< /n:Convert>
< /n:ConvertToRAC>
< /n:RConfig>
As database user, execute the following:
$ rconfig ConvertToRAC_AdminManaged.xml
Converting Database "test" to Cluster Database. Target Oracle Home:
/u02/app/oracle/database/12.1. Database Role: PRIMARY.
Setting Data Files and Control Files
Adding Database Instances
Adding Redo Logs
Enabling threads for all Database Instances
Setting TEMP tablespace
Adding UNDO tablespaces
Adding Trace files
Setting Fast Recovery Area
Updating Oratab
Creating Password file(s)
Configuring Listeners
Configuring related CRS resources
Starting Cluster Database
..

RAC12c Theory Guide v8.0

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

RAC12c Theory Guide v8.0

Uploaded by

Copyright:

Available Formats

Oracle 12c: RAC

Copyright © 2018, Uclid IT School. All rights reserved.

V8.0 www.ucliditschool.com Page 2 of 132

V8.0 www.ucliditschool.com Page 3 of 132

Introduction to Oracle RAC

The following figure shows a typical cluster.

V8.0 www.ucliditschool.com Page 4 of 132

Typically, clusterware includes capabilities that:

 Allow the cluster to be managed as a single entity (not including OS requirements), if

V8.0 www.ucliditschool.com Page 5 of 132

The following block diagram shows the RAC model.

V8.0 www.ucliditschool.com Page 6 of 132

History of Oracle RAC

V8.0 www.ucliditschool.com Page 7 of 132

Components of Oracle RAC

Shared Disk system

V8.0 www.ucliditschool.com Page 8 of 132

Oracle Kernel Components

V8.0 www.ucliditschool.com Page 9 of 132

Tool for Managing RAC

V8.0 www.ucliditschool.com Page 10 of 132

 Oracle Automatic Storage Management Configuration Assistant (ASMCA): ASMCA is a

 NETCA: Configures the network for your Oracle RAC environment.

V8.0 www.ucliditschool.com Page 11 of 132

Oracle Clusterware Concepts

The services provided by Oracle Clusterware include:

About OCR & Voting Disks

V8.0 www.ucliditschool.com Page 12 of 132

Storing Voting Disks on ASM

V8.0 www.ucliditschool.com Page 13 of 132

Oracle Cluster Registry (OCR)

What is stored in OCR?

 cluster database instance to node mapping

Who updates OCR?

V8.0 www.ucliditschool.com Page 14 of 132

 Enterprise Manager (EM)

 Database Configuration assistant (DBCA)

 Database Upgrade Assistant (DBUA)

 Network Configuration Assistant (NETCA)

How is the info stored in OCR?

V8.0 www.ucliditschool.com Page 15 of 132

 Oracle Cluster Registry file (OCR)

Oracle Local Registry

The OLR stores:

By default, OLR is located at Grid_home/cdata/host_name.olr on each node. The

V8.0 www.ucliditschool.com Page 16 of 132

How is GPnP profile used by clusterware?

The order of searching the ASM SPfile is:

About Cluster Interconnect

V8.0 www.ucliditschool.com Page 17 of 132

SQL> select * from GV$CLUSTER_INTERCONNECTS;

About Virtual IP (VIP)

Single Client Access Name (SCAN)

V8.0 www.ucliditschool.com Page 18 of 132

SCAN and Local Listeners

V8.0 www.ucliditschool.com Page 19 of 132

V8.0 www.ucliditschool.com Page 20 of 132

About Oracle ASM

V8.0 www.ucliditschool.com Page 21 of 132

About Oracle ASM Instances

V8.0 www.ucliditschool.com Page 22 of 132

V8.0 www.ucliditschool.com Page 23 of 132

About Oracle ASM Disk Groups

About Mirroring and Failure Groups