Best Practices Virtualizing Oracle Databased On Nutanix
Best Practices Virtualizing Oracle Databased On Nutanix
Copyright
Copyright 2019 Nutanix, Inc.
Nutanix, Inc.
1740 Technology Drive, Suite 150
San Jose, CA 95110
All rights reserved. This product is protected by U.S. and international copyright and intellectual
property laws.
Nutanix is a trademark of Nutanix, Inc. in the United States and/or other jurisdictions. All other
marks and names mentioned herein may be trademarks of their respective companies.
Copyright | 2
Oracle on vSphere
Contents
1. Executive Summary.................................................................................5
2. Introduction.............................................................................................. 6
2.1. Audience.........................................................................................................................6
2.2. Purpose.......................................................................................................................... 6
3
Oracle on vSphere
8. Conclusion..............................................................................................27
8.1. References................................................................................................................... 27
8.2. About Nutanix...............................................................................................................28
List of Figures................................................................................................................ 29
List of Tables.................................................................................................................. 30
4
Oracle on vSphere
1. Executive Summary
The Nutanix Enterprise Cloud provides a complete datacenter infrastructure solution for Oracle
databases, eliminating the complexities and inefficiencies of traditional multitier datacenter
environments. Whether you are virtualizing critical or tier-1 Oracle databases or running them on
bare metal, Nutanix solutions bring the predictable performance, scalability, and cost benefits of
web-scale architecture to your transactional and analytical Oracle database environments.
The hypervisor-agnostic Nutanix solution (VMware vSphere, Microsoft Hyper-V, or Nutanix AHV)
delivers powerful self-healing, data protection, and disaster recovery capabilities to keep your
databases and applications running and your critical data well protected. The Nutanix Enterprise
Cloud provides near-instantaneous local and remote backups using snapshots for offloading
RMAN (Recovery Manager) backups to tape and disk or WORM (write once, read many) for
offsite backup. You can also use a Nutanix clone snapshot to easily refresh production-based test
and development Oracle instances.
This best practices guide recommends Nutanix cluster settings for running Oracle databases, as
well as ESXi settings for VMs, networking, volumes (VMDKs), Linux OS, and Oracle ASM and
database.
Nutanix is an Oracle Platinum Partner.
1. Executive Summary | 5
Oracle on vSphere
2. Introduction
Unless otherwise stated, the solution described in this document is valid on all supported AOS
releases.
2.1. Audience
This best practices guide is part of the Nutanix Solutions Library for Oracle. We wrote it for IT
administrators and solutions architects responsible for designing, managing, and supporting
Oracle Database deployments on Nutanix infrastructures. Readers should be familiar with
VMware vSphere, Oracle Database, the Linux operating system, and the Nutanix Enterprise
Cloud.
Tip: For more information on Linux system administration or Oracle installation and
setup, please refer to documentation from the appropriate software vendor.
2.2. Purpose
This document provides design, configuration, and optimization guidelines for a single instance of
Oracle Database or RAC running on Nutanix with VMware ESXi. In this document, we cover the
following topics:
• Overview of the Nutanix solution.
• Nutanix cluster best practices.
• VMware ESXi best practices.
• Linux operating system best practices.
• Oracle Database best practices.
Upon completing this document, the reader should be able to design, architect, and deploy a
high-performing and highly available Oracle Database solution on the Nutanix platform.
For information specific to Oracle on Nutanix AHV, please refer to the Oracle on AHV best
practices guide.
2. Introduction | 6
Oracle on vSphere
Version
Published Notes
Number
1.0 September 2014 Original publication.
2.0 February 2017 Updated with current Nutanix platform information.
Updated provisioning recommendations and noted
2.1 May 2017 VMware ESXi 6.0.0 support for adding disks online in
Oracle RAC environments.
Updated platform overview and Nutanix Platform
2.2 January 2018
Guidance section.
3.0 November 2018 Updated product information and Best Practices section.
3.1 December 2018 Updated Best Practices section.
Added Nutanix Era section and updated the OLTP
3.2 March 2019
Scenario Detail and OLAP Scenario Detail tables.
4.0 June 2019 Major updates throughout.
2. Introduction | 7
Oracle on vSphere
The following table shows the Nutanix storage pool and container configuration. Always ensure
that your environment has ample storage capacity, so it can tolerate the loss of a node during
failure or maintenance.
Note: Do not enable deduplication for Oracle databases. Only enable erasure
coding if space is a constraint.
4.3. Networking
• Use low-latency 10 GbE, 25 GbE, or 40 GbE switches.
• Establish redundant 10 GbE, 25 GbE, or 40 GbE uplinks from each Nutanix node.
• Ensure adequate throughput between Nutanix nodes and Oracle database VMs.
• Check for any pause frames that could impact replication and VM communication.
• Use dedicated NICs (at least 10 GbE) for Oracle RAC Cache Fusion (heartbeat) networks on
each Nutanix node.
• Do not enable jumbo frames unless absolutely necessary.
• If you are using Nutanix Volumes for Oracle data or log disks:
⁃ Set up an IP address for iSCSI data services on the Nutanix cluster.
⁃ Set up a tagged vLAN to isolate Nutanix Volumes iSCSI traffic.
⁃ Use iSCSI CHAP if required.
Note: If you are upgrading AOS, be sure to check the release notes for your
target AOS version. Consider engaging Nutanix Support if you encounter network
connectivity issues when hosting or migrating Oracle RAC VMs.
Nutanix recommends a leaf-spine network architecture, which is designed for linear scaling.
A leaf-spine architecture consists of two network tiers: an L2 leaf and an L3 spine based on
nonblocking switches. This architecture maintains consistent performance without any throughput
reduction due to a static maximum of three hops from any node in the network.
The figure below shows the design of a scale-out leaf-spine network architecture, which provides
20 GbE active throughput from each node to its L2 leaf and scalable 80 GbE active throughput
from each leaf to its spine switch, allowing you to grow from one Nutanix block to thousands
without any impact to available bandwidth.
Parameter Configuration
Network adapter VMXNET3
Minimum of 3 PVSCSI (OS + DB + redo), 4 for larger or high-
Storage adapter
performance databases
OS and app disks Thin provisioned, disk mode = dependent
Database (ASM) disks for
Thin provisioned, disk mode = independent persistent
standalone*
* Using in-guest iSCSI requires Nutanix Volumes. For more information, refer to the Nutanix
Volumes best practices guide.
• For small Oracle Database VMs, keep vCPUs <= to the number of cores per each physical
NUMA node.
• Keep vCPU numbers simply divisible by NUMA node sizes for easy scheduling.
• Leave hyperthreading sharing at the default policy (any).
• For tier-1 workloads, lock Oracle Database VM memory at a minimum reserve sufficient to
cover both the system global area (SGA) HugePages and the Program Global Area (PGA).
• Use the following calculation to size Oracle Database VM memory:
⁃ VM Memory = Oracle SGA + Oracle PGA + OS memory + VM overhead
• Use Paravirtual SCSI controllers (PVSCSI) and VMXNET3 NICs.
• If you are using Nutanix Volumes for Oracle data or log disks:
• Use separate volume groups for data and logs.
• Use four to eight vDisks for data per volume group, and two to four vDisks per volume group
for logs.
• If you are using Oracle ASM for volume management, map a disk group to a Nutanix volume
group.
• Do not use resource pools unless absolutely necessary. If using resource pools, ensure that
shares are sized correctly.
• Use antiaffinity rules to prevent multiple Oracle RAC nodes in the same RAC cluster from
running on the same Nutanix node.
limits.conf Settings
Following are the shell settings for the “grid” and “oracle” users. You can use the settings as
shown or adjust them to fit your environment. For more information about these settings, refer to
Oracle Database 12cR1 installation documentation and the Oracle kernel parameters page cited
previously.
Note: The memlock setting is reserved for enabling HugePages. Set it to a number
slightly lower than the actual memory. Following is an example for a VM with 32
GB of physical memory. Please refer to the later Configure HugePages for Oracle
Databases section for information on enabling HugePages.
OLTP DSS
SGA_TARGET = (total_mem * 0.8) * 0.8, SGA_TARGET = (total_mem * 0.8) * 0.5,
where total_mem is the total amount of where total_mem is the total amount of
physical memory available on the system physical memory available on the system
OLTP DSS
PGA_AGGREGATE_TARGET = (total_mem * PGA_AGGREGATE_TARGET = (total_mem *
0.8) * 0.2, where total_mem is the total amount 0.8) * 0.5, where total_mem is the total amount
of physical memory available on the system of physical memory available on the system
Note: Use these recommendations as guidelines only. You must monitor the
memory usage and adjust accordingly based on your workload.
• If you are running Oracle 11.2.0.2 or later, you can set USE_LARGE_PAGES to ONLY, which
prevents the database from starting if it is not backed up by HugePages.
• Disable the Transparent HugePages option if it is not already disabled. Transparent
HugePages differs from HugePages in that it dynamically allocates memory via a
khugepaged kernel thread, rather than at boot time, as HugePages does. Transparent
HugePages does not work well with Oracle databases, however; it can cause performance
problems and node reboots in RAC installations.
• Set transparent_hugepage=never.
• Reboot.
block size for the Oracle datafiles or online redo logs. This process can become cumbersome to
manage if there are multiple databases.
The Nutanix Enterprise Cloud eliminates all such problems associated with RAID and block
size. Once you create the required number of volumes for your Oracle database, you’re ready to
deploy.
The table below shows the minimum recommended number of VMDKs for Oracle. The
appropriate number of vDisks depends on the expected size of the Oracle database and the
number of Nutanix nodes in your cluster. We based the following example on a four-node Nutanix
cluster.
Number
Purpose Comment
of VMDKs
1 Boot disk Can be used with LVM or Standard partition
Database datafiles / control Can be used with Oracle ASM or file system
8
files / redo log files with LVM
Can be used with Oracle ASM or file system
4 Database archive log files
with LVM
Database RMAN backup Can be used with Oracle ASM or file system
4
files with LVM
There are two ways to create volumes on Nutanix for your Oracle Database VM with ESXi:
1. VMDKs.
2. Nutanix volume groups for in-guest iSCSI.
6.5. VMDKs
The VMDKs option simplifies VM administration, as it does not require in-guest iSCSI setup with
volume groups. When creating a VM, simply add VMDKs and assign them to the proper PVSCSI
based on the information provided previously in the ESXi Recommended Settings section. This is
the best option if you have a database that does not have intensive I/O requirements.
Note: If the volume group you want to share is already attached to a VM, you must
detach it, change the shared parameter to true, and reattach it.
For more information and how-to guidance on the iSCSI recommended settings, refer to the
Linux on Nutanix AHV best practices guide.
Note: If you want to use Nutanix Volumes for your Oracle Database, please refer to
the Nutanix Volumes best practices guide. As an additional resource, the Linux on
Nutanix AHV best practices guide provides detailed instructions and covers setting
up iSCSI ifaces.
Either put this command in the /etc/rc.local file, so that it runs the next time the server reboots,
or use UDEV rules. For UDEV rules, you can create a file with the below content under the /etc/
udev/rules.d directory.
ACTION=="add", SUBSYSTEMS=="scsi", ATTRS{vendor}=="NUTANIX ", ATTRS{model}=="VDISK", RUN+="/
bin/sh -c 'echo 1024 >/sys$DEVPATH/queue/max_sectors_kb'"
ACTION=="add", SUBSYSTEMS=="scsi", ATTRS{vendor}=="NUTANIX ", ATTRS{model}=="VDISK", RUN+="/
bin/sh -c 'echo 60 >/sys$DEVPATH/device/timeout'"
• After Oracle installation, change the disk io_scheduler to noop. The default is deadline.
[root@oracle1 ~]# lsscsi | grep NUTANIX | awk '{print $NF}' | awk -F"/" '{print $NF}' | grep
-v "-" | while read LUN
do
echo noop > /sys/block/${LUN}/queue/scheduler
done
If you put this command in the /etc/rc.local file, it runs the next time the server reboots.
Alternatively, you can put it in the grub configuration file. Use the following steps for GRUB2
configuration.
• Edit the /etc/default/grub file, add “elevator=noop” to the GRUB_CMDLINE_LINUX line, and
save the file.
• Disable transparent_hugepage.
• If the Linux kernel supports the blk_mq (block multiqueue) option (kernel versions 4.12
and later), add the parameter “scsi_mod.use_blk_mq=1” to enable it and remove the
elevator=noop option.
Note: Linux kernel versions 4.12 and later offer new scheduler options within
blk_mq: mq-deadline, BFQ, and Kyber. Based on Nutanix performance testing, we do
not currently recommend using any of these schedulers. Select the “none” option.
• Create the file system. In this example, we are creating an ext4 file system.
[root@localhost ~]# mkfs.ext4 /dev/vgdata/vol1
• Mount the ext4 file system. If you are creating an XFS file system, please refer to the Mount
Options table below.
[root@localhost ~]# mount /dev/vgdata/vol1 /u01/oradata -o
noatime,nodiratime,discard,barrier=0
Note: You can use xfs_admin for xfs or e2label for ext4 to label your file systems
with a friendly name. Use the LABEL= option in the /etc/fstab file for ease of
management.
The table below lists the recommended mount options for ext4 and XFS file systems.
Settings Values
ASM Allocation Unit (AU) size 1 MB
ASM OCR disk group redundancy Normal/High
ASM DATA disk group redundancy External
ASM FRA disk group redundancy External
Nutanix supports ASM disks using UDEV devices, ASMLib devices, or ASMFD (ASM Filter
Driver) devices. ASMLib is available with Oracle 11g and 12c; ASMFD is available with Oracle
12cR1 and later versions. The ASMFD feature rejects write I/O requests that are not issued by
Oracle software. This filter ensures that users with administrative privileges cannot inadvertently
overwrite Oracle ASM disks, thus preventing corruption in those disks and files within the disk
groups.
Nutanix performance testing shows that ASMFD and UDEV yield similar results, so you can
deploy whichever type of device you are most comfortable with on your Nutanix cluster. Refer to
Oracle documentation for more information on how to deploy Oracle ASM.
change the CSS misscount timeout to 60 seconds to avoid node eviction in case of network or I/
O contention. For more information, please refer to Oracle MOS ID 294430.1 and KB 6843.
8. Conclusion
The Nutanix platform offers the capability to run almost any workload. You have the ability to run
both ORADB and other VM workloads simultaneously on the same platform. The database’s
CPU and storage requirements drive density for Oracle Database deployments. Test validation
has shown that increasing the number of ORADB VMs on the Nutanix platform is preferable
to scaling the number of Oracle instances, to take full advantage of Nutanix performance and
capabilities.
From an I/O standpoint, the Nutanix platform handled the throughput and transaction
requirements of a demanding Oracle database given DSF-localized I/O, server-attached flash,
and intelligent tiering. When you need to virtualize more Oracle databases, you can simply scale
out the Nutanix platform and add more database VMs to gain capacity and performance. During
the load tests, we also subjected the Oracle RAC environment to VMware vMotion migrations to
demonstrate the robustness of the platform and to prove that convergence of all networking, even
when pushed to the saturation point, does not disrupt client connections.
We determined pod sizing after carefully considering performance as well as accounting for
the additional resources needed for N+1 failover capabilities. The Oracle on Nutanix solution
provides a single high-density platform for Oracle, VM hosting, and application delivery. This
modular, pod-based approach enables such deployments to easily be scaled. You can start small
and pay as you grow for any workload.
The appendix includes detailed validation and benchmarking for Oracle Database on Nutanix. It
also includes the scripts and testing configuration required to reproduce the successful results.
For further discussion of the best practices for Oracle on Nutanix, please contact us on our twitter
@Nutanix, or visit with us on the Nutanix NEXT Community.
8.1. References
1. VMware vSphere Networking best practices guide
2. VMware vSphere Storage best practices guide
3. Nutanix Volumes best practices guide
4. Oracle on AHV best practices guide
5. Oracle Contracts
6. Understanding Oracle Certification, Support, and Licensing on VMware Environments
7. VMware Expanded Oracle Support Policy
8. TSANet
8. Conclusion | 27
Oracle on vSphere
8. Conclusion | 28
Oracle on vSphere
List of Figures
Figure 1: Nutanix Enterprise Cloud................................................................................... 8
29
Oracle on vSphere
List of Tables
Table 1: Document Version History................................................................................... 7
30