RAC : 11gR2 Clusterware Startup Sequence

Here is the brief explanation that how the clusterware brings up step by step .

In Unix and Linux operating systems, there is a master daemon process named INIT that functions to start up additional system background processes. The INIT process first spawns the init.ohasd process, which in turn starts up the Oracle High Availability Services Daemon (OHASD). In turn, the OHASD daemon then spawns additional Clusterware processes at each startup level as shown next:

1. When a node of an Oracle Clusterware cluster start/restarts, OHASD is started by platform-specific means. OHASD is the root for bringing up Oracle Clusterware. OHASD has access to the OLR (Oracle Local Registry) stored on the local file system. OLR provides needed data to complete OHASD initialization.

2. OHASD brings up GPNPD and CSSD. CSSD has access to the GPNP Profile stored on the local file system. This profile contains the following vital bootstrap data;

        a. ASM Diskgroup Discovery String

        b. ASM SPFILE location (Diskgroup name)

        c. Name of the ASM Diskgroup containing the Voting Files

3. The Voting Files locations on ASM Disks are accessed by CSSD with well-known pointers in the ASM Disk headers and CSSD is able to complete initialization and start or join an existing cluster.

4. OHASD starts an ASM instance and ASM can now operate with CSSD initialized and operating. The ASM instance uses special code to locate the contents of the ASM SPFILE, assuming it is stored in a Diskgroup.

5. With an ASM instance operating and its Diskgroups mounted, access to Clusterware’s OCR is available to CRSD.

6. OHASD starts CRSD with access to the OCR in an ASM Diskgroup.

7. Clusterware completes initialization and brings up other services under its control.

When Clusterware starts three files are involved.

1. OLR – Is the first file to be read and opened. This file is local and this file contains information regarding where the voting disk is stored and information to startup the ASM. (e.g ASM DiscoveryString)

2. VOTING DISK – This is the second file to be opened and read, this is dependent on only OLR being accessible.

ASM starts after CSSD or ASM does not start if CSSD is offline (i.e voting file missing)

How are Voting Disks stored in ASM?

Voting disks are placed directly on ASMDISK. Oracle Clusterware will store the votedisk on the disk within a disk group that holds the Voting Files.
 
Oracle Clusterware does not rely on ASM to access the Voting Files, which means Oracle Clusterware does not need of Diskgroup to read and write on ASMDISK. It is possible to check for existence of voting files on a ASMDISK using the V$ASM_DISK column VOTING_FILE.
 
So, voting files not depend of Diskgroup to be accessed, does not mean that the diskgroup is not needed, diskgroup and voting file are linked by their settings.

3. OCR – Finally the ASM Instance starts and mount all Diskgroups, then Clusterware Deamon (CRSD) opens and reads the OCR which is stored on Diskgroup.

So, if ASM already started, ASM does not depend on OCR or OLR to be online. ASM depends on CSSD (Votedisk) to be online.

There is an exclusive mode to start ASM without CSSD (but it’s to restore OCR or VOTE purposes)
As per Oracle Documentation

The full description, the really unreadable diagram and/or any updates to this you can find it in MOS Document 1053147.1

Level 1: OHASD Spawns:

• cssdagent – Agent responsible for spawning CSSD.
• orarootagent – Agent responsible for managing all root owned ohasd resources.
• oraagent – Agent responsible for managing all oracle owned ohasd resources.
• cssdmonitor – Monitors CSSD and node health (along wth the cssdagent).

Level 2: OHASD rootagent spawns:

• CRSD – Primary daemon responsible for managing cluster resources.
• CTSSD – Cluster Time Synchronization Services Daemon
• Diskmon
• ACFS (ASM Cluster File System) Drivers

Level 2: OHASD oraagent spawns:

• MDNSD – Used for DNS lookup
• GIPCD – Used for inter-process and inter-node communication
• GPNPD – Grid Plug & Play Profile Daemon
• EVMD – Event Monitor Daemon
• ASM – Resource for monitoring ASM instances

Level 3: CRSD spawns:

• orarootagent – Agent responsible for managing all root owned crsd resources.
• oraagent – Agent responsible for managing all oracle owned crsd resources.

Level 4: CRSD rootagent spawns:

• Network resource – To monitor the public network
• SCAN VIP(s) – Single Client Access Name Virtual IPs
• Node VIPs – One per node
• ACFS Registery – For mounting ASM Cluster File System
• GNS VIP (optional) – VIP for GNS

Level 4: CRSD oraagent spawns:

• ASM Resouce – ASM Instance(s) resource
• Diskgroup – Used for managing/monitoring ASM diskgroups.
• DB Resource – Used for monitoring and managing the DB and instances
• SCAN Listener – Listener for single client access name, listening on SCAN VIP
• Listener – Node listener listening on the Node VIP
• Services – Used for monitoring and managing services
• ONS – Oracle Notification Service
• eONS – Enhanced Oracle Notification Service
• GSD – For 9i backward compatibility
• GNS (optional) – Grid Naming Service – Performs name resolution

The New Features of Oracle Active Data Guard 12c

The New Features of Oracle Active Data Guard 12c

Here are the major ones:

1. Far Sync. This feature allows you to failover to a standby database even if it is remotely located in another site thousands of miles away. You do this without any data loss and without negatively affecting the performance of the primary database. Far Sync also keeps things simple and your costs down.

How does Far Sync works ?

The principle is fairly simple, the Far Sync instance receive data synchronously from the primary database and then forward it asynchronously to up de 29 remote destinations.

The far sync database is not a standard database, it only contains a specific controlfile, a spfile and standby redologs. This database must be placed near the primary database to guarantee an optimal network latency during synchronous replication. But be careful, don’t place this database on the same geographical place than the primary, because if your primary database experiences a geographical disaster, your Far Sync will be impacted too, and some data could be lost.

In case of an outage on the primary database, the standard failover procedure applies and the far sync instance guarantee that no data is lost during the failover. “Far Sync Standby” cannot be used as the target. Note that it is possible to create a “Far Sync Standby” on both sides in case the database roles change.

2. Fast Sync. Data Guard Fast Sync (SYNC NOAFFIRM) with Oracle Database 12c provides an easy way of improving performance in synchronous zero data loss configurations. It enables all sessions waiting for a remote RFS write to proceed as soon as the write is submitted, not when the write completes. This offers the best performance since it reduces the overall SYNC remote write wait event

Data Guard maximum availability supports the use of the noaffirm  redo transport parameter. A standby database returns receipt acknowledgment to its primary database as soon as redo is received in memory. The standby database does not wait for the Remote File Server (RFS) to write to a standby redo log file.
 
This noaffirm parameter provides increased primary database performance in Data Guard and SYNC for redo transport. FastSync isolates the primary database in a maximum availability configuration from any performance impact due to slow I/O at the standby database.

This feature gives you the easiest way to improve the database performance when you have synchronous no data loss setups. When a standby database gets a redo in memory, it will check with the primary database, even without getting a disk I/O to a redo log file in the standby. Fast Sync then effectively reduces the total round trip time between the standby and primary, resulting in better primary database performance. It also prevents fluctuations and outliers in standby I/O performance from impacting application response time. Fast Sync can make it practical to increase the distance between the primary and any Data Guard synchronous destination to provide greater geographic protection.

You should know, however, that there is a minute chance that data loss would happen when you use Fast Sync. But Oracle assures you that these instances would be so rare and it has to happen when both databases fail milliseconds within each other.

3. Real-time Cascade. Real-time Cascade allows your standby database to immediately transmit a redo to another standby database even when it has yet to be archived in the redo log file. Before 12c, redo is only cascade after the standby redo log file has finished being archived locally.

4. SYSDG. SYSDG is an administration privilege wherein the user can only perform basic administration actions and tasks. As such, SYSDG privileges are limited to STARTUP, SHUTDOWN, ALTER DATABASE, ALTER SESSION, ALTER SYSTEM, CREATE RESTORE POINT, CREATE SESSION, DROP RESTORE POINT, FLASHBACK DATABASE, SELECT ANY DICTIONARY, DELETE, and EXECUTE.

Other minor features that were introduced in Oracle Active Data Guard 12c:

Data Manipulation Language operations are now permitted on global temporary tables on the standby.
 

You no longer have to add the USING CURRENT LOGFILE clause when you initiate real-time apply. You can now use sequences in a Data Guard environment.

You no longer need to shut down all other primary database instance if you need to switchover from an Oracle Real Application Clusters primary database to a physical standby. Application continuity is now supported as well.

Beside these new features the Data Guard Broker commands have been extended. One of them are in my point of view very useful: The “validate database” command checks if the database is ready for a role transition.

 

PHYSICAL STANDBY DATABASE RELATED PROCESSES

On the Primary Database:

LGWR : The log writer process on the primary database writes entries from the redo log buffer to the online redo log file. When the current online redo log file is full, it triggers the archiver process to start the archiving activity. In some cases, the log writer process writes redo entries to the online redo log file of the primary database and the standby redo log file of the standby database. Usually, in this kind of arrangement the LGWR works as the log transport agent that is setup to achieve high data protection modes.

LNS : The LogWriter Network Service (LNS) reads the redo being flushed from the redo buffers by the LGWR and sends the redo over network to the standby database. The
main purpose of the LNS process is to free up the LGWR process from performing the redo transport role.

ARCH  : The archiver processes archives the ORL files to archive log files. Up to 30 ARCH processes can exist, and these ARCH processes are also used to fulfill gap resolution requests. Note that one ARCH process has a special role in that it is dedicated to local redo log archiving only and never communicates with a standby database.

Fetch Archive Log (FAL) The FAL process has two components: FAL Client and FAL Server. Both processes are used for archive gap resolution. If the Managed Recovery Process (MRP) on the standby database site detects an archive gap sequence, it initiates a fetch request to the FAL client on the standby site. This action, in turn, requests the FAL server process on the primary database to re-transmit the archived log files to resolve the gap sequence.

On the Standby Database:

RFS : The main objective of the Remote File Server process is to perform a network receive of redo transmitted from the primary site and then writes the network buffer (redo data) to the standby redo log (SRL) files.

ARCH : The archive processes on the standby site perform the same functions performed on the primary site, except that on the standby site, an ARCH process generates archived log files from the SRLs.

MRP : The managed recovery process coordinates media recovery management. Remember that a physical standby is in perpetual recovery mode.

Logical Standby Process (LSP) - The LSP applies the redo records from archived redo logs to the logical standby database. The Oracle database log miner engine is used by the logical standby process for the SQL apply operations. Using the log miner engine, the LSP process recreates the SQL statements from redo logs that have been executed on the primary database. These statements are then applied to the standby database to keep it current with the primary database.

---

Basically we can categorize physical standby database  into three major components:

1) Data Guard Redo Transport Services

– To transfer the redo that is generated by the primary database to the standby database.

2) Data Guard Apply Services  

– To receive and apply the redo sent by Redo Transport Services to the standby database.

3) Data Guard Role Management Services

– To assist in the database role changes in switchover and failover scenarios.

This service works in the background and takes care of switchover/failover scenarios

Oracle 12c RAC New Features

Oracle 12c RAC New Features

Oracle ASM Flex

This will help reduce the overhead on the database server by running an ASM instance remotely. Instances can use the remote ASM node for any planned or unplanned downtime as well. All the metadata requests can be covered by non-local ASM instances. In addition to that this feature removes the risk of single point of failure, of having only one ASM instance for storage management. Now if a local ASM instance fails, the database instance can now connect directly to any of the other surviving remote ASM instances in the cluster.

ASM Disk Scrubbing

This monitors all the disks in the ASM disk groups and will discover logical corruptions. Normally these corruptions were discovered when an RMAN backup job ran. Disk Scrubbing will try to automatically recover those logical corruptions without the DBA even knowing!

Shared Password file in ASM

A single password file can now be stored in the ASM diskgroup and can be shared by all nodes. No need to have individual copies for each instance.

ASM Multiple Diskgroup Rebalance and Disk Resync Enhancements

Resync Power limit – Allows multiple diskgroups to be resynced concurrently.
Disk Resync Checkpoint – Faster recovery from instance failures.

Grid Infrastructure Rolling Migration support for one-off’s

When applying a one-off patch to the ASM instance, the databases that it is serving can be pointed to use a different ASM instance.

Oracle Clusterware Flex Cluster

This feature may appear similar to the ASM Flex feature but actually it is not. This is another type of cluster that has been introduced in Oracle 12c. We need to understand the two main components Hub Nodes and Leaf Nodes.

Hub Nodes are nodes that you currently see in the 11g RAC architecture. Every node is a full-fledged node with the required Clusterware software, share storage with a voting disk, interconnect network etc. components. On the other hand the leaf nodes are lightweight nodes with no shared storage and minimal Clusterware software. A leaf node will be connected to a Hub Node.

Grid Home Server

This new feature will allow you to have a single Golden Oracle Home on one of nodes and all other nodes to be a client of that Golden Home. You will only have to patch the single golden Oracle Home and rest will take it from there.

Application Continuity

This helps minimize the application downtime caused by temporary failures in the infrastructure and/or the database servers. This piece sits between the application and the database working at the JDBC driver layer. If any failure occurs and is recoverable, it will be recovered automatically while being transparent to the application. The application will only observe a minor latency delay in the transactions and the failure will automatically be recovered transparently. Additionally Oracle guarantees the successful completion of the in-flight transactions, eliminating the chance of duplicate transactions. The purpose of leaf nodes is to include application servers and other servers with additional software running on the Oracle 12c Clusterware infrastructure. These leaf nodes will not have any database instances running on them. If a leaf node goes down then there will be no impact on Hub nodes either. This allows the flexibility to run leaf nodes on Virtual Machines while Hub nodes can run on the actual physical machines.

IPv6 Support

IPv6 was supported in Oracle database 11gr2 but was only available for a standalone database. In 12c Database, clients can also now connect to database in RAC environments, using the IPv6 protocol. The interconnect however still only supports IPv4. This feature helps the customers meet the PCI, SOX and other Security Compliance standards.

Multiple SCAN’s Per Subnet

Now you can configure multiple SCAN’s per subnet mask, per cluster. This obviously is made available to provide redundancy.

GHCTL

The new ghctl utility will improve patching process.

UI auto runs root.sh

Oracle UI will execute the root.sh script on all nodes. You don’t have to do this on all nodes manually.

Deprecated Features

While it is good to be familiar with the new features it equally if not more to be aware of what features of RAC are being deprecated so plans can be made to move away from those Deprecated feature and alternatives chosen.

Oracle Restart

Oracle Restart feature which was provided as a part of the Oracle Grid Infrastructure has been deprecated. It will be de-supported in future versions.

RAW/Block Storage Devices

Oracle Database 12c and Oracle Clusterware 12c, no longer support raw storage devices. The files must be moved to Oracle ASM before upgrading to Oracle Clusterware 12c.

Oracle Database 12c – RMAN New Features

Oracle Database 12c – RMAN New Features:

Oracle Database 12c has new enhancements and additions in Recovery Manager (RMAN).  The Recovery Manager continues to enhance and extend the reliability, efficiency, and availability of Oracle Database Backup and Recovery. In this article series, I will be explaining the new features and how it will help Oracle community. In this article I will cover: SQL Interface Improvements SYSBACKUP Privilege

Oracle Database 12c has new enhancements and additions in Recovery Manager (RMAN).  The Recovery Manager continues to enhance and extend the reliability, efficiency, and availability of Oracle Database Backup and Recovery. In this article series, I will be explaining the new features and how it will help Oracle community.

Part I will cover:
• SQL Interface Improvements
• SYSBACKUP Privilege
• Support for multitenant container and pluggable databases
• DUPLICATE enhancements

Part II will cover:
• Multisection Backup Improvements
• Restoring and Recovering Files Over Network
• Storage Snapshot Optimization
• Active Database Duplication Improvements

Part III will cover:
• Cross-Platform Backup and Restore Improvements
• Recovering Tables and Table Partitions using RMAN Backups
• Unified auditing and RMAN

Oracle Database 12c New Features

During this Oracle Database 12c new features article series, I shall be extensively exploring some of the very important new additions and enhancements introduced in the area of Database Administration, RMAN, High Availability and Performance Tuning.

During this Oracle Database 12c new features article series, I shall be extensively exploring some of the very important new additions and enhancements introduced in the area of Database Administration, RMAN, High Availability and Performance Tuning.

Part I covers:
1. Online migration of an active data file
2. Online table partition or sub-partition migration
3. Invisible column
4. Multiple indexes on the same column
5. DDL logging
6. Temporary undo in- and- outs
7. New backup user privilege
8. How to execute SQL statement in RMAN
9. Table level recovery in RMAN
10. Restricting PGA size

Part 2 covers:
1. Table partition maintenance enhancements
2. Database upgrade improvements
3. Restore/Recover data file over the network
4. Data Pump enhancements
5. Real-time ADDM
6. Concurrent statistics gathering

Part 3 covers:
1. Additions/Enhancements in ASM
2. Additions/Enhancements in Grid Infrastructure
3. Additions/Enhancements in Real Application Cluster (database)

Part 4 covers:
1. How to truncate a master table while child tables contain data
2. Limiting ROWS for Top-N query results
3. Miscellaneous SQL*Plus enhancements
4. Session level sequences
5. WITH clause improvements
6. Extended data types

Oracle Mutex

Mutexes are objects that exist within the operating system to provide access to shared memory structures. They are similar to latches, which will be covered in following chapters, as they are serialized mechanisms used to control access to shared data structures within the Oracle SGA.
Serialization provides benefits via mutexes in that they are required to avoid having database objects being read during modification and to provide consistency as part of the relational database ACID (Atomicity, Consistency, Isolation and Durability) model.

Mutexes can be used and defined in various ways. Each data structure within Oracle which is under the protection of a mutex can also have its own mutex such as a parent cursor may have its own separate mutex as well as each child cursor can also have its own mutex. Structures within Oracle can be protected by multiple mutexes so that each mutex will protect a different area of the database structure. While latches and mutexes are similar regarding both being serialization mechanisms and providing data protection, mutexes differ from latches in the following ways.

Mutexes are smaller and operate faster than latches because they contain fewer instructions than those in a latch get operation. Secondly, mutexes take up fewer memory resources and space than latches. Mutexes also provide less chance of contention within the database than latches do which means that mutexes provide greater protection of data structures and flexibility than latches.
Another key feature of mutexes is that they can be referenced across many sessions concurrently by using shared mode. Mutexes also function in a dual role as both a serialization item similar to a latch and also as pin operator by preventing objects from aging out of the Oracle memory buffers. Since both latches and mutexes are independent mechanisms, a process within Oracle can hold both a latch and mutex at the same time.

Oracle 10g release 2 and beyond replaced some latch mechanisms with the mutex approach, claiming that they are faster and more efficient than traditional locking mechanisms.  To improve cursor execution speed and hard pare time within the library cache, mutexes replace library cache latches and library cache pins. Oracle claims that mutexes are faster and use less CPU, which is important for CPU-bound database where large data buffers remove I/O as a primary source of contention. 
Oracle also claims that a mutex allows for improved concurrency over the older latch mechanism because the code path is shorter.