Friday, June 23, 2017

12cR2 RAC Upgrade from 12cR1 RAC


12.2.0.1 Upgrade from 12.1.0.2 (Apr-2017 PSU)


 GRID INFRASTRUCTURE UPGRADE

Starting with this Oracle Grid Infrastructure 12c Release 2 (12.2), the Oracle Grid Infrastructure software is available as an image file for download and installation. In order to start the installation or upgrade we must extract the Grid Infrastructure image software into the directory where you want your Grid home to be located.  Here in our case we used " /u01/app/12.2.0.1/GRID/ ", and then run the gridSetup.sh script to start the Oracle Grid Infrastructure installation.

Point need to remember before installation.

  • Starting from 12cR2 it is now mandatory that we have 33+ GB of free disk space in the ASM disk groups, without we can't process upgrade or any new installation.
  • I have applied Apr-2017 PSU on my 12.1.0.2 RAC Cluster, so this time I am not facing any kind of issue for rootupgrade.sh on every nodes.
  • In my other environment where we have not applied the APR-2017 PSU, we faced failure  on rootupgrade.sh on upgrade and  root.sh in fresh installation where each server bounce was necessary then after rerun the root.sh/rootupgrade.sh.  
Pre-validation before starting upgrade:

[oracle@rac131 ~]$ . oraenv
ORACLE_SID = [+ASM1] ?
The Oracle base has been set to /u01/app/oracle
  
[oracle@rac131 12.2.0.1]$ crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [12.1.0.2.0]

[oracle@rac131 12.2.0.1]$  crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [12.1.0.2.0]

[oracle@rac131 12.2.0.1]$ crsctl query crs softwareversion -all
Oracle Clusterware version on node [rac131] is [12.1.0.2.0]
Oracle Clusterware version on node [rac132] is [12.1.0.2.0]

[oracle@rac131 12.2.0.1]$ crsctl query crs administrator
CRS Administrator List: oracle root

[oracle@rac131 12.2.0.1]$ crsctl query crs softwarepatch
Oracle Clusterware patch level on node rac131 is [3924561976].

[oracle@rac131 12.2.0.1]$  crsctl query dns -servers
CRS-10018: the following configuration was found on the system:
CRS-10019: There are 1 domains in search order. They are:
ora.com
CRS-10022: There are 1 name servers. They are:
192.168.1.100
CRS-10020: number of retry attempts for name lookup is: 4
CRS-10021: timeout for each name lookup is: 5

[oracle@rac131 12.2.0.1]$  crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   e54d8c16d0d74f24bf3e8571390bb749 (ORCL:CRS01) [OCR_VOTING]
Located 1 voting disk(s).

[oracle@rac131 12.2.0.1]$ crsctl query crs softwarepatch rac131
Oracle Clusterware patch level on node rac131 is [3924561976].

[oracle@rac131 12.2.0.1]$ crsctl query crs softwarepatch rac132
Oracle Clusterware patch level on node rac132 is [3924561976].


[oracle@rac131 ~]$ crsctl stat res -t
----------------------------------------------------------------
Name           Target  State        Server   State details                                                                                             
----------------------------------------------------------------
Local Resources
----------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac131   STABLE
               ONLINE  ONLINE       rac132   STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac131   STABLE
               ONLINE  ONLINE       rac132   STABLE
ora.OCR_VOTING.dg
               ONLINE  ONLINE       rac131   STABLE
               ONLINE  ONLINE       rac132   STABLE
ora.asm
               ONLINE  ONLINE       rac131   Started,STABLE
               ONLINE  ONLINE       rac132   Started,STABLE
ora.net1.network
               ONLINE  ONLINE       rac131   STABLE
               ONLINE  ONLINE       rac132   STABLE
ora.ons
               ONLINE  ONLINE       rac131   STABLE
               ONLINE  ONLINE       rac132   STABLE
----------------------------------------------------------------
Cluster Resources
----------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac131   STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac131   STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac131   STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       rac132   169.254.104.131 10.                                                                                        1
                                             0.10.132 20.20.20.1                                                                                        3
                                             2,STABLE
ora.ccbprod.db
      1        ONLINE  ONLINE       rac131   Open,STABLE
      2        ONLINE  ONLINE       rac132   Open,STABLE
ora.cvu
      1        ONLINE  ONLINE       rac131   STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       rac132   Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       rac132   STABLE
ora.rac131.vip
      1        ONLINE  ONLINE       rac131   STABLE
ora.rac132.vip
      1        ONLINE  ONLINE       rac132   STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       rac131   STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       rac131   STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       rac131   STABLE
----------------------------------------------------------------


[oracle@rac131 ~]$ asmcmd -p lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  4194304     20472    14408                0           14408              0             N  DATA/
MOUNTED  EXTERN  N         512   4096  4194304     40896    36236                0           36236              0             Y  OCR_VOTING/

[oracle@rac131 app]$ cd /ora_nfs/software/Oracle/database/12.2.0.1/
 [oracle@rac131 12.2.0.1]$ ls -ltrah
total 6.1G
drwxrwxrwx. 3 777 501 4.0K Nov 26  2016 .
drwxrwxrwx. 8 777 501 4.0K Nov 26  2016 ..
drwxrwxrwx. 7 777 501 4.0K Jan 26 21:52 database
-rwxrwxrwx. 1 777 501 2.8G Mar  2 03:20 linuxx64_12201_grid_home.zip
-rwxrwxrwx. 1 777 501 3.3G Mar  2 03:21 DB_soft_12cR2.zip
-rwxrwxrwx. 1 777 501    0 Mar  4 00:13 oracle-database-server-12cR2-preinstall-1.0-2.el7.x86_64.rpm
-rwxrwxrwx. 1 777 501    0 Mar  4 19:36 oracle-database-server-12cR2-preinstall-1.0-1.el7.x86_64.rpm


Unzipping Grid Binary to /u01/app/12.2.0.1/GRID/

[oracle@rac131 12.2.0.1]$ unzip -d /u01/app/12.2.0.1/GRID/ linuxx64_12201_grid_home.zip
Archive:  linuxx64_12201_grid_home.zip
   creating: /u01/app/12.2.0.1/GRID/addnode/
  inflating: /u01/app/12.2.0.1/GRID/addnode/addnode_oraparam.ini.sbs
  inflating: /u01/app/12.2.0.1/GRID/addnode/addnode.pl
  inflating: /u01/app/12.2.0.1/GRID/addnode/addnode.sh
  inflating: /u01/app/12.2.0.1/GRID/addnode/addnode_oraparam.ini
   creating: /u01/app/12.2.0.1/GRID/assistants/
   creating: /u01/app/12.2.0.1/GRID/assistants/jlib/
  inflating: /u01/app/12.2.0.1/GRID/assistants/jlib/initconfig
  inflating: /u01/app/12.2.0.1/GRID/assistants/jlib/appfwk.jar
  inflating: /u01/app/12.2.0.1/GRID/assistants/jlib/rconfig.jar
  inflating: /u01/app/12.2.0.1/GRID/xdk/mesg/jznpt.msb
  inflating: /u01/app/12.2.0.1/GRID/xdk/mesg/lsxja.msb
finishing deferred symbolic links:
  /u01/app/12.2.0.1/GRID/bin/lbuilder -> ../nls/lbuilder/lbuilder
  /u01/app/12.2.0.1/GRID/javavm/admin/libjtcjt.so -> ../../javavm/jdk/jdk8/admin/libjtcjt.so
  /u01/app/12.2.0.1/GRID/javavm/admin/lfclasses.bin -> ../../javavm/jdk/jdk8/admin/lfclasses.bin
  /u01/app/12.2.0.1/GRID/javavm/admin/classes.bin -> ../../javavm/jdk/jdk8/admin/classes.bin
  /u01/app/12.2.0.1/GRID/javavm/admin/cbp.jar -> ../../javavm/jdk/jdk8/admin/cbp.jar
  /u01/app/12.2.0.1/GRID/javavm/lib/sunjce_provider.jar -> ../../javavm/jdk/jdk8/lib/sunjce_provider.jar
  /u01/app/12.2.0.1/GRID/javavm/lib/jce.jar -> ../../javavm/jdk/jdk8/lib/jce.jar
  /u01/app/12.2.0.1/GRID/javavm/lib/security/US_export_policy.jar -> ../../../javavm/jdk/jdk8/lib/security/US_export_policy.jar
  /u01/app/12.2.0.1/GRID/javavm/lib/security/java.security -> ../../../javavm/jdk/jdk8/lib/security/java.security
  /u01/app/12.2.0.1/GRID/javavm/lib/security/local_policy.jar -> ../../../javavm/jdk/jdk8/lib/security/local_policy.jar
  /u01/app/12.2.0.1/GRID/javavm/lib/security/cacerts -> ../../../javavm/jdk/jdk8/lib/security/cacerts
  /u01/app/12.2.0.1/GRID/jdk/bin/ControlPanel -> jcontrol
  /u01/app/12.2.0.1/GRID/jdk/jre/bin/ControlPanel -> jcontrol
  /u01/app/12.2.0.1/GRID/jdk/jre/lib/amd64/server/libjsig.so -> ../libjsig.so
  /u01/app/12.2.0.1/GRID/lib/libclntshcore.so -> libclntshcore.so.12.1
  /u01/app/12.2.0.1/GRID/lib/libjavavm12.a -> ../javavm/jdk/jdk8/lib/libjavavm12.a
  /u01/app/12.2.0.1/GRID/lib/libodm12.so -> libodmd12.so
  /u01/app/12.2.0.1/GRID/lib/libagtsh.so -> libagtsh.so.1.0
  /u01/app/12.2.0.1/GRID/lib/libclntsh.so -> libclntsh.so.12.1
  /u01/app/12.2.0.1/GRID/lib/libocci.so -> libocci.so.12.1
[oracle@rac131 12.2.0.1]$



Starting 12cR2 Grid Infrastructure Upgrade

To use upgrade Oracle Grid Infrastructure for a cluster:
Go to Unzipped directory and start the Grid Setup wizard:
/u01/app/12.2.0.1/GRID/gridSetup.sh
On the Select Installation Wizard page, select Upgrade Oracle Grid Infrastructure



Select the Upgrade Option 


Ensure all node selected  



Select appropriate group 


Select correct Oracle Base 


Here I am running rootupgrade.sh manually, so I am not clicking automatic run configuration scripts 


Fix any found failure  




Checking OCR Integrity Failure

During upgrade I found OLR integrity failure.  I used below approach to fix the issue.

[oracle@rac131 12.2.0.1]$ cat -n /etc/oraInst.loc
     1  inventory_loc=/u01/app/oraInventory
     2  inst_group=oinstall
[oracle@rac131 12.2.0.1]$
[oracle@rac131 12.2.0.1]$
[oracle@rac131 12.2.0.1]$
[oracle@rac131 12.2.0.1]$ cluvfy comp olr -verbose

Verifying OLR integrity

Checking OLR integrity...
Check of existence of OLR configuration file "/etc/oracle/olr.loc" passed
Check of attributes of OLR configuration file "/etc/oracle/olr.loc" passed
PRVG-2031 : Owner of file "/u01/app/12.1.0.2/GRID/cdata/rac131.olr" did not match the expected value on node "rac131". [Expected = "root" ; Found = "oracle"]

WARNING:
This check does not verify the integrity of the OLR contents. Execute 'ocrcheck -local' as a privileged user to verify the contents of OLR.

OLR integrity check passed

Verification of OLR integrity was successful.


[oracle@rac131 12.2.0.1]$ sudo /u01/app/12.1.0.2/GRID/bin/ocrcheck -local
Status of Oracle Local Registry is as follows :
         Version                  :          4
         Total space (kbytes)     :     409568
         Used space (kbytes)      :        972
         Available space (kbytes) :     408596
         ID                       : 2106775746
         Device/File Name         : /u01/app/12.1.0.2/GRID/cdata/rac131.olr
                                    Device/File integrity check succeeded

         Local registry integrity check succeeded

         Logical corruption check succeeded


[root@rac131 12.2.0.1]$ ls -ltrah  /u01/app/12.1.0.2/GRID/cdata/rac131.olr
-rw-------. 1 oracle oinstall 481M Jun 18 11:37 /u01/app/12.1.0.2/GRID/cdata/rac131.olr

Changing the permission to root:oinstall

[root@rac131 12.2.0.1]$ chmod root:oinstall /u01/app/12.1.0.2/GRID/cdata/rac131.olr

Validating permission to root:oinstall

[root@rac131 12.2.0.1]$ ls -ltrah   /u01/app/12.1.0.2/GRID/cdata/rac131.olr

-rw-------. 1 root oinstall 481M Jun 18 11:59 /u01/app/12.1.0.2/GRID/cdata/rac131.olr

[oracle@rac131 12.2.0.1]$ sudo /tmp/GridSetupActions2017-06-18_12-06-46PM/CVU_12.2.0.1.0_oracle/runfixup.sh
All Fix-up operations were completed successfully.







Now Up-gradation of GI is in progress


rootupgrade.sh is pointing so running on both node.  



Run rootupgrade.sh as root user

[oracle@rac131 ~]$ sudo /u01/app/12.2.0.1/GRID/rootupgrade.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/12.2.0.1/GRID

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying oraenv to /usr/local/bin ...
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/12.2.0.1/GRID/crs/install/crsconfig
The log of current session can be found at:
  /u01/app/oracle/crsdata/rac131/crsconfig/rootcrs_rac131_2017-06-18_12-56-54AM.
2017/06/18 12:56:58 CLSRSC-595: Executing upgrade step 1 of 19: 'UpgradeTFA'.
2017/06/18 12:56:58 CLSRSC-4015: Performing install or upgrade action for Oracle
2017/06/18 12:57:25 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer                                                                                    (TFA) Collector.
2017/06/18 12:57:25 CLSRSC-595: Executing upgrade step 2 of 19: 'ValidateEnv'.
2017/06/18 12:57:38 CLSRSC-595: Executing upgrade step 3 of 19: 'GenSiteGUIDs'.
2017/06/18 12:57:39 CLSRSC-595: Executing upgrade step 4 of 19: 'GetOldConfig'.
2017/06/18 12:57:39 CLSRSC-464: Starting retrieval of the cluster configuration data
2017/06/18 12:57:50 CLSRSC-515: Starting OCR manual backup.
2017/06/18 12:57:55 CLSRSC-516: OCR manual backup successful.
2017/06/18 12:58:13 CLSRSC-486:
 At this stage of upgrade, the OCR has changed.
 Any attempt to downgrade the cluster after this point will require a complete cluster outage to restore the OCR.
2017/06/18 12:58:13 CLSRSC-541:
 To downgrade the cluster:
 1. All nodes that have been upgraded must be downgraded.
2017/06/18 12:58:13 CLSRSC-542:
 2. Before downgrading the last node, the Grid Infrastructure stack on all other cluster nodes must be down.
2017/06/18 12:58:13 CLSRSC-615:
 3. The last node to downgrade cannot be a Leaf node.
2017/06/18 12:58:22 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2017/06/18 12:58:22 CLSRSC-595: Executing upgrade step 5 of 19: 'UpgPrechecks'.
2017/06/18 12:58:39 CLSRSC-595: Executing upgrade step 6 of 19: 'SaveParamFile'.
2017/06/18 12:58:48 CLSRSC-595: Executing upgrade step 7 of 19: 'SetupOSD'.
2017/06/18 12:59:03 CLSRSC-595: Executing upgrade step 8 of 19: 'PreUpgrade'.
2017/06/18 12:59:08 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode
2017/06/18 12:59:08 CLSRSC-482: Running command: '/u01/app/12.1.0.2/GRID/bin/crsctl start rollingupgrade 12.2.0.1.0'
CRS-1131: The cluster was successfully set to rolling upgrade mode.
2017/06/18 12:59:12 CLSRSC-482: Running command: '/u01/app/12.2.0.1/GRID/bin/asmca -silent -upgradeNodeASM -nonRolling false -oldCRSHome /u01/app/12.1.0.2/GRID -oldCRSVersion 12.1.0.2.0 -firstNode true -startRolling false '

ASM configuration upgraded in local node successfully.

2017/06/18 12:59:16 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode
2017/06/18 12:59:28 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
2017/06/18 12:59:54 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.
2017/06/18 12:59:56 CLSRSC-595: Executing upgrade step 9 of 19: 'CheckCRSConfig'.
2017/06/18 12:59:57 CLSRSC-595: Executing upgrade step 10 of 19: 'UpgradeOLR'.
2017/06/18 13:00:04 CLSRSC-595: Executing upgrade step 11 of 19: 'ConfigCHMOS'.
2017/06/18 13:00:04 CLSRSC-595: Executing upgrade step 12 of 19: 'InstallAFD'.
2017/06/18 13:00:11 CLSRSC-595: Executing upgrade step 13 of 19: 'createOHASD'.
2017/06/18 13:00:17 CLSRSC-595: Executing upgrade step 14 of 19: 'ConfigOHASD'.
2017/06/18 13:00:33 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.conf'
2017/06/18 13:00:54 CLSRSC-595: Executing upgrade step 15 of 19: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac131'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac131' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2017/06/18 13:01:41 CLSRSC-595: Executing upgrade step 16 of 19: 'InstallKA'.
2017/06/18 13:01:48 CLSRSC-595: Executing upgrade step 17 of 19: 'UpgradeCluster'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac131'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac131' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac131'
CRS-2672: Attempting to start 'ora.evmd' on 'rac131'
CRS-2676: Start of 'ora.evmd' on 'rac131' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac131'
CRS-2676: Start of 'ora.gpnpd' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac131'
CRS-2676: Start of 'ora.gipcd' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac131'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac131'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac131'
CRS-2676: Start of 'ora.diskmon' on 'rac131' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac131'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac131'
CRS-2676: Start of 'ora.ctssd' on 'rac131' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac131'
CRS-2676: Start of 'ora.asm' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac131'
CRS-2676: Start of 'ora.storage' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rac131'
CRS-2676: Start of 'ora.crf' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac131'
CRS-2676: Start of 'ora.crsd' on 'rac131' succeeded
CRS-6017: Processing resource auto-start for servers: rac131
CRS-6016: Resource auto-start has completed for server rac131
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 1.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac131'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac131'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac131'
CRS-2673: Attempting to stop 'ora.asm' on 'rac131'
CRS-2677: Stop of 'ora.asm' on 'rac131' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac131'
CRS-2677: Stop of 'ora.net1.network' on 'rac131' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac131' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac131' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac131'
CRS-2673: Attempting to stop 'ora.crf' on 'rac131'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac131'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac131'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac131'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac131' succeeded
CRS-2677: Stop of 'ora.crf' on 'rac131' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rac131' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac131' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac131' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac131'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac131' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac131'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac131'
CRS-2677: Stop of 'ora.ctssd' on 'rac131' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac131' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac131'
CRS-2677: Stop of 'ora.cssd' on 'rac131' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac131'
CRS-2677: Stop of 'ora.gipcd' on 'rac131' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac131' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.evmd' on 'rac131'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac131'
CRS-2676: Start of 'ora.evmd' on 'rac131' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac131'
CRS-2676: Start of 'ora.gpnpd' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac131'
CRS-2676: Start of 'ora.gipcd' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac131'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac131'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac131'
CRS-2676: Start of 'ora.diskmon' on 'rac131' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac131'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac131'
CRS-2676: Start of 'ora.ctssd' on 'rac131' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac131'
CRS-2676: Start of 'ora.asm' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac131'
CRS-2676: Start of 'ora.storage' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rac131'
CRS-2676: Start of 'ora.crf' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac131'
CRS-2676: Start of 'ora.crsd' on 'rac131' succeeded
CRS-6017: Processing resource auto-start for servers: rac131
CRS-2673: Attempting to stop 'ora.rac131.vip' on 'rac132'
CRS-2672: Attempting to start 'ora.ons' on 'rac131'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac132'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac132' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac132'
CRS-2677: Stop of 'ora.rac131.vip' on 'rac132' succeeded
CRS-2672: Attempting to start 'ora.rac131.vip' on 'rac131'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac132' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac131'
CRS-2676: Start of 'ora.scan1.vip' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac131'
CRS-2676: Start of 'ora.rac131.vip' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'rac131'
CRS-2676: Start of 'ora.ons' on 'rac131' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac131' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.ccbprod.db' on 'rac131'
CRS-2676: Start of 'ora.ccbprod.db' on 'rac131' succeeded
CRS-6016: Resource auto-start has completed for server rac131
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2017/06/18 13:06:26 CLSRSC-343: Successfully started Oracle Clusterware stack
2017/06/18 13:06:26 CLSRSC-595: Executing upgrade step 18 of 19: 'UpgradeNode'.
2017/06/18 13:06:44 CLSRSC-474: Initiating upgrade of resource types
2017/06/18 13:07:48 CLSRSC-482: Running command: 'srvctl upgrade model -s 12.1.0.2.0 -d 12.2.0.1.0 -p first'
2017/06/18 13:07:48 CLSRSC-475: Upgrade of resource types successfully initiated.
2017/06/18 13:08:23 CLSRSC-595: Executing upgrade step 19 of 19: 'PostUpgrade'.
2017/06/18 13:08:29 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded


Run rootupgrade.sh on Node2 (rac132)

[oracle@rac132 ~]$ sudo /u01/app/12.2.0.1/GRID/rootupgrade.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/12.2.0.1/GRID

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying oraenv to /usr/local/bin ...
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/12.2.0.1/GRID/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/oracle/crsdata/rac132/crsconfig/rootcrs_rac132_2017-06-18_01-28-28PM.log
2017/06/18 13:28:30 CLSRSC-595: Executing upgrade step 1 of 19: 'UpgradeTFA'.
2017/06/18 13:28:30 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2017/06/18 13:28:55 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2017/06/18 13:28:55 CLSRSC-595: Executing upgrade step 2 of 19: 'ValidateEnv'.
2017/06/18 13:29:00 CLSRSC-595: Executing upgrade step 3 of 19: 'GenSiteGUIDs'.
2017/06/18 13:29:00 CLSRSC-595: Executing upgrade step 4 of 19: 'GetOldConfig'.
2017/06/18 13:29:00 CLSRSC-464: Starting retrieval of the cluster configuration data
2017/06/18 13:29:10 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2017/06/18 13:29:10 CLSRSC-595: Executing upgrade step 5 of 19: 'UpgPrechecks'.
2017/06/18 13:29:12 CLSRSC-595: Executing upgrade step 6 of 19: 'SaveParamFile'.
2017/06/18 13:29:15 CLSRSC-595: Executing upgrade step 7 of 19: 'SetupOSD'.
2017/06/18 13:29:17 CLSRSC-595: Executing upgrade step 8 of 19: 'PreUpgrade'.

ASM configuration upgraded in local node successfully.

2017/06/18 13:29:22 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
2017/06/18 13:29:49 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.
2017/06/18 13:29:54 CLSRSC-595: Executing upgrade step 9 of 19: 'CheckCRSConfig'.
2017/06/18 13:29:54 CLSRSC-595: Executing upgrade step 10 of 19: 'UpgradeOLR'.
2017/06/18 13:29:56 CLSRSC-595: Executing upgrade step 11 of 19: 'ConfigCHMOS'.
2017/06/18 13:29:56 CLSRSC-595: Executing upgrade step 12 of 19: 'InstallAFD'.
2017/06/18 13:29:57 CLSRSC-595: Executing upgrade step 13 of 19: 'createOHASD'.
2017/06/18 13:29:59 CLSRSC-595: Executing upgrade step 14 of 19: 'ConfigOHASD'.
2017/06/18 13:30:14 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.conf'
2017/06/18 13:30:31 CLSRSC-595: Executing upgrade step 15 of 19: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac132'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac132' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2017/06/18 13:31:11 CLSRSC-595: Executing upgrade step 16 of 19: 'InstallKA'.
2017/06/18 13:31:13 CLSRSC-595: Executing upgrade step 17 of 19: 'UpgradeCluster'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac132'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac132' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac132'
CRS-2672: Attempting to start 'ora.evmd' on 'rac132'
CRS-2676: Start of 'ora.evmd' on 'rac132' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rac132' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac132'
CRS-2676: Start of 'ora.gpnpd' on 'rac132' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac132'
CRS-2676: Start of 'ora.gipcd' on 'rac132' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac132'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac132' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac132'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac132'
CRS-2676: Start of 'ora.diskmon' on 'rac132' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac132' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac132'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac132'
CRS-2676: Start of 'ora.ctssd' on 'rac132' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac132' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac132'
CRS-2676: Start of 'ora.asm' on 'rac132' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac132'
CRS-2676: Start of 'ora.storage' on 'rac132' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'rac132'
CRS-2676: Start of 'ora.crf' on 'rac132' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac132'
CRS-2676: Start of 'ora.crsd' on 'rac132' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: rac132
CRS-2673: Attempting to stop 'ora.rac132.vip' on 'rac131'
CRS-2672: Attempting to start 'ora.ons' on 'rac132'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac131'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac131' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac131'
CRS-2677: Stop of 'ora.rac132.vip' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.rac132.vip' on 'rac132'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac131' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac132'
CRS-2676: Start of 'ora.rac132.vip' on 'rac132' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'rac132'
CRS-2676: Start of 'ora.scan1.vip' on 'rac132' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac132'
CRS-2676: Start of 'ora.ons' on 'rac132' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'rac132' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac132'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac132' succeeded
CRS-2676: Start of 'ora.asm' on 'rac132' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'rac132'
CRS-2676: Start of 'ora.DATA.dg' on 'rac132' succeeded
CRS-2672: Attempting to start 'ora.ccbprod.db' on 'rac132'
CRS-2676: Start of 'ora.ccbprod.db' on 'rac132' succeeded
CRS-6016: Resource auto-start has completed for server rac132
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2017/06/18 13:33:11 CLSRSC-343: Successfully started Oracle Clusterware stack
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 12c Release 2.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2017/06/18 13:33:42 CLSRSC-595: Executing upgrade step 18 of 19: 'UpgradeNode'.
Start upgrade invoked..
2017/06/18 13:34:13 CLSRSC-478: Setting Oracle Clusterware active version on the last node to be upgraded
2017/06/18 13:34:13 CLSRSC-482: Running command: '/u01/app/12.2.0.1/GRID/bin/crsctl set crs activeversion'
Started to upgrade the active version of Oracle Clusterware. This operation may take a few minutes.
Started to upgrade CSS.
CSS was successfully upgraded.
Started to upgrade Oracle ASM.
Started to upgrade CRS.
CRS was successfully upgraded.
Successfully upgraded the active version of Oracle Clusterware.
Oracle Clusterware active version was successfully set to 12.2.0.1.0.
2017/06/18 13:35:26 CLSRSC-479: Successfully set Oracle Clusterware active version
2017/06/18 13:35:46 CLSRSC-476: Finishing upgrade of resource types
2017/06/18 13:36:10 CLSRSC-482: Running command: 'srvctl upgrade model -s 12.1.0.2.0 -d 12.2.0.1.0 -p last'
2017/06/18 13:36:10 CLSRSC-477: Successfully completed upgrade of resource types
2017/06/18 13:36:47 CLSRSC-595: Executing upgrade step 19 of 19: 'PostUpgrade'.
2017/06/18 13:36:57 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded





Click OK to continue the installation



GI Upgrade has successfully completed.



Finally we have upgraded to 12cR2 GI.


Post validation the upgrade:

[oracle@rac132 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   e54d8c16d0d74f24bf3e8571390bb749 (ORCL:CRS01) [OCR_VOTING]
Located 1 voting disk(s).

[oracle@rac132 ~]$ crsctl query cluster site -all
Site 'rac-cluster' identified by GUID 'ae47291ec170cf03bfbd8b2ec3b4456c' in state 'ENABLED' contains nodes 'rac131,rac132' and no disk.


[oracle@rac132 ~]$ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [0].

[oracle@rac132 ~]$ crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [12.2.0.1.0]


[oracle@rac132 ~]$ crsctl query crs softwareversion -all
Oracle Clusterware version on node [rac131] is [12.2.0.1.0]
Oracle Clusterware version on node [rac132] is [12.2.0.1.0]


[oracle@rac132 ~]$  crsctl query crs softwarepatch rac131
Oracle Clusterware patch level on node rac131 is [0].

[oracle@rac132 ~]$  crsctl query crs softwarepatch rac132
Oracle Clusterware patch level on node rac132 is [0].


[oracle@rac132 ~]$ crsctl query dns -servers
CRS-10018: the following configuration was found on the system:
CRS-10019: There are 1 domains in search order. They are:
ora.com
CRS-10022: There are 1 name servers. They are:
192.168.1.100
CRS-10020: number of retry attempts for name lookup is: 4
CRS-10021: timeout for each name lookup is: 5


[oracle@rac132 ~]$ crsctl check cluster -all
**************************************************************
rac131:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
rac132:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************





[oracle@rac132 ~]$ crsctl stat res -t
-----------------------------------------------------------------
Name           Target  State        Server    State details
-----------------------------------------------------------------
Local Resources
-----------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       rac131    STABLE
               ONLINE  ONLINE       rac132    STABLE
ora.ASMNET2LSNR_ASM.lsnr
               ONLINE  ONLINE       rac131    STABLE
               ONLINE  ONLINE       rac132    STABLE
ora.DATA.dg
               ONLINE  ONLINE       rac131    STABLE
               ONLINE  ONLINE       rac132    STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac131    STABLE
               ONLINE  ONLINE       rac132    STABLE
ora.OCR_VOTING.dg
               ONLINE  ONLINE       rac131    STABLE
               ONLINE  ONLINE       rac132    STABLE
ora.chad
               ONLINE  ONLINE       rac131    STABLE
               ONLINE  ONLINE       rac132    STABLE
ora.net1.network
               ONLINE  ONLINE       rac131    STABLE
               ONLINE  ONLINE       rac132    STABLE
ora.ons
               ONLINE  ONLINE       rac131    STABLE
               ONLINE  ONLINE       rac132    STABLE
ora.proxy_advm
               OFFLINE OFFLINE      rac131    STABLE
               OFFLINE OFFLINE      rac132    STABLE
-----------------------------------------------------------------
Cluster Resources
-----------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac132    STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac131    STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac131    STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       rac131    169.254.96.83 10.10.
                                              10.131 20.20.20.131,
                                              STABLE
ora.asm
      1        ONLINE  ONLINE       rac131    STABLE
      2        ONLINE  ONLINE       rac132    STABLE
ora.ccbprod.db
      1        ONLINE  ONLINE       rac131    Open,HOME=/u01/app/o
                                              racle/product/12.1.0
                                              .2/DB_1,STABLE
      2        ONLINE  ONLINE       rac132    Open,HOME=/u01/app/o
                                              racle/product/12.1.0
                                              .2/DB_1,STABLE
ora.cvu
      1        ONLINE  ONLINE       rac132    STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       rac131    Open,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       rac132    STABLE
ora.rac131.vip
      1        ONLINE  ONLINE       rac131    STABLE
ora.rac132.vip
      1        ONLINE  ONLINE       rac132    STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       rac132    STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       rac131    STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       rac131    STABLE
-----------------------------------------------------------------

Now seems every thing is OK..



Moving to Database Upgrade to 12.2.0.1


[oracle@rac131 ~]$ cd /ora_nfs/software/Oracle/database/12.2.0.1/database/
[oracle@rac131 database]$ ls -ltrah
total 44K
-rwxrwxrwx.  1 777 501  500 Feb  7  2013 welcome.html
drwxrwxrwx.  3 777 501 4.0K Nov 26  2016 ..
drwxrwxrwx.  4 777 501 4.0K Jan 26 20:09 install
-rwxrwxrwx.  1 777 501 8.6K Jan 26 20:09 runInstaller
drwxrwxrwx.  2 777 501 4.0K Jan 26 20:09 rpm
drwxrwxrwx.  2 777 501 4.0K Jan 26 21:23 sshsetup
drwxrwxrwx.  2 777 501 4.0K Jan 26 21:23 response
drwxrwxrwx. 14 777 501 4.0K Jan 26 21:25 stage
drwxrwxrwx.  7 777 501 4.0K Jan 26 21:52 .


Click Next...


Select Upgrade Existing Database and Click Next...


Select Oracle RAC Database Installation and Click Next...



Select all nodes and Click Next...


Click Next...


Select the RDBMS HOME and Click Next...


Check appropriate Click Next...


Click Next...


Click Next... 




Run the root.sh as root user 




Run the root.sh as root user  on node 1

[oracle@rac131 database]$ sudo /u01/app/oracle/product/12.2.0.2/DB_1/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/12.2.0.2/DB_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
[oracle@rac131 database]$


Run the root.sh as root user on node 2

[oracle@rac132 ~]$ sudo /u01/app/oracle/product/12.2.0.2/DB_1/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/12.2.0.2/DB_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.

Click OK ..  and type the sys dba password



Click Next....


Click Next....


Click Next....  

Fix the any issue if prompting before upgrade.  In my case I have remove all hidden "_your_parameter_name"  parameter from my parameter in order to start the installation.


Select Enable Parallel Upgrade and all necessary in below list and Click Next....


Since I have taken backup before upgrade so I am skipping this step, Click Next....


Here I am configuring EM database express, if you don't want the you can unchecked the check box ... Click Next....


Validate the summary and Click Next....



In my case Upgrade was taking very long time (14 Hours) as I have set 8GB RAM for my virtual box of my each nodes, as minimum requirement for 12cR2 is 12GB.  So total time taking is depends on system RAM.


My Database got upgraded to 12cR2.  





Validating the upgraded database and pdbs


SQL> SELECT DISTINCT A.INST_ID,
NAME DB_NAME,
A.INSTANCE_NAME, VERSION, HOST_NAME , OPEN_MODE, 
TO_CHAR(STARTUP_TIME,'DD-MON-RRRR HH:MI:SS AM') STARTUP_TIME,
TO_CHAR(CURRENT_SCN) CURR_SCN
FROM GV$INSTANCE A, GV$DATABASE B ;


INST_ID DB_NAME    INSTANCE_NAME  VERSION     HOST_NAME        OPEN_MODE   STARTUP_TIME              CURR_SCN
------- ---------- -------------- ----------- --------------- ----------- ------------------------- ---------
      2 CCBPROD    CCBPROD2       12.2.0.1.0  rac132.ora.com   READ WRITE  18-JUN-2017 09:01:27 PM   5583974
      1 CCBPROD    CCBPROD1       12.2.0.1.0  rac131.ora.com   READ WRITE  18-JUN-2017 09:01:21 PM   5583974


PDBS CHECKING

SQL> SELECT INST_ID,
CON_ID,
NAME,
OPEN_MODE,
OPEN_TIME,
CREATION_TIME,
LOCAL_UNDO,
RESTRICTED,
RECOVERY_STATUS
FROM GV$PDBS  WHERE NAME <>'PDB$SEED';

INST_ID CON_ID NAME OPEN_MODE  OPEN_TIME                        CREATION_ LOCAL_UNDO RES RECOVERY
------- ------ ---- ---------- -------------------------------- --------- ---------- --- --------
      1      3 PCCB READ WRITE 18-JUN-17 11.51.18.229 PM +05:30 18-JUN-17          0 NO  ENABLED
      2      3 PCCB READ WRITE 18-JUN-17 11.51.18.238 PM +05:30 18-JUN-17          0 NO  ENABLED

Validate the upgraded component

SQL>  select COMP_ID,COMP_NAME,VERSION,STATUS,NAMESPACE,CONTROL,SCHEMA from dba_registry ;

COMP_ID    COMP_NAME                           VERSION    STATUS  NAMESPACE CONTROL SCHEMA
---------- ----------------------------------- ---------- ------- --------- ------- ---------------
CATALOG    Oracle Database Catalog Views       12.2.0.1.0 VALID   SERVER    SYS     SYS
CATPROC    Oracle Database Packages and Types  12.2.0.1.0 VALID   SERVER    SYS     SYS
JAVAVM     JServer JAVA Virtual Machine        12.2.0.1.0 VALID   SERVER    SYS     SYS
XML        Oracle XDK                          12.2.0.1.0 VALID   SERVER    SYS     SYS
CATJAVA    Oracle Database Java Packages       12.2.0.1.0 VALID   SERVER    SYS     SYS
APS        OLAP Analytic Workspace             12.2.0.1.0 VALID   SERVER    SYS     SYS
RAC        Oracle Real Application Clusters    12.2.0.1.0 VALID   SERVER    SYS     SYS
XDB        Oracle XML Database                 12.2.0.1.0 VALID   SERVER    SYS     XDB
SDO        Spatial                             12.2.0.1.0 VALID   SERVER    SYS     MDSYS
OWM        Oracle Workspace Manager            12.2.0.1.0 VALID   SERVER    SYS     WMSYS
CONTEXT    Oracle Text                         12.2.0.1.0 VALID   SERVER    SYS     CTXSYS
ORDIM      Oracle Multimedia                   12.2.0.1.0 VALID   SERVER    SYS     ORDSYS
DV         Oracle Database Vault               12.2.0.1.0 VALID   SERVER    SYS     DVSYS
XOQ        Oracle OLAP API                     12.2.0.1.0 VALID   SERVER    SYS     OLAPSYS
OLS        Oracle Label Security               12.2.0.1.0 VALID   SERVER    SYS     LBACSYS
APEX       Oracle Application Express          5.0.4.00.1 VALID   SERVER    SYS     APEX_050000






....Hope this approach will help in your case. :)