Senthil Rajendran
The voice in my head may not be real , but they have some good ideas !!!Senthil Rajendranhttp://www.blogger.com/profile/15572088214110845083noreply@blogger.comBlogger291125
Updated: 2 weeks 1 day ago
Teri Meri Prem Kahani Cover | Keyboard Performance | by Dharun at Improviser Music Studio
My Son Dharun Performing at Improviser Music Studio
Teri Meri Prem Kahani Cover
DarkSide Cover | Keyboard Performance | by Dharun at Improviser Music Studio
My Son Dharun Performing at Improviser Music Studio
DarkSide Cover
DarkSide Cover
Believer | Keyboard Performance | by Dharun at Improviser Music Studio
My Son Dharun Performing at Improviser Music Studio
Believer
Please subscribe to our channel Improviser Music Studio
Despacito | Keyboard Performance | by Dharun at Improviser Music Studio
My Son Dharun Performing at Improviser Music Studio
Despacito
Please subscribe to our channel Improviser Music Studio
Despacito
Please subscribe to our channel Improviser Music Studio
Tchaikovsky | Swan Lake Keyboard Performance | by Dharun at Improviser Music Studio
My Son Dharun Performing at Improviser Music Studio
Tchaikovsky | Swan Lake
Please subscribe to our channel Improviser Music Studio
Shawn Mendes - Treat You Better | Keyboard Performance | by Dharun at Improviser Music Studio
My Son Dharun Performing at Improviser Music Studio
Shawn Mendes - Treat You Better
Please subscribe to our channel Improviser Music Studio
Ekadantaya Vakratundaya Keyboard Performance | by Dharun at Improviser Music Studio
My Son Dharun Performing at Improviser Music Studio
Ekadantaya Vakratundaya
Please subscribe to our channel Improviser Music Studio
EBS 12.2 ADOP Cycle Errors During Validation Cannot open XML file for load
EBS 12.2 ADOP Cycle Errors During Validation Cannot open XML file for load
ADOP cycle will have validation errors in some cases.
*******FATAL ERROR*******
PROGRAM :
(/test/apps/CLONE/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPEvalSrvStatus.pl)
TIME : Wed Nov 13 15:50:36 2019
FUNCTION: TXK::XML::load_doc [ Level 1 ]
MESSAGES:
error = Cannot open XML file for load
errorno = No such file or directory
file =
/test/apps/CLONE/fs_ne/EBSapps/log/adop/6/fs_clone_20191113_153822/CLONE_test/TXK_EVAL_fs_clone_Wed_Nov_13_15_49_52_2014/ctx_files/CLONE_test.xml
*******FATAL ERROR*******
PROGRAM :
(/test/apps/CLONE/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME : Wed Nov 13 15:50:36 2019
FUNCTION: main::validatePatchContextFile [ Level 1 ]
MESSAGES:
message = Access permission error on test
File CLONE_test.xml not readable
If you see the above stack then do the fix as suggested below
- Validate FND_NODES table for valid hostnames
- Validate FND_OAM_CONTEXT_FILES table for run and patch context file
- If a valid node does not have a valid run and patch context file in FND_OAM_CONTEXT_FILES , then it has to be loaded either by running autoconfig from the respective file system or if you do not want to run autoconfig then load the context file using API
$ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload contextfile=$CONTEXT_FILE
Rerun ADOP Cycle.
Have fun with Oracle EBS 12.2
ADOP cycle will have validation errors in some cases.
*******FATAL ERROR*******
PROGRAM :
(/test/apps/CLONE/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPEvalSrvStatus.pl)
TIME : Wed Nov 13 15:50:36 2019
FUNCTION: TXK::XML::load_doc [ Level 1 ]
MESSAGES:
error = Cannot open XML file for load
errorno = No such file or directory
file =
/test/apps/CLONE/fs_ne/EBSapps/log/adop/6/fs_clone_20191113_153822/CLONE_test/TXK_EVAL_fs_clone_Wed_Nov_13_15_49_52_2014/ctx_files/CLONE_test.xml
*******FATAL ERROR*******
PROGRAM :
(/test/apps/CLONE/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME : Wed Nov 13 15:50:36 2019
FUNCTION: main::validatePatchContextFile [ Level 1 ]
MESSAGES:
message = Access permission error on test
File CLONE_test.xml not readable
If you see the above stack then do the fix as suggested below
- Validate FND_NODES table for valid hostnames
- Validate FND_OAM_CONTEXT_FILES table for run and patch context file
- If a valid node does not have a valid run and patch context file in FND_OAM_CONTEXT_FILES , then it has to be loaded either by running autoconfig from the respective file system or if you do not want to run autoconfig then load the context file using API
$ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload contextfile=$CONTEXT_FILE
Rerun ADOP Cycle.
Have fun with Oracle EBS 12.2
EBS 12.2 APPL_TOP Only Clone - ERRORMSG: Invalid APPS database user credentials.
[oracle@testebsop3app01 ~]$ perl /u01/install/APPS/fs1/EBSapps/comn/clone/bin/adcfgclone.pl appltop /u01/install/APPS/fs1/inst/apps/SATURN_testebsop3app01/appl/admin/SATURN_testebsop3app01.xml
Appl_top only clone might result in the below error
START: Instantiate the txkWfClone.sql...
instantiate file:
source : /u01/install/APPS/fs1/EBSapps/appl/fnd/12.0.0/admin/template/txkWfClone.sql
dest : /u01/install/APPS/fs1/inst/apps/SATURN_testebsop3app01/admin/install/txkWfClone.sql
Adding execute permission to : /u01/install/APPS/fs1/inst/apps/SATURN_testebsop3app01/admin/install/txkWfClone.sql
END: Instantiated txkWfClone.sql...
START: Executing /u01/install/APPS/fs1/inst/apps/SATURN_testebsop3app01/admin/install/txkWfClone.sh -nopromptmsg
txkWfClone.sh exited with status 1
ERROR: txkWfClone.sh execution failed, exit code 1
[oracle@testebsop3app01 clone_10091511]$
if you scroll up the logs you may find some more errors
TXK Script Name: txkChkFormsLaunchMethod.pl
Enter the APPS user password:
*******FATAL ERROR*******
PROGRAM : (/u01/install/APPS/fs1/EBSapps/appl/fnd/12.0.0/patch/115/bin/txkChkFormsLaunchMethod.pl)
TIME : Wed Oct 9 14:50:36 2019
FUNCTION: main::validateAppsSchemaCredentials [ Level 1 ]
ERRORMSG: Invalid APPS database user credentials.
ERRORCODE = 1 ERRORCODE_END
.end std out.
stty: standard input: Inappropriate ioctl for device
stty: standard input: Inappropriate ioctl for device
*******FATAL ERROR*******
PROGRAM : (/u01/install/APPS/fs1/EBSapps/appl/fnd/12.0.0/patch/115/bin/txkChkFormsLaunchMethod.pl)
TIME : Wed Oct 9 14:50:36 2019
FUNCTION: [ Level 1 ]
ERRORMSG: *******FATAL ERROR*******
PROGRAM : (/u01/install/APPS/fs1/EBSapps/appl/fnd/12.0.0/patch/115/bin/txkChkFormsLaunchMethod.pl)
TIME : Wed Oct 9 14:50:36 2019
FUNCTION: main::validateAppsSchemaCredentials [ Level 1 ]
ERRORMSG: Invalid APPS database user credentials.
The problem is tnsnames.ora does not have an entry for TWO_TASK
from MT , sqlplus apps/apps@TWO_TASK will fail to connect.
Fix :
run autoconfig on the DB node
run adcfgclone.pl appltop
Appl_top only clone might result in the below error
START: Instantiate the txkWfClone.sql...
instantiate file:
source : /u01/install/APPS/fs1/EBSapps/appl/fnd/12.0.0/admin/template/txkWfClone.sql
dest : /u01/install/APPS/fs1/inst/apps/SATURN_testebsop3app01/admin/install/txkWfClone.sql
Adding execute permission to : /u01/install/APPS/fs1/inst/apps/SATURN_testebsop3app01/admin/install/txkWfClone.sql
END: Instantiated txkWfClone.sql...
START: Executing /u01/install/APPS/fs1/inst/apps/SATURN_testebsop3app01/admin/install/txkWfClone.sh -nopromptmsg
txkWfClone.sh exited with status 1
ERROR: txkWfClone.sh execution failed, exit code 1
[oracle@testebsop3app01 clone_10091511]$
if you scroll up the logs you may find some more errors
TXK Script Name: txkChkFormsLaunchMethod.pl
Enter the APPS user password:
*******FATAL ERROR*******
PROGRAM : (/u01/install/APPS/fs1/EBSapps/appl/fnd/12.0.0/patch/115/bin/txkChkFormsLaunchMethod.pl)
TIME : Wed Oct 9 14:50:36 2019
FUNCTION: main::validateAppsSchemaCredentials [ Level 1 ]
ERRORMSG: Invalid APPS database user credentials.
ERRORCODE = 1 ERRORCODE_END
.end std out.
stty: standard input: Inappropriate ioctl for device
stty: standard input: Inappropriate ioctl for device
*******FATAL ERROR*******
PROGRAM : (/u01/install/APPS/fs1/EBSapps/appl/fnd/12.0.0/patch/115/bin/txkChkFormsLaunchMethod.pl)
TIME : Wed Oct 9 14:50:36 2019
FUNCTION: [ Level 1 ]
ERRORMSG: *******FATAL ERROR*******
PROGRAM : (/u01/install/APPS/fs1/EBSapps/appl/fnd/12.0.0/patch/115/bin/txkChkFormsLaunchMethod.pl)
TIME : Wed Oct 9 14:50:36 2019
FUNCTION: main::validateAppsSchemaCredentials [ Level 1 ]
ERRORMSG: Invalid APPS database user credentials.
The problem is tnsnames.ora does not have an entry for TWO_TASK
from MT , sqlplus apps/apps@TWO_TASK will fail to connect.
Fix :
run autoconfig on the DB node
run adcfgclone.pl appltop
Not All Endpoints Registered
Not All Endpoints Registered
crsctl stat res -t
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_LSTNSOLTP .lsnr
1 ONLINE INTERMEDIATE orcl1 Not All Endpoints Registered,STABLE
ora.LISTENER_LSTNSOLTP .lsnr
1 ONLINE ONLINE orcl2 STABLE
ora.LISTENER_LSTNSOLTP .lsnr
1 ONLINE ONLINE orcl3 STABLE
"Not All Endpoints Registered"
To fix the problem
- compare the listener.ora of all the nodes
- find the differences and sync it up
if you find static entries make sure you stop the listener running with "LSNRCT STOP " otherwise you might end up with duplicate processes
$ ps -ef|grep -i LSTNSOLTP
grid 312080 1 0 03:17 ? 00:00:00 /u01/app/12.1.0.2/grid/bin/tnslsnr LSTNSOLTP -no_crs_notify -inherit
$
-- found static entry in listener.ora
-- fixed listener.ora by removing the static entry
-- used crsctl to start the listener
$ ps -ef|grep -i LSTNSOLTP
grid 166779 1 0 Sep18 ? 00:14:53 /u01/app/12.1.0.2/grid/bin/tnslsnr LSTNSOLTP -no_crs_notify -inherit
grid 267334 1 0 03:07 ? 00:00:00 /u01/app/12.1.0.2/grid/bin/tnslsnr LSTNSOLTP -no_crs_notify -inherit
$
correct way to approach this problem
$ ps -ef|grep -i LSTNSOLTP
grid 312080 1 0 03:17 ? 00:00:00 /u01/app/12.1.0.2/grid/bin/tnslsnr LSTNSOLTP -no_crs_notify -inherit
$
$ lsnrctl stop LSTNSOLTP
$ ps -ef|grep -i LSTNSOLTP
$
Make sure no processes are running
-- fix listener.ora by removing the static entry
-- use crsctl to start the listener
$ ps -ef|grep -i LSTNSOLTP
grid 267334 1 0 03:07 ? 00:00:00 /u01/app/12.1.0.2/grid/bin/tnslsnr LSTNSOLTP -no_crs_notify -inherit
$
crsctl stat res -t
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_LSTNSOLTP .lsnr
1 ONLINE ONLINE orcl1 STABLE
ora.LISTENER_LSTNSOLTP .lsnr
1 ONLINE ONLINE orcl2 STABLE
ora.LISTENER_LSTNSOLTP .lsnr
1 ONLINE ONLINE orcl3 STABLE
Oracle EBS 12.2 - ADOP ad_zd_prep.create_patch_service exact fetch returns more than requested number of rows
Oracle EBS 12.2 - ADOP ad_zd_prep.create_patch_service exact fetch returns more than requested number of rows
Please note , if you get ADOP issues on PROD please read the logs and understand the problem before executing any commands.
SQL> exec ad_zd_prep.create_patch_service;
BEGIN ad_zd_prep.create_patch_service; END;
*
ERROR at line 1:
ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.AD_ZD_PREP", line 342
ORA-06512: at "APPS.AD_ZD_PREP", line 378
ORA-06512: at line 1
A Quick fix without autoconfig
Please note , if you get ADOP issues on PROD please read the logs and understand the problem before executing any commands.
SQL> exec ad_zd_prep.create_patch_service;
BEGIN ad_zd_prep.create_patch_service; END;
*
ERROR at line 1:
ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.AD_ZD_PREP", line 342
ORA-06512: at "APPS.AD_ZD_PREP", line 378
ORA-06512: at line 1
A Quick fix without autoconfig
- SQL> create table fnd_oam_bkup1 as select * from fnd_oam_context_files;
- SQL> truncate fnd_oam_context_files
- move DB context file MT
- Load the Database Context File
- $ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload contextfile=/tmp/test_dbserver.xml
- Load the Run File System Context File
- $ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload contextfile=/u01/test/appl/fs1/inst/apps/test_mtserver/appl/admin/test_mtserver.xml
- Load the Patch File System Context File
- $ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload contextfile=/u01/test/appl/fs1/inst/apps/test_mtserver/appl/admin/test_mtserver.xml
Next Maintenance run Autoconfig on Database and Middletier
SQL> exec ad_zd_prep.create_patch_service; -- this will succeed
or if you are running ADOP prepare will succeed.
Oracle EBS 12.2 - ADOP ad_zd_prep.create_patch_service exceptions
Oracle EBS 12.2 - ADOP ad_zd_prep.create_patch_service exceptions
Please note , if you get ADOP issues on PROD please read the logs and understand the problem before executing any commands.
There are cases where you might have to create a patch service manually. So usually after cloning or fresh install , add node , mass patching , upgrade this crazy error will pop up.
SQL> exec ad_zd_prep.create_patch_service;
BEGIN ad_zd_prep.create_patch_service; END;
*
ERROR at line 1:
ORA-01422: exact fetch returns more than requested number of rows
ORA-06512: at "APPS.AD_ZD_PREP", line 342
ORA-06512: at "APPS.AD_ZD_PREP", line 378
ORA-06512: at line 1
Check the state of the package
SQL> select owner, object_name,status,object_type from dba_objects where object_name like upper ('AD_ZD_PREP');
OWNER OBJECT_NAME STATUS OBJECT_TYPE
---------------------------------------- ---------------------------------------- ------- -----------------------
APPS AD_ZD_PREP VALID PACKAGE
APPS AD_ZD_PREP INVALID PACKAGE BODY
SQL> alter package apps.ad_zd_prep compile body;
Warning: Package Body altered with compilation errors.
SQL> show error
Errors for PACKAGE BODY APPS.AD_ZD_PREP:
LINE/COL ERROR
-------- -----------------------------------------------------------------
463/3 PL/SQL: SQL Statement ignored
463/19 PL/SQL: ORA-00942: table or view does not exist
467/5 PL/SQL: SQL Statement ignored
467/21 PL/SQL: ORA-00942: table or view does not exist
501/5 PL/SQL: SQL Statement ignored
501/21 PL/SQL: ORA-00942: table or view does not exist
Fix : follow this note
12.2 E-Business Suite DBA Upgrade Script ADZDEXRPT.sql Fails On Invalid Object 'AD_ZD_PREP' Or Patch 13543062 Fails With Error: ORA-06508: PL/SQL: could not find APPS.AD_ZD_PREPAD_ZD_PREP Due To Missing Package 'xdb_migrateschema' (Doc ID 2066607.1)
1. Confirm if the package 'xdb_migrateschema' exists using the following select statement:
select owner, object_name from dba_objects where object_name like upper ('xdb_migrateschema');
If it does NOT exist, then use the following scripts to create it and confirm results again using the above query:
conn / as sysdba
@?/rdbms/admin/dbmsxdbschmig.sql
@?/rdbms/admin/prvtxdbschmig.plb
2. Run adgrants.sql.
3. Recompile the package AD_ZD_PREP and ensure it compiles successfully.
4. Continue with the upgrade and confirm online patching is successfully enabled.
select owner, object_name from dba_objects where object_name like upper ('xdb_migrateschema');
If it does NOT exist, then use the following scripts to create it and confirm results again using the above query:
conn / as sysdba
@?/rdbms/admin/dbmsxdbschmig.sql
@?/rdbms/admin/prvtxdbschmig.plb
2. Run adgrants.sql.
3. Recompile the package AD_ZD_PREP and ensure it compiles successfully.
4. Continue with the upgrade and confirm online patching is successfully enabled.
Move the adgrants.sql from MT to DB before running it.
exec ad_zd_prep.create_patch_service -- this should succeed.
Oracle EBS 12.2 - ADOP patching on a downtime mode with [ERROR] Patch service is not exist or running
ADOP patching on a downtime mode with [ERROR] Patch service is not exist or running
Please note , if you get ADOP issues on PROD please read the logs and understand the problem before executing any commands.
When applying adop patches on a downtime mode you might see an error that the patch service is not existing or running. Please note NOT all patches are meant to be applied to a downtime mode so read the readme of the patch to understand the patching process. Exception cases breaking the rule book patches can be applied to a development environment but you have to know how to deal with post adop failure cases.
In this case it is clear that the patch service is not existing.
adop phase=apply patches=1234567 apply_mode=downtime
Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:
Validating credentials.
Initializing.
Run Edition context : /p01/ebs/appl/fs1/inst/apps/ebs_app01/appl/admin/ebs_app01.xml
Patch edition context: /p01/ebs/appl/fs2/inst/apps/ebs_app01/appl/admin/ebs_app01.xml
Patch file system free space: 77.95 GB
Validating system setup.
Node registry is valid.
[ERROR] Patch service is not exist or running
[WARNING] ETCC: The following required database fixes have not been applied to node app01:
bla bla bla
Encountered the above errors when performing database validations.
Resolve the above errors and restart adop.
Fix : connect as APPS user and execute
SQL> exec ad_zd_prep.create_patch_service;
Run ADOP now and it should proceed further.
Please note , if you get ADOP issues on PROD please read the logs and understand the problem before executing any commands.
When applying adop patches on a downtime mode you might see an error that the patch service is not existing or running. Please note NOT all patches are meant to be applied to a downtime mode so read the readme of the patch to understand the patching process. Exception cases breaking the rule book patches can be applied to a development environment but you have to know how to deal with post adop failure cases.
In this case it is clear that the patch service is not existing.
adop phase=apply patches=1234567 apply_mode=downtime
Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:
Validating credentials.
Initializing.
Run Edition context : /p01/ebs/appl/fs1/inst/apps/ebs_app01/appl/admin/ebs_app01.xml
Patch edition context: /p01/ebs/appl/fs2/inst/apps/ebs_app01/appl/admin/ebs_app01.xml
Patch file system free space: 77.95 GB
Validating system setup.
Node registry is valid.
[ERROR] Patch service is not exist or running
[WARNING] ETCC: The following required database fixes have not been applied to node app01:
bla bla bla
Encountered the above errors when performing database validations.
Resolve the above errors and restart adop.
Fix : connect as APPS user and execute
SQL> exec ad_zd_prep.create_patch_service;
Run ADOP now and it should proceed further.
Oracle EBS 12.2 - adop phase=cleanup fails with "library cache pin" waits
adop phase=cleanup fails with "library cache pin" waits
When running cleanup action in adop , it hung for ever.
Lets dig deeper to see the problem
SQL> SELECT s.sid,
s.username,
s.program,
s.module from v$session s where module like '%AD_ZD%';
2313 APPS perl@pwercd01vn074 (TNS V1-V3) AD_ZD
SQL> select event from v$session where sid=2313 ;
EVENT
----------------------------------------------------------------
library cache pin
SQL>
SQL> select decode(lob.kglobtyp, 0, 'NEXT OBJECT', 1, 'INDEX', 2, 'TABLE', 3, 'CLUSTER',
4, 'VIEW', 5, 'SYNONYM', 6, 'SEQUENCE',
2 3 7, 'PROCEDURE', 8, 'FUNCTION', 9, 'PACKAGE',
4 11, 'PACKAGE BODY', 12, 'TRIGGER',
5 13, 'TYPE', 14, 'TYPE BODY',
6 19, 'TABLE PARTITION', 20, 'INDEX PARTITION', 21, 'LOB',
22, 'LIBRARY', 23, 'DIRECTORY', 24, 'QUEUE',
28, 'JAVA SOURCE', 29, 'JAVA CLASS', 30, 'JAVA RESOURCE',
32, 'INDEXTYPE', 33, 'OPERATOR',
34, 'TABLE SUBPARTITION', 35, 'INDEX SUBPARTITION',
40, 'LOB PARTITION', 41, 'LOB SUBPARTITION',
42, 'MATERIALIZED VIEW',
43, ' 7 DIMENSION',
44, 'CONTEXT', 46, 'RULE SET', 47, 'RESOURCE PLAN',
48, 'CONSUMER GROUP',
51, 'SUBSCRIPTION', 52, 'LOCATION',
55, 'XML SCHEMA', 56, 'JAVA DATA',
57, 'SECURITY PR 8 OFILE', 59, 'RULE',
62, 'EVALUATION CONTEXT',
'UNDEFINED') object_type,
lob.KGLNAOBJ object_name,
pn.KGLPNMOD lock_mode_held,
pn.KGLPNREQ lock_mode_requested,
ses.sid,
ses.serial#,
ses.userna 9 me
FROM
x$kglpn pn,
v$session ses,
x$kglob lob,
v$session_wait vsw
WHERE
pn.KGLPNUSE = ses.saddr and
pn.KGLPNHDL = lob.KGLHDADR
and lob.kglhdadr = vsw.p1raw
and vsw.event = 'library cache pin'
10 order by lock_mode_held desc
/
OBJECT_TYPE OBJECT_NAME LOCK_MODE_HELD LOCK_MODE_REQUESTED SID SERIAL# USERNAME
------------------ -------------------- -------------- ------------------- ---------- ---------- ------------------------------
PACKAGE DBMS_SYS_SQL 2 0 3822 60962 APPS
PACKAGE DBMS_SYS_SQL 2 0 2313 27404 APPS
PACKAGE DBMS_SYS_SQL 2 0 2313 27404 APPS
PACKAGE DBMS_SYS_SQL 2 0 3822 60962 APPS
PACKAGE DBMS_SYS_SQL 0 2 3821 14545
PACKAGE DBMS_SYS_SQL 0 2 3821 14545
PACKAGE DBMS_SYS_SQL 0 3 2313 27404 APPS
PACKAGE DBMS_SYS_SQL 0 3 2313 27404 APPS
Fix :
When running cleanup action in adop , it hung for ever.
Lets dig deeper to see the problem
SQL> SELECT s.sid,
s.username,
s.program,
s.module from v$session s where module like '%AD_ZD%';
2313 APPS perl@pwercd01vn074 (TNS V1-V3) AD_ZD
SQL> select event from v$session where sid=2313 ;
EVENT
----------------------------------------------------------------
library cache pin
SQL>
SQL> select decode(lob.kglobtyp, 0, 'NEXT OBJECT', 1, 'INDEX', 2, 'TABLE', 3, 'CLUSTER',
4, 'VIEW', 5, 'SYNONYM', 6, 'SEQUENCE',
2 3 7, 'PROCEDURE', 8, 'FUNCTION', 9, 'PACKAGE',
4 11, 'PACKAGE BODY', 12, 'TRIGGER',
5 13, 'TYPE', 14, 'TYPE BODY',
6 19, 'TABLE PARTITION', 20, 'INDEX PARTITION', 21, 'LOB',
22, 'LIBRARY', 23, 'DIRECTORY', 24, 'QUEUE',
28, 'JAVA SOURCE', 29, 'JAVA CLASS', 30, 'JAVA RESOURCE',
32, 'INDEXTYPE', 33, 'OPERATOR',
34, 'TABLE SUBPARTITION', 35, 'INDEX SUBPARTITION',
40, 'LOB PARTITION', 41, 'LOB SUBPARTITION',
42, 'MATERIALIZED VIEW',
43, ' 7 DIMENSION',
44, 'CONTEXT', 46, 'RULE SET', 47, 'RESOURCE PLAN',
48, 'CONSUMER GROUP',
51, 'SUBSCRIPTION', 52, 'LOCATION',
55, 'XML SCHEMA', 56, 'JAVA DATA',
57, 'SECURITY PR 8 OFILE', 59, 'RULE',
62, 'EVALUATION CONTEXT',
'UNDEFINED') object_type,
lob.KGLNAOBJ object_name,
pn.KGLPNMOD lock_mode_held,
pn.KGLPNREQ lock_mode_requested,
ses.sid,
ses.serial#,
ses.userna 9 me
FROM
x$kglpn pn,
v$session ses,
x$kglob lob,
v$session_wait vsw
WHERE
pn.KGLPNUSE = ses.saddr and
pn.KGLPNHDL = lob.KGLHDADR
and lob.kglhdadr = vsw.p1raw
and vsw.event = 'library cache pin'
10 order by lock_mode_held desc
/
OBJECT_TYPE OBJECT_NAME LOCK_MODE_HELD LOCK_MODE_REQUESTED SID SERIAL# USERNAME
------------------ -------------------- -------------- ------------------- ---------- ---------- ------------------------------
PACKAGE DBMS_SYS_SQL 2 0 3822 60962 APPS
PACKAGE DBMS_SYS_SQL 2 0 2313 27404 APPS
PACKAGE DBMS_SYS_SQL 2 0 2313 27404 APPS
PACKAGE DBMS_SYS_SQL 2 0 3822 60962 APPS
PACKAGE DBMS_SYS_SQL 0 2 3821 14545
PACKAGE DBMS_SYS_SQL 0 2 3821 14545
PACKAGE DBMS_SYS_SQL 0 3 2313 27404 APPS
PACKAGE DBMS_SYS_SQL 0 3 2313 27404 APPS
SQL> select
2 distinct
ses.ksusenum sid, ses.ksuseser serial#, ses.ksuudlna username,ses.ksuseunm machine,
ob.kglnaown obj_owner, ob.kglnaobj obj_name
3 4 5 ,pn.kglpncnt pin_cnt, pn.kglpnmod pin_mode, pn.kglpnreq pin_req
, w.state, w.event, w.wait_Time, w.seconds_in_Wait
-- lk.kglnaobj, lk.user_name, lk.kgllksnm,
6 7 8 --,lk.kgllkhdl,lk.kglhdpar
--,trim(lk.kgllkcnt) lock_cnt, lk.kgllkmod lock_mode, lk.kgllkreq lock_req,
9 10 --,lk.kgllkpns, lk.kgllkpnc,pn.kglpnhdl
from
x$kglpn pn, x$kglob ob,x$ksuse ses
, v$session_wait w
11 12 13 14 where pn.kglpnhdl in
15 (select kglpnhdl from x$kglpn where kglpnreq >0 )
and ob.kglhdadr = pn.kglpnhdl
and pn.kglpnuse = ses.addr
and w.sid = ses.indx
order by seconds_in_wait desc
/
SID SERIAL# USERNAME MACHINE OBJ_OWNER OBJ_NAME PIN_CNT PIN_MODE PIN_REQ STATE EVENT WAIT_TIME SECONDS_IN_WAIT
---------- ---------- ------------------------------ ------------------------------ -------------------- -------------------- ---------- ---------- ---------- ------------------- ---------------------------------------------------------------- ---------- ---------------
2313 27404 APPS apprnrcod05 SYS DBMS_SYS_SQL 0 0 3 WAITING library cache pin 0 701
2313 27404 APPS apprnrcod05 SYS DBMS_SYS_SQL 2 2 0 WAITING library cache pin 0 701
803 46104 APPS orarnrcod05 SYS DBMS_SYS_SQL 2 2 0 WAITED SHORT TIME control file sequential read -1 24
Other errors on the adop logs , you can also use scanlogs to verify the errors.
[ERROR] [CLEANUP 1:1 ddl_id=69120] ORA-04020: deadlock detected while trying to lock object SYS.DBMS_SYS_SQL SQL: begin sys.ad_grants.cleanup; end;
Reference
Adop Cleanup Issue: "[ERROR] [CLEANUP] ORA-04020: deadlock detected " (Doc ID 2424333.1)
Fix :
SQL> select count(1)
from dba_tab_privs
where table_name='DBMS_SYS_SQL'
and privilege='EXECUTE'
and grantee='APPS' 2 3 4 5 ;
COUNT(1)
----------
1
SQL> exec sys.ad_grants.cleanup;
PL/SQL procedure successfully completed.
SQL> select count(1)
from dba_tab_privs
where table_name='DBMS_SYS_SQL'
and privilege='EXECUTE'
and grantee='APPS' 2 3 4 5 ;
COUNT(1)
----------
0
SQL>
now adop cleanup fine without any issues.
Create Databases from Automatic Daily Backups - New Feature - Jan 2019 - Oracle Cloud Infrastructure
New Feature - Create Databases from Automatic Daily Backups
Services : Block Volume
Release Month : Jan 2019
Automatic Daily Backup feature lets you to create automated backups of OCI DB System Databases on a daily basis. With this feature you can now create a new system out of the automated daily backups.
Here is a DB System in OCI that is created and you can see there is a Automated Backup Running
Once the backup is completed you will see that it is ready for the refresh
From the backup section select Create Backup and fill in the required details
Once you start the database creation process you will see the restore process beginning immediately.
The DB System will have your database in provisioning stage
Once restore is done you will be having your new Database up and running.
This is indeed a cool feature. I bet your DBA will love to see this , he can now start looking into other useful stuffs.
Services : Block Volume
Release Month : Jan 2019
Automatic Daily Backup feature lets you to create automated backups of OCI DB System Databases on a daily basis. With this feature you can now create a new system out of the automated daily backups.
Here is a DB System in OCI that is created and you can see there is a Automated Backup Running
Once the backup is completed you will see that it is ready for the refresh
From the backup section select Create Backup and fill in the required details
Once you start the database creation process you will see the restore process beginning immediately.
The DB System will have your database in provisioning stage
Once restore is done you will be having your new Database up and running.
This is indeed a cool feature. I bet your DBA will love to see this , he can now start looking into other useful stuffs.
In-transit encryption for boot and block volumes - New Feature - Jan 2019 - Oracle Cloud Infrastructure
New Feature : In-transit encryption for boot and block volumes
Services : Block Volume
Release Month : Jan 2019
Data is often considered less secured when in movement. It could be across two servers , two data center , between two services, between cloud and on-premise or between two cloud providers. Wherever data is moving , data protection methods should be implemented for in transit data that are critical. While organization care more about data at rest , protecting sensitive data in-transit should also be given high importance as attackers find new methods to steal data.
Encryption is the best way to protect data in-transit. This is done by encrypting the data before sending it , authenticating the end points and decryption once the data is received.
OCI block volume service encrypts all block volumes at rest and their backups as well using AES Advanced Encryption Standard algorithms with 256-bit encryption. Data moving between the instance and the block volume is transferred over an internal and highly secure network. This transfer could be encrypted with this feature announcement for paravirtualized volume attachments on virtual machines.
Optionally you can use the encryption keys managed by the key management service for volume encryption. if there is no service used oracle provided encryption key is used and this is for both data at rest and in-transit.
Here above when you specify the key for the block while creating then the same will be used for in-transit as well.
Consistent device paths for block volumes - New Feature - Jan 2019 - Oracle Cloud Infrastructure
New Feature : Consistent device paths for block volumes
Services : Block Volume
Release Date : Jan 2019
With this feature you can now select a device path that will remain consistent between instance reboots. though this is an optional feature it is recommended to use the device path as you can refer to the volumes when create partitions , creating file systems , mounting file system , you can also specify this option in /etc/fstab file for automatically mounting volumes on the instance boot.
Operating System Linux Images that are released by Oracle prior to November 2018 would not be able to use this feature. Windows based , Custom images and Partner images are not supported.
To verify if consistent device path support is available on your instance , login into your environment and do a "ll /dev/oracleoci/oraclevd*" , if you see a list of devices then it is supported else if you get a message "no such file or directory" then it is not supported.
Attaching a device path in the console is done simply by selecting a device path for the block volume. Once attached you can verify the block volume from the summary page
Device Path : /dev/oracleoci/oraclevdb
update : /etc/fstab --- /dev/oracleoci/oraclevdb1 /oradata ext3 defaults,_netdev,noatime 0 2
mkdir /oradata
mount /dev/oracleoci/oraclevdb1 /oradata
Services : Block Volume
Release Date : Jan 2019
With this feature you can now select a device path that will remain consistent between instance reboots. though this is an optional feature it is recommended to use the device path as you can refer to the volumes when create partitions , creating file systems , mounting file system , you can also specify this option in /etc/fstab file for automatically mounting volumes on the instance boot.
Operating System Linux Images that are released by Oracle prior to November 2018 would not be able to use this feature. Windows based , Custom images and Partner images are not supported.
To verify if consistent device path support is available on your instance , login into your environment and do a "ll /dev/oracleoci/oraclevd*" , if you see a list of devices then it is supported else if you get a message "no such file or directory" then it is not supported.
Attaching a device path in the console is done simply by selecting a device path for the block volume. Once attached you can verify the block volume from the summary page
Device Path : /dev/oracleoci/oraclevdb
After attaching the device then from the operating system you can create a partition using the device path.
fdisk /dev/oracleoci/oraclevdb
mkfs.ext3 /dev/oracleoci/oraclevdb1update : /etc/fstab --- /dev/oracleoci/oraclevdb1 /oradata ext3 defaults,_netdev,noatime 0 2
mkdir /oradata
mount /dev/oracleoci/oraclevdb1 /oradata
Steal Percentage on the Virtual Machine
Recently have been working with a customer who was running high workload on virtual machine. After a complete review , DBAs have identified the database is running non optimized codes/sql/programs. During high peak load we observed very high Steal %
In a virtual environment, CPU cycles are shared across virtual machines on the server. If your virtual machine displays a high %st in top (steal time), this means CPU cycles are being taken away from your virtual machine to serve other purposes. You may be using more than your share of CPU resources or the physical server may be over-sold. Move the virtual machine to another physical server. If steal time remains high, try giving the virtual machine more CPU resources.
In this case this database was not having any noisy neighbor and it was the same machine fighting against itself. The reason was over-provisioning. The underlying hardware had 48 CPU but virtual machine was over provisioned for 64 CPU.
To resolve the issue , virtual machine was migrated from 48 CPU bare metal to 82 CPU bare metal.
The Steal is gone.
it is highly recommended to have a metric collection on the monitoring tool to have Steal monitored on high workload virtual machines. This will help take action and move environment to have better performance.
In a virtual environment, CPU cycles are shared across virtual machines on the server. If your virtual machine displays a high %st in top (steal time), this means CPU cycles are being taken away from your virtual machine to serve other purposes. You may be using more than your share of CPU resources or the physical server may be over-sold. Move the virtual machine to another physical server. If steal time remains high, try giving the virtual machine more CPU resources.
In this case this database was not having any noisy neighbor and it was the same machine fighting against itself. The reason was over-provisioning. The underlying hardware had 48 CPU but virtual machine was over provisioned for 64 CPU.
To resolve the issue , virtual machine was migrated from 48 CPU bare metal to 82 CPU bare metal.
The Steal is gone.
it is highly recommended to have a metric collection on the monitoring tool to have Steal monitored on high workload virtual machines. This will help take action and move environment to have better performance.
ADOPValidations EBS 12.2
ADOPValidations_detailed.log reported a warning message for one of the critical error "Cross Reference"
ValidateFSConfigFilesForCrossRef:
[WARNING]: Found invalid cross references in FS config files.
cross reference is nothing but having incorrect values in the context file of EBS 12.2
for example : run file system context file having path of patch file system
$ echo $CONTEXT_FILE
/prod/inst/fs1/inst/apps/PROD_node1/appl/admin/PROD_node1.xml
$
$ grep -i fs2 /prod/inst/fs1/inst/apps/PROD_node1/appl/admin/PROD_node1.xml
/prod/applmgr/fs2
/prod/applmgr/fs2/EBSapps/comn/adopclone_node1
$
$ grep -i fs1 /prod/inst/fs2/inst/apps/PROD_node1/appl/admin/PROD_node1.xml /prod/applmgr/fs1
/prod/applmgr/fs1/EBSapps/comn/adopclone_node1
/prod/applmgr/fs2/EBSapps/comn/util/jdk/jre/lib/sparcv9/server:/prod/applmgr/fs2/EBSapps/comn/util/jdk/jre/lib/sparcv9:/prod/applmgr/fs2/EBSapps/appl/pay/12.0.0/vendor/quantum/lib:/prod/applmgr/fs2/FMW_Home/webtier/jdk/jre/lib/sparcv9:/prod/applmgr/fs2/FMW_Home/webtier/jdk/jre/lib/sparcv9/native_threads:/prod/applmgr/fs2/EBSapps/appl/cz/12.0.0/bin64:/prod/applmgr/fs2/EBSapps/appl/sht/12.0.0/lib64:/prod/applmgr/fs1/EBSapps/comn/adopclone_node1/oui/lib/solaris
$
here on the patch file system there is a cross reference which can be very costly mistake if you ignore it. so consider having all validations reviewed.
how to fix it then ....
simple
just update the context file and upload it to the database
$ADJVAPRG oracle.apps.ad.autoconfig.oam.CtxSynchronizer action=upload contextfile= logfile=/tmp/patchctxupload.log
then run ADOP operations...
