Senthil Rajendran

  Zero to Hero DevOps Series - CI/CD Workshop

My First Build OIC Journey Starts Here

OIC – Oracle Integration Cloud is a cloud based Integration tool. It seamlessly integrates on-prem applications with Oracle Cloud (SaaS) or other Cloud Apps (as Workday, Salesforce etc.) to other Cloud Apps.

Looks cool , here is my first app driver orchestration in OIC

 My first successful build in Oracle Cloud Services. My DevOps Journey Starts here.

Useful Script to Analyze last 30 days of OutofMemory Error on Oracle E-Business Suite OPP ( Output Post Processor ) Logs

First Spool the logfile that is required for analysis. Here I am spooling from fnd tables. you can use find command to get the last 30 days of OPP logs.

 Grafana for OCI Monitoring in Docker on Windows

 OCI CLI to List all the Database and its Host Lifecycle

 Ignore Python Warning

# compartmentList=$(oci iam compartment list)

/usr/lib/python3.6/site-packages/oci/packages/cryptography/hazmat/bindings/openssl/binding.py:179: CryptographyDeprecationWarning: OpenSSL version 1.0.2 is no longer supported by the OpenSSL project, please upgrade. The next version of cryptography will completely remove support for it.

  utils.CryptographyDeprecationWarning,

# 

here is a way to ignore the warning if you have updated packages

# export PYTHONWARNINGS="ignore"

# compartmentList=$(oci iam compartment list)

#

Well there are many ways to get Oracle EBS Release 12 installed, I am here going to write about Enterprise Deployments. 

Shared vs Non-Shared - when enterprise deployments are considered you may need multiple nodes for your internal Private tier and for your external DMZ tier. The code for EBS has to be available on all the nodes so that it can be accessed by end users through various services like http , forms , concurrent manager... The code becomes easy to maintain if it is Shared and one single repository is used by all these multi-node in an enterprise deployments. Config becomes easy , Patching becomes easy... Having them in the Non-Shared isolates the code and becomes tought to maintain it. Though EBS 12 provides various tools to keep them in sync but it gets complex as you grow. So choosing a Shared file system is a good decision for enterprise deployments.

Shared Storage that are commonly available are NFS/ZFS , in OCI we have FSS 

Here is another option that might be interesting in OCI which is attaching a volume to multiple instances.

- Concurrent write operations to block volumes are not provided so basically you can have only one instance write to it at a time. well there must be a solution for this... not one there are many... Cluster Aware Solution. There are many options listed from differnet vendors which you can find here https://en.wikipedia.org/wiki/Clustered_file_system 

Oracle Cluster File System so called as OCFS2 (version 2) is a Cluster Aware solution provided by Oracle. Similar solutions are provided by GlusterFS , Veritas , GFS2 , VMFS

In OCI there are three simple steps to get this for your Enterprise Deployments

• Create your EBS deployments and attach block volume to the middleware instances as Read/Write-Sharable
• Setup OCFS2 
• Create File Systems on OCFS2
• Mount the File System 
• Install/Clone EBS with Shared APPL_TOP Option

 Correct Way to Make Python3 Default on Linux

# update-alternatives --install /usr/bin/python python /usr/bin/python3 1

# update-alternatives --list|grep -i python

python  auto    /usr/bin/python3

EBS Upgrade Part 10 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

Now the current state of the EBS and DB is 12.1.3 / 19c multi-tenant running on OEL6.

At this stage you can move upgrade the OS from OEL6 to OEL7 and then proceed further. If upgrade is not possible you can perform a outplace migration from OEL6 to OEL7 and then proceed further.

Upgrade from 12.1.3 to 12.2

• Software / Patch Download and Software Staging
• Source Environment Review
• Backup
• Gather Database Stats -- exec dbms_stats.gather_database_stats(estimate_percent => 15);
• Implement fix : Fresh Install of EBS 12.2 File System Failed With RW-50004 On /tmp Directory (Doc ID 2311816.1) -- correct the file system
• Gather System Stats  -- exec dbms_stats.gather_system_stats
• Rapid Insall - Create Upgrade File System  
• ETCC to the base 12.2 ( weblogic bsu slowness fixed by a patch)
• Gather SYS, Fixed Object and Dictionary Statistics (required) -- Page 4-39 -- run this task and perform the validation
• Gather Schema Stats - skip
• Drop event alert triggers in custom schemas - no custom top prefixed with XX found - skipping this step
• OATM verification - verified
• Application DBA (AD) Tasks Matching - $AD_TOP/sql/adutconf.sql - skipping because version is 12.1.3
• Oracle E-Business Suite Integrated SOA Gateway - skipping because integration is not in use
• GUEST user_id - matching to 6 and end_date is null
• Customer Relationship Management Tasks - does not apply to 12.1.3
• Financials and Procurement Tasks - Advanced Collections - IEX module is Inactive in FND_APPLICATIONS
• Financials and Procurement Tasks - Subledger Accounting - XLA.B.3 is Shared - apply Patch 13420532:R12.XLA.B for R12.1.x - Patched
• Human Resource Management (HRMS) - Object Name : INSERT_POSITION_TXN_INFO -- modified the package from LONG to CLOB
• Supply Chain Management Tasks - Install Base is Inactive --skipping
• Supply Chain Management Tasks - Order Management is Shared -- affected column is with NON-NULL status --- verified
• Supply Chain Management Tasks - Product Hub is Inactive --skipping
• Supply Chain Management Tasks - Packing Hierarchy Structure Type --skipping
• Supply Chain Management Tasks - Backing Up User Defined Attributes Data for Items --skipping
• Supply Chain Management Tasks - Checking for Duplicate Records --validated , as per note and patch instructions there are no duplication rows
• select 'exec dbms_ijob.broken('||job||',true);' from dba_jobs where broken='N';  , change db to spfile , restart , unlock all the locked stats table
• Database Initialization Parameters for Oracle E-Business Suite Release 12 (Doc ID 396009.1) - validation
• create table bkup_dba_objects_pre_upg_122 as select * from dba_objects;
• Disable AOL Audit Trail (conditional)
• Shutdown Apps
• Download PSU and Apply Latest PSU to Both the RAC Database [ p29141015_121020_Linux-x86-64.zip] and run datapatch post install
• Install ETCC and Apply all Patches p29408621_12102190416_Linux-x86-64.zip
• Run DB ETCC to verify recommendations
• JDK 1.7 Upgrade
• Init.ora verified
• Set FAILED_LOGIN_ATTEMPTS to UNLIMITED
• Maintenance Mode Enabled
• Trigger Disabled
• recycle bin verification
• Apply AD 12.2 upgrade driver , merge with AD Consolidated Upgrade Patch , for adgrants.sql use the one from the merge driver folder + adadmin post install 
• prepare the CPU + PreInstall driver and apply in preinstall mode
• Review  tablespace  and TEMP
• Apply known issues for 12.2 
• Apply Consolidated Upgrade Patch 12.2.0 after merging with preinstall drivers 
• create table bkup_dba_objects_post_upg_122 as select * from dba_objects; --- 263 invalids
• Disable Maintenance Mode
• DB appsutil and autoconfig
• JRE on DB home
• Configuring the Oracle E-Business Suite Release 12.2 Application Tier
• Enable Online Patching , ETCC ( db only , mt done during staging )
• Gather system stats
• Apply GSCC patch and report review and fix violation action
• create table bkup_dba_objects_post_adop_122 as select * from dba_objects;
• Review and Extend APPS_TS_TX_DATA & APPS_TS_TX_IDX
• Review and run the fix by note 2066607.1 for AD ZD invalid objects
• Apply Online Patching adpatch
• alter system set cluster_database=true scope=spfile; / srvctl stop database -d prod ; srvctl start database -p prod
• Page 4-64 Database Initialization Parameters
• Compile Invalids
• page 4-62 GSCC reports
• DB Parameters settings
• AD and TXK for Release 12.2
• RUP 12.2.10
• Product Interoperatability Patch - oracle workflow

Post Upgrade Task

• Start services , Health Check
• Concurrent Manager Post Upgrade Programs
• check for running job and make sure all finish
• Obsolete Schema drop
• Drop dangling synonyms
• shutdown application
• Online help patch 12.2.0
• Online help patch 12.2.10

NLS task

• apply base NLS patch 12.2.0
• apply NLS and TXK AR patch
• apply NLS patch 12.2.10
• apply base NLS help patch 12.2.0 
• apply base NLS help patch 12.2.10
• wf patch

Post Upgrade Release Task

• Application Customization Migration 
• Additional Licensing
• Patch 28984562  for workflow
• Patch as per 2357700.1  for workflow
• Connection Leak Patches
• Instance Stabilization - oacore sizing 4 per node with 2gb heap , 2 forms servlet engine per node with 2gb heap
• run fs_clone force=yes
• Create Mainfest File for NLS
• hrglobal
• dummy cutover
• Mailer Configuration
• UTL Folder Update
• Node Addition
• LB Configuration

Refer to the Index Page to see the full flow of the upgrade. 

EBS Upgrade Part 9 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

Now the current state of the EBS and DB is 12.1.3 / 11.2.0.4 running on OEL6. Well to get to EBS 12.2 Database 19c there are two ways. 

• The best way is to keep 11.2.0.4 database and upgrade EBS 12.1.3 to EBS 12.2. This is stable and kind of been done for some years now.
• The recent supported way is to get the database upgrade to 19c multi-tenant architecture on EBS 12.1.3 and then run upgrade EBS to 12.2

Refer to the Index Page to see the full flow of the upgrade. 

EBS Upgrade Part 8 - 11.5.9 Database 9i RHEL 4 to EBS 12.2 Database 19c OEL 7

Now we are in 12.1.1 , we can directly move to 12.1.3 or 12.2. This step can be differed if you want to move to 12.2

Upgrade Application 12.1.1 to 12.1.3

• Complete Action as per Document 761570.1
• Database Specific Patches Applied as per Document 761570.1
• Upgrading to the OracleAS 10g 10.1.3.4  Doc ID 728531.1
• Apply R12.AD.B.DELTA.3 Patch 9239089
• mv /dev/random /dev/random_original;ln -s /dev/urandom /dev/random
• Relink Error Fix 2057857.1
• Use AutoPatch to apply  12.1.3 Patch 9239090 + post Patch
• Online Help Patch 9239095
• Apply Patch 9817770 and Patch 9966055
• Run DB and MT Autoconfig with appsutil configuration
• Run adpreclone on DB and MT
• Disable Maintenance Mode
• Start Services and Health Check
• Gather Schema Stats - Run Now and Schedule Every Week
• Configure Database RMAN and MT Backup

Post Upgrade 12.1.3 Open Issues

• Missing System Administrator Responsibility for SYSADMIN , recompile JSP
• CM Not Coming up , rebuild CM view using cpadmin
• Slow Standard Manager , run GSS
• Tablespace Space Addition
• Apply Patch 9707056 as per 1559441.1
• Apply patch as per Note 1581955.1 and 1489097.1
• Set SGA to 8 GB , enable AMM
• WF_DEFERRED rebuild as per 286394.1
• WF_CONTROL rebuild as per 1663093.1
• Start Workflow Agents
• Run Bulk Responsibilities Sync Custom Program
• Run Sync Workflow Concurrent Program

Refer to the Index Page to see the full flow of the upgrade.