Syed Jaffar

Subscribe to Syed Jaffar feed
Whatever topic has been discussed on this blog is my own finding and views, not necessary match with others. I strongly recommend you to do a test before you implement the piece of advice given at my blog.The Human Flyhttp://www.blogger.com/profile/03489518270084955004noreply@blogger.comBlogger301125
Updated: 6 days 23 hours ago

Oracle Exadata Expert's Handbook - is out now

Mon, 2015-07-13 03:59
I am happy to share a good news with the Oracle community today. Our latest book 'Oracle Exadata Expert's Handbook' authored by well-known Oracle gurus is hit the racks on 6-July-2015.

Contents if you wish know what has been covered in this book



Order your copy now at Pearson or Amazon

We hope the book will meet your expectations and will add more knowledge your skill set.  

Toad world's Oracle Pro for January 2015 - truly honored

Fri, 2015-01-09 12:44
Thank you Toad for honoring me with 'Toad World's Oracle Pro for January 2015'. I am indeed truly touched for the honor.


http://www.toadworld.com/platforms/oracle/default.aspx?Redirected=true

I thank my family, all friends, colleagues, Oracle community, my readers, Chris and Steve Hilker from Toad for your support and encouragement.

AIOUG annual Oracle conference - SANGAM14

Wed, 2014-10-29 00:43
All India Oracle User Group (AIOUG) annual Oracle conference Sangam14 is less than 10 days away. This is the largest Oracle conference that take place every year in different cities of India with thousand's of attendees plus over 100 different topics by many Oracle experts across the globe.

This year's SANGAM is scheduled on Nov 7,8,9 in Bangalore city. Don't let the opportunity go vain, avail/grab the opportunity if you are in India. I am super excited about the conference and look forward attending Tom Kyte's 'Optimizer master class', a full day class and also Maria's 'Oracle database in-memory option' session.

My sessions are as follow:



For more details on agenda, speakers, enrollment, visit http://sangam14.info/.

Look forward to seeing you in-person at the conference.

Oracle 11204 Clusterware upgrade - ASM glitch

Tue, 2014-08-19 04:01
Yet another tough challenge thrown at my team right after the disaster recovery (DR) simulation drill which performed barely couple of weeks ago. The new task (challenge) in hands is to upgrade the existing four cluster environments from 11.2.0.2 to 11.2.0.4 as Oracle already stopped supporting v11.2.0.2.

Although last week we had a 3 node successfully upgrade track record, we encountered ASM upgrade troubles whilst running rootupgrade.sh in a new cluster environment (7 nodes). The following error was reported during the course of rootupgrade.sh script execution:

CRS-2672: Attempting to start 'ora.asm' on 'node01' 
CRS-5017: The resource action "ora.asm start" encountered the following error: 
ORA-48108: invalid value given for the diagnostic_dest init.ora parameter 
ORA-48140: the specified ADR Base directory does not exist [/u00/app/11.2.0/grid/dbs/{ORACLE_BASE}] 
ORA-48187: specified directory does not exist 
HPUX-ia64 Error: 2: No such file or directory 
Additional information: 1CRS-2674: Start of 'ora.asm' on 'node01' failed 
CRS-2679: Attempting to clean 'ora.asm' on 'node01
CRS-2681: Clean of 'ora.asm' on 'node01' succeeded 
CRS-4000: Command Start failed, or completed with errors. 

When tried to start-up the ASM instance manually through sqlplus prompt, the following error was thrown:

SQL> 
ORA-32004: obsolete or deprecated parameter(s) specified for ASM instance 
ORA-48108: invalid value given for the diagnostic_dest init.ora parameter 
ORA-48140: the specified ADR Base directory does not exist [/u00/app/11.2.0/grid/dbs/{ORACLE_BASE}] 
ORA-48187: specified directory does not exist 
HPUX-ia64 Error: 2: No such file or directory

Sadly, there wasn't much info available about the nature of this problem. As usual, after giving it 1 hr try with different options, we opened a SR with Oracle support and agreed to rollback the upgrade from the node where the rootupgrade script failed. Luckily, this was the first node we tried and other 6 nodes were just running fine. After rolling back to the previous cluster version, ASM instance error was still persist.

To resolve the ASM instance startup issues, the following action was taken:

  • export diagnostic_dest=/u00/app/oracle
  • From active ASM instance on another node, executed the following statement:
    • SQL> ALTER SYSTEM STOP ROLLING MIGRATION;

Cause:
The problem caused an ASM instance startup issue was reported/logged as a known bug (17449823).

Workaround:
According to the MOS Doc ID (1598959.1), the bug is still being worked by the development team, they suggest the following work around on each node just before running the rootupgrade.sh script:
  • mkdir <New-GI-HOME>/dbs/{ORACLE_BASE} 
Third successful attempt
The upgrade failed in first 2 attempts, and the 3 attempt was successful and we managed to upgrade all 7 nodes from 11.2.0.2 to 11.2.0.4. It was also learnt that CRS_HOME, ORACLE_HOME, ORACLE_BASE was not unset before the runinstaller was initiated. In 3rd attempt with unsetting those parameters, upgrade went successfully.

Addendum (24-Aug-2014)
Couple of new challenges encountered in the last  upgrade task on 10 nodes.

  1. OUI window from which runInstaller was initiated got closed due to PC rebooted.
  2. Although the directory {ORACLE_BASE} created under the new GRID home, the issue were reoccurring.
Here is the solution:
  1. How to Complete 11gR2 Grid Infrastructure Configuration Assistant(Plug-in) if OUI is not Available (Doc ID 1360798.1)
  2. Ensure the diagnostic_dest is updated on ASM Spfile to the new location before running the rootupgrade.sh


References:

  • Things to Consider Before Upgrading to 11.2.0.3/11.2.0.4 Grid Infrastructure/ASM ( Doc ID 1363369.1) 
  • Things to Consider Before Upgrading to 11.2.0.4 to Avoid Poor Performance or Wrong Results ( Doc ID 1645862.1) 
  • GI rootupgrade.sh on last node: ASM rolling upgrade action failed ( Doc ID 1598959.1) 
  • bug 17449823




Disaster Recovery Simulation test - performed 30 databases failover

Wed, 2014-08-06 06:55
Successfully switched (fail-over) the role of over 30 physical standby databases  this morning as part of the Disaster Recovery (DR) simulation test. Fortunately, there were no technical glitches and hassles during the course of testing as  anticipated. It was indeed a great test and very successful one too.

The next  big challenge to the team would be reconstructing and making in sync those 30 physical standby databases whose range from 100GB to 5TB size.

Anyways, my team loving the challenges and true enjoying every moment.





EID Holidays and things to do

Wed, 2014-07-23 03:07
Looking forward to a much anticipated 9 day EID holiday break to complete the to-do-list which I have been carrying for a while now. Determined to complete some of the writing assignments that I have kept pending for a long period of time now. At the same time, will have to seek the possibilities to exploring the new features of v12.1.0.2 and Exadata as we might we going for the combination in the coming weeks for a Data Warehouse project.

Will surely blog about my test scenarios and will share the inputs on Oracle 12c new features.

I wish everyone a very happy and prosperous EID in advance.

Monthly article publication on Toad World portal

Wed, 2014-06-04 06:50
Let me quickly give you an update if you are wondering why I kept clam, not making much buzz on my blog. Well, over the past one year or so, I have been contributing monthly articles to Toad World news letter, and all my articles are published at their portal. I must encourage you all to visit Toad World website where many Oracle Experts/Gurus contributing/sharing knowledge.

Middle East & North Africa (MENA) 2014 OTN Tour - Tunis/Riyadh/Jeddah/Dubai

Mon, 2014-05-19 05:44
First ever MENA OTN Tour is scheduled From 26-May to 1-June 2014 in Tunis/Saudi Arabia/Dubai major cities.

2 Continents, 3 Countries, 5 Cities:50+ Action-Packed Oracle Sessions.
Courtesy of the Oracle Technology Network (OTN) and the ARABOUG, the inaugural 2014 OTN MENA Tour brings a star-studded cast, consisting of some of the world's best Oracle ACEs, ACE Directors and Rock Star Speakers to the region. The tour aims at sharing cutting edge knowledge and independent research in the MENA region, by accomplished Oracle experts from all over the world. This is a 100% FREE event for the benefit of the local Oracle communities.

Speakers
bjoernrostedwardroskeSyedJaffarHussainOsama MustafaMohamedHouriJimCzupyrinskitariqfarooqmikeaultjoelpereshead044InamPic1basheerkhanmohamedcharguiSouhaibAmdouniZarroukHousseMeddingVishnu

Middle East North Africa (MENA) OTN Tour - May 26 - June 1

Mon, 2014-05-12 00:53
Middle East North Africa (MENA) OTN Tour, scheduled through May 26 until June 1 in Tunisia, Saudi Arabia and Dubai countries.

More updates about agenda, registration will be published very shortly. Stay tuned.

Oracle 11g upgrade and SYS.AUD$ obstacles

Sun, 2014-02-16 06:44
During one of the Oracle 10gR2 databases manual upgrade to Oracle 11gR2 couple of days ago, to our own surprise, the upgrade procedure took nearly 5.34 hrs to finish. Although we had done earlier 100's of database manual upgrades comfortably within 2 hrs time window for individual database, the delay indeed surprised us this time. Unfortunately, there is no log that could help us understanding what is really going on. We started investigation the delay after reaching the anticipated upgrade duration.  The only new addition in this upgrade was a standby database in place this time. But, this shouldn't be the genuine reason of the delay, right?
When we queried the v$session , we found the following statement was executing:

update SYS.AUD$ set dbid = 223249361 where dbid is null and rownum <= 10000

It was understood that during the course of Oracle Server component upgrade in 11gR2, the upgrade procedure performs the following actions on the SYS.AUD$ table:

  • Add 11 new columns
  • Updates ntimestamp column
  • Update DBID column with the appropriate value for every <10000 records in a loop
Fortunately the SYS.AUD$ table in the database has over 6.4 million records. And the update statement on the table was for every 10000 records at a time which took 4.30 hrs time to complete. Subsequently the remaining components upgrade procedure took 1 hr time to complete.

When we did our upgrades earlier, auditing option was not enabled on the databases, hence, we never encountered such delayed scenario before. In order to speed up the upgrade length, when SYS.AUD$ contains huge amount records, it is best recommend the following practice:


This would certainly help us when we plan Oracle 12c upgrade in the future.



Bug 18223021 : DISK FILE OPERATIONS I/O ON HP-UX.

Tue, 2014-02-11 01:45
We have been dealing with a strange ASM  behavior over the past few months across all ASM instances in our multiple Oracle 11gR2 (11.2.0.2) Cluster envs on HPUX 11.31 OS. Even a simple and typical ASM task like, adding new disk to ASM disk group, disk group mount/unmount and querying for CANDIDATE asm disks was taking min of 20 minutes, and sometimes infinity time. This behavior caused a lot of performance degradation across all database instance in the cluster, and most of the databases suffered with 'Disk file operations i/o' wait with other consequences.

I must say, Oracle support indeed made a few unsuccessful attempts by suggesting increasing the ASM instance SGA, enabling async i/o on OS, reducing the number of disks etc to address the issue. Finally, they have logged a BUG '18223021' for our issue and we are yet to receive the fix.

Below are a few consequences of this behavior we are confronting:

  • Oracle 10g database instances running on the node where a new ASM disk being added to the existing disk group had suffered with control file locking and database hung issues
  • We ensure the disk group to which a new disk is being added is not mounted on non relative ASM instances (in other nodes)
  • When adding disk takes infinite time on one of the ASM instances, the only way was to shutdown the instance with dismount the ASM disk group subsequently to speed-up and complete the procedure
  • Most databases are suffering with 'Disk File operations I/O'

If you have MOS access, you may refer the BUG for more details.

Stay tuned for the fix and solution...

CRS-1615:No I/O has completed after 50% of the maximum interval

Mon, 2014-01-27 03:01
A few days back on one of the production cluster nodes the clusterware became unhealthy and caused instances termination on the node. If it was a node eviction, everything would have come back automatically after the node reboot, however, in this scenario, just cluster components were in unhealthy state, hence, no instance started on the node.

Upon referring the ocssd.log file, the following error found:

cssd(21532)]CRS-1615:No I/O has completed after 50% of the maximum interval. Voting file /dev/rdisk/oracle/ocr/ora_ocr_01 will be considered not functional in 13169 milliseconds

The node was unable to communicate with rest of the cluster nodes. If so, why didn't my node crashed/evicted? Isn't that question comes to your mind?.

Well, from the error it is clear that the node suffered I/O issues, and was unable to access the voting disk and started complaining (in the ocssd.log) that it can't see other nodes in the cluster. When contacted the storage and OS teams, the OS team was quick to identify the issue on PCI card. For some reasons, the I/O channel to the storage from the node was suspended for 10 seconds and re-established the connection back after 10 seconds. In the mean time, the cluster on the node become unhealthy and couldn't proceed further with any other action.

The only workaround we had to perform was restarting the CSS component using the following command:

crsctl start res ora.cssd -init

Once the CSS was started, everything back to normal state. Luckily, didn't have to do a lot of research why cluster become unhealthy or instances crashed.

AUTO Vs Manual PSU patch - when and why?

Sun, 2014-01-26 03:46
The purpose of this blog entry is to share my thoughts on AUTO Vs Manual PSU patch deployment with when and why scenario, also,  a success story of reducing (almost 50%) over all patching time we achieved in the recent times. Thought this would help you as well.

In my own perspective, I think we are one of the organizations in the Middle East who has a large and complex Oracle Cluster setups in-place. Having six (06) cluster environments, which includes production, non-production and DR, it is always certainly going to be an uphill task and challenging aspect maintaining them. One of the tasks that requires most attention and efforts more often is non-other than the PSU patch deployment for all the six environments at our premises.

We are currently in the process of applying 11.2.0.2.11 PSU patch in all our environments and the challenge in front of us is to bring down the patching time on each server. Our past patching experience says, an AUTO patching on each node needs minimum of 2 hours , and if you are going to patch a 10 node cluster, you need at least 22 hrs time frame to complete the deployment.

AUTO Patching
No doubt AUTO patching is the coolest enhancement from Oracle to automate the entire patching procedure smoothly and more importantly without much human intervene. The downside of the AUTO patch is the following, where AUTO patching for GI and RDBMS homes together fail or can't go with AUTO patch for GI and RDBMS home together:

  1. When you have multiple Oracle homes with different software owners, for example, we have an Oracle EBusiness Suite and typical RDBMS homes(10g and 11g) under different ownership.
  2. If you have multiple versions of Oracle databases running, for example, we have Oracle v10g and Oracle v11g databases.
  3. During the course of patching, if one of the files get failed to copy/rollback the backup copy due to file busy with any other existing OS process, the patch will be rolled back and will automatically restarts the cluster and the other services on the node subsequently.
Perhaps in the above circumstances, one may choose going with the AUTO patch separately for GI and then to the RDBMS homes. However, when you hit the 3 point above, the same thing gonna happen, which is time consuming, of course. While patching on particular node, the AUTO patching on GI home failed 3 times due to unsuccessfully cluster shutdown and we end-up rebooting the node 3 times.

Manual Patching
In contrast, manual patching requires heavy human intervene during the course of patch deployment. A set of steps needs to be followed carefully.

Since we got a challenge to look at all the possibilities to reduce the overall patching time, we started off analyzing various options between AUTO and Manual patch deployment and where the time is being consumed/wasted. We figured out that, after each successful/unsuccessful AUTO patching attempt, the cluster and the services on the nodes will have to restart and this was the time consuming factor. In a complex/large cluster environment with many instances, asm disks, diskgroups, it is certainly going to take a good amount of time to start off everything. This caught our attention and thought of giving a manual patching try.

When we tried manual patching method, we managed to patch the GI and RDBMS homes in about 1 hour time, this was almost 50% less than with the AUTO patching time-frame. Imagine, we finish 6 nodes in about 7 hours time in contrast to 12-13 hours time frame.

In a nutshell, if you have a small cluster environment, say 2 nodes cluster, you may feel you are not gain much with respect to saving the time, however, if you are going to patch a large/complex cluster environment, think of manual method which could save pretty huge patching downtime. At the same time, keep in mind that this method requires DBA intervene more than the AUTO patching method.


Upgrade to Oracle 12c [1Z0-060 ] exam is available now

Fri, 2013-12-13 14:16
A very quick note about the announcement of Upgrade to Oracle 12c - 1Z0-060 exam availability, it is no longer beta now. The exam has two sections, with 51 and 34 question respectively in each section with 64% and 65% passing margin. You must succeed in both sections in order to certify Oracle 12c upgrade exam.
However, I felt the exam fee is little higher, 245$, isn't expensive?

What you guys waiting for, go ahead, and upgrade your certification to the latest Oracle release. Its a good chance to upgrade to the latest release with a single upgrade exam, even if you are certified earlier with Oracle 7.3.

Here is the link for more information about the exam:

http://education.oracle.com/pls/web_prod-plq-dad/db_pages.getpage?page_id=5001&get_params=p_exam_id:1Z0-060

https://blogs.oracle.com/certification/entry/0856_18


Good luck with your exam!

Exadata - a new learning cuvrve

Mon, 2013-10-28 08:42
Perhaps I might be touch late on adopting/exploring/reacting/coping up with the exadata, but, it was always my dream and passion to work with the technologies since the announcement. As we knew Exadata is not something we can download and configure that easily on PC like those of traditional database or RAC software. Luckily and with the immense help from one of my friends, I managed to simulate Exadata setup (faked) on my new Macpro Book, 2 cell servers and 2 Oracle 12c RAC db nodes. Believe me, they are running incredibly fast and I am already on my way to explore/test the capabilities of Exadata.

I have also set some goals so that I will not become lazy or go slow on what I am doing. Planning to appear Exadata Admin and Implementation Specialist certification this December and January. If you are one of me and on the same boat as I am, feel free to contact me to discuss about Exadata stuff.

Have a good day

I am speaking @AIOUG - SANGAM13 conference

Wed, 2013-10-23 07:39
This post is specifically aimed to the folks back in India, mostly in Hyderabad or surrounding places.

Perhaps most of you probably knew that the All India Oracle Users Group (AIOUG) annual conference SANGAM13 will be taking place in Hyderabad this year. I have been trying my luck over the past couple of years to attend and present at least a session at the conference without a success. However, this time I determined not to miss the conference, and successfully proposed two sessions.

I am glad to inform you guys that I an able to make it to the conference this year and also delivering the following two technical sessions:

  • Oracle RAC 12c upgrade - best practices (9/11/2013)
  • Troubleshooting & Managing Clusterware - 360 degrees (9/11/2013)
Having said that, I am indeed super excited to visit Hyderabad after a very long time, as this will be my first visit to the city after almost a decade, yes, you read it correctly!. Though I belong to the same state, I couldn't get a chance to visit Hyderabad over the past 10 years or so. You guys can guess all my excitement. 

I strongly recommend and encourage all of you who live in India or Hyderabad not to miss the opportunity to attend the conference and meet with many world known Oracle Gurus, like, Arup Nanda, Gaja Krishna Vidyanatha, Murali Vallath, also many other local big shots.

For enrollment and more information about the conference, do visit the one of the following websites:


Look forward to seeing you folks at the conference.

Have a good time.

Oracle Open World 2013 - attending & presenting

Thu, 2013-09-12 02:36
Most of us knew Oracle Open World 2013 event is fast approaching and the event will kick start in a few days time. I haven't talked much about the OOW13 event of late, therefore, I am sharing my plans over here.

  • attending Oracle ACE Directors briefing scheduled on 19-20 Sept 2013 at Oracle HQ
  • participating and speaking in the following two Expert panel sessions 
      • Session ID: UGF9774
        Session Title: Oracle RAC SIG Meeting: Expert Panel
        Venue / Room: Moscone West - 3004
        Date and Time: 9/22/13, 15:30 - 16:30
If you are coming to OOW13, you are most welcome and I will be glad to see you in my sessions.

I can't guarantee though I will try to bring all authors of 'Expert Oracle RAC 12c'  book together for an unofficial book singing event at the book shop.

Stay tuned for more updates!!

Good luck and look forward seeing you at OOW13.



Jaffar

Deprecated database init. parameters in 12c

Sun, 2013-08-18 01:39
Here is a very short blog about all the deprecated database initialization parameters in 12c.  It is sometime becomes essential to be familiar with those database features and init. parameters that are either de-supported or deprecated in a new release of Oracle database before proceeding with the upgrade. This is because, some of the legacy applications upgrade decision either directly or in-directly depends on those changes.
To obtain a list of deprecated database parameter in 12c, use the following SQL statement:

SQL> SELECT name from v$parameter WHERE isdeprecated = 'TRUE' ORDER BY name;
 
Here is the list:
 
active_instance_count
background_dump_dest
buffer_pool_keep
buffer_pool_recycle
commit_write
cursor_space_for_time
fast_start_io_target
global_context_pool_size
instance_groups
lock_name_space
log_archive_local_first
log_archive_start
max_enabled_roles
parallel_automatic_tuning
parallel_io_cap_enabled
parallel_server
parallel_server_instances
plsql_v2_compatibility
remote_os_authent
resource_manager_cpu_allocation
sec_case_sensitive_logon
serial_reuse
sql_trace
standby_archive_dest
user_dump_dest
 
I am little bit surprised to see the sec_case_sensitive_logon parameter in the list.

Useful MOS notes on 12c

Fri, 2013-06-28 08:31
Here is the list of useful MOS notes on 12c:


  1. Oracle Database 12c Release 1 (12.1) Upgrade New Features [ID 1515747.1]
  2. NOTE:1520299.1 - Master Note For Oracle Database 12c Release 1 (12.1) Database/Client Installation/Upgrade/Migration Standalone Environment (Non-RAC)
  3. NOTE:1493645.1 - Oracle Database 12c Release 1 (12.1) DBUA : Understanding New Changes With All New 12.1 DBUA 
  4. NOTE:1503653.1 - Complete Checklist for Manual Upgrades to Oracle Database 12c Release 1 (12.1)
  5. Oracle Database 12c Release 1 (12.1) Upgrade New Features [ID 1515747.1]
  6. RMAN RECOVER TABLE Feature New to Oracle Database 12c [ID 1521524.1]
  7. How to Merge Multiple Partitions in Oracle 12C [ID 1482263.1]
  8. How to Drop/Truncate Multiple Partitions in Oracle 12C [ID 1482264.1]
  9. How to Create Interval-Reference Partitioned Tables in Oracle 12c [ID 1519042.1]


Oracle Database 12c Technology Day worldwide

Fri, 2013-06-28 03:30
Oracle quotes that 12c comes with more than 500 new features. Join the Oracle technology day event to learn firsthand how the new multitenant architecture make it easy to deploy and manage database clouds.

You will learn the following:


    • Simplify database consolidation
    • Automatically compress and tier data
    • Improve application continuity
    • Redact sensitive data
Why waiting, select your city and enroll for the event now on http://www.oracle.com/us/corporate/events/database-12c-technology-day/index.html




Pages