Archive for the ‘Bugs’ Category

Oracle Database New Features

In Bugs, DBA, Golden Gate, OOW, Oracle database, Security and auditing on August 28, 2013 at 14:08

Eventually, is out although I always had the feeling that it will come before 12c. Patch number is 13390677. More than 5000 bugs have been fixed!


End date for is 31st of October 2013, which means that it is time to move to either or Or why not to Check the roadmap!

The release notes can be found here. As it will be the terminal patchset of 11gR2, the features introduced are the last we will see in 11g.

Check into detail the New Features of but most of all it is as follows:

1. Oracle Data Redaction is now part of 11gR2
2. The Trace File Analyzer (TFA) and Collector
3. RACcheck is now included
4. The OPTIMIZER_DYNAMIC_SAMPLING initialization parameter is set to the new value of 11
5. Total Recall: there is a new OPTIMIZE DATA clause when creating or altering a flashback data archive
6. The DES, RC4, and MD5 algorithms are desupported
7. New sqlnet.ora parameter SSL_EXTENDED_KEY_USAGE
8. New init.ora parameter: when set to true ENABLE_GOLDENGATE_REPLICATION RDBMS services used by Oracle GoldenGateare enabled.

If you would like to check the bugs fixed in check MOS Patch Set – List of Bug Fixes by Problem Type (Doc ID 1562142.1)!

Note: is not yet available for Exadata.

And finally, here is all the information about my presentations at Oracle OpenWorld 2013, welcome!

Session ID: CON2131
Session Title: DBA Best Practices for Performance Tuning in a Pluggable World
Venue / Room: Moscone South – 308
Date and Time: 9/23/13, 12:15 – 13:15

Session ID: CON1759
Session Title: The Least-Known Features of Oracle Database, Part 2
Venue / Room: Moscone South – 200
Date and Time: 9/24/13, 12:00 – 13:00

Session ID: CON1715
Session Title: The Important Things DBAs Should Do Before and After an Oracle Database Upgrade
Venue / Room: Moscone South – 236
Date and Time: 9/25/13, 17:00 – 18:00

Session ID: CON9863
Session Title: Managing Oracle Engineered Systems
Marriott Marquis – Foothill F
Date and Time: 9/26/13, 15:30 – 16:30


Restoring the Oracle database

In Bugs, Database tuning, DBA, Oracle database, RMAN on January 11, 2012 at 06:08

If you are reading this, there is high chance that someone has been breathing behind your back while you were restoring a database or mildly harassing you over the phone with the only question: when do you think that the database will be open for WebLogic connections?

Indeed when? There is no way to say precisely. You can guess based on the time the backup had taken in average but that is just an estimate. After the restore, there might be a nasty long sequence of archivelog files that has to be applied (with the hope that none is missing) and then you can open with resetlogs. Unless you hit a “feature” or two.

There are 5 major reasons why RMAN restore performance may be poor:

1. Missing statistics: Before a restore even begins the the control file is queried, datafile headers are read, media managers are initialised, the catalog is resynced, PL/SQL is generated and compiled, RMAN metadata either in the control file or the catalog is queried. All this incurs I/O against the control file, the datafile headers and the recovery catalog.

Make sure that the catalog database is analyzed and reorganized on regular basis. It also requires regular maintenance: crosscheck backup, etc.

Generate statistics on the fixed objects, with the GATHER_FIXED_OBJECTS_STATS procedure. This should be done when the database has been running for awhile with generic workload, so the information in the fixed objects reflects a reasonable state of the database load.

Gather the statistics with the following command:

exec dbms_stats.gather_fixed_objects_stats;

Believe it or now, often adding this to the RMAN script might help for both faster backup and restore:

alter session set optimizer_mode=RULE;

2. The backup is hardware multiplexed: all channels write to a single tape device. If the media manager does not support multiplexing only one backuppiece can be returned at a time. Considering the fact that each backuppiece is multiplexed with the rest of the others, this restore would require as many scans as the number of channels allocated of at most the whole backup size each (with a tape rewind after each scan).

3. Individual files or tablespaces are being restored: A restore of a single file needs a scan of potentially the whole database backup depending on where the header of the file being restored is positioned on the tape. The worst situation is when single file is being restored so that the file’s block header is the last block written to the backuppiece.

4. A different number of channels is used for restore compared to backup: The number of channels to be allocated should be equal to the number of physical tape drives. Try to do a couple of things:

Set enough big large pool: set LARGE_POOL_SIZE to at least 256M
Enable the backup_tape_io_slaves by setting BACKUP_TAPE_IO_SLAVES = true

However, some media managers will allow hardware multiplexing to a single tape and are able to parallelize the restore (in the media manager layer) such that the backuppieces are returned from a single tape to multiple channels in parallel. In such a situation it is possible to use an unpublished SET PARALLELMEDIARESTORE OFF command to make RMAN ignore this check.

      RUN { ...

5. The RMAN relation with bugs is not a new one. The “mend-it-or-end-it” principle does not play any role here. 5 years ago, Jaffar posted on his blog Known RMAN Performance Problems. It is worth checking again the MOS note 247611.1

Yet, here are some Oracle or MML bugs not fixed by January 1st, 2012:

Bug 12363733: restoring backups are very slow (reproduced both with asm and filesystem)
Bug 7573468: RMAN duplicate until time very slow
Bug 11835641: 11gr2 – RMAN restore took more time – duplicate or restore is slow
Bug 10210264: RMAN restore is slower following upgrade to 11.2
Bug 11827990: select name from v$datafile wait on “rdbms ipc reply” on rman restore
Bug 12543119: EM not using correct ASM diskgroup to restore datafiles when creating clone
Bug 11892765: please allow RMAN restore to pre-allocate file extents
Bug 9556740: RMAN job failed when using sudo
Bug 9724316: dbclone not choosing the correct controlfile for the clone
Bug 9845029: OEM clone 11gR2 ASM database fails during recovery
Bug 5381095: RMAN: poor restore performance in nocatalog mode
Bug 10093810: RMAN backup need to maintain an index for file header block for each file
Bug 9918138: RMAN restoring a compressed backup running very slow
Bug 6964596: RMAN single channel restore performance is faster than 6 channels
Bug 5462916: RMAN: recover database skip forever tbs with ORA-01157
Bug 2608118: restore of autobackup controlfile take a long time to complete
Bug 1533686: RMAN hangs when restoring via adsm with an ops target db
Bug 2911857: bad performance of RMAN duplication with large number of files (> 1500)
Bug 6770620: RMAN recover of copy database command consumes CPU

Bug 13109129: after upgrade to 11.2 compressed RMAN backups are slow
Bug 13066081: RMAN unregister database is slow/hangs
Bug 12896388: RMAN backup slower with – check logical – option on compressed tablespace
Bug 13500084: RMAN switch command is taking too long
Bug 13064833: RMAN process hangs after finished backup
Bug 13520050: RMAN session does not terminate- loops -spinning on cpu when destination is full

Some of the bugs will never be probably fixed as you can see that they may get, and are closed (or suspended) because of vendor OS problems, problem cannot be replicated, closed as duplicate, information not avaiable, etc.

Or how about those bugs with status 92: closed, not a bug. For example bug 11835641 is actually not a bug? Right.

Additional information: 11.2 Oracle Database Backup and Recovery User’s Guide. Check Part VI, Tuning and Troubleshooting.

Oracle Database new features

In Bugs, DBA, Oracle database on September 24, 2011 at 15:16

Oracle Database 11g Release 3 ( was released on the 23rd of September 2011.

For now, is available only for Linux x86 and Linux x86-64 and we have to download about 5G of data (7 files altogether). is a full installation of the Oracle Database software meaning that you do not need to install before installing

According to the ReadMe for 10404530, the list of bugs fixed for Oracle Database 11g Release 2 ( is in document 1348303.1: List of Bug Fixes by Problem Type available on the My Oracle Support Web site. Check it out please. It is not a short one. Not at all.

The complete list of new features of can be found here. They are by far much less than the new features of, here is a summery of these new features:

Oracle ACFS Snapshot Enhancements
Oracle ACFS Security and Encryption Features
Support for ACFS Replication and Tagging on Windows
Oracle LogMiner Support for Binary XML
SQL Apply Support for Binary XML
Oracle LogMiner Support for Object Relational Model
SQL Apply Support for Object Relational Model
Deprecation of Obsolete Oracle XML DB Functions and Packages
Oracle Warehouse Builder Support for Partition DML
Enhanced Partitioning Support in Oracle Warehouse Builder
Oracle Warehouse Builder External Table Data Pump Support
Oracle Warehouse Builder External Table Preprocessor Support
Compressed Table and Partition Support in Oracle Warehouse Builder
Support for PL/SQL Native Compilation

Starting with Oracle Database 11g Release 2 ( you can enter the Proxy Realm information when providing the details for downloading software updates. The proxy realm identifies the security database used for authentication. If you do not have a proxy realm, then you do not have to provide an entry for the Proxy Username, Proxy Password, and Proxy Realm fields. It is case-sensitive.

The following initialization parameter is new to Oracle Database 11g Release 2 ( AWR_SNAPSHOT_TIME_OFFSET

Starting with Oracle Database 11g Release 3 (, Oracle Universal Installer displays a new screen, Grid Installation Options, which helps in the installation of the examples software on an Oracle RAC database. The examples software is installed in the selected Oracle RAC home on all the nodes where it exists.

Nothing impressive but with a new patchset we do not actually expect new features, rather we hope that all (did I just say all) old bugs have been somehow fixed.

Unfortunately, I am waiting for the accessibility of document 1348303.1: List of Bug Fixes by Problem Type.

Exadata does not currently support Database

When downgrading from release to, the following error is raised when you run @catdwgrd.sql (reference Bug 11811073): ORA-20000: Upgrade from version cannot be downgraded to version.

Data Pump Export operations do not work if the DMSYS schema is not removed as part of the upgrade to release (reference Bug 10007411).

Oracle Database release or upgrade to Oracle Clusterware release is not supported if the or release of Oracle Grid Infrastructure for a cluster is installed in a non-shared Oracle home and the release of Oracle Grid Infrastructure for a cluster is installed in a shared Oracle home (reference Bug 10074804).

Starting with release of Oracle Database, the Data Mining Java API is deprecated.

Starting in release, configuring HTTPS with Oracle XML DB requires that you first set up SSL_CIPHER_SUITES to include SSL_DH_anon (reference Bug 8403366).

After upgrading from or to, deinstallation of the Oracle home in the previous version may result in the deletion of the old Oracle base that was associated with it. This may also result in the deletion of data files, audit files, etc., that are stored under the old Oracle base.

Database links imported from an database into a version prior to (including will not be usable in the import database. Any attempt to use a database link will cause the following ORA-600 error: ORA-00600 [kzdlk_zt2 err], [18446744073709551601]

For features not available or restricted in, click here.

For a complete list of all possible bugs and issues, check on the Open Bugs. was called Oracle Database 11g Release 3 for a short period of time but Oracle fixed the documentation.

Oracle Database Appliance and Automatic Bug Fixing in the Cloud

In Bugs, Database tuning, DB2 database, DBA, Grid Control, Oracle database on September 21, 2011 at 19:13

Mathematician Alfred North Whitehead said: “Civilization advances by extending the number of important operations which we can perform without thinking about them.”

Same holds for the Database!

Oracle have just announced the new “Oracle Database Appliance” with self managing automatic features preconfigured (simple, highly reliable, affordable database system for small/midsize enterprises and departments):

I have just gathered a list of the automated processes offered by Oracle within the Oracle database (in no order whatsoever):

– Automatic Statistics Collection
– Automatic Tuning Optimizer (ATO)
– Automatic Repair in a Data Guard Configuration
– Automatic Undo Management
– Automatic Undo Retention Tuning
– Automatic Shared Memory Management
– Automatic Space Segment Management (ASSM)
– Automatic PGA Memory Management
– Automatic Memory Management (AMM)
– Automatic Degree of Parallelism
– Automatic Storage Management (ASM)
– Automatic RAC Database Startup/Restart
– Automatic Maintenance Tasks
– Automatic Tablespace Point In Time Recovery
– Automatic Workload Repository (AWR)
– Automatic Service Registration
– Automatic SQL Tuning (my favourite!)
– Automatic Database Diagnostic Monitor
– Automatic Segment Advisor
– Automatic Diagnostic Repository (ADR)
– Automatic Checkpoint Tuning
– Automatic Maintenance Jobs
– Automatic Global Index Maintenance During DDL
– Automatic Client Failover
– Automatic OCR Backup Rotation
– Automatic Plan Capture with SPM
– Automatic Refresh of Materialized Views
– Automatic VIP failback
– Automatic Block Recover
– Automatic Disaster Recovery Fails with RMAN
– Automatic Channel Failover
– Automatic Synchronization of Oracle Text Index
– Automatic Registration of the Database with the Default Listener
– Automatic Offlining of the Datafile in Noarchivelog
– Automatic Datatype Conversion
– Automatic Tape Drive Cleaning and a Cleaning Tape in a Tape Library
– Automatic Controlfile Backup
– Automatic Eject Of Tape After Backup Using Oracle Secure Backup
– Automatic BackupSet Failover On Missing or Corrupt BackupPieces
– Automatic BMR (Block Media Recovery)
– Automatic System Tasks
– Automatic Database Performance Monitoring
– Automatic Archiving
– Automatic Propagation in Replication
– Automatic Job Scheduling
– Automatic Resume of DataPump

Quite a list I would say, right?

An excellent paper from Oracle called Oracle Database 11g vs. IBM DB2 UDB V9.7 points out the most important trend of database manageability: the self automation of the database product. Let me quote (part of) the conclusion of the paper:

“The Automatic Database Diagnostic Monitor (ADDM), SQL Advisors and Real Application Testing are just some of the unique Oracle Database 11g features that are yet unmatched by DB2 version 9.7. Oracle Database 11g is the only database product available today that automatically collects and manages historical performance data for self-management purposes, periodically and automatically analyses this data and makes tuning recommendations. Oracle Database 11g is also the only RDBMS with rich software quality management features for real workload testing. These distinct technologies are at the core of the next generation of Oracle databases that represent simplicity, ease of management and software quality management while still providing the most robust, reliable and secure of relational databases.”

How true indeed! But what we would like to see in the future is even more: how about automatic bug fixing in the Cloud? Just like this:

1. First, we set the credentials with Oracle Support: How to set the ‘My Oracle Support’ Preferred Credentials in the Grid Console? [ID 1168603.1]
2. The Oracle database creates an incident and it is transferred to Oracle Support via the Enterprise Manager Support Workbench.
3. Then it is internally verified if the problems is bug related.
4. If it is a bug then Oracle’s own BugDB checks for a patch or workaround which fixes the bug.
5. If there is a patch available, then the patch is automatically uploaded to the clients cloud environment and then applied (online of course)!
6. If there is a workaround with an init.ora parameter, then the “alter system” command is automatically applied, as what Oracle Support can remotely run in the client’s database is controlled by the client with a new init.ora parameter called mos_cloud_permission_level.

That is what I call automation!

Note that something similar is even now offered by Oracle via the SQL Repair Advisor. It is a semi-automatic patching of SQL statements throwing ORA-600 or ORA-7445. But here by patch Oracle mean more of an SQL transformation than a standard patch downloadable from MOS.

The Business of Business is Business

In Bugs, Database tuning, DBA, Grid Control, Oracle database, RAC on July 11, 2011 at 21:58

Winston Churchill said: “Some regard private enterprise as if it were a predatory tiger to be shot. Others look upon it as a cow that they can milk. Only a handful see it for what it really is – the strong horse that pulls the whole cart.”

Have a glimpse at Nasdaq’s article: Upside to Oracle’s Dominant Position in Database Software to double check that namely Oracle is the strong horse. “Oracle continues to maintain its leadership position in the database software market with a share of close to 50%, thanks to constant innovation in its database business and positive feedback from the Exadata line of servers.”

According to Gartner’s market share numbers by relational database management systems, Oracle leads worldwide RDBMS software market share and total software revenue, with its Linux and Unix garnering 74.3% and 60.7% market share respectively.

But for good horses, you need good bug control:

Let me show you an example of a perfectly well working RAC database where all of a sudden you see this “Other” activity in Grid Control:

Drilling down in EM, you witness the unorthodox event “latch: ges resource hash list”. Note that GES resources (GES = Global Enqueue Service) are accessed via a hash array where each resource is protected by a ges resource hash list child latch.

So, what have we done wrong that we have the privilege of witnessing this horrible histogram below (doesn’t the one above look like the top of the head of Bart Simpson)?

Turns out it is Bug 11690639 – High enqueue activity results in “latch: ges resource hash list” waits (Doc ID 11690639.8). Question: is the document ID named after the bug number or is it vice versa?

This is just an example. Many DBAs are nowadays mildly frustrated with the daily routine of bug fixing, looking for workarounds and possible patches. While some years ago the first thing a DBA would do for a problem was to try and tune the database, improve the SQL, etc., now most DBAs try to overrule at first the possibility of a bug.

Bugs are fixed but they re-appear. Fixing a bug can bring another one. Bugs bring systems down. You have covered the infinite availability SLA with RAC, Gloden Gate, Active Dataguard, etc. but one small bug can cause a so severe performance problem that all your efforts and diagnostic tools count for nothing.

How many Database experts talk about bugs at database events? How often do you see blog articles tagged with the word bug? Have you ever tried to look on the Internet for books on database bugs? Check out Martin Decker’s! I enjoy the site a lot!

Oracle keep all their bugs in a database called BugDB. It is a mission critical one and was the first internal application in Oracle to be upgraded with EBR. Check this one out: Online Application Upgrade of Oracle’s Bug DB with EBR!

A Bug DBA is something one would probably not like to be called (unlike a performance tuning DBA) but this phenomenon in the database world is significantly growing its importance and the faster the DBA spots out the bug the less the downtime will be. Unlike using the traditional way where anyone can open an SR with MOS/Metalink and wait for a (possible) resolution.

Bug spotting is of extreme importance nowadays as hours and days of slow performance or downtime can be devastating for the enterprise. And the work of the Bug Buster does not finish after the identification of the bug: then (s)he must decide between recommending patching or using a workaround. And all over again..

Auditing vs. Performance in the Oracle Database

In Bugs, Database tuning, DBA, Oracle database, Security and auditing on May 12, 2011 at 02:24

You show this (part of a) AWR report to the DBA and he proudly concludes: disable auditing, it is killing the performance! And thus, quite often Oracle database auditing is not enabled. And here are the 3 main reasons why auditing is not turned on:

– DBAs, developers, etc. are not familiar with this feature: For those who are not familiar with auditing, I suggest Tim Hall’s and Pete Finnigan’s articles: Auditing in Oracle 10g Release 2 and Introduction to Simple Oracle Auditing.
– Security is not considered important and necessary: For those who do not consider auditing important, I wish them luck. They are anyway not interested in what I have to say..
– Performance is being hit by enabling auditing: For the ones having issues with performance when auditing is enabled, here is something.

There are 3 major reasons why performance suffers when auditing is enabled: too much is being audited, AUD$ still hangs in the SYSTEM tablespace and surprise, surprise: the Oracle bugs.

1. Too much is being audited. If it is a new database, spend some time with all parties involved on what to audit. The reality however is something like that: go-live day is getting closer, oh do we have auditing enabled? How do you enable it, can you give me the command please. And it should not go like that. You first decide on the value of audit_trail and then audit what is really needed, do not audit repetitive commands that generate too many inserts into the AUD$ table for it can grow very fast indeed.

Have a look at this thread from Pete Finnigan’s site called Performance Impact of Auditing.

If it is an existing database, check first what is being audited. To find out system audited stuff run the following:

select * from DBA_PRIV_AUDIT_OPTS
union all
select * from DBA_STMT_AUDIT_OPTS;

Note that the difference between the two views above is very small and I have not found yet a place with explanation about the difference. The documentation says that DBA_STMT_AUDIT_OPTS describes the current system auditing options across the system and by user while DBA_PRIV_AUDIT_OPTS describes the current system privileges being audited across the system and by user. Puzzled? Me too.



For the auditing options on all objects, check DBA_OBJ_AUDIT_OPTS.

Check the Oracle 11gR2 documentation for the Recommended Audit Settings.

2. AUD$ still hangs in the SYSTEM tablespace. The system tablespace might be fragmented. Starting 11gR2, Oracle supports moving the AUD$ table out of the SYSTEM tablespace. But first, noaudit your policy or stop the auditing.

If using 11.2.0 and above follow the documentation instruction.

If still running 11.1.0 or a below, here is how to do it:

create tablespace AUDIT_DATA datafile ...;
create table AUDX tablespace AUDIT_DATA as select * from AUD$;
rename AUD$ to AUD$$;
rename AUDX to AUD$;
create index i_aud2 on AUD$(sessionid, ses$tid) tablespace AUDIT_DATA;

Remember to purge the records on regular basis. Do not just delete them but move them to a centralized auditing repository. Use the new DBMS_AUDIT_MGMT package. Check Tim Hall’s instructionon how to purge audit trail records. In urgent cases, it is safe to run truncate table AUD$;

If you use FGA, remember to move also FGA_LOG$ away from the SYSTEM tablespace:

    audit_trail_type           => DBMS_AUDIT_MGMT.AUDIT_TRAIL_FGA_STD,
    audit_trail_location_value => 'AUDIT_DATA');

And I would recommend this article by Martin Widlake: Why is my SYSTEM Tablespace so Big?! The 3rd SYSTEM table after SYS.AUD$ and SYS.HISTGRM$ that I have seen to grow up is SYS.IDL_UB1$.

3. Oracle bugs. If you enable auditing you might get several bugs for free, most old ones should be fixed in, don’t know about the new ones 🙂

20 years ago, Bug 52646: audit trail degrades performances too much was solved in Oracle 6:

Well, you still have many similar ones. As of today, all these bugs have empty field for “Fixed in Product Version”. And this is not the complete list!

Bug 10347785: huge version count for insert into sys.aud$ causing library cache: mutex x/hang
Bug 504968: ora-600[17051] and database crash when writing to audit table
Bug 11901734: dbms_audit_mgmt audit trail cleanup cannot keep up with aud$ volume
Bug 8236755: ora-00060 occurs while updating aud$ table.
Bug 6159102: export session spins when logging off in repeated update sys.aud$ statements
Bug 6334058: deadlock with ora-00060 while updating sys.aud$ and auditing on select is enable
Bug 4405301: too many entries in aud$ when sequence is queried and audit is by session
Bug 1257564: noaudit does not turn off auditing of database (very nice indeed!)

I wish Oracle will create one bug called “Performance issues with the AUD$ table”, solve it so finally no one complains about the performance of one simple table which in my opinion is not even a real dictionary table.

Question: In RAC, sessions from both/all nodes are being audited. Will AUD$ hot blocks “stuffed” with new data be ping-ponged via the interconnect?

Oracle and computers with zero CPUs

In Bugs, DBA, Grid Control, Oracle database, OS on January 20, 2011 at 20:50

How do you license a computer with zero CPUs? Of course, there are no such computers but still according to Oracle’s Grid Control this is not always the case 🙂 Have a look:

These hardware inventions should not bring us any cost, shouldn’t they 🙂

It is worth asking the question are there any computers with zero IO devices and NULL CPU boards:

Looks like also that the number of CPU boards cannot be visible for some computers, not to mention that although the documentation claims that “MGMT$CSA_COLLECTIONS displays top-level information about all client configurations”, I can still see nothing from this view:

You notice that this is in Grid Control 11g? I remember similar cases in 10g as well, here is an old snapshot of mine:

When will computers run without CPUs? But it is like a person without a brain says a colleague of mine.

And finally, is Oracle wrong or did HP forget to put a ventilator in the computer 🙂

Oracle’s maths

In Bugs, Oracle database on January 5, 2011 at 21:08

I have seen very strange computational bugs with Oracle, well most fixed already to my knowledge, but here is something that still makes me wonder.

The indeterminate form zero on the power of zero has been discussed for almost 200 years. You can google about it and decide for yourself. Here is what Oracle gives for the expression in

Another interesting issue is about remainders when the divisor is zero. Of course, when you divide say twenty with zero, you cannot even speak about what the remainder is, right? Here is what Oracle has to say:

I still wonder is natural logarithm of zero on the power of zero really zero 🙂

First time I noticed this was in Oracle 7, I tried again yesterday in 11gR2:

The quality of the Oracle database product

In Bugs, DBA, Oracle database, RAC, Replication on January 4, 2011 at 10:19

Two major things have recently caught my attention. They are more like strategic and of long term importance for the Oracle database.

1. Real online patching and database upgrades are still happening mostly on power point presentations. They should not be dependent on database types, patch type, version, replication, etc. and should be in DBA terms simple.

Top Google searches for Oracle database live upgrades relate rather to certification upgrades, not database upgrades:

Does Oracle have zero downtime maintenance, have a look:

Is Oracle Golden Gate similar to QREP in IBM’s DB2?

2. Product quality of all database components is getting worse and worse all the time.

– Look at the bug fixes: they are just too many!
– The following three components throw too many errors and bugs: all types of replication, RAC and Grid Control
– ORA-600 pops up too often nowadays!