The Business of Business is Business

Winston Churchill said: “Some regard private enterprise as if it were a predatory tiger to be shot. Others look upon it as a cow that they can milk. Only a handful see it for what it really is – the strong horse that pulls the whole cart.”

Have a glimpse at Nasdaq’s article: Upside to Oracle’s Dominant Position in Database Software to double check that namely Oracle is the strong horse. “Oracle continues to maintain its leadership position in the database software market with a share of close to 50%, thanks to constant innovation in its database business and positive feedback from the Exadata line of servers.”

According to Gartner’s market share numbers by relational database management systems, Oracle leads worldwide RDBMS software market share and total software revenue, with its Linux and Unix garnering 74.3% and 60.7% market share respectively.

But for good horses, you need good bug control:

Let me show you an example of a perfectly well working RAC database where all of a sudden you see this “Other” activity in Grid Control:

Drilling down in EM, you witness the unorthodox event “latch: ges resource hash list”. Note that GES resources (GES = Global Enqueue Service) are accessed via a hash array where each resource is protected by a ges resource hash list child latch.

So, what have we done wrong that we have the privilege of witnessing this horrible histogram below (doesn’t the one above look like the top of the head of Bart Simpson)?

Turns out it is Bug 11690639 – High enqueue activity results in “latch: ges resource hash list” waits (Doc ID 11690639.8). Question: is the document ID named after the bug number or is it vice versa?

This is just an example. Many DBAs are nowadays mildly frustrated with the daily routine of bug fixing, looking for workarounds and possible patches. While some years ago the first thing a DBA would do for a problem was to try and tune the database, improve the SQL, etc., now most DBAs try to overrule at first the possibility of a bug.

Bugs are fixed but they re-appear. Fixing a bug can bring another one. Bugs bring systems down. You have covered the infinite availability SLA with RAC, Gloden Gate, Active Dataguard, etc. but one small bug can cause a so severe performance problem that all your efforts and diagnostic tools count for nothing.

How many Database experts talk about bugs at database events? How often do you see blog articles tagged with the word bug? Have you ever tried to look on the Internet for books on database bugs? Check out Martin Decker’s ora-solutions.net! I enjoy the site a lot!

Oracle keep all their bugs in a database called BugDB. It is a mission critical one and was the first internal application in Oracle to be upgraded with EBR. Check this one out: Online Application Upgrade of Oracle’s Bug DB with EBR!

A Bug DBA is something one would probably not like to be called (unlike a performance tuning DBA) but this phenomenon in the database world is significantly growing its importance and the faster the DBA spots out the bug the less the downtime will be. Unlike using the traditional way where anyone can open an SR with MOS/Metalink and wait for a (possible) resolution.

Bug spotting is of extreme importance nowadays as hours and days of slow performance or downtime can be devastating for the enterprise. And the work of the Bug Buster does not finish after the identification of the bug: then (s)he must decide between recommending patching or using a workaround. And all over again..

DBAs and extreme productivity

In May 2011, Harvard Business Review published Robert Pozen’s two principles on Managing Yourself: Extreme Productivity.

Principle 1: Know Your Comparative Advantage
Principle 2: It’s Not the Time You Spend but the Results You Produce

The article is CEO oriented by the same principles hold for DBAs:

First (principle 1): try to delegate DBA work to others even if you think that you are the most skillful for the task in question. The more you do at the same time the bigger the chance to make a mistake or two. Not to mention that the quality of the work will suffer.

And second (principle 2): DBA work is not measured in hours of work done by the DBA but rather in hours of downtime caused by the work of the DBA.

Many managers have the (wrong) perception that DBA productivity is derived from tools and automated (even routine) tasks. Sure, they help and impact the work but DBA productivity and efficiency are purely based on skills, knowledge and most of all experience.

You can often witness the fact that the great ideas of middle management on how to improve DBA productivity are not eventually that great. And there is a perfectly sane explanation: how do you improve something you have never done? How indeed?

You get your DBA skills as a combination of your knowledge and experience. The broader the knowledge and the longer the experience, the more skillful you are likely to become.

The knowledge part is rather tricky for DBAs. Versions of database vendors change faster than people change cars. Every new version comes with (1) a set of new features and (2) a set of obsolete features. Not to mention the set of new products developed and used around the database.

So how do you keep up with the new information (and information is not knowledge) flooding us day by day? Granted you are expected to work for 8 hours! Did I just say eight? DBAs need at least 20% white time to spend on self-training, external training, seminars and conferences, reading internet forums, white papers and blogs. They need that time to test and prove concepts, read books and magazines, attend web seminars, talk to other DBAs.

But as Oscar Wilde said: “Nothing that is worth knowing can be taught.”

The experience part is even more tricky. Companies are looking for DBAs with experience and very seldom for junior DBAs and on other hand I have personally noticed that long-term DBAs can be often categorized as rusty DBAs. You do not get much time to enjoy and stamp the experience of say Oracle 9 with Oracle 12 knocking on the door.

DBA experience is something very, very relative. You might have spent years and years working for the same company and same set of databases but that helps you specialize in a certain set of tasks. I exclude here Data Center DBAs who have more challenges and feature-facing than even desired.

So does 20 years of DBA experience count? Yes and no. Nobody needs your skills on how to perform a successful upgrade from 7.3.4 to 8.1.5. On the other hand, the routine tasks you have done 100s and 100s of time will certainly be beneficial. What matter is the ability to move out of your DBA comfort zone and look for new challenges and experience. Experience to mess up a database or two.

And I will quote Oscar Wilde again: “Experience is the name everyone gives to his mistakes.”

So, how to become an excellent DBA: a very knowledgeable and experienced one? Try to overcome The Paradox of Excellence. As Tomas and Sara DeLong say: “To achieve continued success, you must open yourself up to new learning experiences that may make you feel uncertain at best and incompetent at worst.”

It is not important how well the DBA knows the old stuff. What matters is how fast (s)he can learn the new things. And as time goes by, the productivity will increase. People will notice it and you wil make the difference.