Few Exadata MOS Docs to review

If you have MOS login credentials and managing Exadata database machines, below is the list of few MOS Doc which is worth reading:

• 888828.1, "Exadata Database Machine and Exadata Storage Server Supported Versions"
• 1070954.1, "Oracle Exadata Database Machine Exachk or HealthCheck"
• 1353073.2, "Exadata Diagnostic Collection Guide"
• 1500257.1, " Exadata Write-Back Flash Cache - FAQ"
• 1553103.1, "dbnodeupdate.sh and dbserver.patch.zip: Updating Exadata Database Server Software using the DBNodeUpdate Utility and patchmgr"
• 1589868.1, "Procedure to check for corrupted root file system on Exadata Storage Servers and Linux database servers"

Oracle EBS Suite blank Login page - Post Exadata migration

As part of EBS database migration on Exadata, we recently deployed a brand new Exadata X5-2L (eighth rack), and migrated an Oracle EBS Database from the typical storage/server technologies.

Below are the environment details:

EBS Suite 12.1.3 (running on two application servers with hardware LOAD BALANCER)
Database : 11.2.0.4, RAC database with 2 instance

After the database migration, buildxml and autoconfig procedures went well on both application and database tiers. However, when the EBS login page is launched, it came out as just blank page, plus, apps passwords were unable to change through the typical procedure. We wonder what went wrong, as none of the procedure gave any significant failure indications, all went fine and we could see the successful completion messages.

After a quick initial investigation, we found that there is an issue with the GUEST user, and also found that the profile was not loaded when the autoconfig was ran on the application server. In the autoconfig log file, we could see the process was failed to update the password (ORACLE). We then tried all workaround, the recommended on Oracle support and other websites. Unfortunately, none of the workarounds helped us.

After almost spending a whole day, investigating and analyzing the root cause, we looked at the DB components and status in dba_registry view. We found the JSserver Java Virtual Machine component INVALID. During Exadata software deployment at RDBMS patch installation phase, there was an error while applying the patch due to conflict between the patches.

Without further wasting a single second, we ran the @catbundle and followed by ultrp.sql.

Guess what, all issues disappeared. We run the autoconfig on the application servers. After which we could change the app user password and we could see the Login page too.

CONCLUSION
When you face EBS blank login page, and none of the workarounds fix the issues, its a good idea to have a look at dba_registry for any INVALID components.

It was quite a nice experience.

This really 

Exadata X5-2L deployment & Challenges

A brand new Exadata X5-2L eighth rack (I knew the latest is X7 now, but it was for a POC, so no worries) has been deployed recently at a customer for Oracle EBS Exadata migration POC purpose. Its wasn't an easy walk in the park as I initially presumed. There were some challenges (network, configuration) thrown during the migration, but, happily overcome and had it installed and EBS database migration completion.

So, I am going to share yet another Exadata bare metal deployment story explaining the challenges I have faced, and how they are fixed.

Issue 1) DB network cable issues:
After successful execution of the elasticConfig, all the Exadata factory IP addresses have been set to client IPs. Though the management network was accessible from the outside, client network was not accessible. When verified with the network team about enabling the ports on the corporate switch, they confirmed that the ports are enabled, however, the connection is showing as not active and asked to us investigate the network cables connected to the DB nodes. When we verified the network cables ports, we didn't find any lights flashing and after an extensive investigation (Switch ports, SFP on Exadata and Corporate switch, checking the cables status), it was found that the cables pin direction was not properly connected. Also, found that the network bonding interfaces (eth4 and eth5) were not up, confirmed from ethtool eth1 command. After fixing the cables, and bringing up the interfaces (ifup eth4 & eith5), we could see that cables are connected properly and we can also see the lights on the ports.

$ethtool eth4 (shows the interfaces were not connected)
Settings for eth4:
        Supported ports: [ TP ]
        Supported link modes:   100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Advertised link modes:  100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Speed: Unknown!
        Duplex: Unknown! (255)
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: external
        Auto-negotiation: on
        MDI-X: Unknown
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: no

Issue 2) Wrong Netmask selection for client network:
After fixing the cable issues, we then continued with onecommand execution. During the validation it failed because of different netmask for client network (under the cluster information section). The customer unfortunately made a mistake in the client network netmask selection for cluster settings, so there was a difference in the client netmask value for client and cluster. This was fixed by modifying the netmask value in the ifcfg-boneth0 file (/etc/sysconfig/network-scripts), restart the network services.

Issue 3) Failed eight Rack configuration (rack type and disk size):
Since the system was delivered somewhere during end of August 2015, no one actually knows exactly the disk size and rack model. The BOQ (Bill of quantity) for the order only shows X5-2 HC storage. So, there was wrong selection in the OEDA for Exadata rack and disk size. Instead of 4TB disk size, it was selected as 8TB disk size and instead of Elastic configuration, fixed Eighth rack was selected. This was fixed by rerunning the OEDA with the correct options.

Issue 4) Cell server IP issues:
There was another obstacle faced while doing the cell connectivity (part of onecommand). Cell server IPs were not modified by the elasticConfig. Fortunately, I found my friend blog on this topic and quickly fixed the issue. This is why I like to blog all the technical issues, who knows, this could solve someone pains.

http://blog.umairmansoob.com/exadata-deployment-error-cell-02559/

Issue 5) SCAN Listener configuration:
Cluster validation failed due to inconsistent values for scan name. During the investigation of various issues, private, public & scan IPs are put in the /etc/hosts file. So, while configuring LISTENER_SCAN2 and SCAN3, this issue happened. This was fairly understandable. Due to 3 entries of scan values in the /etc/hosts file this happened. Upon a quick google about the issue, the following blog helped me to fix the issue

https://learnwithme11g.wordpress.com/2010/09/03/how-to-add-scan-listener-in-11gr2-2/

Finally, I have managed to deploy the Exadata successfully and perform the Oracle EBS database migration . No doubt, this experience really made me strong in network and other areas. So, every challenges comes with opportunity to learn.

I thank those individuals who really write blogs and share their experience to help Oracle community.

There is still one open issues which is yet to be resolved. A slow sqlplus and db startup. I presume this is due to heavy resource utilization over the server. Yet to resolve the mystery. Stay tuned for more updates.

Update : 06-Nov-2017.

sqlplus slowness resolved by increasing the hugepages from 26gb to 96GB.

Whats new in Exadata X7

Oracle announced its new Exadata Database Machine X7 during OOW 2017.  Lets walk through quickly about the key features of X7.


Key features

• Up to 912 CPU core and 28.5TB memory per rack
• 2 to 19 DB servers per rack
• 3 to 18 Storage servers per rack
• Maximum of 920TB flash capacity
• 2.1PB of disk capacity
• Delivers 20% faster throughput from earlier models
• 50% more memory capacity from earlier models
• 10TB size disk. (10TB x 12 = 120TB RAW per storage server). The only system in the market today with 10TB disk capacity
• Increased OLTP performance : about 4.8 million reads and about 4.3 million writes per second
• Featuring an Intel Skylake processor with 24 cores 
• Enhanced Ethernet connectivity: supports over 25GbE
• Delivers in-memory performance from Shared Storage 
• OEDA CML interface
• New Exadata Smart Software : Exadata 18c

 

http://docs.oracle.com/cd/E80920_01/DBMSO/exadata-whatsnew.htm#DBMSO-GUID-2FA29E52-D72B-4235-8B1E-57B38966EB11 

Exadata migration

Had a wonderful Sangam16 conference in India, and received much applaud for the two presentations delivered,  Oracle 12c multitenancy and Exadata migration best practices.

After a very short trip to India, life started to be business as usual again, and become busy. Was fully occupied with multiple assignments: Oracle EBS database health check assessment at a client for 2 days, GI/RDBMS/PSU deployments on Oracle Sun Super Cluster M7, Exadata configuration preparation and 9 databases migration to Exadata during the week-end.

Over the last week-end, we (me and my colleague) were pretty busy with 9 databases migration to Exadata. There were a few challenges , and learned a few new things too. I would like to discuss couple of scenarios that were interesting:

One of the databases had corrupted blocks, and the expdp was keep failing with ORA-01555: snapshot too old: rollback segment number  with name "" too small. Our initial thoughts were tuning undo_retention, increasing the undo tablespace, setting an event, etc. Unfortunately, none of the workarounds helped in the situation. We then cameacross a MOS note which explains that an ORA-01555 with "", no rollback segment name is probably due to corrupted blocks. After applying the solution explained in the note, we managed to export/import the database successfully. My colleague has blogged about the scenario at his blog: http://bit.ly/2fBOxm7

Another database is running on Windows x86 64-bit, and its full of LOBs, hence, the datapumps (expdp) took significant time, as NFS filesystem used to store the  dump file. We then thought of doing direct RMAN restore from source to target, as the database on Windows x86 64bit and Linux x86 64-bit are the same (Litten) Endian formats. As per one of the MOS notes, we can also do the Data Guard setup, and do RMAN restore. However, RMAN recovery would fail with ORA-600, as cross platform redo conversion won't be possible. We are now thinking of taking a cold backup (consistent) and do a complete restore with reset logs option.

Stayed tuned for more updates on this.

Upgrade yourself from DBA to DMA

Be it any profession, learning never ends. From my very own personal perspective and exprience, I strongly beleinv in one who upgrade his skills always stays a step ahead from the rest. To stay competitive, it is necessary to upgrade our skills according to the market demand.

For DBA folks who aim and dream to learn and upgrade their knwoledge with Oracle Exadata technologies, I wrote a 5 series article to Toad, below are the links to the articles:

Exploring Exadata - for experienced non-Exadata DBAs (Part I)

Exploring Exadata - for experienced non-Exadata DBAs (Part II)

Exploring Exadata - for experienced non-Exadata DBAs (Part III)

Exploring Exadata - for experienced non-Exadata DBAs (Part IV)

Exploring Exadata - for experienced non-Exadata DBAs (Part V)

Below are few more Exadata related stuff.

Managing & Troubleshooting Exadata - Upgrading & Patching Exadata

Shrink/Grow Exadata diskgroups

Storage difference between 2 identical Exa boxes. How and why?

Pass on the knolwedge to the community.

Happy reading.

Storage difference between 2 identical Exa boxes. How and why?

We noticed around 1.6TB storage difference between two Eight (1/8) Exadata boxes while configuring Data Guard. Wondered what went wrong. The Exa box configured for DR was around 1.6TB short compare to the other Exa box. Verified the lun, physical disk and grdidisk status on a cell, which showed active/online status. The tricky part on Exadata is, everything has to be active/online across all cell storage servers. We then figured-out that grid disk status on the 3rd cell storage server was inactive. After making them active on the 3rd cell server, everything become normal, i mean, the missing 1.6TB space appeared.

When you work with Exadata, you need to verify all cell storage servers to confirm the issue, rather than just query things over just one cell server.

Shrink/Grow Exadata diskgroups

One of the important tasks that I foresee after an initial Exadata deployment is, mostly prior to DB in production, is to balance/resize the Exadata diskgroups (DATA & RECO).  Generally, the space is distributed as 80(DATA), 20(RECO) or 40(DATA), 60(RECO), depending on the database backup option you choose while deploying. In one of our Exadata setups, we don't need such a huge RECO size, hence, we shrunk the RECO size and increased the DATA diskgroup size. I am pretty sure, many of you might have done and want to do the same. However, shrinking/rebalancing the space is not like a normal ASM resize operation on Exadata, it needs some special consideration and tasks. The following Oracle Support Notes has the better explanation and examples to achieve the task.

Example of dropping RECO diskgroup and adding the space to DATA diskgroup (Doc ID 1905972.1)
NOTE:1465230.1 - Resizing Grid Disks in Exadata: Example of Recreating RECO Grid Disks in a Rolling Manner
How to increase ASM disks size in Exadata with free space in cell disks (Doc ID 1684112.1)
Resizing Grid Disks in Exadata: Examples (Doc ID 1467056.1)

Exadata - a new learning cuvrve

Perhaps I might be touch late on adopting/exploring/reacting/coping up with the exadata, but, it was always my dream and passion to work with the technologies since the announcement. As we knew Exadata is not something we can download and configure that easily on PC like those of traditional database or RAC software. Luckily and with the immense help from one of my friends, I managed to simulate Exadata setup (faked) on my new Macpro Book, 2 cell servers and 2 Oracle 12c RAC db nodes. Believe me, they are running incredibly fast and I am already on my way to explore/test the capabilities of Exadata.

I have also set some goals so that I will not become lazy or go slow on what I am doing. Planning to appear Exadata Admin and Implementation Specialist certification this December and January. If you are one of me and on the same boat as I am, feel free to contact me to discuss about Exadata stuff.

Have a good day

Series: Oracle Exadata Command Reference by Arup Nanda

A very famous and world known Oracle expert Mr. Arup Nanda has written a four series article about Oracle Exadata technologies. The four-part series contains the following:

• Part 1: Jumpstarting on Exadata – a quick primer on Exadata, its components and inner workings. If you are new to Exadata, this installment is definitely for you.
• Part 2: Command Categories, Configuration, and Basic Commands – an overview of the categories of commands you need to master, and a look at the different configuration activities after you get a brand-new Exadata rack
• Part 3: Storage Management – a detailed explanation of the CellCLI and DCLI commands, all their parameters and options, and how they are used for managing storage cells
• Part 4: Metrics and Reporting – a system is not very useful unless you can measure its effectiveness and fine-tune it. This installment will explain the commands related to gathering statistics and reports on Grid Control and how they are used for troubleshooting.

Visit the following URL to read the articles.

http://www.oracle.com/technetwork/articles/oem/exadata-commands-intro-402431.html

Thank you Arup once again for your excellent work and contribution towards the Oracle community. We indeed love and admire your professionalism.

Happy reading...

Jaffar