Overview

In this section we will configure and manage the Exadata storage cells. In exadata the storage cells are separate servers with their own operation system, mini oracle database and of course a storage. The facilitate the smart scan capabilities of exadata, by being able to scan through an oracle block and provide ONLY the necessary rows to a server process. Since they provide only the rows, that information cannot “live” in the SGA, but only in the PGA.

You can see basic architecture of the physical, LUN, celldisk and Grod Disks:

A more detailed view of a disk, you can see also below:

Configuration

In this section we will configure and re-configure certain feature of the cell storage server.

Enable Mail Notifications

To enable Mail notification from a certain cell, we can use the following command:

Enable Mail Notification

However since the mail doesn't exist, we will recieved the following error if we try to validate it:

Validate Mail

You can also validate the whole configuration using the following command:

Validate Exadata Configuration

Management

List Cell Processes

The storage cell server has couple processes:

RS (Restart Server) - Used to start and shut down the Cell server (cellsrv) and Management Server (MS)

MS (Management Server) - Provides exadata cell management and configuration. Cooperates with the cellcli interface. Also send alerts and gather statistics in addition to these collection by the cellsrv

Cellsrv (Cell server) - Primary exadata component, provides most of the exadata storage services. Primary Cellsrv communicates with the oracle database in order to provide simple block requests or smart-scan capabilities. It also implements IO management (IORM) and collects numerous statistics.

RS Server

MS Server

CellSRV

it is important to note that both: MS and CellSRV are children of the RS server (they have RS as a parent).

Examine storage Cell

With the cellCli we can also list the storage cell Luns. P.S. Since I don't have money for real exadata there will be some discrepencies between my output and the real output which you will get in the real exadata: Each exadata consists of:

12 Hard disk based Luns

4 Flash based Luns

Let's firstly start with the examination of tha Hard disk based storage:

Hard Disk Based

To list the status of a storage cell server, we can use the following command:

We can obtain more detailed information using the cellCli interface:

List Lun

The result which you will get on the real exadata will be similary to:

The reason for that is the fact that on my virtualized env, the Cells are mapped to a virtualized disks and virtualized flash devices, where on real exadata, they will be mapped to PCI slot and device number.

We can list a specific LUN in more details, as follows:

List Lun

List Physical Disk

List celldisk

List grid disk

Flash Based

The flash based modules, can be examined as the hard disk based ones:

List the cell disks

Apart from the flashdisk, the exadata has also flashlog, to improve the redo log latency.

Flash log

Drop/Create Flash Cache

We can use the distributed CLI interface, provided with the exadata: “dcli” to run a command on multiple storage cells:

Stop / Start Cell Services

We can Restart all cell services using the cellCli interface as follows:

This action, will NOT cause downtime.

ReConfigure GridDisk

Let's put ourselves a challange, let's change the size of the ASM disks one diskgroup, on oura grid disks from: 928 MB to - 608 MB That is hard process if we want to avoid downtime. In a nutshell, what we will do is:

Drop Disks on Cell Server 1

Reconfigure the Disks on Cell Server 1

Add the new disks to the diskgroup, while dropping the ones on the next Cell server

We will repeat step 1), 2) and 3) until we have all disks with the same size. P.S. Have disks with different size in one Diskgroup is very bad practice, it will cause constant rebalancing. Without further adew, let's get started:

Check Our Diskgroups

Now, we can rebalance and manually bring the disks to 608, but that won't help:

Rebalance

The problem here is the fact that the grid disk on the Storage cell is still 928, as indicated by the “OS_MB” size. To change that, we need to do it on rotations, just like with Redo log files :)

Drop disks in Storage Cell Server 1

Now that we have the disks on Cell Storage Server 1 dropped, we can drop the actual disks from the Storage cell. Go to your storage cell server 1 and drop them:

Drop Grid disks

Now, that we have brand new disks, we have to add them :) Remember never drop more than you can carry

External Redundancy - Drop will cause data loss

Normal Redundancy - Drop more than 1/2 will cause data loss

High Redundancy - Drop more than 2/3 will cause data loss.

In our case, we have Normal Redundancy, so we go one by one just in case.

Verify the Disks

So we have 6 shiny new disks to add to our group, let's add them:

Add the new Disks

Now we can see that we have one disk group with two different sizes:

Check Diskgroups

Again that is very bad practice, so we should continue, as the steps repeat I will slack on explaining :) In a nutshell we have to move the disks from the group with 928 MB disk size to 608 disk size :)

Move the disks

Finally we can check the size of the disk groups again:

Check the results

Flash Cache

Flash cache in Exadata is a memory which stores often accessed data on storage level. With Flash cache you can achieve very good perfmance for reading. There are 3 types of a caches:

Write through

Using the write-through policy, data is written to the cache and the backing store location at the same time. The significance here is not the order in which it happens or whether it happens in parallel. The significance is that I/O completion is only confirmed once the data has been written to both places.

Advantage

Ensures fast retrieval while making sure the data is in the backing store and is not lost in case the cache is disrupted.

Disadvantage

Writing data will experience latency as you have to write to two places every time.

What is it good for?

The write-through policy is good for applications that write and then re-read data frequently. This will result in slightly higher write latency but low read latency. So, it’s ok to spend a bit longer writing once, but then benefit from reading frequently with low latency.

Write-around

Using the write-around policy, data is written only to the backing store without writing to the cache. So, I/O completion is confirmed as soon as the data is written to the backing store.

Advantage

Good for not flooding the cache with data that may not subsequently be re-read.

Disadvsntage

Reading recently written data will result in a cache miss (and so a higher latency) because the data can only be read from the slower backing store.

What is it good for?

The write-around policy is good for applications that don’t frequently re-read recently written data. This will result in lower write latency but higher read latency which is a acceptable trade-off for these scenarios.

Write-back

Using the write-back policy, data is written to the cache and Then I/O completion is confirmed. The data is then typically also written to the backing store in the background but the completion confirmation is not blocked on that.

Advantage

Low latency and high throughput for write-intensive applications.

Disadvantage

There is data availability risk because the cache could fail (and so suffer from data loss) before the data is persisted to the backing store. This result in the data being lost.

What is it good for?

The write-back policy is the best performer for mixed workloads as both read and write I/O have similar response time levels. In reality, you can add resiliency (e.g. by duplicating writes) to reduce the likelihood of data loss.

Management

Drop Flash Cache

We can drop the flash cache only after, we have to stop the cluster.

Drop Cache

Shutdown Services

Shutdown Services

Enable Flash Through Cache

Enable Flash Through Cache

Restart Flash Cache Services

Restart Flash Cache services

Monitoring

We can monitor the flash cache, using several commands

Determine the Flash Cache Type

Determine Type

Determine the amount of Dirty Data

Determine the amount of dirty Data