Tuesday, January 27, 2009

Oracle Architecture -- Memory Structures

Because Oracle uses the system memory to store various information, Memory structures are required to access and control a database. These Memory structures reside in the system's main memory and are used;
  • to execute program code
  • to access the control file for information about the database
  • to access the status of a table (if two users try to modify the same table simultaneously, there must a way to stop one of them)
  • to access the datafiles and redo log files

In order to accomplish these tasks, the Memory structure has two main areas called :

  • System Global Area (SGA) which contains data and control information for a database instance and is accessible to all the user and server processes that are running
  • Program Global Area (PGA) which contains data for one process, accessible within different parts of a process, but not outside of it.
The system global area stores information such as transactions made to the data, data dictionary information, (objects that the user can access, location of these objects, etc) and redo log information. The information stored in the SGA can be accessed by all the sessions and processes.

The program global area stores the variables used in a session, which can only be accessed by processes or programs within that session.


TIP: Memory is also known as buffer.

As an analogy, you can think of SGA as the lounge area in a hotel, accessible to all the guests staying in the hotel similarly to (the SGA, which can be accessed by all the processes running). The PGA can be compared to a hotel room, which can be accessed by only the guests who have reserved that room, and not by others.


When a user issues a command, the relevant data is retrieved from the datafiles and is brought to the SGA. This recently used data is kept in memory (i.e., in the SGA) for some time so that any time the same command is issued by any other user, data can be taken directly from the SGA instead of going to the disc again.

The information stored in the SGA is divided into the following Memory structures:

  • Data buffers
  • Redo log buffers
  • Shared pool
Data buffers-This is the area in the SGA where the transactions made to the data are stored. Data which is read from the datafiles is brought and kept here.

Redo buffers-This is the area in the SGA where any changes made to the data are recorded . Later, these changes are written to the redo log files (on the disc).

Shared pool-This is the portion of the SGA where the command issued by the user, its text, and compiled version is kept. It also contains the data dictionary information.


Processes that Run in Memory

Every program uses a process to communicate with the Oracle database. Two types of processes run in the Oracle database are user and server. This topic discusses these two processes. A process is a series of steps executed by the operating system and has its own space in which it runs. Oracle servers have two processes:


  1. User process: or client process, executes the code in an application by sending the request to the server.
  2. Server process: handles requests from the user, communicates with the database to carry out the request, and gets back to the user process with information.
The user process communicates with the server process through a medium called the program interface, which --

  • understands the requests made on the data;
  • passes the information to the user process;
  • traps and returns errors, if any;
  • converts and translates data between different computers.
The server process is responsible for communicating with the user process and for passing information from the Oracle database. The user and server processes may be a single process or separate. client/server systems execute the user and server processes on different machines.

Background Server Processes

Oracle starts a set of background processes every time the database is opened. These processes increase the performance of the programs run. Whenever an Oracle database is started, then a system global area is allocated and Oracle background processes started. The interacting of the system global area and background processes is called an Instance. There are five mandatory processes in an Oracle Instance:

  1. The process that writes all changes in data to the datafiles. This is called the database writer-DBWn-because it writes the information in the data buffers to the data files.
  2. The process that writes the redo log information to the disk. This is called log writer-LGWR-because it writes the information in the redo buffers to the redo log files.
  3. The process that signals Oracle, precisely DBWn, to write the information stored in the data buffers to the datafiles. This process is called the Checkpoint Process or CKPT.
  4. The process that performs crash recovery in the event of an instance failure is called System Monitor or SMON.
  5. The process that performs recovery when a user or server process fails is called the Process Monitor or PMON.
A detailed description of all these processes is of course beyond the scope of this course, but the foregoing provides you with a good fundamental understanding of how the Oracle database works.

An Instance is depicted in the the figure below.

What follows is a step-by-step working example of Oracle:
  1. The database is started. This involves 3 steps:
  • An instance is allocated (i.e., SGA and background processes are started).
  • The database is identified (i.e., the datafiles and redofiles are identified-This is called mounting the database.)
  • The database is opened (i.e., the user can now access the database).
  1. When a user connects to the database and a session starts.
  2. The user process and the server process-they do a handshake.
  3. A user issues an SQL command: Update emp Set sal = sal *12 where deptno = 20;
  4. The user process passes on this statement to the server process.
  5. The server process receives the statement and checks the SGA to see if the relevant data already exists there. If it does, then it takes the data from there only. Otherwise, it retrieves the data from the datafiles and puts it in the SGA (to be precise, in the data buffers).
  6. The update is completed in the data buffers.
  7. This modification of the data is recorded in the redo buffers.
  8. The user issues a command to make the changes permanent (i.e., issues a commit ).
  9. The redo entries ( i.e., the information in the redo buffers) is written to the redo log files; this is done by the LGWR.
  10. After some time, the changes made in the data buffers are written to the datafiles; this is done by the DBWn.
  11. Throughout the process other background processes run, looking out for situations that require their response.

Oracle Architecture -- Physical Database Structures

The Physical structure of a database contains the operating system files. Information in the Oracle database is stored in the tablespace in the form of the Physical structures which occupy the disk space available to store data. These structures include:
  • Password file, which contain the password information for all users.
  • Parameter file, which contains all the important information necessary to start a database. The parameter file is called init.ora , which you can find in the operating system.
  • Datafiles, which contain the application data being stored, as well as any data necessary to store user-IDs, passwords, and privileges.
  • Redo log files, which store all the transactions made to the database. These files are also called transaction log files.
  • Control files, which store information specifying the structure of the database, such as the name and time of creation of the database and the name and location of the datafiles.

Password File

Since the tasks performed by the database administrator (DBA), are critical and should not be done by any other user, there needs to be a more secure method to safeguard his password.

To authenticate a user logging in as the DBA, two methods can be used. Either an operating system authentication, power can be employed or a password file can be used.

A password file is a file that contains the passwords of the DBA. The password file authenticates the user logging in as the DBA. When a user is granted the privileges of a DBA, his username is automatically added in the password file. The password file contains the passwords in encrypted form.

Parameter File

When you start an Oracle database instance, the start-up process checks for the status of certain parameters. The parameter file, init.ora, is the most important part of a database. Some of the parameters identified by the parameter file include:
  • the amount of system memory area that can be used by an instance
  • the name of the database
  • the maximum number of licensed users
  • the names of the control files
  • the amount of memory space that can be used by the redo log files
  • the frequency with which the redo log is updated

Datafiles

The Oracle database requires one or more datafiles to store the application data. The tablespace is composed of the datafiles. Every datafile is associated with just one tablespace, whereas a tablespace can contain more than one datafile. The data stored in datafiles include:


  1. The data for an application, for example, employee information (first name, last name, address, phone, salary, and department) and product information (product name, unit-price, and supplier-name);
  2. The information that the database needs to manage user-data (for example, since the salary of an employee must contain only numbers, the system datafiles will ensure that characters are not stored.);
  3. The information to the database need to determine the validity of your username and password when you log on to the Oracle database.

Redo Log Files

The Oracle database has a set of two or more redo log files. The main purpose of the redo log is to keep track of changes made to the application data. Any change made to the data is not immediately made permanent in the database. The changes are first stored in the systems' redo log and later made permanent. This makes having redo logs a useful feature because if the system fails (which usually happens when you least expect it), you can restore the changes from the redo log, a fact which means you don't have to lose your work! More precisely, if you have an unexpected power outage that shuts your system down, you will lose all transactions stored in the RAM. However, using the automatically maintained redo log, you can apply the changes made recently and restore all information up to the time of the power failure. Oracle allows you to protect the redo log itself by allowing for multiplexed redo log files that back up the data from the original redo log files in different disks simultaneously.

Control Files

The Oracle database has a control file that contains information about the Physical structure of the database. The information may include:
  • the name of the database;
  • the name and location of the datafiles;
  • the name and location of the redo log;
  • the time when the database was created.

The control file is used every time an instance of an Oracle database is started, to identify the database and determine the redo log that must be opened to start the database. The control file is automatically modified when the database is changed; for example, when a new datafile is created or when the redo log changes. Oracle allows multiplexed control files; that is, you can have many control files in different disks that will be updated simultaneously. The information stored in the control files are also used for database recovery, to identify which redo log must be accessed. The control file always contains the last change number of the last change to the database.

Oracle Architecture---Logical Database Structures

Logical database structures manage the storage of physical data. They include:

TABLESPACES

A database is divided into logical storage units called Tablespaces. A tablespace is a logical construct for arranging different types of data you have in the database. An Oracle database must have at least a system tablespace. It is recommended to have different tablespaces for user and system data. One or more datafiles are created for each tablespace to physically store the data of all Logical structures. The relationship between tablespace and datafiles can be understood as: for every user-table created a System Tablespace is created.
System tablespaces exists purely as a logical unit; where as the user-tablesapces exist as physical spaces.


Oracle has provided a standard, recommended way to arrange different kinds of data in different logical units. They have even recommended conventions for naming these logical units. Lots of people still follow this way of arranging their database data and find it to be especially logical and the most effective way. Oracle has provided us with seven logical tablespaces. Let's look at all of them so as to understand their purpose in the database. The seven logical tablespaces are:
  1. SYSTEM Tablespace: This tablespace is available in every Oracle database. The database cannot function without this logical unit, which stores the SYSTEM data. This data is controlled by Oracle itself and it Inserts, Updates, and Deletes this data internally. This logical unit is also the area where DBA's get most of the internal information about database functioning. This logical unit should never be deleted. The System Tablespace plays a role in almost all the transactions that occur in the database.
  2. DATA Tablespace: This tablespace is the logical area where the user of the database stores all the application data. This is the area where you will store USER data. For example, a payroll application will have all of its names, salary, commission, and tax information stored in this logical unit. Depending on the amount of data, this logical unit could be huge and have many datafiles associated with it. Oracle recommends that you relate the tablespace name with its associated datafile name in some way. For example, if the tablespace name for this logical unit is PAYROLL_DATA , then you name its associated physical datafiles as payroll_data_01.dbf and the second one (if needed) as payroll_data_02.dbf . This way if you see any datafile anywhere on the system, you will have no problem relating it to the corresponding tablespace associated.
  3. INDEX Tablespace: This logical unit is used in conjunction with the DATA tablespace. This stores the indexes needed by the application for a speedy response time. Indexes are logical objects created internally in the database to help in the speedy retrieval of user data and thus help in increasing the overall performance of the Application.
  4. TEMP Tablespace: This is the logical unit that serves a very special purpose in the database. When users retrieve data stored in the database, they sometimes do a lot of sorting and grouping. To give you an example, suppose that a user of the database who serves a payroll application wants to see all employees who are working the Information Technology field, who are in their twenties, and who are earning more than 100K a year. To retrieve this kind of data (i.e., to show names of employees who fit this profile), the Oracle database has a lot of work to do inside. First, it must retrieve data relating to information technology personnel; then it sorts this data again to retrieve the employees between 20 and 30 years old. Again it sorts this data to take only those employees who earn more than 100K. Now if we look at the entire transaction, we see Oracle doing a lot of sorting inside the database. Where does all this happen? It happens in the TEMP tablespace, the logical unit, which allocates space to perform all these tasks.
  5. USERS Tablespace: This Tablespace is created for all users who will be connecting to the database. All the users need a work area, which they can use to access data. Having such a logical unit is very helpful because it restricts users to a specific logical area, preventing access to the rest of the database, a feature making database management much easier.
  6. ROLLBACK Tablespace: Let us suppose you go inside the database and write a command that increases the salary of all the employees by 20%. Immediately after writing this command, you realize you didn't intend to give them a 20% raise, only a 10% raise. So how do you undo? Oracle comes to your help in this area because it stores all data before any change is made, in a special area called rollback area. You can retrieve the old data from this holding area. This logical unit is available to hold all the undo information. The usual name given to this tablespace is ROLLBACK, or RBS.
  7. TOOLS Tablespace: This is an important Tablespace that you should have if you plan to install Oracle's or any other third party tools in your database. Most of these tools need to use the SYSTEM database account in the database to create tables and indexes. Now the SYSTEM database account has the System Tablespace as its default Tablespace for the user to create tables. It is important to keep in mind that you should NEVER create any database objects in the System Tablespace even if you install any tools. Thus, to give the database user SYSTEM some place in the database to create tables and indexes needed for tools you may be installing, you should create a separate Tablespace and make that Tablespace its default Tablespace. Oracle recommends you call that Tablespaces Tools.

Schemas and Schema Objects

A Schema is a collection of database objects. Schema objects refer directly to the data.

Relationship between Schema Objects and Tablespaces:

Objects within the same Schema can be stored in different tablespaces, and the same tablespace can store objects from different Schemas. In a database named EMPLOYEE, a user has created the following objects:

  • Table EMP (Physical space)
  • Table DEPT (Physical space)
  • Index I_EMP (Logical space)
  • Index I_DEPT (Logical space)

The schema contains the objects EMP, DEPT, I_EMP, and I_DEPT; part of the EMPLOYEE database.

Relationship Between Schema Objects and Datafiles:

A schema object can span datafiles. In fact, in Oracle RDBMS, the user does not have the control of specifying in which datafile she wants the data of a particular Schema object to go. If a datafile gets exhausted, then the data of the particular object starts filling up another datafile of that tablespace. The relationship between Schema objects and datafiles are depicted.
Schema objects include structures such as:

  • tables, which you can use to store data;
  • views, which present data from one or more tables;
  • sequences, which generate a list of numbers;
  • synonyms, which provide a mask for other database objects (such as tables or views);
  • indexes and clusters, which can assist in improving the performance when you retrieve data;
  • procedures, functions, packages, and triggers (collectively called Program Units), which are programs that perform specific tasks;
  • database links, which describe the path when using more than one database.

Table: This is the main database object within which the data is stored inside the database. Every table has columns in which it stores different kind of data. Oracle stores information about all the tables that exist inside the database. When you create a table, you specify how many columns it will have and what kind of data (number, string, date, etc.) these columns will contain. To better know how a table is organized and filled, think of a spreadsheet. In a spreadsheet you have columns and rows; similarly in an Oracle table, you have columns and rows. Columns describe the data stored and rows consist of the actual data stored. The following code is an example of how a table is created syntactically in Oracle:

CREATE TABLE EMPLOYEE (first_name varchar2(30), last_name varchar2(30), gender varchar2 (5), salary number);

The above code creates a table called EMPLOYEE, which has four columns: first_name, last_name, gender, and salary. The varchar2 and number are two datatypes of Oracle, which represent the kind of data that a column can have. The varchar2 datatype accepts numbers, letters, and any special characters. The number datatype accepts only numbers.

View: A view is an object that acts as a window, or a filter, to the data in a table. What we mean by a window is that using this object, you get to see only partial data from one or more tables. This object is not a physical object and does not have a storage clause as any other table. A view is based on the underlying table or tables and allows you to see only the columns you specifically include in that view. This object is stored in the database as a query against a table and whenever anything is trying to retrieve data from a view, the underlying query is executed and the data from pre-selected columns are obtained. Let's look at how do we create views in Oracle. To create a view based on the EMPLOYEE table, the syntax would appear as follows:

CREATE VIEW my_view AS SELECT first_name, salary FROM EMPLOYEE;

The above syntax would create a view of the employee table with only first_name and salary as its viewable columns. Thus, if you try to retrieve data from my_view , you will be able to see the first name and the salaries of employees. The chief reasons as to why we create views are: one, to get an additional level of security on the data and to restrict users from being able to see all data; two, to hide the complexity of the data. That is, a view can be based on several tables and you can isolate only the requested from all the base tables. This effectively hides how the data is stored in the underlying base tables from outside users. You can relate this feature to normalizing the database. When you normalize tables, you try to avoid redundant data by creating many tables and establishing a link between tables using Primary and Foreign Keys. A view helps you to retrieve data from normalized tables and make it (data) appear as coming from a single table.

Indexes: Indexes are database objects that help performance by speeding data retrieval. Consider the index given at the end of a book, which you use to look up any topic or word in the entire book, without having to search yourself page by page or line by line. These indexes basically help you find a topic faster. Similarly, the indexes in Oracle help you look in specific rows from data stored in a table. Indexes could be based on one or more columns of a table. An index is used if you want to filter the data based on the Primary Key. The syntax to create an index on the salary column of the EMPLOYEE table is as follows:

CREATE INDEX sal_idx
ON EMPLOYEE (salary);

Synonyms: This object in the database is just a pointer to an object like an alias. Assume there is a table called EMP0934BASE that stores all the employee data. You can create a Synonym called employee which points to the EMP0934BASE table. There are two reasons for creating a Synonym; one is to simplify an objects's name, and the other to disguise the owner of the object. Assume that two users, John and Richard, have access to the table, EMP0934BASE. If John is the owner of the table (that is, the person who created the table), then Richard can access the table only by referencing the table with John's name: John.EMP0934BASE. To avoid this unnecessary need for prefixing the table name with the name of the owner, you can create direct simple synonyms. Once you create a Synonym, Richard no longer needs to prefix it with John's name. There are two kinds of Synonyms: Private and Public. You may want to use synonyms if you want to hide the true name or identity of the table or to give simpler names for the users to use in place of more complicated names.

Privileges: This is a database object related to database security and its objects. All objects created in the database have access rights associated with them. Only users with proper rights can access a database object. There are four main privileges that can be granted on database objects by the owner. They are: SELECT, INSERT, UPDATE, and DELETE. SELECT allows the user to be able to see the data in a table or view. It does not allow the user to manipulate the data stored in it in any way. INSERT, UPDATE, and DELETE are granted to allow other users to add, change, or remove data, respectively. The syntax to grant a privilege in the database is as follows:

GRANT SELECT, INSERT, DELETE
ON EMPLOYEE TO richard;

This way you have granted three privileges on the EMPLOYEE table to the user called Richard.

Roles: Consider a scenario where you have 1,000 users and they all are developers. There are 10 tables in the database and all users need to have SELECT and INSERT privileges on all the 10 tables. To give each user privileges for each table separately would require two days of writing a similar SQL over and over again. There is, though, another more efficient way. You can create a role (let's call it development ) and grant all the necessary privileges to this single role and then simply grant this role to all the users. This way, if you want to provide UPDATE privilege on one table to all the users, you can simply grant it to the role instead of granting it to each and every user. Thus, a role makes the management of privileges more transparent and more manageable. The following figure shows how the role called development fits in the picture. You grant privileges to the role, then grant the role to all the users.

Data Blocks

A data block is the smallest unit of Input/Output used by the Oracle database. Though the underlying operating system stores the data in arrays of bits and bytes, Oracle writes and reads in chunks of database blocks. One datablock corresponds to a specific number of bytes of physical database space on disk. The size of data block for any database is fixed at the time of creation of the database; to change it you will have to recreate the database. Thus the data block size for that database should be chosen very carefully. Some values of the data block size are 2KB, 8KB, 16KB, and 32KB. Oracle recommends a size of 8KB for a typical Online Transaction Processing (OLTP) database.

Extents

An extent is the next level of data storage. One extent consists of a specific number of contiguous data blocks allocated for storing a specific type of information. One noteworthy aspect of extents is that storage values for all physical objects created in the database are in number of extents. One or more extents in turn make up a segment. When the existing space in a segment is completely used, Oracle allocates a new extent for the segment. When you try to create any physical object in the database, storage parameters expressed in terms of extents define every segment you wish to create for that object. Thus, in the storage clause of the CREATE TABLE statement, you can determine how much space is initially reserved for a table's data segment.

Segments

Segment is the level of logical database storage above an Extent. A segment consists of a set of extents assigned for a specific Logical structure. Each table's data is stored in its own single segment. Similarly, each index's data is stored in a single segment. The different types of segments are the data segments, index segments, rollback segments, and temporary segments.

TIP: You can remember the order of levels of the logical storage structures through the acronym 'DEST' (Datablocks, Extents, Segments, Tablespaces).

More extents are automatically allocated by Oracle to a segment if its existing extents become full. An important point to be remembered is that although an Extent consists of contiguous data blocks (i.e., data blocks adjacent to each other on the same datafile), a segment may not consist of contiguous Extents; thus, the Extents constituting a segment may lie on different datafiles.

There are four main types of segments that are used in the Oracle database. They are:

  1. Data segment: This type of segment is created by Oracle when you create a non-clustered table or a cluster inside an Oracle database. Thus, a non-clustered table, or a cluster, has a single data segment to hold all its data.
  2. Index Segment: Like the data segment, every index in the Oracle database has an index segment to hold all its data. A CREATE INDEX command creates this kind of segment.
  3. Temporary Segment: Many commands used to communicate with the database create a temporary segment which Oracle uses for a transaction. These segments are automatically created and dropped by Oracle.
  4. Rollback Segment: These are the undo segments which Oracle has for its use to store the undo information. Every database has several of these segments. The database can function only if there is at least a single rollback segment.

Logical Structures:

Schema Objects --> Need storage space
Data Blocks --> Stored in one or more datablocks
Extends --> One or more continuous data blocks form an extent
Segments --> A set of extends form a segment; when full Oracle creates/allocates space.
Tablespace --> Includes above structures and form a database

Oracle Architecture

A reasonably strong theoretical understanding of Oracle architecture will prove to be of great practical benefit later in the developing of applications to work with the Oracle database. While at first, it might seem that to work with a database system as well-developed as Oracle, all you really have to know is how to record and retrieve data, the practical reality is that knowing how Oracle works "under the hood" will pay off.

Once you understand the internal structure and mechanism of the Oracle database system, you will find yourself developing much more efficient applications-more efficient in terms both of processing speed and space requirements. If you have a reasonably strong technical understanding of how Oracle actually works, when creating objects such as tables, for example, you will have a much better sense of how much space to allocate to the object, where to put the object, and so forth.

To begin with, we need to understand that Oracle, like any database, stores data its user tells it to store. In addition, though, Oracle stores other information-data for its own use. Let's examine these two species of data stored and used.

The first type of data, the data the user stores for his or her own application's use, is appropriately termed USER data. This data can be stored and used in many ways, as well as customized according to the user's needs. You, the user, can decide the type of the data and the amount you want to store. You enjoy essentially complete control over the user data.

The second type of data, which Oracle spontaneously generates for itself, which the system uses, updates, or deletes on its own, is referred to as SYSTEM data. What is remarkable about Oracle's system data is that Oracle is able to generate and manage this type of data on its own in order to keep itself functioning smoothly and consistently. This data, which the user has little or no control over, Oracle generates and stores in internal tables during the initial installation. The sort of information stored as system data includes the number of users, their passwords, their privileges, the number of tables the user is creating, the kind of data being stored, how much data is stored, and so on. How Oracle actually functions as a database depends largely upon its system data. Finally, although users cannot access system data, Oracle database administrators routinely work with this data in order to fine-tune Oracle's performance.

The Oracle Database has three types of structures in which it stores and maintains User and System data.

  1. A Logical structure, that maps the data to the Physical structure.
  2. A Physical structure, that is part of the operating system's file structure.
  3. A Memory structure, where all the processing takes place.

The Logical structures control how the data must be stored in the database. There are five Logical structures, namely, tablespaces, segments, extents, data blocks, and schema objects. and five Physical structures, namely, parameter files, password files, datafiles, redo log files, and control files that reference data.

Every program uses a process to communicate with the Oracle database. These processes run in the System's Random Access Memory (RAM). In addition to Logical and Physical structures, there are two main Memory structures, namely, the System Global Area (SGA) and the Program Global Area (PGA). The Oracle database uses these memory areas to store information before they are made permanent in the database.

The purpose of a database is to both store and manipulate information. The Oracle database controls this dual process efficiently by maintaining the Physical and Logical structures separately, a feature allowing for modification of physical data without the need for accessing the Logical structure, and vice-versa. By way of illustration, consider the following: assume that you have a database table for a retail outlet that stores information about its salespeople. Initially, you decide to store only the first name, the last name, the address, the phone number, and the social security number of each salesperson. Later on, you decide to store the city, state, and zip code information in separate columns instead of including them with the address. Since the Logical and Physical structures in an Oracle database are maintained separately, you can easily accomplish such changes by just adding the columns to the Logical structure (which is a table, in this case), and then inserting the data. This data will be updated in the file-system, a Physical structure. What about the reverse of this? If you have to change the contents of the data (something you will do often), you may end up changing the Logical structure as well each time.

Logical structures control how the actual data will take up available storage space. The two different sets of Logical structures available to support this purpose are:

  • Tablespaces, segments, extents, and data blocks, each of which provide a different level of organization and storage units-tablespaces being the highest level.
  • Schema objects, which reference data in the database.

The Physical structures occupy the disk space available to store data. These structures include:

  • Password File, which contains the password information for all users.
  • Parameter File, which contains all the important information necessary to start a database.
  • Data Files, which contain all the data in the Oracle database.
  • Redo Log Files, which keep track of changes made to the data.
  • Control files, which contain information about the Physical structure, such as the database name, location, etc., and also the system change number for the last change made to the database, allowing for the restoration of the database to its last change before a crash.

Memory areas are used to manage the storage of data. There are two memory areas:

  • The System Global Area (SGA), which contains data and control information for a database instance and is accessible to all users and server processes that are running.
  • Program Global Area (PGA), which contains data for one process, and which therefore is accessible within different parts of a process, but not outside of it.

Saturday, January 24, 2009

Data Normalization

Data Normalization is a process of refining a database model so that its structures conform to certain standards for the design of tables. The tables and fields you define make up the structure of the database. Good data normalization results in data tables that make sense in a fundamental way. Well-normalized tables are easy to understand when looked at. It is easy to see the kind of data stored and the types of updates needed to be performed. It should be easy to create data entry routines and simple reports directly from normalized tables. In fact, the rule of thumb is: If it's hard to work with a data table, more normalization is probably needed.

Normalization ensures that your database design is the most effective, or in other words, the least likely to change. A normalized database also reduces the chances of repetitive data occurring.

There are five rules involved in normalization. Each of these rules is referred to as a normal form. When you refine a data model to comply with one of these rules, the model is said to be normalized to that rule's degree. For instance, once the design is refined to comply with the first normal form, the design is normalized to the first degree.

Following are some of the advantages of normalization:

  • Applying normal form rules to your application sometimes reveals new requirements or rules that have been overlooked.
  • Normalization leads to accurate and complete reflection of business requirements and rules.
  • Normalization minimizes redundancy by means of referential integrity.
  • Normalization creates a logical scheme that minimizes the potential for unintentional results.
Normalizations Rules

First Normal Form (1NF) (Eliminate Repeating Groups)
  • All the columns in a table must be single-valued and no repeating groups are allowed. In other words, each column must contain one and only one piece of information, and no two rows can contain the same record of information.
  • Example of Single Valued: In the Customer table, a column cannot store both the First Name and Phone Number of a customer. Instead, one column stores the first name, and the other column stores the phone number of a customer.
Second Normal Form (2NF)(Eliminate Redundant Data)
  • All the non-key attributes in a table must depend only on the primary key. In other words, no attribute may depend on another attribute in a table, if the latter is not a primary key.
pid empname deptid deptname deptlocation
100 John Smith 10 Accounts Florida
101 Adam Scott 20 Security Virginia
102 John Doe 10 Accounts Florida
103 Mary Jones 20 Security Virginia
104 Elizabeth London 30 Technical Texas

Third Normal Form (3NF) (Eliminate Keys Not Dependent on Primary Key)

  • Transitive dependencies are not allowed. In other words, any column which is not a key in the table should not depend on another column not a key in the table.
orderid orderdate customerid productid product_price
100 20-JUN-99 200 12 20.00
101 30-AUG-99 201 13 12.00
102 25-JAN-98 201 12 20.00
103 07-JUL-99 202 14 15.00
104 15-MAR-00 203 15 14.00
105 24-APR-00 204 16 23.00
106 12-JUN-00 205 17 34.00

Fourth Normal Form (4NF)


  • Isolate Independent Multiple Relationships

Fifth Normal Form (5NF)

  • Isolate Semantically Related Multiple Relationships

Friday, January 23, 2009

Introduction to Data Modeling

Data Modeling is analogous to an architect's initial developing of a blueprint for a building. Before starting actual construction, the architect must first develop a design based upon the owner's requirements-how many rooms, the number of facilities, parking spaces, etc. The architect must also keep in mind various legal requirements and building codes. Data Modeling involves a similar sort of process, of integrating input from end users, determining and documenting their requirements. Based upon all your gathered information, and incorporating business rules and practices, you construct the data model, or prototype.

The application development process should have a data model. A database inadequately designed can prove to be a disaster, both to the business and to the application itself. The time you spend in analyzing user requirements to develop a solid data model will prove well worth the effort. A data model lets you present user requirements in their proper perspective. In the database design process, you design customized tables to accurately reflect elements that the users wish to maintain and which accurately identifies their characteristics. You also define relationships between the tables, and a structure to integrate all the information in the most efficient manner. Following are the processes to analyze user-requirements, design tables, identify attributes, and establish relationships between tables.

Several software tools are available for modeling a database. Some of these tools, called CASE tools (computer-aided software engineering tools), can be used not only to draw diagrams but also to create documentation of the design. The three main components of these diagrams, called Entity/Relationship diagrams, or E/R Diagrams, are:

  • Entities
  • Attributes
  • Relationships

While designing a database, you need to follow several steps, specifically:

  • Identify the need for the database,
  • Analyze existing information,
  • Identify a list of Entities (Tables),
  • Identify a list of Attributes (Columns),
  • Identify relationships between Entities (Tables),
  • Normalize the design.

Let's consider the database requirements of a certain Mrs. Cruz, who runs a Book Store. Currently she keeps track of all her books in an Excel spreadsheet. Her business is growing and she finds she can no longer keep up with the demand, especially because of increasing difficulties she is having maintaining her Excel spreadsheet. The spreadsheet has no capability of searching by title, author, or ISBN number. Currently, customer orders are manually logged into a registry as they are received. In order to automate all of these features, Mrs. Cruz should design a database for her book store. To help her design such a database, let us walk through the step-by-step process of database design, as outlined above. We will need to help in two ways:

  1. Identify the need for a database. The need for the database is usually defined by the client's needs. For example, Mrs. Cruz' needs will include:
  • Keeping track of a list of books
  • Keeping track of customers buying the books
  • Producing a monthly sales report
  • Keeping track of all book orders
  1. Analyze existing information. This step will involve asking such questions as: Is there an already existing database? If not, how is current data being stored? And so on. This will help in designing tables and columns.

It is important to understand the client's requirements and to prepare specifications, or requirements, document. This document can be submitted to the client so that the client and you agree with the stated requirements. While preparing this document, it is very important to communicate with the end-users of the database and also to understand, as fully as possible, business rules and practices.

The next step is to identify entities and attributes based on the requirements gathered in the above steps. You will identify tables and columns for this database in further topics.

Entities

Entity refers to information of significance in the end-user's working environment. An entity can be considered any substantive object of significance to the business, about which you maintain information. An entity may or may not be a tangible or actual object, such as a person, place, or thing; it may be an abstract activity, like an appointment or an operation. It must, however, be uniquely identifiable. Moreover, every instance or occurrence of the entity must be separate and distinctly identifiable from all other instances of that type of entity. Tables are used to represent entities in a database.

Following are the guidelines to represent an entity:

  • Entities are represented in round-cornered boxes, called softboxes:
  • Entity names appear in uppercase letters.
  • Entity names are nouns, for e.g., CUSTOMER, EMPLOYEE, DEPARTMENT, etc.
  • Entity names are singular.

Consider the bookstore example we discussed in the previous topic and identify the entities. Since the bookstore needs to keep track of all the books, you can identify BOOKS as an entity. The bookstore also keeps track of customers and the orders placed by them, so that we will need to establish two more entities called CUSTOMER and ORDERS. You can represent these entities in the E/R Diagram.

Consider another example. You need to design a database for a bank. In this case, you need to keep track of all the accounts for each customer. You will need two entities called ACCOUNT and CUSTOMER, assuming that a customer can have more than one account. Let's say the bank may have several branches and that you need to keep track of all customers individually and their account information by branch. Thus, you would need an entity called BRANCH.

Attributes

Attributes are characteristics or properties of an entity. In other words, any property that helps to describe an entity is called an attribute. An attribute might be either required or optional. When it's required, it must have a value; when it's optional, it may or may not have an assigned value. When an attribute has no value, it is said to be null. Zero and blank spaces are by definition notnull. Examples of typical attributes include name, gender, order date, and quantity. Attributes are analogous to columns in databases.

Following, are the conventional guidelines for accurately representing the attributes of an entity:

  • Attributes should help to describe the entity.
  • Attributes should appear inside softboxes, located below the name of the appropriate entity.
  • Attributes appear in lowercase letters (for example: name, address, phone_number, gender, etc.).
  • Attributes are written in the singular.

Consider the book store example, where you defined three entities CUSTOMER, ORDERS, and BOOKS. Let us identify the attributes for each of these entities. For the CUSTOMER entity, you need to record first name, last name, middle initial, address, city, state, zip, and phone number. For the BOOKS entity, you need to record ISBN, title, description, author, year published, and publisher. For the ORDERS entity, you need to keep track of the order number, order date, and shipment type. You can represent these attributes for each of the entities as shown below:

Let's consider another example-a bank database. We will identify the attributes for the entities CUSTOMER, BRANCH, and ACCOUNT. For the BRANCH entity, you can keep track of branch number, address, city, state, and zipcode. For the CUSTOMER entity, you can keep track of first name, last name, middle initial, address, city, state, and phone number. For the ACCOUNT entity you will need the account number, account type, account balance, and date opened.

Relationships

In Relational databases, relationships can exist between data held in different tables. The relationship between tables can be defined according to business requirements and rules. Relationships can be technically defined as bi-directional associations between two or more entities (tables). The four types of relationship are:

  • One-to-One
  • One-to-Many
  • Many-to-Many
  • Recursive

Relationships are represented by lines connecting two entities. The following figure illustrates the usual relationship-diagramming conventions:

Representing the relationship in the E/R Diagram depends on carefully considering how the two entities actually relate to each other. For example, consider the relation between the MOVIE and VIDEO_RENTAL entities. The video store may have tapes that have never been rented. In this instance the relation from MOVIE to VIDEO_RENTAL would be optional, which means you would use the dashed line near the VIDEO_RENTAL 's end. On the other hand, anything rented from the store must be a video. This cannot be optional, so we use the solid line near the MOVIE end.

One-to-One Relationships

In One-to-One relationships, the two entities (or tables) are related to each other such that for every row in table X, there exists only one row in the table Y and for every row in table Y, there exists only one row in the table X.

For example, observe the relationship between two tables - EMPLOYEE and EMPLOYEE_CONFIDENTIAL

Each EMPLOYEE can have only one confidential record stored in EMPLOYEE_CONFIDENTIAL table. Each confidential record in EMPLOYEE_CONFIDENTIAL table relates to only one employee in EMPLOYEE table. Hence, there is a One-to-One relationship between these two tables.

Consider the Book Store example discussed earlier. Let's say you want to keep track of certain confidential information relating to customers' credit card accounts, information you will maintain in a different entity called CUSTOMER_CREDITCARD. You need to keep track of only one record for each customer, and let's say it is not mandatory to have such confidential information for each and every customer. You can accomplish this simply by introducing a new, additional entity called CUSOTMER_CREDIT, and by relating it to each CUSTOMER entity as a One-to-One relationship, with an optional relationship for confidential information. This relationship can be represented by dashed line pointing towards the entity called CUSTOMER_CREDIT.

Now consider the Bank database discussed previously. Let's say each branch of the bank also has some confidential information such as a branch security code, locker code, and that every branch must have one and only one confidential record stored separately. In this case, you would need an entity called BRANCH_SECURITY related to the BRANCH entity on a One-to-One basis, with a mandatory relationship on both sides.

One-to-Many Relationships

In One-to-Many relationships, the two entities (or tables) are related to each other such that for every row in table X, there exists one or more rows in table Y, and for every row in table Y, there exists only one row in table X.

For example, observe the relationship between two tables--EMPLOYEE and DEPARTMENT.

Each department can have many employees while each employee can only work in one department. In other words, for each row in the DEPARTMENT table, there could be many corresponding rows in the EMPLOYEE table.

Consider again our Book Store example. A customer can place an order for books any number of times. Furthermore, let's say a customer is entered in the database only if he/she has ordered at least one book. Finally, an order will always relate to only one customer. Based on such business rules and practices, you will have a mandatory One-to-Many relationship between the two entities, CUSTOMER and ORDERS.

Now consider again the example of the Bank database. Let's say each branch of the bank will have many accounts, and that you could have a branch without any accounts. Based on such a business rule or practice, you will have a One-to-Many relationship between BRANCH and ACCOUNT, with an optional relation on the ACCOUNT side, since a branch can exist without any accounts having been opened. Also, let's say a customer can have one or many accounts and that each customer must have at least one account. It follows that you will have a mandatory One-to-Many relationship between CUSTOMER and ACCOUNTS.

Many-to-Many Relationships

In Many-to-Many relationships, the two entities (or tables) are related to each other such that and for every row in table X, there exists one or more rows in table Y, and vice-versa.

When two entities have a Many-to-Many relationship, the storage of repetitive data results, eventually requiring enormous space and becoming increasingly difficult to maintain.You can avoid the Many-to-Many relationship by breaking it into two One-to-Many relationships.

Observe below, for example, the relationship between two tables - ORDERS and PRODUCTS.

In this example, we have a table that stores order information and another table that stores product information. Each order could involve many products and each product could be involved in many orders, leading to a Many-to-Many relationship.

In the Book Store example, where a book can be in one or many orders, and an order can include one or many books, every order should show at least one book ordered. A book in the book's entity may or may not have any orders, meaning it may not have been bought by any customer. In this case the relationship is optional. Therefore, it follows that you have a Many-to-Many relationship between ORDERS and BOOKS, with an optional relation.

Recursive Relationships

In Recursive relationships, an entity has two columns that are related to each other with a hierarchical relationship between the columns. Recursive relationships can, in turn, be One-to-One, One-to-Many or Many-to-Many.

For example, in the employee table, the managers are also considered employees of the company. One manager could be managing several other employees. The Manager column in the EMPLOYEE table thus refers back to the EmployeeID column of the EMPLOYEE table. Such a relationship within a table is called a Recursive relationship.

Identifiers

Identifiers enforce business rules in a database. Identifiers enable you to maintain the integrity of the database. The two identifiers are:

  • Primary Key
  • Foreign Key

A Primary key is a column or combination of columns whose values uniquely identify a row in a table. Each and every table in the database must have a primary key. The two properties of the primary key are that:

  • its value is unique
  • its value is not null

Therefore it is said that a primary key's value is definitely known and is unique in the entire table. For example, the VIDEO table can have a unique identifier called video_id. Attributes are identified by their role in the table or entity as follows:

  • Unique attributes that are a part of the primary key are represented with the pound symbol, #.
  • Mandatory attributes are represented by the * symbol.
  • Optional attributes are represented by the letter, o.

A Foreign Key is a column in a table that refers to a primary key of another table. For example, movie_id (foreign key) in the VIDEO_RENTAL table refers to movie_id (primary key) in the MOVIE table. Foreign keys are generally represented as (fk) in the E/R Diagram.

Wednesday, January 21, 2009

Relational Database Model

Introduction

Inherent limitations of the Hierarchical and Network database models slowly led (with the advent of microcomputer systems) to the increased popularity of Relational database models. Dr. E.F. Codd, who invented the Relational model in the early 1970's while heading a research project sponsored by IBM, based the new model on principles of relational algebra in order to solve problems associated with storing large volumes of data. The fundamental improvements in the Relational model are founded upon the design principles of the table, in which all the data is stored. In the Relational model, each table has a unique name, so that all the user needs to know is the name of the table, and not where or how the data is stored. In traditional databases, by contrast, you would have had to understand how the database was structured, in order to reach the sought-after data. By contrast, now to find the table or data you are looking for in the Relational model, you use a new, specialized language, called Structured Query Language (SQL), in order to "talk" to the database; for example, you can now type a command essentially asking the database to show you all the rows in the video table. This methodology has made the understanding and functionality of database principles easier and more accessible, and, hence, more popular. Furthermore, database administration is now a lot easier, too. Tables in this model can be used not only for storing the actual data, but also for managing access rights to the database; thereby, ensuring data integrity. Everything in the Relational model can be stored in tables. Today, the Relational model has been implemented in a large number of databases, also known as Relational Database Management Systems (RDBMS).
The Relational database model, thus presents the following advantages:



  • Ease of Understanding: data is stored in a way that even end-users understand at least at the conceptual level;

  • Visible Relationships: the relationships between entities are clearer, no longer embedded or hidden from the end user;

  • Data Independence: modifications may be made to the data without making global modifications to the application's structure.
The Relational database stores its data using, of course, what is known as the Relational model. In a Relational database, the data is stored in a set of related tables. The term relational is used because relationships are established between multiple tables. A system that manages a Relational database is referred to as a Relational Database Management System (RDBMS). The data contained in such a database is stored across several tables. To access and manipulate this data, a user types in (or enters) specific commands in a language understood by the database. Structured Query Language (or SQL) is one of the languages used to communicate with Databases.


Object-Relational Database Management Systems

Over the last two decades, a new style of programming has developed, called Object-Oriented Programming (OOP), which differs significantly from traditional programming. Basically, data structures processed by OOP are much more complex than those processed by traditional programming languages. OOP data structures more accurately approximate real-world entities and thus help in representing data as it actually exists in the real world. Because these data structures have remained difficult to incorporate into existing relational DBMS products, a new category of DBMS products, called object-oriented database systems, have evolved, and continue to evolve. They provide certain significant advantages over RDBMS's.
Oracle has extended its RDBMS model to an Object-Relational model, which makes it possible to store object data structures within the Relational model. The Object-Relational model allows users to create user-defined data structures (called object types) and to apply them within the Relational model.
In ORDBMS, users themselves are free to define additional kinds of data types in ways which specify both the structure of the data and the ways of operating (methods) on it. This flexible approach adds value to the data stored in the database because, again, both the structure of the data and the ways of operating on it can be independently specified.
Such user-defined data types are therefore termed object types.
Let us consider a purchase order system. Its tables may contain such data as customer, item, purchase order, etc. The application would need dynamically created status information about the purchase order, such as current value of the shipped or unshipped items, and the total money value of a purchase order. For this, we would normally have to create some sort of data logic within the application (e.g., with triggers), but now, in an ORDBMS, we can simply define the purchase order itself as a datatype (called an object type). This object type will specify those attributes that make up the purchase order and will also specify the operations (called methods) we perform on the data unit or object type (i.e., the purchase order), such as determining the total money value of an individual purchase order. The application thus will not be required to calculate the total value of the purchase order since the logic of the purchase order's structure and behavior is already included in the database schema. The application will no longer need to know the details and thus will not need to keep up with most changes.
So, by using object types, the application need only contain application logic not data logic.


Object Types

An object type is a user-defined data type which encapsulates the data structure along with the operation needed to manipualte it. An object type represents a real-world entity such as a car dealer, employee, and address. By using object types, programs can better reflect the world they seek to emulare.

An object type has three components:

  • Name -- the name of the object type
  • Attribute -- characteristics representing the structure of the real-world entity
  • Methods -- representating behaviours of the real-world entity, and operations the applacation can perform on the real-world entity

OJO: Object types are analogous ti IC's that can be plugged into various electronic devices. To plug an object type into your program, you need only know what it does, not how it works.

Advantages of an Object-Relational Model over Strictly Relational Model

  • An ORDMS by design bundles the data structure with the operation performed on it. Consequently, the application is no longer burdened with variations on how to operate on the data object. For instance, in the purchase order example, the application need not write code to calculate the sum of the line items in order to calculate the total cost to the customer. By contrast, while an RDBMS provides a very efficient way of storing and retrieving data, every application developer must first write the necessary code for an operation to be performed on the data.
  • Because object types store data in its more natural form and allow applications to retrieve it that way, ORDBMS better represents the real world.
    Since the logic of the object type's structure and behavior is contained within the database schema itself, application logic need not account for most details nor track most changes. Thus, an application need only contain application logic and not data logic.
  • Object types can be used across applications, allowing users to be concerned with only what they do, not how they do it.
  • User-defined types make it much easier to work with multimedia data such as sound, video, graphics, etc.
  • The ORDBMS approach greatly enhances transmission efficiency. For example, a client-side application can request a purchase order from the server and receive all relevant data in a single transmission.



Monday, January 19, 2009

Basic Methodology Common to Both Hierarchical and Network Models

In both Hierarchical and Network models, relationships must be predefined-and therefore fully anticipated before implementing the database. These relationships are represented by means of separate data structures, such as indexes or linked lists. By contrast, in the Relational model, as we will see, relationships are established through data values.

Interestingly, transaction processing systems actually have two specific advantages over systems that rely on storing relationships within their data. The first advantage relates to performance efficiency. When relationships are predefined, data structures can be selected and fine-tuned according to an anticipated workload, thereby optimizing performance. The second advantage relates to control. By control we mean that different users (or applications) can be assigned different, limited tasks. For example, an order-entry program need not access the accounts-payable data. A particular user (or program) is therefore given a restricted view of the data, a feature leading in this one respect to a greater central control.

Hierarchical Database Model

As the name suggests, the Hierarchical database employs a hierarchical structure. The effects of such a structure include:

  • All data relationships are assimilated into hierarchies;
  • Files are arranged in tiers, or layers, of relationships;
  • Relationships are analogous to the parent-child relationship, though, as we will see, it is significant that in the Hierarchical model, while the parent may have more than one child, the child may have only one parent;
  • Data access may occur only through predefined relationships, a feature rendering the Hierarchical database model relatively rigid. Just as you cannot create a subfolder without having a main, or parent, folder in Windows Explorer, so you cannot have information about a child in a Hierarchical database without having the corresponding information about the parent.
  • Repetition of data is required within the database, an inevitable redundancy making the Hierarchical database model very inefficient.

The most common example of the Hierarchical database model is Data Language I (DL/I), implemented on the IMS (Information Management System), and developed by IBM.

Network Database Model

This model is based upon the networking of data relationships and was developed to overcome the inefficiency of the Hierarchical database model. In a Network data model, multiple computers are connected, enabling them to share information and to communicate. Although there have been several kinds of Network models, the major one is the CODASYL DBTG model, named after the committee that developed it. It was the DBTG model that introduced the terms Data Definition Language (DDL) and Data Manipulation Language (DML). The Hierarchical model did not include a DDL, so the CODASYL model was an immediate improvement over the Hierarchical model and over earlier database systems that had only data manipulation capabilities.

Program and Data View Relationship

The Network database model provides for two alternative views of the database:

  • Schema-complete logical view of the database (the entire database)
  • Subschema-a subordinate view of the database, the limited perspective an application program sees; a subset of the database.

Users interact with the database exclusively by means of an application program (no independent interactive query language exists). Moreover, applications view the database through a subschema, as shown below:

Various programs might share subschemas, or each may have its own.

The fundamental building blocks for such a database include:

  • Data items-Data items make up a field equivalent to a column in a table.
  • Record-A record is a collection of data items.
  • Set-A set is a One-to-Many relationship between records. Sets have owners and members. The owner of a set is the parent and the members are the children.

In the Network model, children are allowed to have multiple parents, a feature providing for improved relationships and resulting in a reduction in data redundancy.