50 New Features of SQL Server 2008

• Transparent Data Encryption
Enable encryption of an entire database, data files, or log files, without the need for application changes. Benefits of this include: Search encrypted data using both range and fuzzy searches, search secure data from unauthorized users, and data encryption without any required changes in existing applications.

• Extensible Key Management
SQL Server 2005 provides a comprehensive solution for encryption and key management. SQL Server 2008 delivers an excellent solution to this growing need by supporting third-party key management and HSM products.

• Auditing
Create and manage auditing via DDL, while simplifying compliance by providing more comprehensive data auditing. This enables organizations to answer common questions, such as, "What data was retrieved?"

• Enhanced Database Mirroring
SQL Server 2008 builds on SQL Server 2005 by providing a more reliable platform that has enhanced database mirroring, including automatic page repair, improved performance, and enhanced supportability.

• Automatic Recovery of Data Pages
SQL Server 2008 enables the principal and mirror machines to transparently recover from 823/824 types of data page errors by requesting a fresh copy of the suspect page from the mirroring partner transparently to end users and applications.

• Log Stream Compression
Database mirroring requires data transmissions between the participants of the mirroring implementations. With SQL Server 2008, compression of the outgoing log stream between the participants delivers optimal performance and minimizes the network bandwidth used by database mirroring.

• Resource Governor
Provide a consistent and predictable response to end users with the introduction of Resource Governor, allowing organizations to define resource limits and priorities for different workloads, which enable concurrent workloads to provide consistent performance to their end users.

• Predictable Query Performance
Enable greater query performance stability and predictability by providing functionality to lock down query plans, enabling organizations to promote stable query plans across hardware server replacements, server upgrades, and production deployments.

• Data Compression
Enable data to be stored more effectively, and reduce the storage requirements for your data. Data compression also provides significant performance improvements for large I/O bound workloads, like data warehousing.

• Hot Add CPU
Dynamically scale a database on demand by allowing CPU resources to be added to SQL Server 2008 on supported hardware platforms without forcing any downtime on applications. Note that SQL Server already supports the ability to add memory resources online.

• Policy-Based Management
Policy-Based Management is a policy-based system for managing one or more instances of SQL Server 2008. Use this with SQL Server Management Studio to create policies that manage entities on the server, such as the instance of SQL Server, databases, and other SQL Server objects.

• Streamlined Installation
SQL Server 2008 introduces significant improvements to the service life cycle for SQL Server through the re-engineering of the installation, setup, and configuration architecture. These improvements separate the installation of the physical bits on the hardware from the configuration of the SQL Server software, enabling organizations and software partners to provide recommended installation configurations.

• Performance Data Collection
Performance tuning and troubleshooting are time-consuming tasks for the administrator. To provide actionable performance insights to administrators, SQL Server 2008 includes more extensive performance data collection, a new centralized data repository for storing performance data, and new tools for reporting and monitoring.

• Language Integrated Query (LINQ)
Enable developers to issue queries against data, using a managed programming language, such as C# or VB.NET, instead of SQL statements. Enable seamless, strongly typed, set-oriented queries written in .NET languages to run against ADO.NET (LINQ to SQL), ADO.NET DataSets (LINQ to DataSets), the ADO.NET Entity Framework (LINQ to Entities), and to the Entity Data Service Mapping provider. Use the new LINQ to SQL provider that enables developers to use LINQ directly on SQL Server 2008 tables and columns.

• ADO.NET Data Services
The Object Services layer of ADO.NET enables the materialization, change tracking, and persistence of data as CLR objects. Developers using the ADO.NET framework can program against a database, using CLR objects that are managed by ADO.NET. SQL Server 2008 introduces more efficient, optimized support that improves performance and simplifies development.

• DATE/TIME
SQL Server 2008 introduces new date and time data types:
o DATE—A date-only type
o TIME—A time-only type
o DATETIMEOFFSET—A time-zone-aware datetime type
o DATETIME2—A datetime type with larger fractional seconds and year range than the existing DATETIME type
The new data types enable applications to have separate data and time types while providing large data ranges or user defined precision for time values.

• HIERARCHY ID
Enable database applications to model tree structures in a more efficient way than currently possible. New system type HierarchyId can store values that represent nodes in a hierarchy tree. This new type will be implemented as a CLR UDT, and will expose several efficient and useful built-in methods for creating and operating on hierarchy nodes with a flexible programming model.

• FILESTREAM Data
Allow large binary data to be stored directly in an NTFS file system, while preserving an integral part of the database and maintaining transactional consistency. Enable the scale-out of large binary data traditionally managed by the database to be stored outside the database on more cost-effective storage without compromise.

• Integrated Full Text Search
Integrated Full Text Search makes the transition between Text Search and relational data seamless, while enabling users to use the Text Indexes to perform high-speed text searches on large text columns.

• Sparse Columns
NULL data consumes no physical space, providing a highly efficient way of managing empty data in a database. For example, Sparse Columns allows object models that typically have numerous null values to be stored in a SQL Server 2005 database without experiencing large space costs.

• Large User-Defined Types
SQL Server 2008 eliminates the 8-KB limit for User-Defined Types (UDTs), allowing users to dramatically expand the size of their UDTs.

• Spatial Data Types
Build spatial capabilities into your applications by using the support for spatial data.
o Implement Round Earth solutions with the geography data type. Use latitude and longitude coordinates to define areas on the Earth's surface.
o Implement Flat Earth solutions with the geometry data type. Store polygons, points, and lines that are associated with projected planar surfaces and naturally planar data, such as interior spaces.

• Backup Compression
Keeping disk-based backups online is expensive and time-consuming. With SQL Server 2008 backup compression, less storage is required to keep backups online, and backups run significantly faster since less disk I/O is required.

• Partitioned Table Parallelism
Partitions enable organizations to manage large growing tables more effectively by transparently breaking them into manageable blocks of data. SQL Server 2008 builds on the advances of partitioning in SQL Server 2005 by improving the performance on large partitioned tables.

• Star Join Query Optimizations
SQL Server 2008 provides improved query performance for common data warehouse scenarios. Star Join Query optimizations reduce query response time by recognizing data warehouse join patterns.

• Grouping Sets
Grouping Sets is an extension to the GROUP BY clause that lets users define multiple groupings in the same query. Grouping Sets produces a single result set that is equivalent to a UNION ALL of differently grouped rows, making aggregation querying and reporting easier and faster.

• Change Data Capture
With Change Data Capture, changes are captured and placed in change tables. It captures complete content of changes, maintains cross-table consistency, and even works across schema changes. This enables organizations to integrate the latest information into the data warehouse.

• MERGE SQL Statement
With the introduction of the MERGE SQL Statement, developers can more effectively handle common data warehousing scenarios, like checking whether a row exists, and then executing an insert or update.

• SQL Server Integration Services (SSIS) Pipeline Improvements
Data Integration packages can now scale more effectively, making use of available resources and managing the largest enterprise-scale workloads. The new design improves the scalability of runtime into multiple processors.

• SQL Server Integration Services (SSIS) Persistent Lookups
The need to perform lookups is one of the most common ETL operations. This is especially prevalent in data warehousing, where fact records need to use lookups to transform business keys to their corresponding surrogates. SSIS increases the performance of lookups to support the largest tables.

• Analysis Scale and Performance
SQL Server 2008 drives broader analysis with enhanced analytical capabilities and with more complex computations and aggregations. New cube design tools help users streamline the development of the analysis infrastructure enabling them to build solutions for optimized performance.

• Block Computations
Block Computations provides a significant improvement in processing performance enabling users to increase the depth of their hierarchies and complexity of the computations.

• Writeback
New MOLAP enabled writeback capabilities in SQL Server 2008 Analysis Services removes the need to query ROLAP partitions. This provides users with enhanced writeback scenarios from within analytical applications without sacrificing the traditional OLAP performance.

• Enterprise Reporting Engine
Reports can easily be delivered throughout the organization, both internally and externally, with simplified deployment and configuration. This enables users to easily create and share reports of any size and complexity.

• Internet Report Deployment
Customers and suppliers can effortlessly be reached by deploying reports over the Internet.

• Manage Reporting Infrastructure
Increase supportability and the ability to control server behaviour with memory management, infrastructure consolidation, and easier configuration through a centralized store and API for all configuration settings.

• Report Builder Enhancements
Easily build ad-hoc and author reports with any structure through Report Designer.

• Forms Authentication Support
Support for Forms authentication enables users to choose between Windows and Forms authentication.

• Report Server Application Embedding
Report Server application embedding enables the URLs in reports and subscriptions to point back to front-end applications.

• Microsoft Office Integration
SQL Server 2008 provides new Word rendering that enables users to consume reports directly from within Microsoft Office Word. In addition, the existing Excel renderer has been greatly enhanced to accommodate the support of features, like nested data regions, sub-reports, as well as merged cell improvements. This lets users maintain layout fidelity and improves the overall consumption of reports from Microsoft Office applications.

• Predictive Analysis
SQL Server Analysis Services continues to deliver advanced data mining technologies. Better Time Series support extends forecasting capabilities. Enhanced Mining Structures deliver more flexibility to perform focused analysis through filtering as well as to deliver complete information in reports beyond the scope of the mining model. New cross-validation enables confirmation of both accuracy and stability for results that you can trust. Furthermore, the new features delivered with SQL Server 2008 Data Mining Add-ins for Office 2007 empower every user in the organization with even more actionable insight at the desktop.

Working with SQL Table Partitions

When a database table grows in size to the hundreds of gigabytes or more, it can become more difficult to load new data, remove old data, and maintain indexes.
Even the data that we want to load or remove may be very few records, making INSERT and DELETE operations on the table becomes impractical.

In this Scenario, How can we face this issue and How can we achieve Performance ?

Partitioning a large table divides the table and its indexes into smaller partitions, so that maintenance operations can be applied on a partition-by-partition basis, rather than on the entire table.

In addition, the SQL Server optimizer can direct properly filtered queries to appropriate partitions rather than the entire table.

Table partitioning was introduced in SQL Server 2005 Enterprise Edition and enhanced in SQL Server 2008 Enterprise.

Every table in SQL Server 2005 and SQL Server 2008 may be looked upon as partitioned, in the sense that each table is given a default partition in the sys.partitions catalog view. However, a table is not truly partitioned unless it is created using a partition scheme and function, which ordinary database tables by default do not have.

In SQL Server, there are two ways to partition:
a. You can manually subdivide a large table's data into multiple physical tables.
b. you can use SQL Server's table partitioning feature to partition a single table.

1st way: Again there are two primary ways to partition data into multiple tables:

i. Horizontal Partitioning: where selected subsets of rows are placed in different tables
When a view is created over all the tables, and queries directed to the view, the result is a partitioned view.

Disadvantages: In a partitioned view, you have to manually apply the required constraints and operations, and maintenance can be complex and time-consuming.

ii. Vertical Partitioning: where the columns of a very wide table are spread across multiple tables containing distinct subsets of the columns with the same number of rows. The result is multiple tables containing the same number of rows but different columns, usually with the same primary key column in each table.

Often a view is defined across the multiple tables and queries directed against the view. SQL Server does not provide built-in support for vertical partitioning, but the new sparse columns feature of SQL Server 2008 can be a better solution for tables that require large numbers of columns

2nd Way: SQL Server's table partitioning differs from the above two approaches by partitioning a single table: here Multiple physical tables are no longer involved.

When a table is created as a partitioned table, SQL Server automatically places the table's rows in the correct partition, and SQL Server maintains the partitions behind the scenes.
You can then perform maintenance operations on individual partitions, and properly filtered queries will access only the correct partitions. But it is still one table as far as SQL Server is concerned.

A partitioned table is a unique kind of table in SQL Server. It depends on two pre-existing objects, the partition function and partition scheme, which are used only for partitioned tables and indexes


A partitioned table has a column identified as a partitioning column, and that column is referenced when the table is created. The partitioned table must be created on a partition scheme, which defines the filegroup storage locations for the partitioned table. The partition scheme in turn depends on a previously created partition function that defines the number of partitions the table will have and how the boundaries of the partitions are defined


Planning for Table Partitioning:

In order to successfully partition a large table, you must make a number of decisions. In particular, you need to:

Plan the partitioning:
Decide which table or tables can benefit from the increased manageability and availability of partitioning.
Decide the column or column combination upon which to base the partition.
Specify the partition boundaries in a partition function.
Plan on how to store the partitions in filegroups using a partition scheme.

Choosing a Table to Partition:

There is no firm rule or formula that would determine when a table is large enough to be partitioned, or whether even a very large table would benefit from partitioning.
Sometimes large tables may not require partitioning, some conditions where partitioning a table could benefit are:
Index maintenance on the table is costly or time-consuming and could benefit from re-indexing partitions of the table rather than the whole table at once.
Data must be aged out of the table periodically, and the delete process is currently too slow or blocks users trying to query the table.
New data is loaded periodically into the table, the load process is too slow or interferes with queries on the table, and the table data lends itself to a partition column based on ascending date or time.
In other words, make sure that the large table will actually benefit from partitioning; don’t partition it just because it’s big.


Choosing a Partition Column:

The partition column choice is critical, because after you have partitioned a table, choosing a different partition column will require re-creating the table, reloading all the data, and rebuilding all the indexes. Some things to note about the partition column:
The partition column must be a single column in the table (either a single column or a computed column).
If you have a combination of columns that form the best partition column, you can add a persisted computed column to the table that combines the values of the original columns and then partition on it.
In a clustered table, the partition column must be part of either the primary key or the clustered index. If the partition column is not part of the table's primary key, the partition column might allow NULL. Any data with a NULL in the partition column will reside in the leftmost partition.
You should also try to choose a partitioned column that will be used as a filter criterion in most of the queries run against the table. This enables partition elimination, where the query processor can eliminate inapplicable partitions from the query plan, and just access the partitions implied by the filter on the queries.

After you have identified the table and its partition column, you can then go through the steps of partitioning a table or index, which are:
Create or use an existing partition function that sets the correct range boundaries.
Create or use an existing partition scheme tied to the partition function.
Create the table using the partition scheme.

Benefits of Partitioned Tables and Indexes:

SQL Server’s partitioned tables and indexes offer a number of advantages when compared with partitioned views and other forms of partitioning data:
SQL Server automatically manages the placement of data in the proper partitions.
A partitioned table and its indexes appear as a normal database table with indexes, even though the table might have numerous partitions.
The table can be managed at the partition and filegroup level for ease of maintenance.
Partitioned tables support easier and faster data loading, aging, and archiving, as illustrated in the example above.
Application queries that are properly filtered on the partition column can perform better by making use of partition elimination and parallelism.
In cases where partitioned data will not be modified, you can mark some or most of a partitioned table's filegroups as read-only, making management of the filegroups easier.
In SQL Server 2008, you can compress individual partitions as well as control lock escalation at a partition level.

Partitioning large tables has some challenges:

There is a maximum of 1,000 partitions for a table.
You must manage filegroups and file placement if you place partitions on individual filegroups.
The metadata-only operations (SWITCH, MERGE, and SPLIT) can be blocked by other DML actions on the table at the time, until a schema-modification lock can be obtained.
Managing date or time-based data can be complex.
You cannot rebuild a partitioned index with the ONLINE option set to ON, because the entire table will be locked during the rebuild.
Automating changes to partitioned tables, as in a sliding window scenario, can be difficult, but Microsoft provides some tools to assist in automating the process.

Table Partitioning Best Practices:

Table partitioning is useful on very large data tables for primarily two reasons.
The major reason for partitioning is to gain better management and control of large tables by partitioning them. To gain better management of large tables, you can:
Rebuild and reorganize indexes by partition.
Use partition-aligned indexed views in switching operations.
Use a sliding window strategy for quickly bringing in new data and archiving old data.
Additionally, SQL Server's query optimizer can use partition elimination and parallelism to increase appropriately filtered query performance against partitioned tables. To make use of partition elimination:
Ensure that indexes are aligned with the partitioned table, and that indexed views are partition-aligned.
Ensure that queries against the partitioned tables have filters based on the partition column.
On data warehouse joins, keep the join column simple (such as an integer or date) and explicit, so as to take advantage of bitmap filtering for star joins.
In general, to take full advantage of table partitioning, you should:
Make sure that the configuration of max degree of parallelism is set sufficiently high to take advantage of parallel operations, or else add a MAXDOP query hint to fine-tune the degree of parallelism.
Maintain an empty partition on both ends of the partitioned table and ensure that only empty partitions are split and merged in a sliding window scenario.
Remember that RANGE RIGHT may be more convenient than RANGE LEFT in partition functions, especially when you are specifying date ranges.
Use data types without fractional components as partition columns, such as a date or an integer.
Always use a standard language-independent date format when specifying partition function boundary values.
Use an integer-based date and date dimension in data warehouses.
Use a single column of the table as the partitioned column whenever possible. If you must partition across more than one column, you can use a persisted computed column as the partitioning column. But then to achieve partition elimination, you must control queries to ensure they reference the partition column in their filters.
Use SWITCH with MERGE to drop partition data: Switch out the partition and remove the partition's boundary value using MERGE.
Use TRUNCATE TABLE to delete partition data by switching a partition out to a staging table and truncating the staging table.
Check for partition elimination in query plans.
Place read only data on read-only filegroups to reduce locking and simplify recovery for piecemeal restores.
Spread filegroups across all disks for maximum I/O performance.
Automate sliding window scenarios using available tools.

Example:

USE [master]
GO

CREATE DATABASE [SampleDB] ON PRIMARY
(
NAME = N'SampleDB',
FILENAME = N'C:\All Projects\TestDB\SampleDB.mdf' ,
SIZE = 3048KB ,
MAXSIZE = UNLIMITED,
FILEGROWTH = 1024KB
),

FILEGROUP [Filegroup1]
(
NAME = N'File1',
FILENAME = N'C:\All Projects\TestDB\File1.ndf' ,
SIZE = 2048KB ,
MAXSIZE = UNLIMITED,
FILEGROWTH = 1024KB
),

FILEGROUP [Filegroup2]
(
NAME = N'File2',
FILENAME = N'C:\All Projects\TestDB\File2.ndf' ,
SIZE = 2048KB ,
MAXSIZE = UNLIMITED,
FILEGROWTH = 1024KB
),

FILEGROUP [Filegroup3]
(
NAME = N'File3',
FILENAME = N'C:\All Projects\TestDB\File3.ndf' ,
SIZE = 2048KB ,
MAXSIZE = UNLIMITED,
FILEGROWTH = 1024KB
),

FILEGROUP [Filegroup4]
(
NAME = N'File4',
FILENAME = N'C:\All Projects\TestDB\File4.ndf' ,
SIZE = 2048KB ,
MAXSIZE = UNLIMITED,
FILEGROWTH = 1024KB
)
LOG ON
(
NAME = N'SampleDB_log',
FILENAME = N'C:\All Projects\TestDB\SampleDB_log.ldf' ,
SIZE = 1024KB ,
MAXSIZE = 2048GB ,
FILEGROWTH = 10%
)
GO

USE SampleDB
GO

CREATE PARTITION FUNCTION PF2_Left (int)
AS RANGE LEFT FOR VALUES (0, 10, 20);

CREATE PARTITION FUNCTION PF2_Right (int)
AS RANGE RIGHT FOR VALUES (0, 10, 20);

GO


CREATE PARTITION SCHEME PS2_Right
AS PARTITION PF2_Right
TO (Filegroup1, Filegroup2, Filegroup3, Filegroup4)


CREATE TABLE PartitionedTable
(
PartitionColumnID int not null,
Name varchar(100) not null
)
ON PS2_Right(PartitionColumnID);


INSERT PartitionedTable
select * from (values (31,'Amit'),(34,'Rahul'),(32,'Rajesh'),(33,'Ajay'),(28,'Ketan'))as tab(ID,NAME)


select * from PartitionedTable


More Examples:

REATE PARTITION FUNCTION myRangePF1 (int)
AS RANGE LEFT FOR VALUES (1, 100, 1000);
GO

CREATE PARTITION SCHEME myRangePS1
AS PARTITION myRangePF1
TO (test1fg, test2fg, test3fg, test4fg);



CREATE PARTITION FUNCTION myRangePF2 (int)
AS RANGE LEFT FOR VALUES (1, 100, 1000);
GO

CREATE PARTITION SCHEME myRangePS2
AS PARTITION myRangePF2
TO ( test1fg, test1fg, test1fg, test2fg );


CREATE PARTITION FUNCTION myRangePF3 (int)
AS RANGE LEFT FOR VALUES (1, 100, 1000);
GO

CREATE PARTITION SCHEME myRangePS3
AS PARTITION myRangePF3
ALL TO ( test1fg );



CREATE PARTITION FUNCTION myRangePF4 (int)
AS RANGE LEFT FOR VALUES (1, 100, 1000);
GO

CREATE PARTITION SCHEME myRangePS4
AS PARTITION myRangePF4
TO (test1fg, test2fg, test3fg, test4fg, test5fg)

--Creating a RANGE LEFT partition function on an int column


CREATE PARTITION FUNCTION myRangePF1 (int)
AS RANGE LEFT FOR VALUES (1, 100, 1000);

Values
col1 <= 1 col1 > 1 AND col1 <= 100 col1 > 100 AND col1 <= 1000 col1 > 1000



--Creating a RANGE RIGHT partition function on an int column


CREATE PARTITION FUNCTION myRangePF2 (int)
AS RANGE RIGHT FOR VALUES (1, 100, 1000);

Values
col1 < 1 col1 >= 1 AND col1 < 100 col1 >= 100 AND col1 < 1000 col1 >= 1000




--Creating a RANGE RIGHT partition function on a datetime column

CREATE PARTITION FUNCTION [myDateRangePF1] (datetime)
AS RANGE RIGHT FOR VALUES ('20030201', '20030301', '20030401',
'20030501', '20030601', '20030701', '20030801',
'20030901', '20031001', '20031101', '20031201');

DROP PARTITION FUNCTION myRangePF;

Best Practices for Building a Large Relational Data Warehouse

Consider partitioning large fact tables
Build clustered index on the date key of the fact table
Choose partition grain carefully
Design dimension tables appropriately
Write effective queries for partition elimination
Efficiently delete old data
Efficiently load the initial data
Manage statistics manually
Consider efficient backup strategies

Data Extract from XML file to SQL Server using SSIS

Data Extraction from XML file to SQL is simple just need to consider 2 things.
1. We must have XML file as a source file.
2. We must have XSD file as a source file.

XML file contains the tags and values where as XSD file contains the definition (rules) of the file so both the files are required for data population. If we gets only XML file as a source then we need to created XSD file for XML. Rest all data mapping formalities are same.

Remove HTML and XML Tags

select * , Left(Replace(Replace(Replace(Replace(Ltrim(tag),'<',''),'',''),'/',''),
Case when CHARINDEX('{0}',(Replace(Replace(Replace(Replace(ltrim(tag),'<',''),'',''),'/',''))) = 0
Then LEN(Replace(Replace(Replace(Replace(Ltrim(tag),'<',''),'',''),'/',''))
Else
CHARINDEX('{0}',(Replace(Replace(Replace(Replace(ltrim(tag),'<',''),'',''),'/','')))-1
END)
from TableName