Backup and recovery best practices for Microsoft Exchange Server 2007 with HP servers and HP StorageWorks array and tape products
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . Solution conﬁguration . . . . . . . . . . . . . . . . . . . . . . Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . Testing objectives . . . . . . . . . . . . . . . . . . . . . . NTBackup with Exchange ESE streaming backup API . . . . . . . Backing up using 4 concurrent streams to 4 disks . . . . . . Backing up using 4 concurrent streams to 4 tape drives . . . . Backing up using 16 concurrent streams to 16 disks . . . . . Backing up using 16 concurrent streams to 4 disks . . . . . . Backing up using 32 concurrent streams to 32 disks . . . . . Backing up using 10 concurrent streams to 10 tape drives . . . Testing summary . . . . . . . . . . . . . . . . . . . . Exchange 2007 backups and VSS . . . . . . . . . . . . . . Exchange 2007 continuous replication and backups . . . . . Incremental backups . . . . . . . . . . . . . . . . Passive database backups . . . . . . . . . . . . . . Off-host database backups . . . . . . . . . . . . . . VSS and streaming backups . . . . . . . . . . . . . . . Best practices and results . . . . . . . . . . . . . . . . . . . . Monitor server workload . . . . . . . . . . . . . . . . . . . Keep mailbox databases to a manageable size . . . . . . . . . Use storage groups to gain a proper concurrency setting for backups Know the impact of multiplexing backups . . . . . . . . . . . . Be careful with VSS concurrency . . . . . . . . . . . . . . . Perform incremental backups to save time and space . . . . . . . Three important things to back up . . . . . . . . . . . . . . . Minimize LUN presentation for backup volumes . . . . . . . . . Beneﬁts of using streaming backups . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . Appendix A Bill of materials . . . . . . . . . . . . . . . . . . . For more information . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2 3 7 7 7 8 9
10 1 1 12 13
14 15 15 16 16 16 16 18 18 18 18
18 18 19 19 19 19 20 21 23
Backup and recovery operations are a critical activity with respect to Microsoft Exchange Server 2007. With messaging commonly viewed as a mission critical application in most companies, the ability to backup all or part of your Exchange data when required and within the speciﬁed backup window is vital. Microsoft is encouraging Exchange 2007 administrators to back up by utilizing their new embedded host-based Volume Shadowcopy Service (VSS) technology. However, many users will choose to continue to deploy online streaming backups either as a ﬁrst step in implementation, or as a valid method to simplify processes or leverage concurrency. This paper provides users in enterprise environments the data to effectively understand the options as well as the limitations of implementing online streaming backups into their infrastructure. Host-based VSS backups are also addressed and will be tested in depth at a later point. This paper is primarily directed to HP users in an enterprise environment currently running or planning to run Exchange 2007. This paper provides: • Best practice recommendations based on extensive lab test results; encompassing conﬁguration, design, process, and considerations • Backup and recovery recommendations for online streaming backup users, focusing on both disk and tape as targets (D2D2T and D2T) • A comparison of performance with Exchange 2003 • Supporting conﬁguration recommendations for HP servers and for HP StorageWorks 8000 Enterprise Virtual Array (EVA8000) • Recommendations for the use of multiple streams and storage group conﬁgurations to shorten backup windows • A discussion of backups with VSS, including advantages and new features • A discussion of the advantages and disadvantages of utilizing streaming versus host-based VSS for backups By leveraging the recommendations and best practices, administrators have the opportunity to shorten backup windows and efﬁciently load the server, thereby reducing cost and maximizing hardware and personnel resources.
The testing environment consisted of Microsoft Exchange 2007 utilizing HP ProLiant BladeServers and the HP StorageWorks EVA8000. The HP StorageWorks 6000 Enterprise Virtual Array (EVA6000) with Fibre attached technology (FATA) drives, and HP StorageWorks Enterprise Systems Library (ESL) e-Series 712e tape library were both used as dedicated backup and restore devices. Conﬁguration of the servers and SAN-attached devices was based on Microsoft Exchange 2007 planning and architecture documents, HP StorageWorks 4x00/6x00/8x00 Enterprise Virtual Array conﬁguration best practices white paper, available on http://h71028.www7.hp.com/ERC/downloads/4AA0-2787ENW.pdf, and HP StorageWorks Enterprise Backup Solution design guide, available on http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdf. For a list of hardware and software used in this project, see “Bill of materials” on page 21. During this test cycle, Windows Server 2003 NTBackup was the only released backup application supporting Exchange 2007. NTBackup was used for online backups with the goal of reducing backup windows, allowing more time for online maintenance and defragmentation. A reduction in restore times as a result of these solutions can allow more time for recovery operations, thus increasing the potential to meet tighter Recovery Time Objectives (RTOs).
Note Although NTBackup was used to conduct the testing discussed in this paper, the best practices developed through this testing can be applied to other backup applications to reduce backup windows and recovery time.
Exchange Mailbox Servers
The Microsoft Exchange 2007 servers consisted of HP ProLiant BL480c G1 BladeServers running Microsoft Windows Server 2003 R2 x64 Enterprise Edition. Each server was conﬁgured with two dual-port Fibre Channel mezzanine cards. The Fibre Channel cards were based on Emulex LP1 105 Series adapters and were installed using the HP host bus adapter (HBA) Smart Component drivers. The HBA drivers used were the HP Storport mini-port drivers along with the Microsoft Windows 2003 Storport driver. Storport drivers were used to create a compatible conﬁguration in which disk and tape can be accessed through the same HBA. For more tape compatibility information, see the Enterprise Backup Solution (EBS) compatibility matrix and the HP StorageWorks Enterprise Backup Solution design guide. The c7000 Blade enclosure provided the Fibre Channel and Ethernet connections to the ProLiant c-Class BladeServers. The enclosure also provided redundant power and cooling features along with Integrated Lights Out (iLO) connectivity. The iLO connectivity made it possible to deploy and manage remote servers from their consoles. All management features in the c-Class enclosure were interfaced through the c-Class Onboard Administrator (OA). In order to size the Exchange environment, the HP Sizing and Conﬁguration Tool for Microsoft Exchange Server 2007 and the Microsoft TechNet article Microsoft Exchange Server 2007 (available on http://technet.microsoft.com/en-us/library/bb124558.aspx) provided guidance on server CPU and memory sizing.
The EVA8000 storage array was set up in a 2C12D conﬁguration with 146 300-GB 10K Fibre Channel disk drives, which were used for the Exchange stores. An EVA6000 was set up in a 2C8D conﬁguration with 80 500-GB 10K FATA disk drives and was used as a target for D2D backups. The virtual disks created on the EVA8000 were built with a redundancy of Vraid 1 and the virtual disks created on the EVA6000 were built with a redundancy of Vraid 0. All Exchange database and log data was stored on the EVA. To verify the Exchange storage size, we used the Microsoft Exchange 2007 Storage Planning Calculator and HP Exchange 2007 Storage Calculator (http://h71019.www7.hp.com/ activeanswers/Secure/51 1755-0-0-0-121.html).
The HP StorageWorks ESL e-Series 712e tape library was used as the Nearline storage device. The ESL was conﬁgured with 16 HP StorageWorks Ultrium 960 (LTO3) Fibre Channel tape drives, 4 e2400-FC 4-Gb Fibre Channel Interface Controllers, and one Interface Manager.
Backup application server
NTBackup was run on the Exchange servers. In every test case backups were run from the Exchange server to the SAN attached storage.
Note A dedicated backup server is recommended when using a third-party backup application.
The HP StorageWorks Command View general purpose server running on an HP ProLiant DL580 G1 server was conﬁgured with Microsoft Windows Server 2003 Enterprise Edition SP1. HP StorageWorks Command View EVA 6.0 and Command View TL 1.80 were used to manage the EVA and tape devices, respectively.
Two independent fabrics consisted of HP StorageWorks Edge Switch 4/32 B-Series switches running ﬁrmware version 5.1.0b. All links to and from servers and storage devices were 4 Gb.
A single Active Directory (AD) Enterprise utilizing a domain controller and backup domain controller consisted of HP ProLiant BL20p servers running Microsoft Windows Server 2003 R2 Enterprise Edition and were conﬁgured as AD, DNS, DHCP, and Global catalog servers.
Application load simulation
Microsoft Exchange Load Generator (LoadGen) was used as the tool for creating the AD users and populating their mailboxes. LoadGen was run on the Exchange servers themselves. Users were
initialized with 100 MB and 1 GB mailbox sizes. Mailbox sizes were created by editing the LoadGen XML conﬁguration ﬁle for a speciﬁc distribution and weight of test messages.
Database storage layout
For this testing there were two Exchange 2007 mailbox servers. The ﬁrst server hosted 5,000 users with 100-MB mailboxes and the second server hosted 5,000 users with 1-GB mailboxes. For both servers one database LUN and one transaction log LUN were presented for every storage group. The server hosting the 5,000 100-MB mailboxes was conﬁgured with 32 storage groups with one mailbox database per storage group. Each 100-MB user mailbox database hosted 156 users and had a total mailbox database size of approximately 16 GB. Total server storage was approximately 520 GB. The server hosting the 5,000 1-GB mailboxes was conﬁgured with 50 storage groups, with one mailbox database per storage group. Each 1-GB user mailbox database hosted 100 users and had a total mailbox database size of approximately 100 GB. Total server storage was approximately 5 TB.
Figure 1. Test environment conﬁguration
1. Exchange LoadGen network clients 2. Management server 3. Exchange servers (BL480c) 4. Active directory and backup servers 5. 32-Port 4Gb B-Series Fibre Channel switches
6. EVA6000 (FATA) 7. EVA8000 8. ESL e-Series 712e 9. Local area network
Exchange 2007 offers many new features that directly and indirectly support high performance and feature rich backup and recovery solutions. Since online streaming backups through the Extensible Storage Engine (ESE) API were de-emphasized by Microsoft, much work has been done by backup application vendors to support backing up and restoring Exchange 2007 using VSS. VSS offers a way to create a consistent point-in-time copy of the database. Now that Cluster Continuous Replication (CCR) or Local Continuous Replication (LCR) provides standby copies of the Exchange database, better and simpler Exchange backups are possible. Some of the more indirect enhancements in Exchange 2007 that support backup and recovery include: • The 64-bit operating environment • The number of storage groups available • Exchange management console • Recovery storage groups • Database portability • Increased retention in the deleted items dumpster Customers using Exchange 2007 are expected to raise mailbox quotas, which will cause Exchange servers to host more data. Because there will be more data to back up and no more time in which to perform the backups, it is important to consider ways to accelerate the process. This paper focuses on testing performed with Exchange 2007 in an HP server and storage end-to-end backup solution. The goals of the testing were to: • Demonstrate ways of minimizing the backup windows through the use of storage group conﬁgurations with streaming backups. • Develop backup and recovery best practices. NTBackup was used to test Exchange streaming backups.
NTBackup with Exchange ESE streaming backup API
Since Microsoft has de-emphasized their support of the ESE streaming backup API, new features and functionality will not be available for this backup method. ESE streaming backup has changed very little in its support of Exchange 2007 and works in much the same way it does with Exchange 2003 to pull data out of the Exchange database and move it to backup media. We conducted the following tests to understand the effect of concurrency on backup performance: 1. 2. 3. 4. Backing up using 4 concurrent streams to 4 disks Backing up using 4 concurrent streams to 4 tape drives Backing up using 16 concurrent streams to 16 disks Backing up using 16 concurrent streams to 4 disks
Backing up using 32 concurrent streams to 32 disks Backing up using 10 concurrent streams to 10 tape drives
Each test is analyzed separately. Backing up using 4 concurrent streams to 4 disks We used NTBackup to backup Exchange in order to allow comparison between previous Exchange 2003 test data and the new Exchange 2007 test data. Exchange 2003 imposed a maximum of four storage groups. Typically, a storage group in Exchange is assigned to a volume partition on the server. If there were four partitions for the storage groups then it would be possible to back up using up to four concurrent streams. To enable a fair comparison, Exchange 2007 was conﬁgured with 5,000 users, each with a 100-MB mailbox, split among four storage groups and four mailbox databases per storage group, as was Exchange 2003 in the previous tests. In previous backup testing using dedicated disk backup partitions on an EVA6000 with FATA drives, Exchange 2003 was able to achieve a performance of 283 GB/hour or 520 GB in 1 hour and 50 minutes. When performing Exchange 2007 backups, using the same conﬁguration, similar performance was observed, therefore demonstrating that not much has changed in the streaming API for Exchange. Performance did not increase with the change to a 4-Gb storage area network (SAN) infrastructure and two additional Fibre Channel host bus adapters as well as an additional 28 GB of memory. Figure 2 highlights results of the test.
Figure 2. 4 streams to disk (D2D)
Note the page pool memory utilization and processor utilization in Figure 2. Knowing how hard the server is working to perform backups is a key indicator of Exchange server efﬁciency and can indicate potential room for improvement. With only 5% processor utilization and less than 70 MB of page pool memory in use, this server has plenty of room for a larger backup workload, assuming backups are done during scheduled backup windows where there is minimal user activity. The lack of performance improvement between Exchange 2003 and Exchange 2007, with its additional memory and faster storage networking infrastructure, indicates that the performance bottleneck must lie somewhere in the delivery of the data from Exchange to the backup media. It is possible that the streaming backup API limits the speed at which data is delivered to the backup media, due to operations such as consistency checks and small read block sizes of 64 KB.
Note HP recommends that you to use the ﬁle un-buffered switch with NTBackup jobs to disk, even on 64-bit Windows Server 2003. Not using this switch will cause processor and memory to be over-utilized resulting in poor backup performance and consequently extended backup windows. For more information, see the Microsoft support website: http://support.microsoft.com/kb/839272/.
Backing up using 4 concurrent streams to 4 tape drives Later generation tape technology has proven to be more efﬁcient than disk for high capacity backups due to the tape device’s embedded memory buffers and hardware compression. Therefore, we tested Exchange 2007 backups to tape in order to provide data for comparison. Leveraging the same test conﬁguration as the previous disk to disk testing with four streams, and using LTO3 tape devices instead of the FATA drives in the EVA, should improve performance given the memory and compression of the tape drive. Figure 3 shows that performance using tape with a four stream, four storage group conﬁguration was 15% better than when using disks. The same 520-GB database was backed up to tape in 1 hour and 30 minutes. Note that the increased performance corresponds to higher processor and memory utilization than that observed when using disks in the previous test.
Figure 3. 4 streams to tape
Backing up using 16 concurrent streams to 16 disks Although the Exchange online streaming backup API has not helped to improve the performance of Exchange 2007 backups with NTBackup, there are other features in Exchange 2007 that might improve backup performance. The primary focus of this testing was to increase the number of storage groups, resulting in an increase in the number of concurrent streams to backup media, in order to observe the effect on backup times and systems. Exchange 2007 was conﬁgured to spread 5,000 100-MB user mailboxes across 32 storage groups and one mailbox database per storage group. This allowed for the conﬁguration of NTBackup to send 16 concurrent streams to the EVA6000 for disk to disk (D2D) backups. Each backup stream moved two storage groups to the backup target. Figure 4 shows that the server achieved a 220% performance increase over four stream backups, with a transfer rate of approximately 624 GB/hour.
Figure 4. 16 streams to disk
Notice that the server used more CPU and memory to perform the 16 stream backups versus the four stream backups. One of the beneﬁts of using 64-bit Windows Server is the ability to use more than the previous 32-bit limitation of 4 GB of memory. Because this Exchange 2007 64-bit server had 32 GB of memory, kernel memory, or pool page bytes, no longer has the 32-bit, Exchange 2003 limitation of 180 MB. Even though the Exchange 2007 server did not go above this limit, pool page bytes serves as an indication of how hard the server had to work to get the backup done. CPU did not go above 50% processor time. Backing up using 16 concurrent streams to 4 disks The number of D2D target disk partitions was reduced to four from 16 to see if the number of target disks had an effect on overall throughput of the backup job. This will make it easier for storage administrators to manage the proliferation of logical units (LUNs) for Exchange by reducing the number of LUNs required for backup. Figure 5 shows that when the target backup disks were reduced to four, performance stayed the same; however, the server had to work a little harder to maintain the backup ﬁle images on the respective backup disk LUNs. Note how server memory gradually increased as backup ﬁle sizes grew on the limited number of backup disks.
Figure 5. 16 streams to 4 disks
Even with the slight growth in memory and CPU utilization in this test compared to the previous one, the workload did not strain the server and performance was maintained. This test shows the advantages in using disk-based backup technology over tape; multiple streams could be sent to a single disk target without the need for multiplexing or interleaving data streams. Multiplexing is a concern for some backup administrators because of the potential for increased restore time as data is de-multiplexed. Backing up using 32 concurrent streams to 32 disks In this test, the number of storage groups was increased to 32, each with a single mailbox database. Each of the 32 backup streams targeted a dedicated backup LUN on the EVA6000. Managing so many LUNs and partitions became increasingly complex. Figure 6 shows the decrease in backup performance which resulted from the server having to manage so many streams.
Figure 6. 32 streams to disk
Because NTBackup maintains a single job and graphical interface for each stream, the desktop on this server hosted 32 applications running concurrently, resulting in an increased system memory load. Backup performance dropped by 29% from 624 GB/hour in the 16 streams tests to 445 GB/hour. Note that page pool bytes increased dramatically from the 100–150 MB range in previous tests, to just over 400 MB. Although the upper limit for page pool bytes can be much higher on a 64-bit operating system, the change demonstrates the amount of work the server had to do to manage the multitude of backup streams. As shown in Figure 6, server load has a direct impact on backup performance. In this test, the performance diminished somewhere between 16 and 32 streams. Backing up using 10 concurrent streams to 10 tape drives Due to the efﬁciency of physical tape devices like the LTO3, high capacity backup jobs can achieve high throughput. The Exchange 2007 database was conﬁgured with 5,000 1-GB mailbox users across 50 storage groups. Ten backup streams were conﬁgured with NTBackup, with ﬁve storage groups per stream being sent serially to the tape device. Because physical tape devices offer improved performance due to hardware compression and data buffering, performance of this solution was approximately 1,330 GB/hour which represents an increase in performance of 470% over the four stream backups. It took 3 hours and 50 minutes to back up the 5-TB database. Figure 7 shows the moderate impact on the server resources.
Figure 7. 10 streams to tape
The test demonstrates the beneﬁt of using tape for high concurrency backups. Even higher performance would be expected if more tapes were used to enable more streams with this workload. Testing summary The test data shows that using more storage groups in Exchange 2007 will allow for higher backup concurrency which, in turn, should provide for higher performance backups. See Figure 8. Understanding server utilization during backups helps to identify opportunities for raising concurrency and backup performance. As seen in Figure 6, 32 concurrent streams to 32 disks, performance diminished as the server kernel memory became over-utilized. To reduce the number of backup LUNs required to achieve proper concurrency remember that multiple streams can be sent to a single disk volume, without the requirement to interleave the data, which would cause restores to take longer.
Figure 8. Number of storage groups verses performance with D2D backups
Exchange 2007 backups and VSS
VSS has proven useful in enabling the creation of a point-in-time copy of open ﬁles and databases. Microsoft has improved the VSS framework, making it easier for storage, servers and applications to work together. Previous VSS solutions were complex and costly because they required detailed knowledge of how the VSS storage providers, Exchange VSS writers, and backup application VSS requestors worked together. Few popular backup applications supported Exchange 2007 when it was released although some vendors had committed to support later in the year. With the growth in VSS and with Microsoft’s decision to move Exchange backups towards a VSS-based solution, the situation has improved; backup application providers are now taking their VSS integration seriously. Because VSS is a serial framework and concurrent backups of the same storage group are prohibited, the backup jobs must be conﬁgured carefully. Because VSS is also a volume level application, administrators should be cautious when running concurrent backups of storage groups located on the same partition. Because there will be limitations on concurrency, it is important to conﬁgure the backup applications so as to offset the jobs thus offsetting the VSS creation processes. The new Exchange Store Consistency Check Reference capability (chksgfiles.dll) allows the backup application to programmatically verify the integrity of Exchange storage groups and mailbox databases. This enables the backup application to balance the speed of the integrity check against the workload placed on the server. With earlier versions, VSS backups could not be restored to Exchange Recovery storage groups. With Exchange 2007, VSS backups can now be restored to an Exchange Recovery Storage Group on any Exchange server in the organization. Exchange 2007 continuous replication and backups Continuous replication capabilities, which provide log shipping services to remote storage or remote hosts, enhance availability in Microsoft Exchange 2007. Local Continuous Replication (LCR) provides for the database to be replicated to alternate storage on the same host, with log replay keeping the passive database up to date. Cluster Continuous Replication (CCR) provides for logs to be shipped to a passive cluster node where the replicated logs are replayed into the passive database. Although the beneﬁts for quick recovery are apparent, continuous replication also provides for some important possibilities with respect to other backup and recovery operations. Three capabilities which contribute to backup and recovery are discussed further: • Ability to perform daily incremental backups with weekly full backups
• Ability to minimize the reliance of backup windows by backing up the passive LCR or CCR copy of the database • Ability to perform off-host backups with the passive cluster node in CCR Incremental backups In the absence of continuous replication, daily full backups of the Exchange database are highly recommended because merging incremental with full backups extends recovery time and exposes the organization to the risk of losing data. Continuous replication reduces the risk of data loss and provides for recovery without having to restore from backups. Due to the reduced reliance on backups as the recovery source, incrementals are now a recommended strategy when using continuous replication which in turn will help to minimize backup windows. Passive database backups Using a passive rather than active copy of a database as the backup data source reduces contention for server and storage resources, assuming the passive copy of the database is on a different set of disks. VSS must be used to back up the database replicas because Microsoft has added new functionality to Exchange 2007 but has not enhanced the ESE streaming backup API. There are now two VSS writers that can be responsible for Exchange in the VSS framework: the Store writer and the Replication writer. In either passive or active database backups, restores will always be written to the active database or in other words, to the active server running the store process. The VSS integration is critical in maintaining database consistency through the use of replication services and checksum veriﬁcation of the VSS copy of the active or replica databases. If full or incremental backups of the database are done by using the continuous replica as a source, the replication service will keep the store header up to date in both the active and passive copies of the database to ensure database divergence does not occur. The replication service also plays a key role in how many logs will be truncated as a result of a full or incremental backup. Log truncation on the active database requires that the log ﬁles have been replayed into the active and passive databases and have been backed up using either a full or incremental backup. Off-host database backups When using CCR, a passive database cluster node can take responsibility for Exchange backups so that the active database node can reserve resources for the user and replication loads. The overhead incurred on the passive node should have little to no impact on the active server. This solution may alleviate pressures on enterprise backup windows or time for database maintenance activities. As with passive database backups, restores must be done to the active store. Incrementals can also be used with CCR off-host solutions. VSS and streaming backups VSS is deﬁning the way servers, storage, and applications work together to protect data. This technology is critical for applications that require consistent point-in-time copies. The ﬂexibility that VSS solutions now provide with respect to off-host backups and fast snapshots sets a new standard for Exchange backups. VSS has many beneﬁts. However, it is a relatively new technology and has some limitations. VSS limits in running concurrent streams and its dependence on data at the volume level have a negative impact on performance. Early testing has demonstrated VSS limits at approximately 8 concurrent jobs. Offsetting the start times for VSS backups so that multiple jobs do not start at
the same time can mitigate backup failures caused by VSS timeouts. In addition, because VSS is dependent on data at a volume level, running concurrent jobs from databases or storage groups on the same partition can cause backup failures and VSS snapshot creation timeouts. As VSS becomes a more mature backup framework, it is expected that many of these issues will be resolved. Meanwhile, streaming backups for Exchange 2007 should be considered as a way around the concurrency limitations imposed by VSS. Although not as fast as VSS in single stream applications, streaming backup is a strong alternative when concurrency is involved.
Best practices and results
Monitor server workload
Server administrators should note how Exchange servers perform during backups. Servers may have room to reconﬁgure backup jobs to allow for more concurrency which, in turn, should provide for better performance. In backing up 32 concurrent streams using NTBackup, 400 MB of page pool memory was used. This indicates that the server was over-taxed causing a decline in performance. Keeping page pool memory below 180 MB appears to contribute to efﬁciency and performance.
Keep mailbox databases to a manageable size
Because Exchange 2007 allows for many more mailbox databases, it is possible to allocate fewer users to each mailbox database. Keeping mailbox databases to a manageable size will enable faster restoration and will decrease server workload. Microsoft recommends keeping mailbox database sizes below 100 GB if continuous replication is not being used. Microsoft’s recommendation is based on the time it takes to perform a full recovery. A full recovery involves recovering the ﬁles that were backed up, and then replaying log ﬁles into the database to bring it to the point in time at which it was backed up. The smaller the database ﬁle, the faster the recovery of a database will be.
Use storage groups to gain a proper concurrency setting for backups
Use backup job concurrency to move more data. Typically, concurrency is applied at the volume level, meaning that most backup applications work well with concurrent streams when each stream is coming from a different disk volume. Exchange 2007 storage groups typically reside on a dedicated LUN and provide a good way for Exchange to take part in backup concurrency. Whether using streaming backups or VSS backups, more storage groups are available for higher degrees of concurrency. The data included in this report suggests that 16 concurrent streams provides for the fastest backups. For such high levels of concurrency consider using disk-to-disk backups as there would need to be 16 available tape drives to be able to have 16 concurrent backup streams to tape.
Know the impact of multiplexing backups
It is common to send multiple backup streams to the same tape device in order to keep the tape device streaming quickly and keep concurrency levels high when only a few tape drives are available. In most cases multiplexing can improve backup performance, but can cause restores to take much longer. For example, when four 25-GB database ﬁles are backed up in 4 streams to the same tape device using multiplexing, the backup ﬁle on tape will be a 100-GB backup ﬁle. In order to restore a single 25-GB database ﬁle, all 100 GB of the backup ﬁle must be restored. If fast backups are a requirement and only a few tape drives are available, consider staging the primary backups to a disk volume, and then moving the disk-based backups to tape. Note that there can be multiple concurrent streams to disk, without multiplexing the data streams.
Be careful with VSS concurrency
Creating VSS point-in-time copies is mostly a serial process. Be aware of how backup jobs are set up when using VSS integrated backups. If too many backup jobs are starting at the same time, the VSS process can time out. If a VSS job times out, the backup job can fail and will need to be restarted. Since many backups will use the VSS point-in-time copy as a method for backing up the database
to disk or tape, the point-in- time copy creation will be followed by the bulk data movement to the backup destination. By offsetting backup jobs with pauses (for example, 5 minutes) in between, the VSS point-in-time copy creation can take place on its own, while the bulk data movement will happen in a more concurrent fashion. VSS concurrency appears to become problematic at around 8 streams. Finally, if there are multiple Exchange storage groups per LUN, be sure that storage groups from the same LUNs are not backed up concurrently as VSS is a volume level service; backing up two storage groups from the same volume at the same time could cause conﬂicts.
Note Microsoft has released an update rollup package that resolves some VSS snapshot issues in Windows Server 2003. For more information, see http://support.microsoft.com/?kbid=940349 .
Perform incremental backups to save time and space
If Exchange 2007 continuous replication is being used, incremental backups can be used instead of daily full backups. Incremental backups help to maintain transaction log truncation, while reducing the amount of repeated data backed up every night. If using incremental backups, expect a longer recovery time to restore the full backup as well as every incremental in order to get the database to a point-in-time recovery. Also, plan for enough time to perform log replay after the ﬁles have been restored.
Three important things to back up
Although System State and the Exchange 2007 database are the obvious requirements for backup policies, consider also backing up the EVA conﬁguration data as a part of the regular backup items. EVA conﬁguration data can be collected by running HP StorageWorks Storage System Scripting Utility, which is included in Command View EVA or can be downloaded as a separate update from HP. Saving this conﬁguration data from the EVA will allow for a rapid replacement of the array, should the entire array need to be replaced.
Minimize LUN presentation for backup volumes
Although it is important to have a larger number of LUNs for the Exchange storage groups to backup using concurrent streams, the disks that are being backed up can be fewer than the total number of storage group volumes. Although fewer target volumes can be used, the server will have to work harder as it manages multiple, large, growing ﬁles on a ﬁle system.
Beneﬁts of using streaming backups
Streaming backups will beneﬁt from using concurrency to improve Exchange server backups. Although there are some limitations on how fast data can be streamed when using this backup method, the ability to perform concurrent backups outweighs any performance limitations. Streaming backups are being de-emphasized by Microsoft, however, they can be less complicated than VSS integrated backups.
This paper presented results and best practices derived from online streaming backup and recovery testing, and reported VSS backup testing observations. Both online and VSS backups can have a valid place in Exchange 2007 backup design. The speciﬁc recommendations resulting from this work demonstrate that early planning can have a large impact on the ability to perform efﬁcient, timely, and effective backup and restorations. Planning considerations: • Server workload is important to monitor. If the workload is low, consider reconﬁguring your backup jobs for more concurrency, leading to better performance. • Larger database sizes might not be better. Mailbox size needs to be manageable for faster restoration and decreased workload. • Leverage storage groups to maximize concurrency, conﬁguring storage groups on dedicated LUNs. Findings: • Based on four storage groups (the limit for Exchange 2003), no performance improvement was observed for disk-to-disk backup between Exchange 2003 and Exchange 2007. Disk-to-tape showed an approximate 15% performance improvement. • By taking advantage of the new Exchange 2007 storage group limits, users can increase concurrent streams to backup media. Testing supports that 16 concurrent streams provide the fastest backups. By referencing the lab-proven results from this paper and giving consideration to the derived best practices and recommendations, users are given the supporting information necessary to plan and implement an effective and efﬁcient Exchange 2007 backup and recovery strategy to meet their speciﬁc requirements.
Appendix A Bill of materials
The following table describes the speciﬁc environment utilized in testing and is included for reference purposes. BL480C Servers
Quantity 1 1 1 2 2 2 1 1 4 4 4 24 12 8 4 1 1 1 1 2 2 2 Part number AF002A AF002A 001 403321-B21 406740-B21 412138-B21 412140-B21 412142-B21 H4396A 416669-B21 U2426A 416669-B21 397413-B21 375861-B21 403621-B21 U2426A AF062A AF062A B01 AF054A AF054A 0D1 252663-D75 252663-D75 0D1 AE371A Description HP Universal Rack 10642 G2 Shock Rack Factory Express Base Racking HP BLc7000 1 PH 2 PS 4 Fan Full ICDC Kit HP BLc 1Gb Enet Pass Thru Mod Opt Kit HP BLc7000 Encl Pwr Sply IEC320 Option HP BLc7000 Encl Single Fan Option HP BLc7000 Encl Mgmt Module Option No Additional Support HP ProLiant BL480c G1 5160 4G 2P Svr Microsoft & Novell OE HP ProLiant BL480c G1 5160 4G 2P Svr HP 4GB FBD PC2-5300 2x2GB Kit HP 72GB 10K SAS 2.5 Hot Plug Hard Drive HP BLc Emulex LPe1 105 FC HBA Opt Kit Microsoft & Novell OE HP 10K G2 600W Stabilizer Kit Include with complete system HP 10642 G2 Sidepanel Kit Factory integrated HP 40A HV Core Only Corded PDU Factory integrated Brocade BladeSystem 4/24 SAN Swt Powr Pk
Quantity 1 80 2 1 2 2 Part number AD522A 364437-B23 T4256D AF002A T3724C T3732A Description EVA8000 2C12D FATA Drives, 500 GB FC 10K HP EVA4K/6K/8K 6.0 Controller Media Kit HP Universal Rack 10642 G2 Shock Rack HP Command View EVA v5.0 Media Kit HP CV EVA5000/8000 Unlim use per EVA LTU
Quantity 1 168 168 2 1 2 2 Part number AD522A 364622-B23 364622-B23 T4256D AF002A T3724C T3732A Description EVA8000 2C12D HDD, 300GB FC 1 10K Fact ALL HDD, 146GB FC 1 10K Fact ALL HP EVA4K/6K/8K 6.0 Controller Media Kit HP Universal Rack 10642 G2 Shock Rack HP Command View EVA v5.0 Media Kit HP CV EVA5000/8000 Unlim use per EVA LTU
Quantity 1 4 16 4 1 Part number AA934C AA938A AD595A AD576A SAN Zero drive, 712 LTO cartridge base library ESL E-Series Drive Cluster ESL E-Series Ultrium 960 FC Drive Upgrade Kit ESL E-Series e2400-FC 4G Interface Controller Interface Manager (comes with Enterprise Library)
Quantity 2 64 64 Part number A7393A A6515A SAN HP StorageWorks SAN Switch 4/32 HP Short Wave Optical Transceivers 50 micron SW Fiber Patch cords
Exchange client hardware 18 1 8 DC5000 Desktops, 6 DL320 Servers, 4 DL360 Servers CV EVA and CV general purpose server Networking 4 J4897A HP ProCurve 2724 Gig-E switch
For more information
This section lists references and their online locations. HP Links
Note Some of the following links are secure websites that require an HP Passport registration. HP Passport is a single login service that lets you register with HP Passport-enabled websites using a single user identiﬁer and password of your choice.
• HP Passport registration https://passport2.hp.com/hpp/newuser.do • HP Exchange 2007 Storage Calculator http://h71019.www7.hp.com/activeanswers/Secure/51 1755-0-0-0-121.html • HP Sizing and Conﬁguration Tool for Microsoft Exchange Server 2007 http://h71019.www7.hp.com/activeanswers/Secure/483374-0-0-0-121.html • HP Exchange 2007 ProLiant Sizer http://www27.compaq.com/sb/Exchange2007/index.asp • Customer Focused Testing solutions http://www.hp.com/go/hpcft • Enterprise Backup Solution (EBS). From the EBS page, click a quick link or select EBS123. http://www.hp.com/go/ebs • HP SAN documentation http://www.compaq.com/products/storageworks/san/documentation.html • HP StorageWorks Enterprise Backup Solution near online backup-restore solution implementation guide http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00230303/c00230303.pdf • HP StorageWorks Enterprise Backup Solution design guide http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00775232/c00775232.pdf • Enterprise Backup Solution Hardware/Software Compatability Matrix www.hp.com/go/ebs • HP support documentation (performance and troubleshooting) http://www.hp.com/support/pat • HP StorageWorks SAN design guide http://www.hp.com/go/sandesign • HP IT resource center (see the forums)
http://itrc.hp.com/ Microsoft Links: • NTBackup and Exchange 2007 http://technet.microsoft.com/en-us/library/aa998870.aspx • Exchange 2007 Backups described http://technet.microsoft.com/en-us/library/bb124515.aspx • NTBackup and VSS http://msexchangeteam.com/archive/2004/06/25/166104.aspx • Exchange Store Consistency Check Reference http://msdn2.microsoft.com/en-us/library/bb204219.aspx • Exchange 2007 TechNet article http://technet.microsoft.com/en-us/library/bb124558.aspx • You had me at EHLO… Microsoft Exchange Team Blog http://blogs.msdn.com/exchange • TechNet Webcast: High Availability and Clustering in Exchange Server 2007 https://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032313388&culture=en-us • Backup Process Used with Clustered Exchange Server 2003 Servers http://www.microsoft.com/downloads/details.aspx?FamilyId=63FA9270-563F-4627-A0A08A07E02CF9BF&displaylang=en
© 2007 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation. 4AA1-6077ENW, October 2007