Backup Exec B2D Job Rate Very Slow (Solved)
Problem with your backup time more than 24 hours because of slow job rate?
If you ever found that your backup to disk (B2D) time increase significantly as this picture, you may want to read this hub further. Firstly let speak about my media server specification and tape backup drive:
- Server HP DL 380 G5 with 4GB RAM and harddisk 2x 36GB (RAID1)
- MSA 400 with 6 x 146GB harddisk (RAID5)
- External tape backup HP Ultrium LTO 4
One day suddenly i found 1 harddisk on DL380 dead (we have 2 harddisk configured as RAID1) and causing degraded performance, since then i found that my daily backup job completion time increase from 9 hours to 36 hours. I checked the speed and found it only about 300MB/min , before it was 1600MB/min. Our daily data to backup is range from 700 to 800 GB.
36 hours is only for backup to disk job, i have another job which backup from disk to tape and this will took about 9 hours, means in total will be 45 hours (almost 2 days!). this is absolutely crazy.
With this speed i got big problem, i cannot run daily backup. Daily backup cannot run more than 24 hours as the next day backup job will not run.
Troubleshoting backup job rate problem
As this happen after 1 disk failed (RAID1), i quite confidence that by replacing with new harddisk the problem will gone. After replacing harddisk i wait for the system to full recovered and hope that the performance will back to normal. The harddisk recovery already fully complete but B2D job rate not change still about 300MB/min.
I tried to restart server several times but the result still same. Actually i ever experience this with external tape backup LTO4 and i resolve that issue by deleting the device on system, restart server and re-create/re-install tape driver. Base on that experience, i erase B2D files from backup exec console. I tried to run backup job again but the problem still exist.Oh yes, as additional information our B2D files divided into several 10GB files.
I begin scratching my head :-) as the problem should resolved ASAP because of our business nature.
Product related to tape backup
Culprit of this problem
Under my confusion i tried to defragment harddisk and when analyzing found that my B2D files were fragmented. We have 45% files defragmented and most of them are B2D files as below picture.
I started run defragmentation to that harddisk/storage media and it took very long time (maybe 1 day) because the size and quantity of files to be defragmented.
How to defragment harddisk?
To defragment harddisk is quite simple and straigt forward. On your server (mine user Win 2003 server), click Start, all programs, accessories, system tools, disk defragmenter
To run harddisk defragmenter you should have at least 15% of free space. If you click analysis it will analyze whether you need to defrag your harddisk or not without running harddisk defragmentation. For 3.2 TB as mine will need sometimes to defrag.
Result after dragment harddisk
My long waiting for defragment was paid. Not only B2D speed increase to 2,200MB/min. Backup time to tape also increase to 2,700MB/min from 1,300MB/min. This only by doing harddisk defragmentation.
Still Slow After Defrag Your Disk?
The solution above works fine after several months but suddenly i found the same problem again, job rate is very slow about 350MB/sec the normal is about 2000MB/sec.
After some testing, from defragmenting disk, deleting backup media (file), manually create new job, i found that the problem is coming from disk itself. But no disk fails at all.
When i check with HP Array Configuration Utility (ACU) i found that there is warning about battery, this is battery of smart array card, not server battery as my other experience.
Fragmented B2D files can be serious problem with backup time because harddisk head must looking for data in another location. Ideally when reading a file harddisk head read in contiguous location.
Fragmented files also decrease job rate tape device, as harddisk read files from different sector.
Do harddisk defragmentation when job rate become slow, normal speed for my media server is about 2000MB/Min. It may differ with yours depend on your hardware capability.