Defrag lunacy

What is this sh*t? I thought I'd do a super-defrag on one of my backup drives, so I formatted the drive and then ran my backup program to produce a nicely compacted drive with no fragments. The backup program copied the files one at a time, in alphabetical order by directory, subdirectory, and file, took most of the night to do it.

The result was a nicely compacted disk with 70% fragmentation! (see the attached photo.)

How is this even possible? Did my backup program choose sectors pseudo-randomly based on astrological signs or something?

WTF.

tanstaafl.

(To see the attachment in readable form, right-click the file and choose "View Image")

db

Edit: Um, nevermind. I totally misread your post.

Alphabetical? Um, okay.

Try using a better defrag program, or it's successor.

Wow. That is strange. I wonder if the fact that it is USB attached may have a role in explaining this. Otherwise, I'd tend to doubt of the defrag tool you're using. Have you tried other defrag software?

Could well be USB attached. There's some windows setting (or was on XP) about "optimize for quick removal" (vs optimize for performance). The upshot of this was that it flushes the FAT to the drive very often by default on removable drives, which:

a) Slows things down hugely
b) Could result in allocations for files being cut down to actual size part-way through a copy, which means it'd then be likely to fragment when the file copy continued.

I tend to like this GUI version of jkdefrag.

I do not have good luck with the Windows defrag software.

The problem isn't the defrag program. I didn't use the defrag program. I formatted the disk to erase it, then used my backup program to copy the files, one at a time, in alphabetical order from the source disk to the newly erased backup disk.

I only used the defrag program to look at the freshly written disk. I was surprised to find out that it was 70% fragmented.

tanstaafl.

I think Hugo may be on the right track here. The fragmented drive is a USB external drive optimized for quick removal. I have changed it to be optimized for performance, and will format it it again and repeat the experiment tonight. I'll also turn off the real-time virus checker to speed up the process. At least, I think it will...

I'll let you know in the morning how it works out.

tanstaafl.

I'm with you.

It's strange that a newly formatted drive, would show such fragmentation, after being reloaded with files in the manner described.

I can only offer that maybe your software was not waiting for the each file to finish copying before starting the next. Those kind of overlapping copies could produce that result.

It would also thrash your cpu. Kind of like a denial of service attack.

If overlapping copies are taking place, Process Monitor will show it.

When I was looking into fragmentation issues possibly affecting my PVR recording quality (it does), I noticed that writing files serially to the drive didn't necessarily produce a nice compact unfragmented disk.

I ended up installing defragmenting software that monitors the drives and defragments whenever they're not being used. Along with keeping a good amount of free space on each drive, I no longer seem to have any issues when writing out three or four HD recordings at one time.

As promised, I re-ran the procedure. The results this time are stranger than before.

It looks as though the about same amount (number of bytes) of the hard drive is fragmented, but this time "only" 40% of the files are fragmented. Whoopee, only 17,000 fragmented files instead of 34,000.

As Hugo suggested, the throughput on the backup was hugely improved by setting the Optimize for Performance flag, and probably turning off the virus checker helped a bit too.

I did watch the CPU activity with Task Manager the first time I did this, and it was thrashing, short duration spikes up to 95-100%, then back down to about 30%, then back up to 95%, every five seconds or so.

I suppose the next place to look is my backup software, a freeware program that I like called "Karen's Replicator". It has a simple UI and is not loaded down with features I don't need. Why does commercial backup software have to be so complex? Really, all I need to do is copy any files from the source drive that are either newer or not on the destination drive...

I'll try doing it again, this time from a c:> prompt using xcopy. I think that it will be a long process, though, because if I recall correctly the copying is much slower, probably to do with RAM caching or something.

tanstaafl.

You did, in making the disk analysis and getting a response on how many files were fragmented. I don't completely trust that result you got. I would still suggest you at least give it a try with another defrag program and have it analyze (not defrag) the disk after the backup.



I recommend Robocopy. It is command line, but it is very streightforward to create a .cmd file and perform backup with all sorts of options. What you describe can be done very easily with robocopy. It is free, already included in 7, but freely downloadable from MS website. It is very very reliable. I've been using it for my backups in the last 5 years, and nothing compares to it .

Microsoft Sync Toy is also very useful for that sort of backup and very easy to use. Shame about its name...

http://www.microsoft.com/downloads/detai...;displaylang=en

How sure are you that it copied them one at a time? That's a severely non-optimal strategy for a backup program; unless the destination drive is much slower than the source drive, it can go faster by having lots of simultaneous outstanding requests against the source drive. Ironically this is especially the case if the source drive is fragmented: because Windows and/or the drive will elevator-sort the requests, the average seek between requests is shorter (quicker) the more files are being copied simultaneously.


Or possibly because, unlike a decent backup program, it copies the files one at a time?

Peter

This doesn't bear on the fragmentation issue but, in the IDE days, didn't copy performance suffer if both drives were on the same ribbon?

Or maybe I'm thinking about copies from one diskette to another.

I'm expecting that this isn't a problem with sata>sata or usb>sata copies.

Reasonably sure. As it copies the files, it displays the directory path and file name of each file as it is being copied. Since many of my files are large (50+ megabytes) the copy process takes long enough that I can read the filenames as they go by.

Since the source drive is an internal SATA drive, and the destination drive is an external USB drive, I suspect that the destination drive is indeed much slower than the source drive.

tanstaafl.

OK. Same results.

tanstaafl.

Copy and Xcopy are painfully slow for other reasons in addition to the fact that they do the job sequentially. I've been using XXCOPY for writing batch files that do drive-to-drive synchronization tasks and I'm quite happy with its speed and flexibility. I've also been known to use Beyond Compare sometimes solely for the purposes of doing drive-to-drive backup synchs.

I'm looking forward to trying out that Microsoft Sync Toy mentioned in this thread. I hadn't known about it until now.

Microsoft's Robocopy will run a lot faster than xcopy or copy.

Ok, I am lost, then. It must be USB or the backup software itself.

Out of curiosity, I'd suggest to try
robocopy c:\source k:\destination /MIR
just as a quick try, after of course deleting everything on destination.

Robocopy can be found here.
And, I'd try the same on two internal SATA drives.

Have you tried the same thing with an internally connected drive? Doesn't it also fragment?

No, I don't get excessive fragmenting on the internal drives, nor the one external SATA drive.

I formatted (again!) the offending 500GB USB external drive, and this time instead of using my backup software I used the Microsoft SyncToy software. It also copies the files one at a time, takes even longer to run than my backup software because it does a lot of file prep before it starts copying.

Results below...

tanstaafl.