Unoffical empeg BBS

Quick Links: Empeg FAQ | RioCar.Org | Hijack | BigDisk Builder | jEmplode | emphatic
Repairs: Repairs

Topic Options
#332541 - 28/04/2010 20:48 Defrag lunacy
tanstaafl.
carpal tunnel

Registered: 08/07/1999
Posts: 5546
Loc: Ajijic, Mexico
What is this sh*t? I thought I'd do a super-defrag on one of my backup drives, so I formatted the drive and then ran my backup program to produce a nicely compacted drive with no fragments. The backup program copied the files one at a time, in alphabetical order by directory, subdirectory, and file, took most of the night to do it.

The result was a nicely compacted disk with 70% fragmentation! (see the attached photo.)

How is this even possible? Did my backup program choose sectors pseudo-randomly based on astrological signs or something?

WTF.

tanstaafl.

(To see the attachment in readable form, right-click the file and choose "View Image")

db


Attachments
Disk Frag.jpg




Edited by tanstaafl. (28/04/2010 20:50)
_________________________
"There Ain't No Such Thing As A Free Lunch"

Top
#332544 - 28/04/2010 21:10 Re: Defrag lunacy [Re: tanstaafl.]
wfaulk
carpal tunnel

Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
Edit: Um, nevermind. I totally misread your post.

Alphabetical? Um, okay.

Try using a better defrag program, or it's successor.


Edited by wfaulk (28/04/2010 21:11)
Edit Reason: I'm an idiot
_________________________
Bitt Faulk

Top
#332546 - 28/04/2010 21:13 Re: Defrag lunacy [Re: tanstaafl.]
Taym
carpal tunnel

Registered: 18/06/2001
Posts: 2504
Loc: Roma, Italy
Wow. That is strange. I wonder if the fact that it is USB attached may have a role in explaining this. Otherwise, I'd tend to doubt of the defrag tool you're using. Have you tried other defrag software?
_________________________
= Taym =
MK2a #040103216 * 100Gb *All/Colors* Radio * 3.0a11 * Hijack = taympeg

Top
#332548 - 28/04/2010 21:54 Re: Defrag lunacy [Re: Taym]
altman
carpal tunnel

Registered: 19/05/1999
Posts: 3457
Loc: Palo Alto, CA
Could well be USB attached. There's some windows setting (or was on XP) about "optimize for quick removal" (vs optimize for performance). The upshot of this was that it flushes the FAT to the drive very often by default on removable drives, which:

a) Slows things down hugely
b) Could result in allocations for files being cut down to actual size part-way through a copy, which means it'd then be likely to fragment when the file copy continued.

Top
#332555 - 28/04/2010 23:31 Re: Defrag lunacy [Re: altman]
Dignan
carpal tunnel

Registered: 08/03/2000
Posts: 12338
Loc: Sterling, VA
I tend to like this GUI version of jkdefrag.

I do not have good luck with the Windows defrag software.
_________________________
Matt

Top
#332557 - 28/04/2010 23:56 Re: Defrag lunacy [Re: wfaulk]
tanstaafl.
carpal tunnel

Registered: 08/07/1999
Posts: 5546
Loc: Ajijic, Mexico
Originally Posted By: wfaulk
Try using a better defrag program, or it's successor.


The problem isn't the defrag program. I didn't use the defrag program. I formatted the disk to erase it, then used my backup program to copy the files, one at a time, in alphabetical order from the source disk to the newly erased backup disk.

I only used the defrag program to look at the freshly written disk. I was surprised to find out that it was 70% fragmented.

tanstaafl.
_________________________
"There Ain't No Such Thing As A Free Lunch"

Top
#332559 - 29/04/2010 00:16 Re: Defrag lunacy [Re: altman]
tanstaafl.
carpal tunnel

Registered: 08/07/1999
Posts: 5546
Loc: Ajijic, Mexico
Originally Posted By: altman
Could well be USB attached. There's some windows setting (or was on XP) about "optimize for quick removal" (vs optimize for performance). The upshot of this was that it flushes the FAT to the drive very often by default on removable drives, which:

a) Slows things down hugely
b) Could result in allocations for files being cut down to actual size part-way through a copy, which means it'd then be likely to fragment when the file copy continued.


I think Hugo may be on the right track here. The fragmented drive is a USB external drive optimized for quick removal. I have changed it to be optimized for performance, and will format it it again and repeat the experiment tonight. I'll also turn off the real-time virus checker to speed up the process. At least, I think it will...

I'll let you know in the morning how it works out.

tanstaafl.
_________________________
"There Ain't No Such Thing As A Free Lunch"

Top
#332560 - 29/04/2010 00:20 Re: Defrag lunacy [Re: tanstaafl.]
gbeer
carpal tunnel

Registered: 17/12/2000
Posts: 2665
Loc: Manteca, California
I'm with you.

It's strange that a newly formatted drive, would show such fragmentation, after being reloaded with files in the manner described.

I can only offer that maybe your software was not waiting for the each file to finish copying before starting the next. Those kind of overlapping copies could produce that result.

It would also thrash your cpu. Kind of like a denial of service attack.

If overlapping copies are taking place, Process Monitor will show it.
_________________________
Glenn

Top
#332562 - 29/04/2010 00:51 Re: Defrag lunacy [Re: gbeer]
hybrid8
carpal tunnel

Registered: 12/11/2001
Posts: 7738
Loc: Toronto, CANADA
When I was looking into fragmentation issues possibly affecting my PVR recording quality (it does), I noticed that writing files serially to the drive didn't necessarily produce a nice compact unfragmented disk.

I ended up installing defragmenting software that monitors the drives and defragments whenever they're not being used. Along with keeping a good amount of free space on each drive, I no longer seem to have any issues when writing out three or four HD recordings at one time.
_________________________
Bruno
Twisted Melon : Fine Mac OS Software

Top
#332569 - 29/04/2010 08:38 Re: Defrag lunacy [Re: gbeer]
tanstaafl.
carpal tunnel

Registered: 08/07/1999
Posts: 5546
Loc: Ajijic, Mexico
Originally Posted By: gbeer
I'm with you.

It's strange that a newly formatted drive, would show such fragmentation, after being reloaded with files in the manner described.

I can only offer that maybe your software was not waiting for the each file to finish copying before starting the next. Those kind of overlapping copies could produce that result.

It would also thrash your cpu. Kind of like a denial of service attack.

If overlapping copies are taking place, Process Monitor will show it.

As promised, I re-ran the procedure. The results this time are stranger than before.

It looks as though the about same amount (number of bytes) of the hard drive is fragmented, but this time "only" 40% of the files are fragmented. Whoopee, only 17,000 fragmented files instead of 34,000.

As Hugo suggested, the throughput on the backup was hugely improved by setting the Optimize for Performance flag, and probably turning off the virus checker helped a bit too.

I did watch the CPU activity with Task Manager the first time I did this, and it was thrashing, short duration spikes up to 95-100%, then back down to about 30%, then back up to 95%, every five seconds or so.

I suppose the next place to look is my backup software, a freeware program that I like called "Karen's Replicator". It has a simple UI and is not loaded down with features I don't need. Why does commercial backup software have to be so complex? Really, all I need to do is copy any files from the source drive that are either newer or not on the destination drive...

I'll try doing it again, this time from a c:> prompt using xcopy. I think that it will be a long process, though, because if I recall correctly the copying is much slower, probably to do with RAM caching or something.

tanstaafl.


Attachments
Disk Frag.jpg


_________________________
"There Ain't No Such Thing As A Free Lunch"

Top
#332570 - 29/04/2010 08:57 Re: Defrag lunacy [Re: tanstaafl.]
Taym
carpal tunnel

Registered: 18/06/2001
Posts: 2504
Loc: Roma, Italy
Originally Posted By: tanstaafl.
The problem isn't the defrag program. I didn't use the defrag program.


You did, in making the disk analysis and getting a response on how many files were fragmented. I don't completely trust that result you got. I would still suggest you at least give it a try with another defrag program and have it analyze (not defrag) the disk after the backup.

Quote:
Why does commercial backup software have to be so complex? Really, all I need to do is copy any files from the source drive that are either newer or not on the destination drive...


I recommend Robocopy. It is command line, but it is very streightforward to create a .cmd file and perform backup with all sorts of options. What you describe can be done very easily with robocopy. It is free, already included in 7, but freely downloadable from MS website. It is very very reliable. I've been using it for my backups in the last 5 years, and nothing compares to it smile .
_________________________
= Taym =
MK2a #040103216 * 100Gb *All/Colors* Radio * 3.0a11 * Hijack = taympeg

Top
#332571 - 29/04/2010 09:34 Re: Defrag lunacy [Re: Taym]
andy
carpal tunnel

Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
Microsoft Sync Toy is also very useful for that sort of backup and very easy to use. Shame about its name...

http://www.microsoft.com/downloads/detai...;displaylang=en
_________________________
Remind me to change my signature to something more interesting someday

Top
#332574 - 29/04/2010 10:46 Re: Defrag lunacy [Re: tanstaafl.]
peter
carpal tunnel

Registered: 13/07/2000
Posts: 4180
Loc: Cambridge, England
Originally Posted By: tanstaafl.
used my backup program to copy the files, one at a time, in alphabetical order

How sure are you that it copied them one at a time? That's a severely non-optimal strategy for a backup program; unless the destination drive is much slower than the source drive, it can go faster by having lots of simultaneous outstanding requests against the source drive. Ironically this is especially the case if the source drive is fragmented: because Windows and/or the drive will elevator-sort the requests, the average seek between requests is shorter (quicker) the more files are being copied simultaneously.

Quote:
I'll try doing it again, this time from a c:> prompt using xcopy. I think that it will be a long process, though, because if I recall correctly the copying is much slower, probably to do with RAM caching or something.

Or possibly because, unlike a decent backup program, it copies the files one at a time?

Peter

Top
#332582 - 29/04/2010 13:26 Re: Defrag lunacy [Re: tanstaafl.]
gbeer
carpal tunnel

Registered: 17/12/2000
Posts: 2665
Loc: Manteca, California
This doesn't bear on the fragmentation issue but, in the IDE days, didn't copy performance suffer if both drives were on the same ribbon?

Or maybe I'm thinking about copies from one diskette to another.

I'm expecting that this isn't a problem with sata>sata or usb>sata copies.
_________________________
Glenn

Top
#332589 - 29/04/2010 18:21 Re: Defrag lunacy [Re: peter]
tanstaafl.
carpal tunnel

Registered: 08/07/1999
Posts: 5546
Loc: Ajijic, Mexico
Originally Posted By: Peter
How sure are you that it copied them one at a time? That's a severely non-optimal strategy for a backup program; unless the destination drive is much slower than the source drive,


Reasonably sure. As it copies the files, it displays the directory path and file name of each file as it is being copied. Since many of my files are large (50+ megabytes) the copy process takes long enough that I can read the filenames as they go by.

Since the source drive is an internal SATA drive, and the destination drive is an external USB drive, I suspect that the destination drive is indeed much slower than the source drive.

tanstaafl.
_________________________
"There Ain't No Such Thing As A Free Lunch"

Top
#332590 - 29/04/2010 18:55 Re: Defrag lunacy [Re: Taym]
tanstaafl.
carpal tunnel

Registered: 08/07/1999
Posts: 5546
Loc: Ajijic, Mexico
Originally Posted By: taym
I would still suggest you at least give it a try with another defrag program and have it analyze (not defrag) the disk after the backup.

OK. Same results.

tanstaafl.


Attachments
MyDefrag.jpg


_________________________
"There Ain't No Such Thing As A Free Lunch"

Top
#332591 - 29/04/2010 20:43 Re: Defrag lunacy [Re: peter]
tfabris
carpal tunnel

Registered: 20/12/1999
Posts: 31597
Loc: Seattle, WA
Quote:
Quote:
I'll try doing it again, this time from a c:> prompt using xcopy. I think that it will be a long process, though, because if I recall correctly the copying is much slower, probably to do with RAM caching or something.

Or possibly because, unlike a decent backup program, it copies the files one at a time?


Copy and Xcopy are painfully slow for other reasons in addition to the fact that they do the job sequentially. I've been using XXCOPY for writing batch files that do drive-to-drive synchronization tasks and I'm quite happy with its speed and flexibility. I've also been known to use Beyond Compare sometimes solely for the purposes of doing drive-to-drive backup synchs.

I'm looking forward to trying out that Microsoft Sync Toy mentioned in this thread. I hadn't known about it until now.
_________________________
Tony Fabris

Top
#332598 - 30/04/2010 00:45 Re: Defrag lunacy [Re: tfabris]
Shonky
pooh-bah

Registered: 12/01/2002
Posts: 2009
Loc: Brisbane, Australia
Microsoft's Robocopy will run a lot faster than xcopy or copy.
_________________________
Christian
#40104192 120Gb (no longer in my E36 M3, won't fit the E46 M3)

Top
#332611 - 30/04/2010 14:33 Re: Defrag lunacy [Re: tanstaafl.]
Taym
carpal tunnel

Registered: 18/06/2001
Posts: 2504
Loc: Roma, Italy
Originally Posted By: tanstaafl.
Originally Posted By: taym
I would still suggest you at least give it a try with another defrag program and have it analyze (not defrag) the disk after the backup.

OK. Same results.
tanstaafl.


Ok, I am lost, then. It must be USB or the backup software itself.

Out of curiosity, I'd suggest to try
robocopy c:\source k:\destination /MIR
just as a quick try, after of course deleting everything on destination.

Robocopy can be found here.
And, I'd try the same on two internal SATA drives.
_________________________
= Taym =
MK2a #040103216 * 100Gb *All/Colors* Radio * 3.0a11 * Hijack = taympeg

Top
#332615 - 30/04/2010 15:59 Re: Defrag lunacy [Re: Taym]
hybrid8
carpal tunnel

Registered: 12/11/2001
Posts: 7738
Loc: Toronto, CANADA
Have you tried the same thing with an internally connected drive? Doesn't it also fragment?
_________________________
Bruno
Twisted Melon : Fine Mac OS Software

Top
#332639 - 01/05/2010 03:17 Re: Defrag lunacy [Re: hybrid8]
tanstaafl.
carpal tunnel

Registered: 08/07/1999
Posts: 5546
Loc: Ajijic, Mexico
Originally Posted By: hybrid8
Have you tried the same thing with an internally connected drive? Doesn't it also fragment?


No, I don't get excessive fragmenting on the internal drives, nor the one external SATA drive.

I formatted (again!) the offending 500GB USB external drive, and this time instead of using my backup software I used the Microsoft SyncToy software. It also copies the files one at a time, takes even longer to run than my backup software because it does a lot of file prep before it starts copying.

Results below...

tanstaafl.


Attachments
Defrag.jpg


_________________________
"There Ain't No Such Thing As A Free Lunch"

Top