#332541 - 28/04/2010 20:48
Defrag lunacy
|
carpal tunnel
Registered: 08/07/1999
Posts: 5546
Loc: Ajijic, Mexico
|
What is this sh*t? I thought I'd do a super-defrag on one of my backup drives, so I formatted the drive and then ran my backup program to produce a nicely compacted drive with no fragments. The backup program copied the files one at a time, in alphabetical order by directory, subdirectory, and file, took most of the night to do it.
The result was a nicely compacted disk with 70% fragmentation! (see the attached photo.)
How is this even possible? Did my backup program choose sectors pseudo-randomly based on astrological signs or something?
WTF.
tanstaafl.
(To see the attachment in readable form, right-click the file and choose "View Image")
db
Attachments
Edited by tanstaafl. (28/04/2010 20:50)
_________________________
"There Ain't No Such Thing As A Free Lunch"
|
Top
|
|
|
|
#332544 - 28/04/2010 21:10
Re: Defrag lunacy
[Re: tanstaafl.]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
Edit: Um, nevermind. I totally misread your post.Alphabetical? Um, okay. Try using a better defrag program, or it's successor.
Edited by wfaulk (28/04/2010 21:11) Edit Reason: I'm an idiot
_________________________
Bitt Faulk
|
Top
|
|
|
|
#332546 - 28/04/2010 21:13
Re: Defrag lunacy
[Re: tanstaafl.]
|
carpal tunnel
Registered: 18/06/2001
Posts: 2504
Loc: Roma, Italy
|
Wow. That is strange. I wonder if the fact that it is USB attached may have a role in explaining this. Otherwise, I'd tend to doubt of the defrag tool you're using. Have you tried other defrag software?
_________________________
= Taym = MK2a #040103216 * 100Gb *All/Colors* Radio * 3.0a11 * Hijack = taympeg
|
Top
|
|
|
|
#332548 - 28/04/2010 21:54
Re: Defrag lunacy
[Re: Taym]
|
carpal tunnel
Registered: 19/05/1999
Posts: 3457
Loc: Palo Alto, CA
|
Could well be USB attached. There's some windows setting (or was on XP) about "optimize for quick removal" (vs optimize for performance). The upshot of this was that it flushes the FAT to the drive very often by default on removable drives, which:
a) Slows things down hugely b) Could result in allocations for files being cut down to actual size part-way through a copy, which means it'd then be likely to fragment when the file copy continued.
|
Top
|
|
|
|
#332555 - 28/04/2010 23:31
Re: Defrag lunacy
[Re: altman]
|
carpal tunnel
Registered: 08/03/2000
Posts: 12338
Loc: Sterling, VA
|
I tend to like this GUI version of jkdefrag. I do not have good luck with the Windows defrag software.
_________________________
Matt
|
Top
|
|
|
|
#332557 - 28/04/2010 23:56
Re: Defrag lunacy
[Re: wfaulk]
|
carpal tunnel
Registered: 08/07/1999
Posts: 5546
Loc: Ajijic, Mexico
|
Try using a better defrag program, or it's successor. The problem isn't the defrag program. I didn't use the defrag program. I formatted the disk to erase it, then used my backup program to copy the files, one at a time, in alphabetical order from the source disk to the newly erased backup disk. I only used the defrag program to look at the freshly written disk. I was surprised to find out that it was 70% fragmented. tanstaafl.
_________________________
"There Ain't No Such Thing As A Free Lunch"
|
Top
|
|
|
|
#332559 - 29/04/2010 00:16
Re: Defrag lunacy
[Re: altman]
|
carpal tunnel
Registered: 08/07/1999
Posts: 5546
Loc: Ajijic, Mexico
|
Could well be USB attached. There's some windows setting (or was on XP) about "optimize for quick removal" (vs optimize for performance). The upshot of this was that it flushes the FAT to the drive very often by default on removable drives, which:
a) Slows things down hugely b) Could result in allocations for files being cut down to actual size part-way through a copy, which means it'd then be likely to fragment when the file copy continued. I think Hugo may be on the right track here. The fragmented drive is a USB external drive optimized for quick removal. I have changed it to be optimized for performance, and will format it it again and repeat the experiment tonight. I'll also turn off the real-time virus checker to speed up the process. At least, I think it will... I'll let you know in the morning how it works out. tanstaafl.
_________________________
"There Ain't No Such Thing As A Free Lunch"
|
Top
|
|
|
|
#332560 - 29/04/2010 00:20
Re: Defrag lunacy
[Re: tanstaafl.]
|
carpal tunnel
Registered: 17/12/2000
Posts: 2665
Loc: Manteca, California
|
I'm with you. It's strange that a newly formatted drive, would show such fragmentation, after being reloaded with files in the manner described. I can only offer that maybe your software was not waiting for the each file to finish copying before starting the next. Those kind of overlapping copies could produce that result. It would also thrash your cpu. Kind of like a denial of service attack. If overlapping copies are taking place, Process Monitor will show it.
_________________________
Glenn
|
Top
|
|
|
|
#332562 - 29/04/2010 00:51
Re: Defrag lunacy
[Re: gbeer]
|
carpal tunnel
Registered: 12/11/2001
Posts: 7738
Loc: Toronto, CANADA
|
When I was looking into fragmentation issues possibly affecting my PVR recording quality (it does), I noticed that writing files serially to the drive didn't necessarily produce a nice compact unfragmented disk.
I ended up installing defragmenting software that monitors the drives and defragments whenever they're not being used. Along with keeping a good amount of free space on each drive, I no longer seem to have any issues when writing out three or four HD recordings at one time.
|
Top
|
|
|
|
#332569 - 29/04/2010 08:38
Re: Defrag lunacy
[Re: gbeer]
|
carpal tunnel
Registered: 08/07/1999
Posts: 5546
Loc: Ajijic, Mexico
|
I'm with you. It's strange that a newly formatted drive, would show such fragmentation, after being reloaded with files in the manner described. I can only offer that maybe your software was not waiting for the each file to finish copying before starting the next. Those kind of overlapping copies could produce that result. It would also thrash your cpu. Kind of like a denial of service attack. If overlapping copies are taking place, Process Monitor will show it. As promised, I re-ran the procedure. The results this time are stranger than before. It looks as though the about same amount (number of bytes) of the hard drive is fragmented, but this time "only" 40% of the files are fragmented. Whoopee, only 17,000 fragmented files instead of 34,000. As Hugo suggested, the throughput on the backup was hugely improved by setting the Optimize for Performance flag, and probably turning off the virus checker helped a bit too. I did watch the CPU activity with Task Manager the first time I did this, and it was thrashing, short duration spikes up to 95-100%, then back down to about 30%, then back up to 95%, every five seconds or so. I suppose the next place to look is my backup software, a freeware program that I like called "Karen's Replicator". It has a simple UI and is not loaded down with features I don't need. Why does commercial backup software have to be so complex? Really, all I need to do is copy any files from the source drive that are either newer or not on the destination drive... I'll try doing it again, this time from a c:> prompt using xcopy. I think that it will be a long process, though, because if I recall correctly the copying is much slower, probably to do with RAM caching or something. tanstaafl.
Attachments
_________________________
"There Ain't No Such Thing As A Free Lunch"
|
Top
|
|
|
|
#332570 - 29/04/2010 08:57
Re: Defrag lunacy
[Re: tanstaafl.]
|
carpal tunnel
Registered: 18/06/2001
Posts: 2504
Loc: Roma, Italy
|
The problem isn't the defrag program. I didn't use the defrag program. You did, in making the disk analysis and getting a response on how many files were fragmented. I don't completely trust that result you got. I would still suggest you at least give it a try with another defrag program and have it analyze (not defrag) the disk after the backup. Why does commercial backup software have to be so complex? Really, all I need to do is copy any files from the source drive that are either newer or not on the destination drive... I recommend Robocopy. It is command line, but it is very streightforward to create a .cmd file and perform backup with all sorts of options. What you describe can be done very easily with robocopy. It is free, already included in 7, but freely downloadable from MS website. It is very very reliable. I've been using it for my backups in the last 5 years, and nothing compares to it .
_________________________
= Taym = MK2a #040103216 * 100Gb *All/Colors* Radio * 3.0a11 * Hijack = taympeg
|
Top
|
|
|
|
#332571 - 29/04/2010 09:34
Re: Defrag lunacy
[Re: Taym]
|
carpal tunnel
Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
|
Microsoft Sync Toy is also very useful for that sort of backup and very easy to use. Shame about its name... http://www.microsoft.com/downloads/detai...;displaylang=en
_________________________
Remind me to change my signature to something more interesting someday
|
Top
|
|
|
|
#332574 - 29/04/2010 10:46
Re: Defrag lunacy
[Re: tanstaafl.]
|
carpal tunnel
Registered: 13/07/2000
Posts: 4180
Loc: Cambridge, England
|
used my backup program to copy the files, one at a time, in alphabetical order How sure are you that it copied them one at a time? That's a severely non-optimal strategy for a backup program; unless the destination drive is much slower than the source drive, it can go faster by having lots of simultaneous outstanding requests against the source drive. Ironically this is especially the case if the source drive is fragmented: because Windows and/or the drive will elevator-sort the requests, the average seek between requests is shorter (quicker) the more files are being copied simultaneously. I'll try doing it again, this time from a c:> prompt using xcopy. I think that it will be a long process, though, because if I recall correctly the copying is much slower, probably to do with RAM caching or something. Or possibly because, unlike a decent backup program, it copies the files one at a time? Peter
|
Top
|
|
|
|
#332582 - 29/04/2010 13:26
Re: Defrag lunacy
[Re: tanstaafl.]
|
carpal tunnel
Registered: 17/12/2000
Posts: 2665
Loc: Manteca, California
|
This doesn't bear on the fragmentation issue but, in the IDE days, didn't copy performance suffer if both drives were on the same ribbon?
Or maybe I'm thinking about copies from one diskette to another.
I'm expecting that this isn't a problem with sata>sata or usb>sata copies.
_________________________
Glenn
|
Top
|
|
|
|
#332589 - 29/04/2010 18:21
Re: Defrag lunacy
[Re: peter]
|
carpal tunnel
Registered: 08/07/1999
Posts: 5546
Loc: Ajijic, Mexico
|
How sure are you that it copied them one at a time? That's a severely non-optimal strategy for a backup program; unless the destination drive is much slower than the source drive, Reasonably sure. As it copies the files, it displays the directory path and file name of each file as it is being copied. Since many of my files are large (50+ megabytes) the copy process takes long enough that I can read the filenames as they go by. Since the source drive is an internal SATA drive, and the destination drive is an external USB drive, I suspect that the destination drive is indeed much slower than the source drive. tanstaafl.
_________________________
"There Ain't No Such Thing As A Free Lunch"
|
Top
|
|
|
|
#332590 - 29/04/2010 18:55
Re: Defrag lunacy
[Re: Taym]
|
carpal tunnel
Registered: 08/07/1999
Posts: 5546
Loc: Ajijic, Mexico
|
I would still suggest you at least give it a try with another defrag program and have it analyze (not defrag) the disk after the backup. OK. Same results. tanstaafl.
Attachments
_________________________
"There Ain't No Such Thing As A Free Lunch"
|
Top
|
|
|
|
#332591 - 29/04/2010 20:43
Re: Defrag lunacy
[Re: peter]
|
carpal tunnel
Registered: 20/12/1999
Posts: 31597
Loc: Seattle, WA
|
I'll try doing it again, this time from a c:> prompt using xcopy. I think that it will be a long process, though, because if I recall correctly the copying is much slower, probably to do with RAM caching or something. Or possibly because, unlike a decent backup program, it copies the files one at a time? Copy and Xcopy are painfully slow for other reasons in addition to the fact that they do the job sequentially. I've been using XXCOPY for writing batch files that do drive-to-drive synchronization tasks and I'm quite happy with its speed and flexibility. I've also been known to use Beyond Compare sometimes solely for the purposes of doing drive-to-drive backup synchs. I'm looking forward to trying out that Microsoft Sync Toy mentioned in this thread. I hadn't known about it until now.
|
Top
|
|
|
|
#332598 - 30/04/2010 00:45
Re: Defrag lunacy
[Re: tfabris]
|
pooh-bah
Registered: 12/01/2002
Posts: 2009
Loc: Brisbane, Australia
|
Microsoft's Robocopy will run a lot faster than xcopy or copy.
_________________________
Christian #40104192 120Gb (no longer in my E36 M3, won't fit the E46 M3)
|
Top
|
|
|
|
#332611 - 30/04/2010 14:33
Re: Defrag lunacy
[Re: tanstaafl.]
|
carpal tunnel
Registered: 18/06/2001
Posts: 2504
Loc: Roma, Italy
|
I would still suggest you at least give it a try with another defrag program and have it analyze (not defrag) the disk after the backup. OK. Same results. tanstaafl. Ok, I am lost, then. It must be USB or the backup software itself. Out of curiosity, I'd suggest to try robocopy c:\source k:\destination /MIR just as a quick try, after of course deleting everything on destination. Robocopy can be found here. And, I'd try the same on two internal SATA drives.
_________________________
= Taym = MK2a #040103216 * 100Gb *All/Colors* Radio * 3.0a11 * Hijack = taympeg
|
Top
|
|
|
|
#332615 - 30/04/2010 15:59
Re: Defrag lunacy
[Re: Taym]
|
carpal tunnel
Registered: 12/11/2001
Posts: 7738
Loc: Toronto, CANADA
|
Have you tried the same thing with an internally connected drive? Doesn't it also fragment?
|
Top
|
|
|
|
#332639 - 01/05/2010 03:17
Re: Defrag lunacy
[Re: hybrid8]
|
carpal tunnel
Registered: 08/07/1999
Posts: 5546
Loc: Ajijic, Mexico
|
Have you tried the same thing with an internally connected drive? Doesn't it also fragment? No, I don't get excessive fragmenting on the internal drives, nor the one external SATA drive. I formatted (again!) the offending 500GB USB external drive, and this time instead of using my backup software I used the Microsoft SyncToy software. It also copies the files one at a time, takes even longer to run than my backup software because it does a lot of file prep before it starts copying. Results below... tanstaafl.
Attachments
_________________________
"There Ain't No Such Thing As A Free Lunch"
|
Top
|
|
|
|
|
|