I have quite a large collection of audio books stored on a 2-TB WD20EARS hard drive. (31,759 files in 4512 folders.) The books are stored in a hierarchical structure:

F:>Library
........UNHEARD
.............Author
..................Book Title
.......................[filename].mp3
.........iPOD PENDING
.............Author
..................Book Title
.......................[filename].mp3
........HEARD
.............Author
..................Book Title
.......................[filename].mp3

When I have listened to the books in the iPod, I delete the books from the iPod, move the "iPOD PENDING" books into the "HEARD" directory system, choose new books from "UNHEARD" and move them into "iPOD PENDING" and then copy them into the iPod.

I have a large Excel spreadsheet that I use as an index to the books on the hard drive. The spreadsheet contains Author, Title, a brief synopsis of each book, and a flag indicating whether or not I have listened to the book. As new books are added to the collection, I keep the index file up to date. I am very careful with this.

So...two days ago I cleared the iPod, moved the listened-to books into the "HEARD" directory, and using the synopses in the Excel index file chose the new books to load into the player. I was surprised when one of the books I had chosen from the index was not on the hard drive. Had I somehow failed to add it to the index? No, the title was in the index, the files weren't in the computer.

How odd, I thought. When I went to look for that book, I noted that it was one book out of 19 by the same author, Michael Connelly. (Harry Bosch books). Of the 19 Connelly books, 15 were missing, including some that I had already listened to so I know they had been there before.

Now it's kind of tricky to try and figure out what isn't present in a file system, but with the help of a program called "Agent Ransack" I was able to create a complete file/directory listing of my F:> hard drive, format it into something that Excel was happy with, and sort it into the same order as my index file. Then I put the two spreadsheets side by side and went down them line by line, looking for differences. Since my index spreadsheet only goes to folder level (one folder for each book) I only compared author/book title, not individual .mp3 files. I looked for discrepancies, and found some.

Aside from some mis-spellings and typographical errors (which I corrected), I found 41 books that I am quite certain used to be on my hard drive and now are gone. That would be at least 41 folders and about 400 files. (If the missing book was the only one by that author, then the "Author" folder and the Title folder would be gone, so for some books two folders could be missing, not just one.)

Even more disturbing, I found about half a dozen occurrences where Title sub-folders and their files had somehow migrated to be inside incorrect Author folders.

The SMART data for that drive show no obvious problems, so I think the drive may be OK physically. I am concerned about corruption in the file management table (or whatever it is called) on the drive.

What should I do now?

tanstaafl.

If the drive were the cause of the missing data, I think you'd probably notice the operating system also complaining about it at some point.

Since it didn't (did it?), then the operating system itself might be the primary suspect.

Cheers

Restore from backup. You have backup, right?

I don't know how long the files have been missing, but since I do regular backups, the files are gone from the backup disk as well as the data disk.

tanstaafl.

I guess what I'm asking is, is there a utility (like fsck in Linux) that can check and repair a Windows filesystem? Back in my MS-DOS days I used chkdsk. Is that appropriate on a Windows Vista system?

tanstaafl.

Assuming it is an NTFS partition, Windows will typically know at boot time that the filesystem is damaged. You can kick off a check manually though, right click on a drive in Explorer, select "Properties" and then use the "Error-checking" option on the "Tools" tab.

That's 9 folders and 78 files gone already. File copies are not backups: they don't save you from accidental file deletions, whether manual or caused by software bugs.

Peter

Done. No errors found.

I guess this is just one of those inexplicable Windows mysteries.

tanstaafl.

Now that's interesting, not to mention a bit frightening. But what are my options? Should I uncheck the box in my backup software that says "Replicate Folder and File Deletions?"? I don't think that would work for me because my source drive is not static data storage. There are folders being moved from one directory to another every time I load my iPod, and folders in the "iPOD PENDING" directory are duplicated in the "HEARD" directory until I am done with them on the iPod, at which point they are deleted. My backup drive would fill up pretty fast under those circumstances. I guess I could uncheck the box until the drive filled up, then uncheck it one time for the next backup, but there would be no easy way to determine whether the source drive was indeed missing any folders/files.

If it were just the 15 Michael Connelly books that turned up missing, I could convince myself that I had somehow deleted that directory by mistake. But I know I didn't make that mistake 26 different times on 26 different authors!

As regards the "...9 folders and 78 files gone already.", it is worse than that, since in the past two months I have added at least 18 folders and 200 files to that drive. No, wait, those new files are being held on a different drive pending additional processing.

I don't think (but can't be totally sure) that any single files have disappeared, it seems as though it is always an entire folder full of files that is missing. There is no apparent pattern to the deletions, some of the missing data was relatively new, some was years old. I did note that alphabetically 39 of the 41 missing folder names started with the first five letters of the alphabet.

I'll just start comparing my drive contents to my Excel index file a bit more often. It is a BIG job (hours of tedium) to do that, but I'll work on simplifying/automating it somewhat.

tanstaafl.

Would it work if you never moved the files/folders themselves, but had shortcuts to each title in the unheard/ipod/heard folders and moved those shortcuts around instead? That way, proper incremental backups would stay small (a bunch of shortcuts and whatever genuinely new titles had turned up), and it'd be easier to programmatically keep track of the main library folder, as nothing should ever get deleted from it.

Peter

What Peter said.

Your backup software should never be deleting content from the backup. It should merely remove the file reference from the latest incremental snapshot, and you need to keep enough snapshots to take you back several weeks or even months.

Personally, I take a backup set out of rotation every 3 months or so. I ought to be able to restore anything from the last 2 years. Whether I can find it (or even remember I had it) is another question entirely.

I think you would benefit from a Windows Home Server. I think I've told you this before, but it's able to do some pretty smart backups, including snapshots. This effectively enables you to return to the state of 'several backups ago', which seems to me like exactly what you need. (well, what you did need anyway)

I've used one for years and I can't recommend it enough.

I haven't been ignoring you, peter. I've been trying to wrap my mind around this concept for the past week, and there must be something fundamental that I don't understand.

Wouldn't that just add another layer of complexity to the directory stucture? Instead of looking at a directory, I would be looking at a pointer that points to the directory... not too sure what the advantage is, other than reducing the amount of physical file movement, which isn't all that great to begin with. Average of a gigabyte per week, which amounts to six one-hundredths of one percent of the data on the drive.

Would I have to go through and directory by directory create the shortcuts? That's over 4600 of them. Or is there some automated process that would do them for me?

Then there is the question of loading the files into the iPod. I use Copy Trans Manager (no iTunes on MY computer!, thank you Tony Fabris) and I don't think it would work to try and drag-and-drop shortcuts instead of directories/files into my iPod. So I'd still have to track down the actual directories, one at a time, rather than just drag the whole iPod Pending directory into the Shuffle.

Finally, I think I might have a clue as to what happened to the disappeared directories/files. My backup drive is a 2-TB drive in an external e-SATA dock. Something is not quite right with that setup, and on at least four occasions the computer has locked up during backup, forcing me to do a reboot with the backup software nominally still running. The dock can also use USB-2, and when connected that way I have never had a problem, so I have disconnected the e-SATA and switched to USB. I know that a forced reboot during processing can cause Bad Things to happen...

I was able to streamline and to some extent automate the F:> audit process, and what took over six hours the first time can now be done in 22 minutes. There were, unsurprisingly, more than 100 small errors, differences between my Excel index file and the directory structure in the F:> drive, misspellings, missing middle initial in an author's name, that sort of thing. I have recovered all but nine of the missing books (I guess the Alaska State Library System is my ultimate fail-safe backup! )and I am scheduled to get those in the next few weeks.

If you want some entertainment, look at the attached file. It is my "roadmap" for auditing my F:\Library.

tanstaafl.

This is probably it. eSATA is pretty much the same as having the hard drive plugged in internally. Depending on how Windows settings were set, it was probably using internal drive settings that use write caching and other methods to speed up the drive. These settings are fine as long as you can guarantee the drive will never be removed when in operation (or ensure the system doesn't power off/crash), and time is given to safely flush any caches (in main RAM, on the controller in the computer, and on the drive) out to the disk.

USB within Windows defaults to assuming the drive may be pulled at any time. Less caching ends up being used, and more confirmation is done to ensure the data is written to the disk as soon as possible. It's slower (even when you ignore the speed difference between USB2 and eSATA), but much safer.

But you implied that it's the amount of physical file movement that stops you taking incremental backups all the time. If none of the files ever actual moves, you can untick "Replicate file deletions" or whatever it's called, and incremental backups will remain small.
You (or someone) would have to write a very simple script that did it.

Peter