Any ext3 gurus out there?

Posted by: julf

Any ext3 gurus out there? - 17/12/2004 16:56

Arrrgghh... The classical case - a lot of work in setting up a system, twiddling everything in place, coaxing all pieces of software to work together - and finally, after weeks of work, just before doing a major backup, the system crashes and does not boot. Seems like a hard failure of disk (getting "partial read") in the ext3 filesystem journal. Any hints? good tools?
Posted by: Daria

Re: Any ext3 gurus out there? - 17/12/2004 17:45

i suppose force-mounting it ext2 is right out?
Posted by: mcomb

Re: Any ext3 gurus out there? - 17/12/2004 19:18

Quote:
i suppose force-mounting it ext2 is right out?


I'd force an fsck first (fsck -f). If that doesn't show anything try mounting as ext2. Then fall back to killing the ext3 info with...

tune2fs -O ^has_journal /dev/xyzX

Ext3 is just ext2 with a journal node so any ext2 tools should work fine at that point (if they didn't before).

-Mike
Posted by: julf

Re: Any ext3 gurus out there? - 18/12/2004 12:06

Quote:
I'd force an fsck first (fsck -f). If that doesn't show anything try mounting as ext2. Then fall back to killing the ext3 info with...

tune2fs -O ^has_journal /dev/xyzX


Unfortunately none of these work - they all barf on the hard seek errors, so I guess the hard disk is just physically too toasted. Looking forward to some heavy patchwork with a file system debugger - not! But first off to cold, dark Helsinki for a few days...
Posted by: tman

Re: Any ext3 gurus out there? - 19/12/2004 00:14

How about getting an identical or bigger disk and just doing a sector by sector copy? At least that way the sectors with errors will just be blanked out and fsck should be able to do a better job.
Posted by: julf

Re: Any ext3 gurus out there? - 26/12/2004 12:46

Quote:
How about getting an identical or bigger disk and just doing a sector by sector copy? At least that way the sectors with errors will just be blanked out and fsck should be able to do a better job.

Thanks! That's precisely what I was going to do as soon as I got back home - and now that I finally am (will post in a separate thread about my Christmas adventure) I will!
Posted by: julf

Re: Any ext3 gurus out there? - 05/01/2005 12:32

Quote:
How about getting an identical or bigger disk and just doing a sector by sector copy? At least that way the sectors with errors will just be blanked out and fsck should be able to do a better job.

OK, finally managed to set up another system with a big enough disk - and turns out a *huge* amount of sectors are toast. Oh well, just 3 weeks worth of work gone...

Anyway, I would like to understand what happened. The disk keeps giving me this:

hdb: dma_intr: status=0x51 { DriveReady SeekComplete Error }
hdb: dma_intr: error=0x40 { UncorrectableError }

Anyone (mlord?) with enough familiarity with the linux driver to tell what's going on?
Posted by: mlord

Re: Any ext3 gurus out there? - 05/01/2005 13:49

Hi Julf,

The message means pretty much what it says: "UncorrectableError" is a hard sector failure on the media. This can often be corrected by overwriting the entire disk with, say, zeros: cat /dev/zero >/dev/hda

As part of the overwrite, the drive firmware will fix the errors and try to remap the bad sectors automatically.

But first, it would be good to have a look at the S.M.A.R.T. logs, and run a low-level drive test. The S.M.A.R.T. data may tell WHY the sectors went bad, but most likely it won't.

Under Linux, smartmontools are needed, and the "smartctl" command in particular.

The IBM Drive Fitness Test (self-booting diskette image) does the same stuff, and can also low-level reformat IBM drives.

Cheers
Posted by: julf

Re: Any ext3 gurus out there? - 05/01/2005 14:07

Quote:

The message means pretty much what it says: "UncorrectableError" is a hard sector failure on the media.


Assumed as much, but wondered about the "DriveReady SeekComplete Error" part.
Quote:
This can often be corrected by overwriting the entire disk with, say, zeros: cat /dev/zero >/dev/hda

As part of the overwrite, the drive firmware will fix the errors and try to remap the bad sectors automatically.


Hmm. After having had thousands of bad sectors I think I won't use the disk as anything but paperweight.
Quote:

But first, it would be good to have a look at the S.M.A.R.T. logs, and run a low-level drive test. The S.M.A.R.T. data may tell WHY the sectors went bad, but most likely it won't.


The interesting thing would be drive temperature - will try checking it at some point.

Anyway, thanks and cheers!
Posted by: mlord

Re: Any ext3 gurus out there? - 05/01/2005 15:21

Yes, the S.M.A.R.T. log info about drive temperature could be very interesting to look at!

The other stuff DriveReady SeekComplete Error from my driver message is just an english translation of the low-level status bits, in this case the only important one is Error, which is then expanded upon on the following line UncorrectableError (media error).

Cheers
Posted by: mlord

Re: Any ext3 gurus out there? - 05/01/2005 15:24

Quote:
Hmm. After having had thousands of bad sectors I think I won't use the disk as anything but paperweight.


There may not be that many bad sectors -- again, only the S.M.A.R.T. logs will know for sure. Linux tends to overreport bad sectors, due to incomplete low-level error handling in (all) drivers. If there's one bad sector in a single sequential I/O for, say, 1000 sectors, Linux will usually report read-errors for the bad-sector and all which followed it in the same I/O request, not realizing the err in its ways..

Cheers
Posted by: julf

Re: Any ext3 gurus out there? - 06/01/2005 07:19

Quote:

There may not be that many bad sectors -- again, only the S.M.A.R.T. logs will know for sure. Linux tends to overreport bad sectors, due to incomplete low-level error handling in (all) drivers. If there's one bad sector in a single sequential I/O for, say, 1000 sectors, Linux will usually report read-errors for the bad-sector and all which followed it in the same I/O request, not realizing the err in its ways..


OK, showing my age (as someone who grew up with Good Old UNIX - as opposed to all this new-fangled Linux stuff ) - does the Linux driver/kernel do buffering even when you do a basic dd with BS=512? Is there a linux equivalent of a raw driver?
Posted by: julf

Re: Any ext3 gurus out there? - 06/01/2005 07:20

Quote:

The other stuff DriveReady SeekComplete Error from my driver message is just an english translation of the low-level status bits

Ahh. Yes, that makes sense. Thanks!
Posted by: mlord

Re: Any ext3 gurus out there? - 06/01/2005 12:14

The kernel normally buffers pretty much everything. If one uses O_DIRECT when opening files, then the page cache will be bypassed, and I/O will happen directly on the userspace buffers. This is great for doing bulk copies and the like. But the kernel still batches sequential sectors together, as otherwise I/O throughput would be terrible (think, 1MB/sec rather than 80MB/sec from a modern drive).

For sector-by-sector copy/recovery of a partially bad disk, I prefer to use a very low-level driver API to force sector-at-a-time whenever an error is reported. I wrote a throwaway tool for this once, but have lost it. It used the IDE_TASKFILE interface (ioctls).

Cheers
Posted by: julf

Re: Any ext3 gurus out there? - 06/01/2005 13:11

Quote:
I wrote a throwaway tool for this once, but have lost it.

That sounds all too familiar . I sure there are some real gems on all those old, mouldy QIC-24 cartridges in the basement... :-/
Posted by: bonzi

Re: Any ext3 gurus out there? - 06/01/2005 14:01

Quote:
Quote:
I wrote a throwaway tool for this once, but have lost it.

That sounds all too familiar . I sure there are some real gems on all those old, mouldy QIC-24 cartridges in the basement... :-/

How true. That thought (and sheer nostalgia) kept my 2400' 1600 bpi reel-to-reel tapes alive until few years ago
Posted by: julf

Re: Any ext3 gurus out there? - 06/01/2005 14:28

Quote:

How true. That thought (and sheer nostalgia) kept my 2400" 1600 bpi reel-to-reel tapes alive until few years ago

Ahh, think mine went mouldy and disintegrated from being stored in an unheated attic long ago...