Dead Mk2?

Posted by: tfabris

Dead Mk2? - 22/03/2005 21:06

Heh, did that thread title, combined with this user name, get your attention?

Thought I'd cash in on Meatballman's other thread popularity...

My player isn't dead, it's not even very ill. At least I don't think it is. But it's done something strange a few times recently.

The player app freezes up mid-song. The visuals freeze and the music stops. Almost as if someone had pressed the pause key, but without the on-screen indication of a pause.

But pressing and holding the top button still puts it into standby mode (so I can pull it from the dash and reboot it without a speaker pop). After the reboot, all works well and it even seems to remember which song it was on and remember the correct point to continue playing again.

Details:

It only started doing this recently, I made no changes to the player between the time it worked fine and the time it started doing this.

I didn't add any bad songs to the player, as far as I know. The fact that it works fine after a reboot seems to indicate it's not a bad song (if it were a bad song it'd re-cache the same songs after a reboot).

It only seems to do it after a certain amount of time playing in the car. 30-60 minutes or so.

It does not seem to be temperature-related, it still happens when I've got the cooling fan turned on. When I pull it from the sled, it doesn't feel unusually hot (it's been much hotter and ran fine).

Once, it did it while GPSApp was running in the foreground. GPSApp continued to run for quite a while, but eventually it also froze.

It does not do this on AC power, only DC power in the car. So I am unable to monitor its serial port output during the failure.

It does not do this while sitting on the passenger seat and playing from a cig-lighter adapter into a tape-input adapter in a 4wd pickup truck on a bumpy ride up to the slopes.

I tried adding [hijack] stalk_enabled=0 to the config.ini, no fix to the behavior.

I checked for the "missed interrupts" message on the serial port when it was on home power, no missed interrupts reported.

I have not seen any indications of hard disk trouble. I mean, other than this if you count this as an indication.

Software version is 2.0 final with Hijack 413.

Any other ideas?
Posted by: pgrzelak

Re: Dead Mk2? - 22/03/2005 21:13

Can you force DC power using one of Mark's docks and watch the serial port there? The fact that it does not do it on AC, even when on the seat, makes me think that it is sled / wiring related. Is your sled easy to remove, to look for loose wires on the loom (especially the serial link ones)?

Also, if not already there, turn on the HDD indicator, to see if you are getting any starved thread conditions.

Additional: Any hardware modifications? Especially on the memory end of things. Does your memory test run cleanly?
Posted by: SE_Sport_Driver

Re: Dead Mk2? - 22/03/2005 21:23

Jiffies.
Posted by: tfabris

Re: Dead Mk2? - 22/03/2005 21:30

Good ideas, Paul, I'll try those.

No hardware mods to the memory, but I've got the lit-button hack and I've added a fan which is running off the 3v rail.
Posted by: pgrzelak

Re: Dead Mk2? - 22/03/2005 21:38

In a way, it almost sounds like a memory leak, building until the player app can no longer allocate and crashes. That would explain why the GPSapp would still run for a while but eventually crash too. No explaination of why only on DC power, though. Unless there is something in your config.ini tied to DC vs. AC. I am also thinking it might be a bad memory hit, but (again) no explaination why only on DC.

I really wish you could capture a console log. It really seems like the player app is going down - hard. Or getting blocked by a resource request that cannot be met.

Does it happen with a vanilla config.ini?
Posted by: mcomb

Re: Dead Mk2? - 22/03/2005 22:26

Quote:
In a way, it almost sounds like a memory leak


Thats the same thought I had. Maybe it only happens on DC because Tony only has GPSapp running on DC and GPSapp is leaking mem? Or the player is at fault, but it only shows up on DC because there is more spare memory running on AC without GPSapp?

-Mike
Posted by: tfabris

Re: Dead Mk2? - 22/03/2005 22:32

I have "Launcher" running on DC, and choose GPSapp or emphatic or whatever as needed. So they don't take up RAM when not running. But it's happened without those things running before.

Right now, I'm running it forced-DC with a serial terminal open, not happening yet. Will try other suggested diagnostics later if this one doesn't pan out.
Posted by: mlord

Re: Dead Mk2? - 23/03/2005 13:50

Standby for a new "Tony Edition" general release of Hijack.

Cheers
Posted by: tfabris

Re: Dead Mk2? - 25/03/2005 01:38

Wow, cool, Mark, thanks!

I have no idea what you're planning, but I guess you've got some ideas?

More information on this topic came out the last couple of days:

- I had it happen when running in AC mode once. While it was connected to a cigarette lighter 12v adapter riding on the passenger seat of the pickup truck on its way to the slopes. So DC power is not the factor I originally thought it would be.

- I noticed that the player app does not actually freeze. All that freezes is the playback of the song, but with no pause indicator. I can open the player's main menu and futz around with it. Pressing pause/unpause and hush/unhush on the remote does nothing.

Does any of that help?
Posted by: mlord

Re: Dead Mk2? - 25/03/2005 01:45

Hijack v419 has the "Tony features" you need, I think.

cheers
Posted by: tfabris

Re: Dead Mk2? - 25/03/2005 01:51

/me grabs anxiously...

Thank you!!!!!

In case it really is a hard disk fault, is one of the popups a hard disk trouble indicator? Version note just said "various" hard/soft faults.
Posted by: tfabris

Re: Dead Mk2? - 25/03/2005 01:56

I see my question was answered in an adjacent thread. THANK YOU!
Posted by: tfabris

Re: Dead Mk2? - 30/03/2005 19:25

I haven't seen the problem happen since the new kernel was installed.

It's a weird feeling, actually HOPING for a hardware flaw, just so I can see a new message.
Posted by: tfabris

Re: Dead Mk2? - 05/04/2005 12:37

Interesting follow-up.

I had the problem happen again recently. I was driving home, talking on the cell phone, and I'd had the player in standby mode. The player had booted into standby mode, since it had been in standby mode in the office.

I finished the phone conversation, woke the player, and it started playing some music. After only a few bars of the song, it did that freeze. Visuals froze, music stopped. Note that the time-dither effect of the visuals was still running, but the visuals didn't "move".

No error messages on the screen at all.

It's possible that the error happened when it was trying to fill the cache, and therefore it happened when the player was booted in standby mode, and therefore not visible on the deactivated VFD display. Sigh.

I tried something, too. I tried pressing down-down-down to get it to reshuffle a new playlist, and it gave me the hourglass and then locked up the user interface. I couldn't press/hold the top button to put it into standby any more. However when I got home and shut off the ignition, it correctly went into standby. When I plugged the player back in indoors, it resumed the playlist it had been playing prior to my down-down-down attempt. (In other words, the down-down-down thing did nothing at all to it.)
Posted by: mlord

Re: Dead Mk2? - 05/04/2005 21:28

Mmm.. if it's a disk error, the message will not appear until after the read's timeout and fail mutiple tries (lousy error handling for this situation, I know..).

I can't think of anything else it might be, but..
Posted by: tfabris

Re: Dead Mk2? - 10/04/2005 16:12

Okay, here's something fun and interesting.

The player was playing a song. The song played fine all the way through.

Just before the end of the song, I decided I wanted to hear it a second time. So I pressed the "Left arrow" button on the faceplate. The song started over from the beginning as expected.

Then, about two bars into the second replay of the song, it did the same freeze. No error messages on the screen. At any time, even prior to this happening.

Now, here's my thinking. If this had been hard disk trouble and the freeze happens when it tries to play a song that didn't get fully cached, you'd think that this song would already be in the player's cache, right? Since I already heard the whole song and had simply started it over.

So I'm wondering if we're still looking at disk trouble here? Or maybe something else?
Posted by: mlord

Re: Dead Mk2? - 10/04/2005 16:39

Dunno. The player s/w tries to handle caching internally, and exactly how that works is a mystery to anyone without access to the source code. It would make sense for them to toss the data after it's been sent to the audio jacks.
Posted by: peter

Re: Dead Mk2? - 10/04/2005 17:33

Quote:
Now, here's my thinking. If this had been hard disk trouble and the freeze happens when it tries to play a song that didn't get fully cached, you'd think that this song would already be in the player's cache, right?

Only if you are certain that the cache was not refilled at any point during the song's first play. If it had been, already-played blocks except the first few would have been discarded.

Peter
Posted by: tfabris

Re: Dead Mk2? - 10/04/2005 22:08

Ah, so I'm misunderstanding the way the cache works. It doesn't always keep the current song in memory. If it does read-ahead during the song, then it makes room by discarding already-played portions of the current song. Makes sense.

So OK, we're still looking at hard disk trouble possibly then?
Posted by: SonicSnoop

Re: Dead Mk2? - 11/04/2005 00:37

Have another drive to just throw in and see if it still does it?
Posted by: tfabris

Re: Dead Mk2? - 11/04/2005 00:50

Good question, good idea. I don't have an extra drive with sufficient capacity (40gb) to trade into it. The freeze happens so intermittently that I might go a week or more before seeing it, so I also don't want to go down to one of my 10gb spares because that would leave me with a paltry subset of my true collection.

But I wonder if there is a tool, either on the player or that I could add to the player, that will nondestructively test the hard disk sectors?

I know so little about the linux tool set, for all I know the right parameters to FSCK would do it, and I'd have no idea.
Posted by: Roger

Re: Dead Mk2? - 11/04/2005 06:32

Quote:
But I wonder if there is a tool, either on the player or that I could add to the player, that will nondestructively test the hard disk sectors?


Assuming that the fsck program on the player supports it, you can pass the '-c' switch to fsck.ext2, which will cause a read-only block test. If you pass it twice, you get a non-destructive read-then-write test.
Posted by: peter

Re: Dead Mk2? - 11/04/2005 10:26

Quote:
Assuming that the fsck program on the player supports it, you can pass the '-c' switch to fsck.ext2, which will cause a read-only block test. If you pass it twice, you get a non-destructive read-then-write test.

Fsck does this by running the "badblocks" program, which you can also run directly. But badblocks looks for individual sectors with hard errors, which is not the problem you have -- or at least, not the main problem. Your experience with the song that wouldn't re-play says that a sector which read right the first time, failed the second time. Running badblocks on AC power won't diagnose whatever it is (heat? vibration? bad solder joint?) that makes the whole drive sporadically cease to exist (from the player's point of view) when in the car.

Peter
Posted by: mlord

Re: Dead Mk2? - 11/04/2005 10:54

Quote:
badblocks looks for individual sectors with hard errors, which is not the problem you have -- or at least, not the main problem. Your experience with the song that wouldn't re-play says that a sector which read right the first time, failed the second time.


That's really hard to know for sure. It could easily be a bad sector in the current *physical* track that's mucking things up, outside of the sectors actually needed to play the song.

A bad block scan is what needs to be done. To begin with, that could be as simple as "cat /dev/hda >/dev/null" -- and then sit back and wait for 14 hours.

Cheers
Posted by: tfabris

Re: Dead Mk2? - 11/04/2005 13:03

Quote:
To begin with, that could be as simple as "cat /dev/hda >/dev/null" -- and then sit back and wait for 14 hours.

Cool, I'll give this a shot. Would it be just as effective if I do it without mounting the drives Read Write?
Posted by: pgrzelak

Re: Dead Mk2? - 11/04/2005 13:04

It should be. It is a read check, and this will go through the entire drive as a device file instead of as a filesystem, so it should not matter.
Posted by: peter

Re: Dead Mk2? - 11/04/2005 13:23

Quote:
That's really hard to know for sure. It could easily be a bad sector in the current *physical* track that's mucking things up, outside of the sectors actually needed to play the song.

But surely it would have to be an intermittently bad sector, or it would have failed the first time it played the track too? For a song to get a few seconds in and then stop, it must have blocked the audio thread, which means a read() of the next 64K of MP3 data -- exactly the same read which had been issued the first time the track was cached -- has blocked. But yeah, a bad-blocks scan will at least help eliminate some types of disk failure as suspects. If we're lucky it'll show up heat-related problems too.

Peter
Posted by: tfabris

Re: Dead Mk2? - 11/04/2005 13:54

So if I did the "cat" to dev/null command, and there was something wrong, would I expect to see error messages at the console?

Because I'd be doing it with the player software turned off, so the VFD wouldn't be showing, so I wouldn't see hijack hardware error messages (if any. But I've never seen hijack messages on this player yet anyway).
Posted by: peter

Re: Dead Mk2? - 11/04/2005 14:04

Quote:
So if I did the "cat" to dev/null command, and there was something wrong, would I expect to see error messages at the console?

I think so, yes. Most "disk going away" error conditions end up with a "lost interrupt" message going to the console. There shouldn't be any that retry indefinitely.

Peter
Posted by: mlord

Re: Dead Mk2? - 12/04/2005 00:18

Yuh, you'll see "read error" messages for every bad sector, on the serial port.

I think my driver will retry each error a few times (3?) before giving up and going on to the next.

Cheers
Posted by: tfabris

Re: Dead Mk2? - 17/04/2005 04:34

Hmph. I think it's a dying disk or some other disk related hardware problem.

I got about 75% of the way through a 2-gig upload, then the upload stalled and started timing out on each song. Opening the serial port, I see "hda: Lost Interrupt" every few seconds.

Grr.

I'll open the player and look for obvious problems, then shop for a new disk drive.
Posted by: mlord

Re: Dead Mk2? - 17/04/2005 12:17

Lost Interrupt messages are normally due to a bad IDE header and/or cable.

Cheers
Posted by: tfabris

Re: Dead Mk2? - 17/04/2005 13:53

Yeah, I'm gonna open the player in a few minutes and check for that stuff. Sigh. Thought I was past all that, having already been through it and solved it.