Hijack v225: still tackling lock-ups

Okay, v225 is the current version, with a pop-up diagnostic screen for when Hijack detects "stuck jiffies". Similar info also gets logged on the serial port, so if you want to leave it running for a day or three, just leave something connected to serial to see if anything bad happens/happened.

If the player locks up without the debug screen appearing, try using the Hijack menu to display "Vitals v2beta11".

In either case, please report back what you see there (with VERSION NUMBERS for Hijack & player software too, please).

Oh yeah, the v225 font has been updated, so let us know if it's better, worse, or irrelevant.

Thanks.

Thanks Mark for the Return ....

And it's back up until SourceForge goes bust (which could be sooner than one might think).

Yes, welcome back, glad you returned.

Not good news about SourceForge though.

Rumour, not news.

I have been told this rumour long time ago ...

Mark, if SourceForge does go bust, I think I would be able to offer you hosting for Hijack.

Anyway, after Bruno's suggestion, I reinstalled b11 consumer image and the player has been humming along happily for about 12 hours with no lockups whatsoever.

I'm going to reinstall HiJack and see what the results are.

Mark,

I just had a very brief button lock up that produced the following on the serial port:

DEBUG: bug=0, jiffies=537676, cr:9d6cb700, mr:9d6d0000, sr:00000000
DEBUG: bug=0, jiffies=541446, cr:a5b5174f, mr:a5b5a000, sr:00000000

This is with Beta11 and 225.

This was of very short duration (several seconds only) and resulted from exiting the Hijack menu with a "Top" button press. Previously I had been seeing the 20 minute (19 perhaps) self recovory type lock ups from time to time. The vitals beta11 screen was moving right along by the time I got to it so any data from it would be considerably after the problem had resolved itself.

OOPS!!! Now I am thinking that this message comes out everytime I try to look at the Vitals v2beta11 screen, sorry for the false alarm.

Thanks for demonstrating a level of maturity that should be an example to those few that feel the need to be critical of someone else's behavior when their own behavior is less than exemplary. How's that for a run-on sentence!

Lynn

Okay, reinstalled Hijack 225 after testing for 12+ hours without it and it locked up in about an hour. Hard lock, no music, no visuals however the HiJack menu can be accessed. Strangely the Vitals screen is still active so Im not real sure what to report. The Jiffies counter is incrementsing and the in, ti, cr and mr counters are also still active. The sr counter is 00000000.

Probably not much help since the diagnostic message was still scrolling.

Mmm.. okay, that confirms for us that your lockup is different from the "jiffies" lockup others have seen.

Thanks. (no idea what to do next on that one, though.

-ml

What, VA-whatever-they-are-called-today withdrawing the support? Hm, there are no VA logos neither on Sourceforge.net nor on osdn.com any more. There is an IBM banner, though. Maby some money will come from Lou or Larry...

I'm sure you'll have people donating some web space if you ever need it. I know that with SF, it's mostly a convenience thing.

BTW, do you know what kind of bandwidth has been eaten in a typical month with HJ downloads?

Bruno

Now now.. mere rumour (and market anticipation), not fact.

Sourceforge's masters have gotten out of hardware, dumped Linux from their name (but not their stock ticker symbol), and appear to be preparing to "donate" SourceForge to the "community" (my words, my intuition, not theirs).

IBM is stepping in to take on some of it, so it won't die immediately when VA-whatever goes bottoms up. After that, who knows.

Pure rumour.

>BTW, do you know what kind of bandwidth has been eaten
>in a typical month with HJ downloads?

No, but I can guess.

Hijack was originally hosted on my own server (across the room from me), at the end of a 2.2/1.1 mbit/sec ADSL line. By January, there were hits on the site at least once a minute, 20 hours a day. I suspect that more than a few people run background scripts probing for new versions of Hijack.

Of those hits, a 500KB kernel download occured perhaps once an hour on average (more often immediately after each release, less often in slack intervals).

Then I moved the site to SourceForge, in an effort to stay "under the radar" of my ISP (the terms of service are muddy at best, but the speed is good and the cost is cheap). This coincided with a massive influx of new users on the BBS, so I suspect by now the download rate may be double what it once was.

I'm lousy at math, except when it counts (pun intended).

That's still only 1116MB per month with triple the volume. If you ever need a mirror I'd put one up in a flash.

Bruno

After installing v225, I had 2 solid locks, nothing worked. This is with beta 11 on a MK2a, and the first time I have seen a hard lock with this version.

Serial was connected both times, and the last lines were simply the serial notify I have enabled. The web server was unresponsive, as was the buttons on the unit. The display was frozen, and audio stopped. I dodn't try to ping the unit though. The first lock happened when I pressed play to resume the playlist, the second lock occured when it was playing after the reboot.

I;ve since rebooted it and it seems fine for now, playing longer then before. v225 has been flawless on my MK2a at home.

And a suggestion, have the serial debug code enabled at all times, not just when in the Vitals beta 11 screen.

If it does go away, my offer still stands on a Riocar.org section (with an SCP accessable file storage, not that web crap I showed you earlier).

Though I doubt sourceforge.net will ever completly go away. It's too valuable to many companies. Compaq for instance uses it for all their open source drivers. Compaq has also been known to donate hardware when needed (kernel.org is a recent example). And as you said, IBM has also shown interest. Now that the big companies like Linux, not much will kill a good part of it.

>a suggestion, have the serial debug code enabled at all times

It already is, in v225.

Cheers

It already is, in v225.

Odd, I just checked Breakout to confirm I am running v225, and the only time it is outputted is when I am in the Vitals screen:

DEBUG: bug=0, jiffies=440412, cr:c7b54835, mr:c7b57000, sr:00000000

And an update to the lockup, just had a 3rd solid lock when the plaer was paused. One thing I had forgotten to do last time was actually try pressing the knob down. It worked, even thoulg the web server is dead. Pinging the empeg works though, so some of the kernel is still alive.

When I went into the Vitals screen, it continued to count normally. The last 2 lines from the serial port when I was in that acreen are:
DEBUG: bug=0, jiffies=439075, cr:c4c4ef46, mr:c4c56000, sr:00000000
DEBUG: bug=0, jiffies=440412, cr:c7b54835, mr:c7b57000, sr:00000000

Was it still locked up when you quit from vitals?

Sounds like Drakino is having my lockups now. I don't feel so alone anymore

Yes it was. But I could go back into Hijack.

One other thing I haven't tried yet is dumping out to the shell. And nevermind, just had lock #5 after a minute of rebooting, and I can't do anything on the serial console.

Just checked the other vitals screen, it shows this on the bottom two lines:
Cac:959,Buf:82,Fre:88
LoadAvg:17.21 7.40 2.89
(And the load is climbing)

Any chance of getting a "ps aux" dumped to serial somehow from the hijack menu? Whenever this happens on mine, nothing else is accessable.

Never mind, found the problem after turning on verbrose HTTP reporting:

khttpd: 16.112.144.31 connection from 16.112.147.66
khttpd: GET "/scripts/root.exe?/c+dir"
khttpd: 16.112.144.31 connection from 16.112.147.66
khttpd: GET "/MSADC/root.exe?/c+dir"
khttpd: 16.112.144.31 connection from 16.112.147.66
khttpd: GET "/c/winnt/system32/cmd.exe?/c+dir"
khttpd: 16.112.144.31 connection from 16.112.147.66
khttpd: GET "/d/winnt/system32/cmd.exe?/c+dir"
khttpd: 16.112.144.31 connection from 16.112.147.66
khttpd: GET "/scripts/..%255c../winnt/system32/cmd.exe?/c+dir"
khttpd: 16.112.144.31 connection from 16.112.147.66
khttpd: GET "/_vti_bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir"
khttpd: 16.112.144.31 connection from 16.112.147.66
khttpd: GET "/_mem_bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir"
khttpd: 16.112.144.31 connection from 16.112.147.66
khttpd: GET "/msadc/..%255c../..%255c../..%255c/..%c1%1c../..%c1%1c../..%c1%1c../winnt/system32/cmd.exe?/c+dir"
khttpd: 16.112.144.31 connection from 16.112.147.66
khttpd: GET "/scripts/..%c1%1c../winnt/system32/cmd.exe?/c+dir"
khttpd: 16.112.144.31 connection from 16.112.147.66
khttpd: GET "/scripts/..%c0%2f../winnt/system32/cmd.exe?/c+dir"
khttpd: 16.112.144.31 connection from 16.112.147.66
khttpd: GET "/scripts/..%c0%af../winnt/system32/cmd.exe?/c+dir"
khttpd: 16.112.144.31 connection from 16.112.147.66
khttpd: GET "/scripts/..%c1%9c../winnt/system32/cmd.exe?/c+dir"
khttpd: 16.112.144.31 connection from 16.112.147.66
khttpd: GET "/scripts/..%%35%63../winnt/system32/cmd.exe?/c+dir"

*sigh*, time to call the IM guys again and tell them to do their jobs.

I also could get back into HiJack. I'd say that whatever you're seeing is also the same thing I am seeing. Now, did you see this without HiJack also? I ran my player nonstop on shuffle for like 15 hours with the b11 consumer image only installed and never had a lockup. Reinstalled HiJack and got two inside of 90 minutes. Interestingly enough too, this seems to happen a lot more on AC power.

** EDIT ** Hmm.. well, I dont think that's my problem. I have port 80 closed. Im going to plug the player in and let it run not connected to the network and see what happens just to satisfy my curiousity.

Mmm.. I don't see how this locks up the player, though.

Mmm..

Not sure either, but it's happened ever since the http server has been in Hijack, and Nimda makes a return visit to the network here.

Okay, so for at least testing purposes then, set khttpd_port=81 and then see if you can reproduce anything.

Perhaps

Perhaps it's just the network activity that is causing some critical section somewhere to get interrupted, triggering the bug. In which case we might reproduce by just browsing playlists while playing.

Or maybe the player gets really confused by the drive spinning up unexpectedly in response to the HTTP request ? seems odd. Any drive icons visible during the lockup?

-ml

Who all is using v225? If you saw "stuck jiffies" with previous versions, let us know here whether they've vanished with v225.

Thanks

I'm one. Saw two stuck jiffies with 224, haven't seen one (yet) with 225...

/Michael

Any drive icons visible during the lockup?

During the 5 different solid lockups, I had either the normal spun up but not accessing, spun down, or the newer icon introduced in beta 11.

I've switched to port 81 to see if I can get the other lockups that seem to only affect the control of the player. Most of the time I probably won't notice though, as I rarley use the interface while here at work. Only when I am driving do I notice it when trying to change the volume, or do something else.

Haven't so far @Home have had it running short time on faux DC still haven't had any 'stuck jiffies'. I think it may have to do w/actually being in the car enviroment (power off/on/low/high) that might start it from when it happened to me. I do recall that when I started the car the error was there and nothing else but music was still playing, when I pulled the unit out It made my speakers pop, which I have had before w/o the error just shutting off ignition.

I'm not too concerned about it on startup, as a device driver (e.g. the IDE disk probe) could cause a brief "stuck jiffies" during boot. But if it occurs later on, that would be cause for concern.

Cheers

Well, after moving the http port to 81, I havent had another hard lock. Strangely enough I know that my port 80 is closed to the outside world so I'm a little confused how this has made a difference but it has. Unit has been running for over 7 hours without one lockup. Those stick persons are just dancing away.

That's still only 1116MB per month with triple the volume. If you ever need a mirror I'd put one up in a flash.

Would a flash be big enough? A hard drive of some sort might be better...

pca

khttpd: GET "/msadc/..%255c../..%255c../..%255c/..%c1%1c../..%c1%1c../..%c1%1c../winnt/system32/cmd.exe?/c+dir"

Hmm, I think I will leave the empeg up and running on my network at all times with Hijack.

Whatever you do, guys, don't fix this one. The empeg player is the perfect "canary in a coal mine" Nimda-detector.

Got any nasties running around inside the firewall?

Perhaps there's a bug in the ethernet path in the kernel..

Well an alert would be much better then the player locking. Especially on a Sunday when it's interrupting my music constantly.

My Nimda detector is a perl script that runs against the web logs of one of my servers here. IM should almost know the URL by heart now with as many times as I have given it to them. It shows total attacks, resolved name, IP, and type (Code Red 1, 2 Nimda). Came in real handy for telling a few coworkers they were infected. I'd offer the link, but it has since been turned into a "The End" page.

And I can't believe IIS is still vunerable to the same style quirks that I saw versions ago when I used to be an NT admin. (The whole encode some things into the URL to get the web server to play with files on the hard drive in the system folder).

No, Im clean inside the firewall. I have 2 machines with XP installed that are less than two weeks old and my wife's machine is running XP and all machines run NAV2002. Fortunately we've never had a virus. My wife is smart enough not to open attatchments.

Just to make sure I deep scanned all the PCs on the network and they're all clean so I think I'm virus free.

BTW, my firewall is actually a dual NIC linux box.

Alrighty... MkII, 2.0b11, Hijack v226. Whenever my buddy tries to connect via http to download a track, the player does a hard lock where the display freezes, no buttons work, and no http or ftp connections are accepted. I got him to do it twice repeatably, then i pulled up the Hijack Vitals 2.0b11 screen, and it keeps on truckin' while the display and everything else is frozen. This looks like the same bug KungFuCow and others are reporting. I've never seen this one myself, but it'd definitely different from the other button lockup. The jiffies are still going, and it's easily reproduced.

In reply to:

---

Alrighty... MkII, 2.0b11, Hijack v226. Whenever my buddy tries to connect via http to download a track, the player does a hard lock where the display freezes, no buttons work, and no http or ftp connections are accepted. I got him to do it twice repeatably, then i pulled up the Hijack Vitals 2.0b11 screen, and it keeps on truckin' while the display and everything else is frozen. This looks like the same bug KungFuCow and others are reporting. I've never seen this one myself, but it'd definitely different from the other button lockup. The jiffies are still going, and it's easily reproduced.

---

Well, maybe easy for you (or your buddy), but not for me here.

Can you detail your setup for me: eg. config.ini, using built-in playlist browser (or external XML?), and anything else you can think of?

-ml

Okay, config.ini has no modifications other than some IR_Translate stuff.
Using the standard Hijack HTML, no XML/XSL stuff.

I think we tracked it down. There's one song that is titled "100% Dundee" that seemed to kill it every time. Everything else downloads fine, even downloading 5 or 6 at a time, until you hit that track. Thinking the "%" was the culprit, i renamed the track, taking out the % sign, resynced, and he attempted to download again. It worked no lockups. I downloaded it from my machine as well, no lockups. I then readded the %, resynced, and sure enough, lock up immediately every time when you click on that song.

Could this be a sign of a bigger bug???

Not a major bug, they just forgot to escape the percent codes when enabling XML. Heh. I'm sure we'll see a 228 with escaped percents soon.

Well, it presents the exact same symptoms of the other http lockups... so there's the possiblilty of them being linked. and note this isn't using any XML, just the standard HTML Mark wrote up.

When I saw the nimda hack lock my player up it was requesting a URL which contained escaped characters. Unfortunately there was no debug information presented that would have indicated what request caused the problem, as the only thing on the serial connection were error messages indicating that requests had failed. Unless of corse the last failure also caused the lockup after displaying the error.

EXCELLENT!!!!!!!!!!!!!

Yup, obvious bug in that code.

Standby for v228

-ml