Cache glitch in 2.01?

Greetings!

No worries. This is a simple fix, and I will test adjusting the size of the ReserveCache option when I return to the office tomorrow morning. I am currently not able to get a console (or other) connection to the player at the moment due to insufficient connections...

I made the mistake of typo'ing 160 when I meant 16. My player just kept starting and crashing. Blah.

And I see that my wife never sent out the box with the empeg motherboard and memory to mlord, either.

Greetings!

Testing. But I don't think you will like the results...

Yesterday, I did not notice any problems with the ReserveCache=180. Today, I tried lowering that value to more reasonable numbers. Suffice it to say, I am leaving it at 180. I started to notice increasing instances of the drive icon showing the "W" (audio thread waiting for data) at around 160, and started hearing glitches around 145.

When I saw the glitching (Ws every second or so, with a big audio glitch around 17 seconds) at 6, 8, 10, 12, I decided to do the binary search approach. This brought me to a breakdown point of around 145-150...

Please note, I am not your typical user. My database is unusually large, and my FID count is off the map. So sane computations may not help. Of course, all of this is complicated by VBR files - I could tell that some files were more "trouble" than others, based on the bitrate and the amount of data needed / rate of cache drain. Also, I noticed that the glitches seemed to be made worse (to the ear) by having auto volume adjust in place - perhaps the wait / glitch in the audio thread is being interpreted as valid audio by the algorithm.

For me, a value of 180 for ReserveCache (a level where I only see an occasional W for very complex tracks) is an acceptable tradeoff, but I will still try downgrading to 2.00 this weekend.

Testing methodology:

(previous test complete)
q
rw
rwm
vi /drive0/var/config.ini
(change setting)
rom
ro
sync
hard reboot player (remove from sled)
start player playing
skip ahead at least six tracks (to make sure nothing cached)
play for at least 30 seconds, noting any Ws or glitches
repeat.

Anything audible failed, more than 3 or 4 Ws (depending on the complexity / bitrate of the track) was considered marginal. At 160 and below, I started seeing more Ws but no audio glitches. 150 brought glitches to more complex songs, 145 glitches in more songs than not.

SW: Player 2.01 and hijack 413 only. Auto Volume Adjust (low)
Highest FID: 0x7E4D (32,333)
empeg:/empeg/bin# ls -al /drive0/var
total 4484
drwxrwxr-x 2 root root 4096 Aug 27 15:55 .
drwxr-xr-x 6 root root 4096 Aug 27 15:34 ..
-rw-r--r-- 1 root root 933 Sep 3 05:10 config.ini
-rw-rw-r-- 1 root root 4415935 Aug 27 15:54 database
-rw-rw-r-- 1 root root 137480 Aug 27 15:55 playlists
-rw-rw-r-- 1 root root 376 Aug 27 15:54 tags

As near as I can guess from observing the player software, each ReserveCache increment (in v3alpha8 at least) corresponds to 64kB of memory. So a ReserveCache=180 setting means that Paul is "throwing away" about 11MB of his new memory, bringing the 48MB player down to 37MB or so.

Seems a tad excessive. Linux memory management overhead is about 5% or so on most kernels, so I would expect a setting of ReserveCache=32 to be more than good enough.

If you could give that value a try, it might help us understand it better.

EDIT: I suppose we should also allow for Pauls BIG database here, another 5MB or so, giving a total of ReserveCache=112 for his player.

Also, with current Hijack versions, there really should be no opportunity for the disk I/O to starve the audio playback threads (important bits of this code are attached here for scrutiny), so I don't think it's the reading ahead that should be the culprit here.

EDIT: Ah, nevermind.. I see that Paul has already tried values between this and 180 without success

Cheers

Peter, how large are the chunks of memory which correspond to increments of the ReserveCache parameter? My observations suggest to me that each increment is 64kB, but it would be quite useful here to know for sure.

Thanks

Ooooh... I just got it to happen here on my 32MB player with v2.01. If it's repeatable now, I oughta be able to fix it..

-ml

If this is of any help, one observation is that the glitching seems to occur (first and worst instance) at about 17 seconds. In cases where I see this with the drive status displayed, I can see the audio thread waiting (W icon) well before the 17 second mark, in very short bursts. It is only at that 17 second point (plus or minus 1 second) that there is a big audio thread starve (W icon that stays on for more than a few frames - 1/75ths of a second). This resolves, but there is a blatent audio glitch and it seems to knock the auto voladjust off (perceived sound level changes and fluctuations) for a second or so. Then it smooths out.

I will get repeated instances later in the song, but usually not as bad as that first hit.

Cool! Repeatable is good!!!

Okay, ReserveCache fixes all but the very first glitch for me, even if I completely comment-out the Hijack CPU-yielding hack.

Based on various observations (a bit too complex to explain here), what is happening on my player is a very obvious "low memory" condition, which is causing various threads to block on memory allocation when they normally would never have to block.

This seems to only happen when the disk read-ahead in the player is busy, which makes sense since that activity will tie up a lot of pages in the kernel for holding the disk reads before they get copied to userspace, making even the most trivial of memory allocations result in a process blocking.

For example, opening an FTP connection to my in-kernel kftpd was NOT POSSIBLE during the disk I/O, due to lack of memory for the new thread's stack allocation (it needs 12kB total: 8 for stack, plus 4 for a parameter/buffer block). I saw other indicators of extreme low memory as well.

Pity we don't have O_DIRECT in this kernel version --> O_DIRECT forces disk reads directly into the userspace buffers, bypassing the page cache completely for apps which roll their own, like the player.

As a sidebar, I recently added the "--direct" flag to my hdparm utility, to permit benchmarking with/without direct userspace I/O, and it the difference with the RAID controller I'm working on is most impressive: a 4-drive RAID10 jumps from 70MB/sec to 130MB/sec, and a 4-drive RAID0 whooshes along at 206MB/sec. On a slow CPU using 64-bit/66-Mhz PCI, that is.

Now, for the first glitch, which still can happen. I don't hear it, but I see the W icon flash, which means the software knows it had a glitch. The Hijack CPU-yield on disk I/O is imperfect --> it misses the first FID that is read, so it is possible for a large initial track that the readahead of that same track may still glitch, on both v2.03 and v3alpha. I'm unlikely to fix that one.

Cheers

Okay. I follow. Basically the cache is so large that the call to fill it starves out memory requests from pretty much everything else. Especially with an extra large cache, because the kernel memory used for the initial read / processing takes lots of memory away from other kernel processes. Shrinking the cache makes the problem less of an issue because there is less to fill and the transfer happens quickly enough and are small enough that the other processes are less likely to get choked out by it.

Nice RAID transfer rate jump, by the way!!! Significantly greater throughput!!! COOL!

Edit: And since it is totally dependent on memory size and kernel, the upgrade to 100GB drives is not the issue, other than the side effect of a larger database. And it was not an issue previously with the 80s because the database size was smaller and I was running under 2.00 / older hijack. The only other test I would be curious about is if this would also happen under 2.00 - to see if the memory enhancements made there changed matters or if it is an inherent problem.

Paul, is there any update on this?

I'm running 3a10 with a 64mb memory card and I'm getting more memory related errors than I had before with 2.00final and no extra RAM. I'm needing to play with ReserveCache settings that I never needed before. I'm wondering what benefit I'm getting from the memory card upgrade.

For instance, switching tracks on the player does not display a new "Vcb: 0x428fa000" via serial. If the player is running, I can't connect via emplode. When the player boots up, it usually boots to audio but with no display (LED is still pulsing, blank screen). Usually turning the player ap off makes things like ftp work (rather than time out). After a boot, the menus crawl very very slowly. I never had any of these issues before. Is this stuff that I'll get with 2.01 or are these alpha issues?

I'm now at a ReserveCache of 180, up from 50, up from no setting at all. These changes only helped me from getting:

VM: do_try_to_free_pages failed for kswapd...
VM: do_try_to_free_pages failed for kswapd..
no room for private writable mapping
error: -12

It just doesn't seem like the alphas or 2.01 work too well with extra RAM? Or do I just need to tweak this stuff more? I was going to switch to 2.01 and give up Crossfading, but after reading your post, maybe I shouldn't expect too much of a difference.

Well, if I remember correctly the 3.0 alphas have a larger memory footprint than the 2.0 variations. I have not tried the alphas yet, so I cannot give you a comparison point.

I am running 2.01, I have my ReserveCache set to 180, and that seems to work for me. I am able to console into the player and ftp without a problem. I have never seen the slow menu problem, personally, so I suspect it is alpha related.

The easiest test is to drop to 2.0, delete and rebuild your databases. Because this version is not aware of any extra memory, you eliminate that from the equation. You would still see some benefit at the kernel level, because hijack will still access it, but the player application will not try to grab from the extra RAM for its cache. It is a good check to see if the behavior changes.

If this works, then move up to 2.01 and compare the differences. If you are running more background applications than I am (I just run hijack normally) you may need to set aside more memory for those.

By the way, I am sorry to read in the other thread that your empeg is smoking again.

My player should be back to me this week.. I currently have my drives in a backup player and am thinking of installing 2.01 on the drives so that when the player arrives, I can just swap the drives. Is there a problem running 2.01 on a player with only 16mb? Is the cache sequence too aggressive for 16mb?

I'd like to get it running stable on 2.01 before I switch to 3.0. Also, I doubt my player will handle 3.0 with 16mb and the size of my database. My drives probably won't see any 3.0 variant until I know my hardware issues are behind me and the backup is on the shelf.

Paul, I'm running 2.01 now with the memory card. With no ReserveCache setting, I was getting a lot of low memory errors over the serial output. I then tried 180 and the errors went away but I was still getting an interupted audio signal at the beginning of some tracks. Lots of fast W's too. Then I switched to a setting of 200 and the audio is better (bur still stutters sometimes). The W's are still coming (quick ones) but not as often.

What's odd however is that emplode can not find the player at all unless I turn the player off first... have you had this happen? If I keep bumping the ReserveCache up, might it fix it? 200 seems high already...

EDIT: I can't access the player via html interface, emplode (even to simply add songs to the queue, or ftp) if the player AP is running... I'm upto 220 ReserveCache now. This is going to make DJing impossible if I can't add songs to the playing order remotely.

Hmmm... How much extra memory did you add? I am at 48MB. Unfortunately, I suspect you may have to downgrade to 2.0 if the reservecache option becomes too high or impractical for you. Worst case, with 2.0, the memory is only availble to the kernel because the player application is not aware of it.

I have not had trouble with emplode not detecting the player, but if there is enough memory starvation, I can see where it would happen. Try bumping a little higher and see if that helps you. If it doesn't downgrade to 2.0 and use the RAM for additional background kernel apps.

ksock_rw is part of the Hijack FTP/HTTP daemon.

That's the problem - you have more RAM. The caching routine is designed around the normal 16MB. It is extremely aggressive, in that it completely tries to fill the cache each time. The issue is that it performs the disk reads and buffers the data into the kernel memory space, flooding out other processes. While memory support was added to the 2.01 kernel, there were no known changes to the caching mechanism. Thus the aggressive caching tuned to 16MB becomes a psychotic serial killer with 64MB.

The error you are seeing might be another symptom of this memory limitation. Question - do you notice the same behavior if the player is paused? This is just a test to see if it is the cache (assuming that the paused player fills the cache and then rests once it is full) or the player application in general.

How about if I have an even larger ReserveCache and only leave the player about 16mb or so?

Putting the playing into Pause resolves the issue. Emplode finds it just fine. Of course, I can only put the play into Pause via the player itself because hitting the pause button in emplode or empegface doesn't work.

So am I basically out of luck until a software release takes advantage of the extra memory?

The W's indicate that a thread is waiting for disk. It implies you either have a remarkably slow disk (like the original Toshibas shipped in 1999) or that the disk is generating a lot of errors. It has very little to do with the amount of memory you have installed, or the amount you are reserving for cache.

If you get audio stutters, the most likely place for this to happen is near the start of a track, within the first 5 seconds of playback. The playback cache spits out what it has cached and is expecting more data from the disk, which it is not getting in time - hence a brief pause. No amount of reserved memory will help in this situation if the data is simply not being delivered on time.

Other, continuous audio stutters occuring rapidly during playback also point to a failing disk or one with high latency.

ksock is a kernel socket operation. The socket error -32 is FFE0, the meaning of which as an Errno in Unix escapes me at the moment.

This implies you have an IP stack fault of some sort. Have you had serious electrical problems on this unit? It may imply subtle electronic damage which may also account for the apparent slowness of the disk, as the ethernet chip and the IDE interface are on the same buffered databus/5V power rail. You may have a damaged bi-directional databus buffer chip which is causing data corruption.

Well crap.

Things are fine with 2.00 except for serial output giving me memory errors... Maybe I can fence some stuff on eBay and afford to mail it to you... If only the damn Euro would fall.

What kind of memory errors with 2.00???