My files are large, between 900MB and 10GB for raw MPG recordings and half that for files that have been transcoded (only a tiny handful at this time on my PVR's main recording volumes).

The built-in defrag tool in Windows XP doesn't do anything for these volumes. I mean, it moves a few bytes around, but in the grand scheme of things, your disk is pretty much left exactly how it was before you started. There's no benefit I can find for using the built-in tool.

I'm currently in the process of a lengthy defrag using a free tool called MyDefrag. It's currently operating in the "consolidate free space" mode and running its third pass. I had previously run the "defrag only" mode over night which did a decent job, but didn't pool the free space (or even most of it) together.

Does anyone have any experience with any other tools that perform these types of functions in perhaps a lesser amount of time? I'm not really looking to optimize the speed of existing files, but rather to pool together the majority of free space so that newly written content doesn't need to be fragmented, possibly decreasing write latency.

When my PVR is busy, it's busy. Last night I had 5 tuners recording at once. 3x ATSC (720p and 1080i) and 2 NTSC (480i) from 9-10pm. During another hour there were 3 and at another time only 2 going at one time. Thursday is the busiest night of the week for recording, by far.

I believe I've found an issue which initially I thought might be a fluctuating ATSC signal, causing some pixelation and fraction-of-a-second audio drop out. However, I'm now fairly confident this is happening when the system is recording many streams at one time. The more complex the stream, the more likely the issue.

My evidence is anecdotal at this time, but I've started some empirical testing. First, at a specific time, I've verified that a group of channels are all producing rock-solid audio and video by observing the channels for a decent amount of time, one at a time. Then loading up the tuners each with a recording task, I've observed the issue. Not as pronounced as the worst it's ever been, but my hypothesis still stands.

Sage has a new setting which allows it to prioritize a balanced recording session, where it attempts to spread your concurrent recordings to multiple of your available hard drives. With the defrag I'm hoping to give each drive the best chance it has, essentially trying to remove it from the equation or at least normalize its impact - as if the drive were new and empty.

The system currently has three recording volumes in it. Two are single partitioned drives used for nothing else but recording. The third is actually a large partition on the same drive used for booting/Windows. Ideally I'd like to get another drive into this system, dedicated, like the first two, to nothing but recording. Perhaps most ideal would be to set up the recording to go to a striped volume and then moved (during some low-use period) to a storage volume.

I also had commercial detection active at one point which was also creating 3 tiny (under 1KB) files for each recording, but I've since disabled that feature while trying to track this issue down, along with deleting all the commercial marker files.

If the defrag testing goes well, I'll set up a defrag maintenance schedule to run every night while nothing is being recorded.

I like the automatic features of the recent Diskeeper versions. In a developer environment with lots of perforce syncing, clean and dirty builds and so on, it's been the only defragmenter that was install and forget. It makes sure to get out of the way when real IO is happening, and takes advantage of downtime to clean things up and optimize. Other tools might offer similar, but only after lots of configuration and tweaking.

I can't speak for how well it works with large files though. Usually the largest files in my environment are PDB files, usually around the 100-300MB range.

My favorite tool is an earlier version of MyDefrag. I don't like the changes they made.

You can find the 1.10 version here. Just run it, say yes to the additional downloads, and pick the default defrag mode.

Plus I like the portability of JKDefrag GUI better.

I use MyDefrag in its screensaver mode. Seems to do the job.

Mmm..
It really shouldn't be a fragmentation issue.

I wonder if that operating system/version can be tuned to handle the recording load better.

I know this isn't helpful for your specific setup, but on the Myth box here, we can have six ATSC streams (4HD + 2SD) plus two NTSC streams all recording simultaneously, with two commercial flaggers running, and one playback running. Glitch free.

The storage is a pair of 750GB SATA drives in RAID0, formatted with XFS. A third drive holds the system and database.

If your O/S allows separating the journal file, then putting the journal on a separate/dedicated drive might help considerably. We haven't needed to do that here yet, though.

Cheers

The Sage guys claim to have tested 8 HD tuners without an issue onto a single standard SATA drive. To test out their end of the pipeline I'l be enabling logging to try and trace any possible issues.

I have no idea how to improve the efficiency of the general disk system in Windows XP unfortunately. Apart from possibly helping writes by defragmenting as we're discussing already.

NTFS Log file - Can be adjusted using chkdsk /L:size

MyDefrag has a feature specifically to consolidate free space: FastFill.

I personally use UltimateDefrag. It has a crazy amount of options.
Here is a video giving a tutorial on using UltimateDefrag.

I'm giving Diskeeper a try right now. Typically convoluted setup and UI, but I like the idea of its background defrag and ability to stop its business when it detects the disk being used by other processes.

I'm giving it a shot on the two volumes that I didn't pre-defrag with the other software and will hopefully know by sometime tomorrow how it's done.

I used Diskeeper in the past and a few times per month it would just freak out and defrag my drives until I stopped it.

I bought UltimateDefrag within a day of using the trial version. The settings allow you to pick files / folders and put them in the order you have listed. It can also have files marked as an archive/*zip placed in the middle or end of the disk.

I would expect all of those tools to simply make things worse, unless the system can be left alone (no new recordings or commericial skip jobs) for week or so.

It would be quicker and more effective to simply have a second set of drives, and periodically file copy everything from one set to the other, then switch them.

I doubt that these MS-DOS defrag tools are designed to handle vastly huge files like these. And "pausing in the middle" of a defrag to allow something else to run.. well that just makes the fragmentation incredibly worse than before.

And again, I seriously doubt that disk fragmention is the problem here. The access pattern simply isn't the kind that should produce enough fragmentation to matter. Not even on Windows.

Cheers

I believe that Defraggler recently won the reader's pick for best defrag program on Lifehacker. I haven't tried it, but it's from the same people who do CCleaner, and I use that program all the time.

It probably depends on how smart the PVR is. Either it creates the files the giant final size it needs to start with (possibly causing write latency even if on a background thread, as even reserving that many pages would cause an I/O burst), or it streams gradually into each one. That latter -- if there are five programmes being recorded simultaneously -- is surely the Nick Flynn of fragmentation-producing access patterns.

Peter

When I looked at each drive, prior to any defrag work, they were all badly (very badly) fragmented.

I mentioned that I had been testing MyDefrag. According to its file-use map, it has successfully pooled all of the unused space and everything looked beautiful.

Two recordings later, I could see that the previously empty space had the two new files scattered all over it, not recorded in a nice linear fashion.

Fragmentation is quite apparent on these disks, however, I don't know if it's responsible or even a contributor to the problem I'm trying to remedy.

The filesystem has a massive influence here too: if it performs delayed allocation, with a decent buffer-before-flush interval, then things will be written out in largish contiguous chunks regardless.

I doubt that fragmentation is the cause of the pixelated recordings. Surely the planet's most sold O/S can handle a few measly TV streams!

See, even as it's writing data out, the disk is still seeking around for metadata reads/writes. So fragmentation or not, it's never going to be doing a lot of sequential I/O. Especially not with 4-8 streams going at once.

This is more of a head-scheduling issue than a fragmentation issue.

Cheers

I think at this point, more than anything, I'm just taking away questions so that I can put this more firmly in SageTV's court.

In an unrelated search I stumbled upon someone else in their forums having some recent (similar) issues with their recordings, claiming also that recording with the native capture card's app didn't cause the problems. But it's the only thread I've located about it, so it doesn't appear to be a widespread (and therefore easy to test) problem.

A while after we first began using MythTV here, we had some recording glitches with multiple streams going simultaneously.

The cause was the code performing "fsync()" after each measly tiny write. At worst, it ought to have been using "fdatasync()" instead. At best it shouldn't have been doing either, instead leaving the decision to the operating system.

As a temporary workaround, I just commented out the unnecessary fsync/fdatasync calls. MUCH improved. And since then, a MythTV developer checked in with us on one of the kernel mailing lists, and an even better and more modern solution using "sync_file_range()" was discussed and implemented.

It's quite good, now.

Cheers

Oh, if there was a nice small, completely silent and powerful front-end box with an attractive and usable UI for MythTV, I'd probably stop using SageTV tomorrow.

Has anything been done on this front yet? Tegra2 is the new talk of the chipset town when it comes to HD 1080p decoding and output.

I'd even consider building a box myself so long as it's not going to cost as much as a dedicated PC, nor look like one. From what I know, there's absolutely no traction on supporting SoC solutions for MythTV and I haven't heard of any generic hardware available for self-build that rivals the specs nor compactness of the Boxee box from dLink (Tegra2)

This link mentions only SD-capable front ends: http://www.mythtv.org/wiki/Choosing_Frontend_Hardware

And the conclusion is that all the solutions they discuss pretty much stink.

A number of the new Atom based mini-PCs look quite feasible for the job. They could boot/run from a USB stick, or over the network, and have the full power of MythFrontend at each station.

But MythTV remains a do-it-yourself solution, not an off-the-shelf product that can be bought.

Cheers

Pretty much any modern Nvidia GPU >= 9500GT will do 1080p without breaking a sweat.

The GT220 and up seem to be the best current models. I'm looking for a fanless one before I replace the 9500GT here with one, oddly enough to improve *analog* playback, not HDTV.

Cheers

Ooohhh... Fanless GT240 Card for under C$100... Want!

Edit: Ordered!
Though I will have to nibble away the sheet metal of the MythTV PC case so that the protruding heat-sink will fit.

So, here's an image of what (a portion of) a single recording does to a large empty block on the HD. What's responsible for this framentation? I don't think I could have done a better job of scatting little bits of the recording all over the drive if I had done it manually myself.

Wow.. that's crap! How much total data do you figure that picture represented (approx) ?

I just checked on the MythTV box here. Large recordings seem to be written out in mostly 16MB fragments, with the odd fragment of larger sizes, and some fragments that are gigabytes long.

Eg.

Code:

Recording "3183_20091127010000.mpg" -- 5,761,731,692 bytes.

 filesystem blocksize 524288, begins at LBA 0; assuming 512 byte sectors.
 byte_offset  begin_LBA    end_LBA    sectors
           0  970052608  974181375    4128768 (2,064MB)
  2113929216  966864896  966897663      32768    (16MB)
  2130706432  966832128  966864895      32768    (16MB)
  2147483648  966799360  966832127      32768    (16MB)
  ..
  [129 fragments not shown, 16MB each]
  ..
  4328521728  962538496  962571263      32768    (16MB)
  4345298944  962505728  962538495      32768    (16MB)
  4362076160  962472960  962505727      32768    (16MB)
  4378853376 1013089280 1014531071    1441792   (720MB)
  5117050880 1112882176 1113406463     524288   (260MB)
  5385486336 1114871808 1115606015     734208   (367MB)

I think the smaller 16MB fragments are the result of multiple things recording at the same time.

Edit: ahh.. I just checked the XFS manpages, and the 16MB value is the allocation group size (agsize). If I reformat and increase that value, those smallest fragments should end up larger in the future. Max allowable setting for agsize is 1TB.

-ml

Yep, looks like NTFS. Combine that with an app doing small writes, and a drive gets wrecked quickly. I'll have to post a shot of what happens when I sync down a days worth of build changes at work on Windows.

All of these defrag tools use the same system calls that Microsoft provides to move blocks around. There's nothing MS-DOS about them, and they can certainly handle large files.

I don't know why you think that pausing a defrag will make it worse. It's just moving blocks around.

I'm not really sure if you intended to be a joke or what.

Pausing in the middle of a defrag, allows the "priority" app to then start writing to the disk... quite possibly into the middle of the "new" destination area for the defrag.. requiring the whole thing to be done almost from scratch again.

And with files in the 5-50GB range (as on our PVR here), any attempt to defrag would take so long that it could never really hope to complete before the next use of the PVR.

Cheers

Or, since we know that it's not writing sequentially in the first place, when it gets time it can move that newly written data somewhere else.

That said, it would be nice if the Windows implementation of NTFS didn't suck so hard in the first place.

The defrag seems to have helped the recording process.

Last night during multiple records, there was no pixelation within the recorded files that I've so-far been able to verify. I have a couple more to check out still.

Another (unrelated) problem did come up though. One of my tuners now fails to tune one particular channel, while it remains perfectly functional on the other tuner. This has happened before and went away after a reboot I think. Testing that now...