Big PC hard-drive question.

I wouldn't have believed it, but I've manage to somehow use up all my HD space, 4x60Gb & 1x45Gb IBM, sitting on highpoint and promise controllers (DVD and CD-RW are on the mobo's IDE controller). My ideal setup was:

1st drive OS partitions
2nd drive Games (shares channel with 3rd drive)
3rd drive WAV (shares channel with 2nd drive)
4th drive Files, docs, MP3, etc
5th drive swap file and backup of OS partitions

However I've now got into MP3 in a big way (blame my purchase on the eMpeg car player), and given how often changes are made to MP3 encoding technology, I now keep archival WAV files which presently takes up 3x60Gb drives with my current CD collection. So obviously I had to jiggle things about in my setup. It looks like I'll need to buy a hard-drive or two to get it back to way I like it, and to complicate things I anticipate that I'll be getting into AV in the next few months (so I'll need to get firewire card soon). So my options are:

1) Buy a couple of 120Gb, thus staying within current IDE limitations
2) Buy a Maxtor 180Gb or 2 inc. of promise controller
3) External solution, i.e. firewire/USB2/hot-swappable
4) Something I haven't thought of

r.e.
1) Simple solution, current rig has the space/juice to do that, however without adding another controller card I would have to make certain drives share an IDE channel. I can't afford to have 3 PCI slots taken up with controller cards. Question: Can the adaptec 2400A card just be used as just a plain dumb IDE controller card controlling 4 seperate drives, or does it have to be used in a RAID setup?
2) As above I'm loathe to have to add another controller card (hmmm maybe I can replace one of mine with the new promise and live with just sharing a couple of channels amongst the least intensively accessed drives).
3) I've seen a firewire breakout box which can house either a hard-drive/DVD/CD-RW IDE unit, and it houses a psu/fan/firewire socketry. Presumably it shouldn't have a hard-drive capacity limitation. I like this solution because at least when/if the need arises you can just plonk in a larger hard-drive at a later date. However I have no experience how effective such a solution would be.
4) ...............

There just seems to be too many choices so I'm just seeking advice as to the pros/cons, plus anything I've overlooked.

I too have been looking for a similar solutions. I was thinking about getting something like this: http://www.ubid.com/actn/opn/getpage.asp?AuctionId=6495561
Quantum Snap Server. This way all of the data on it could be available to any computer on my network. This would be very nice for sharing mp3's to my AudioTron network mp3 player. Currently my main computer has to be on to use it.

Even thought this particular one only has 120GB, it consist of 4 30 GB ide drives. I read somewhere (I can't find it now), that you can easily replace the drives with larger drives. If you did 4x120, you should be set for a while.

How about a DVD-R drive? 4.7GB at about $5 a disc. Not too bad, pricewise, and eminently swappable. Or you could go with a FireWire/USB/other swappable hard drive solution where you could swap the physical drive out of the connecting box and have hard drives lying around as removable media. Or, since both FireWire and USB support many devices, you could just keep adding new drives to your system when you run out of space, only using the one controller that you probably already have.

don't those hold like 9 gig or is it just 4.7 a side and the 9 gig ones are two sided ?

The only DVD-R drives (or media) I've seen are 4.7 GB. They definitely only write on one side, so it might be possible that there are discs you could flip over (hmmm... flashbacks to Apple ][s and hole punchers....), but I've never seen any. The DVD standard allows for dual-layer discs that actually have two recording surfaces on each side of equal capacity, making a total of about 9GB (about 17GB if you take both sides into consideration), but no current consumer-level DVD-R drive (or media) supports dual-layer. Almost all (if not all) commerical DVDs (movies, that is) are recorded dual-layer, so the drives can read them, just not write them.

It's 4.7 a side, some burners can do both which is 9.4.

I think I quite like the sound of this. However the main concerns are cost, 160Gb worth of discs (at £15-20 per disc) would cost close to £600 + £600 (cost of the drive). Ouch! Price aside the other thing is I tend to just drag and drop all my wav's onto lame to batch encode them (fyi this takes 3-4 days), and leave the PC to do it's thing. With 4.7Gb per disc, I would have to make frequent visits every so often to keep on feeding the encoder.

Apple is selling packs of 5 DVD-R discs for $25US, whichs works out to about £3.50 each. And the drives are listed (realistically) for as low as $300 (£210) on pricewatch.com. Now, I'm not familiar with getting computer stuff in the UK, but surely it can't be that much more expensive.

The cheapest DVD recordable drive is £400, and the cheapest media is £14 in the uk (Dabs.com), but they are neither of the same format. It really is that expensive over here. They don't call us ripp-off Britain for nothing.

You could try using a lossless encoder, such as FLAC. That'll save you a bundle of space.

Thanks Roger, but that really doesn't address my requirement, and that's a repository of archive WAV files, which can then be used to encode into whatever format I choose (FLAC, MP3, MP3pro, WMA, ATRAC, etc)

I suppose so. It's just something I was personally considering for archiving my music. I suggested it because it's lossless, but still uses less space. If you want to keep the original WAV files, then it's not a lot of use...

Personally, if I ran out of disk space, I'd just add another computer...

If you have the cash, buy NAS (not SAN, but NAS) boxes. Not that much $$$. Add to your network. Otherwise, build a cheap fileserver to put on your network, and utilize whatever drive gives best $$$ per MB. Keep checking the bargain websites (www.techbargains.com, www.anandtech.com forums, etc).

Josh, you lost me there. This falls into the option 4. bracket.

I found out who the company is that makes the drive enclosures for Firewire and USB2 (pity they don't do one which has both connections
Firewire http://www.adstech.com/products/PYRO1394DriveKit/specifications/PYRODriveKitProdSpec.asp?pid=API-800
USB2 http://www.adstech.com/products/USB_2_Drivekit/intro/USB2DriveKit.asp?pid=USBSX-804

There is a thread in "Off Topic" for external drive solutions... it links to another thread too...

But there is a product by "Addonics" www.addonics.com that houses 3.5" or 2.5" drives and allows you to connect via usb, usb2.0, firewire, pcmcia, etc. All you have to do is swap the cable. Pretty cool.

There's a good Snap hack page here.

It looks like a good solution, I may get a low capacity one and upgrade if they auction low enough...

now that looks more like it.

I think you're missing Roger's point slightly, in that FLAC is a lossless codec. That is, it can be used to regenerate the ``original'' WAV exactly, and it takes up much less space. There are also plugins for many audio playback utilities, like WinAMP. And there are other lossless codecs besides FLAC. Take a look at the http://flac.sourceforge.net/comparison.html]FLAC comparison page.

how about all the track info like album, year, etc. if I back up the wav files the I would have to re-enter all the info would FLAC be any different in that respect ?

Sorry wfaulk, the point of this thread is concerned about the convenience of having the raw ingredients to hand before encoding. You say that you can use a FLAC file regenerate the original WAV file exactly. But that would be adding an extra 2 steps. I.e. to make an MP3 at some later date the abridged rites of passage of a mp3 files would be ripp, encoded to flac, regenerate to WAV, encode to mp3 (instead of just ripp & encode to mp3). It's simply not worth going to through all that hassle/time just for potentially halving the WAV file size, anyway I would have to a hard drive large enough to convert my FLAC files back to WAV so that I can encode them in the manner which I stated, so that brings me back to square one. So while it's an interesting proposition, the aggravation, time taken, and plain inconvenience really does rule it out as a practical solution.

The reference FLAC decoder will skip over any ID3v2 or ID3v1 tags found. This is probably more useful than WAV files, since the defined tag format for WAV is actually quite restricted.

anyway I would have to a hard drive large enough to convert my FLAC files back to WAV

the aggravation, time taken, and plain inconvenience really does rule it out as a practical solution.

You could do them in batches -- you'd need to if you moved the WAVs onto DVD. Automating the process wouldn't be that hard, anyway.

That's what shell scripts are for .

To summarise:

Anyway, you don't want to, so the solutions you need to look at involve more storage space. In your case, obvious solutions are:

• More hard disks. Complicated -- cases are only so large, and you'll need more disk controllers. Take a look at the 3ware ones while you're casting around, though.
  
• Another computer, with the disks in it. This is probably my preferred solution.
  
• NAS, like having another computer, but low-maintenance, supposedly.
  
• Tape backup. Still relatively cost-effective for archival purposes. Looking less attractive by the day.
  
• DVD-RAM or DVD-RW -- I don't know enough about this to comment sensibly.
  

The disadvantage of the first three is that there's no sensible backup strategy -- do you really want to rip all of those CDs again in the event of a disk failure? -- whereas the last two are their own backup strategy.

I raised the lossless encoder issue because it's something I've been looking at, and I thought it could be of help to you. It's certainly likely to help other people who are looking at archival solutions.

the point of this thread

Everyone knows that threads in this forum never stay on-topic. Jammy Dodgers anyone?

Um, instead of converting all your flac files to wav and then to mp3, why don't convert one flac to wav, then the wav to mp3, then delete the wav, and move onto the next file? That way instead of needing a spare180Gb, you only need an spare 10Mb to do the encoding.

I imagine converting flac to wav would be a very quick process, so it wouldn't slow down your batch processing much. You could quite easily pipe the wav data from the flac decoder to the lame encoder to do it all in one step too.

Thanks Roger,

From what I can gather, It seems that snapserver might be a solution like nas, but with in built backup as well (when using Raid1). Unfortunately it's not cheap, and even if I were to hack it, to accept larger drives, I still don't know whether or not it has a barrier limit of 137Gb.

My bad, I was quoting re-recordable disc prices. Write once media is priced at @ £5 a disc

All the lossless codecs I've used are quite fast, both in compression and decompression, and I'm using a 333Mhz machine. In fact, I wouldn't even be saving intermediate files. Something like:

for i in `find . -name \*.flac -print`; do

  flac -dc $i | lame -b128 -h - - > ${i%.flac}.mp3

done

You could buy a cheap PC and run Linux or a BSD or Solaris or something on it using a volume manager to make a RAID filesystem and share it using Samba or NFS.

BTW, NAS==Network Attached Storage, which just means a remote filesystem accessible over your normal data network, a la SMB or NFS or AppleShares (whatever they're called) or whatever. SAN==Storage Area Network, which is a network devoted solely to storage media, usually connected to multiple host machines. Definitely not a consumer project.

How long would it take to convert into FLAC and then reconvert into WAV files for say about 100Gb worth of WAV music files?

Seriously, it takes seconds to do the transform either way on around four minute songs. Lemme grab some stats....

Hmmm. Seems to take longer than I remember. I compressed Pink Floyd's Money (6:22) in about 1 minute. It decompresses in about 30 seconds. I could swear that I got faster times before. It's still well faster than encoding mp3s, so as long as you do it via a pipe, it'll be just as fast. (Keep in mind I'm doing this on a P2 333.)

Regardless, unless you're listening to the WAV files on your home machine, FLAC will save you space, which is still pertinent to your quest for more drive space. And if you use WinAMP to listen to your WAVs, there's a plugin that works perfectly for FLAC.

Money.wav = 64.3Mb

That means a 100Gb worth of WAV files would take about 26 hours to compress, and a further 13hours to decompress it to WAV again. But you're right this is one way of reducing hard-drive requirements, but it doesn't deal with my particular requirements.

Although I wasn't planning in using the wav files as a listening source, I might consider using FLAC in conjunction with a Lacie pocketdrive as a possible mobile solution. Presumably there aren't any plans to incorporate FLAC support into the empeg? Mind you with ever increasing affordable hard-drive capacities, are we approaching the time when compression technologies will be just a footnote in history? Just imagine music files which don't need encoding, with endless debates about which encoder is best. Or DVD/Digital TV but without digital compression artifacts.

All files need encoding. WAV is a form of encoding. The audio on the CD is encoded in a certain way. Phonograph records are encoded in a certain manner. Even the signals going over the wires to your speakers must be encoded in a certain way. And, as I'm not sure that you're completely following, FLAC introduces zero artifacts (barring bugs). It's like ZIP, except optimized for music. You get back out exactly what you put in, bit for bit.

What I'm wondering is why you're so interested in WAV files. I understand that it would take a while to compress them (which could potentially be obviated by encoding them directly to FLAC when ripping), but if you're not listening to them directly, why are you interested in having them? If your reason is that you might need to reencode them into mp3s (or WMAs or whatever), FLAC decompression still works much faster than mp3 compression, so if you just pipe the output of flac to the input of lame, since flac is presenting data faster than lame can accept it, you're not losing any time, other than, possibly, a few milliseconds per song for it to present its initial data. That is, the bottleneck would be lame, not flac.

Argh. I just bid on a Snap server because of this thread. Thanks guys

LoL... I was tempted to as well. See what you people do ... Actually what I want is my gateway (cobalt cube) microserver... it was really cheesy but with a few hacks the whole box was open to me. Too bad I had to to give it back to gateway after they let me try it out. I shoulda kept it till they discontinued it.. hehe

-Greg

I was just looking at Snap servers. Given that you can get a 160GB hard drives for about $300 and a bare bones system for about $100 (running a free-of-cost OS, be it Linux, FreeBSD, OpenBSD, Solaris, whatever), that means that you could get a 320GB of mirrored NAS for $1300. The Snap 4100 with 240GB retails for $3000 (Pricewatch minimum seems to be about $2500). Oops; I'm sorry, actually, that Snap server would only provide 120GB in a mirrored configuration. And the homebuilt one would be way more upgradable. So that's $4/GB versus $20/GB.

Hope you're getting a good deal at the auction.

I can't find a Linux box 1U in size with 120GB of capacity for (max bid of) $505.

That's 4x30GB Drives. If the hack pages are right, as time goes, I buy a new drive, swap, and I have more space.

1 yr warranty refurbished. Warranty is from Quantum.

Seemed Like a deal to me. Of course now that I've told everyone here about that I'm sure I'll get outbid. They only have 5 a tthat price.

$505 sounds like a good deal. Most folks don't actually have racks or cabinets at home, so I wasn't taking that into consideration. The homebrewed machine could have an assload of drives in it, though (but you might have to get a real kickass power supply). Think about it -- 4 drives on the motherboard, 4 more per PCI slot. Get a motherboard with enough PCI slots (and, again, a kickass power supply, or multiple ones) and you could have 24 drives in the machine, physical space allowing. Plus, it's easy to fix if it breaks. For this purpose, parts is parts.

This is the way I do it, and from free (or in some cases very cheap) second hand kit from 486's and ancient SGI Personal IRIS's through to my latest Athlon boxes I have a total of 1.1ish Terabytes available for use. I have all of it shared using Samba and NFS and it works beautifully. Admittedly I put more deep archive stuff on the older, slower disks and all the latest stuff on decent UltraSCSI hardware.


It makes for easy arrangements and editing - and I use the fast disks for video editing as well.


The point of this post - use old discarded kit. Rack it up in your attic (well I have to so my wife doesn't complain:) and network it all together and Bob's your uncle

I think you have to sit down and spend half the time that has been dedicated to this thread, to decide whether or not you have a "golden ear" and are capable of telling the difference between a quality MP3 and a ripped WAV. I would be willing to bet that you can't. In car, with a portable and any headphones or even on your home system (unless there's a problem with either source or some other colouring of the sound during playback due to equipment).

Personally I would find it a huge waste of time, space and money to keep WAV files when I can keep the CDs as back up. Rip times are very fast (certainly when compared to the amount of time wasted doing any of the things mentioned in this thread). The time-consuming process is the lossy encoding.

The time required to decompress the FLAC file is, more than likely, a result of the file sizes we're dealing with - try moving a file of the same size around.

Bottom line, given unlimited bandwidth and storage space, we could keep raw tracks around. Lossless compressed would still make transfers that much faster. And lossy compression, would still have benefits and possibly no audible differences. Some people forget about the "audible" part.

Bruno

That means a 100Gb worth of WAV files would take about 26 hours to compress, and a further 13hours to decompress it to WAV again. But you're right this is one way of reducing hard-drive requirements, but it doesn't deal with my particular requirements.

Are your requirements
1) Spending money on a bigass raid array.
2) Archiving your music collection in a lossless format suitable generating mp3s.

Don't let things get out of perspective. How long would it take to mp3 encode 100Gb of WAV data on an old p2 ? The 26 hours would be a one off thing as in the future you would rip straight to flac instead of to wav.

So is an extra 13 hours a significant delay compared to the time taken to mp3 encode 100Gb of wav on a p2 -333?

On a modern system you'd probably be looking at 3 seconds to decode the flac and 40 seconds for a high quality mp3 encode. And don't forget that by piping the commands togeather allot of this is going on in parallel, as much of the time taken in the flac decode process will be i/o.

If you can half you current storage requirements this way, then by the time you have doubled your music collection you can probably just go out and buy a 10Tb disc and be set for the next 10 years. Maybe buying another 10Gb in the meantime for AV stuff if you do start messing with that.

My requirements are the ability to encode mp3 to play on my eMpeg (primarily) and mp3pro for burning onto CDRW and taking to work, and low bitrate wma to use on my portable player. Since getting envolved with encoding music files, I've re-encoded several times, firstly with Xing, then with lame 3.88 , and since then lame 3.91 to take advantage of improving technologies. I don't have to, but I choose to.

My beef is not the 26 hours it would take to encode to FLAC, that as you say is a one off, I just posted that as information for the forum. It's the time to take to decode that is bothersome. I have to consider that what is an already long process to produce the end-product I want, will be even longer because of this extra step converting FLAC to WAV, and that I need the space anyway to give me the convenience (heaven forbid) of encoding all my mp3's in one hit. UNLESS there is a way to directly encode from FLAC to mp3/mp3pro/wma files, then convenience and WAV files are my requirements.

Until 2 days ago I didn't even know about the existence of FLAC. As a technology, going on what's been said, it's the equivalent of WinZip, and I'm not faulting it. It's just at present the raw ingredient used by the encoders seems to be WAV. Hopefully one day either one day mp3/mp3pro/wma encoding methods will have matured so that it doesn't change as it does, or these encoders will accept FLAC as the source material. Until then spending a couple of hundred pounds on a few disk is IMHO worth it.

...will be even longer because of this extra step converting FLAC to WAV, and that I need the space anyway to give me the convenience (heaven forbid) of encoding all my mp3's in one hit. UNLESS there is a way to directly encode from FLAC to mp3/mp3pro/wma files, then convenience and WAV files are my requirements.


as wfaulk posted earlier in the thread:

for i in `find . -name \*.flac -print`; do
flac -dc $i | lame -b128 -h - - > ${i%.flac}.mp3
done

No space wasted (all in memory) for wav data. As the flac decoding is far quicker than the mp3/whatever encoding, it'll just pile up the data in the pipe to lame while lame will start encoding as soon as it gets the first data - very little extra time spent compared to encoding directly from wav. Can be run on any level in your archive, from the whole tree to a single directory/album.

If you'd encapsulate the command in a script, it'd even look like it was a single step flac -> mp3 conversion

The above loop is unixy, but it should be possible to do something similar on other platforms (don't recall if you've mentioned what you're running)

/Michael

t's just at present the raw ingredient used by the encoders seems to be WAV

Well as mentioned somewhere, it's easy to send your flac files straight to lame without having to first generate intermediatory WAV files, just pipe the two commands togeather. The extra time for the flac decode should be insignificant compared to the time for a high quality mp3 encode.