I have been employed (yes -- money involved and everything!) to scan several thousand pages into PDF files. To accomplish this, the employer has lent me her Fujitsu FI-5120C scanner. This is a pretty impressive machine, up to 25 pages per minute through the ADF.

I downloaded the appropriate TWAIN driver and installed it, and then (to once again demonstrate my ignorance) found out that the TWAIN driver isn't where the UI to do the scanning is... I have to have software that talks to the TWAIN driver.

The good news is... the scanner talks to my computer. I was able to scan a page from Microsoft Word into *.doc format, so hardware, connectivity, and TWAIN driver are all in order.

What I need is software, that is, the application to scan from. Adobe Acrobat would be the obvious choice since I am aiming for PDF output. But I am not thrilled about the idea of buying Acrobat 9 for $299, and waiting 10 days for it to arrive.

What are my options? Preferably I want software I can download rather than wait for shipping, and I hope not to have to spend hundreds of dollars. The software must be capable of scanning multiple pages into a single document file, and this scanner does support duplexing.

Any help or suggestions will be greatly appreciated.

tanstaafl.

A brief search of my favorite free-ware help site got me to this forum entry:
http://www.techsupportalert.com/content/scan-multi-page-pdf.htm
Possibly a couple ideas there.

A little googling got me here:
http://www.snapfiles.com/get/scan2pdf.html

Only have the 5 user reviews to go on if it's ok...

:-)

Thank you!!!

tanstaafl.

You can download a fully functional (I believe) 30-day trial from Adobe.

Stu

I've always been a fan of Vuescan. Not free, but not horribly expensive either.

Best of luck to you. It's a good thing it scans that fast, otherwise this would be quite a chore. Are any/many of the pages double-sided? My old office had a Scansnap that would scan both sides of every sheet that went through, then at the end it would just delete all the blank sides. Very nifty machine (I might have to think about picking one up).

Its a tad expensive but the best software I've found is Kofax VRS if you're doing document imaging. We use it at work and its great for doing scans of paper docs. It'll remove the fuzzy blob caused by coloured backgrounds and sharpen text etc...

Is the finished .pdf supposed to be real, editable text? Or just a raster scan of the originals (images) ??

If it's just images, then xsane on Linux is totally free, and more than up for the job. It will scan to image files, which are then easily imported into an OpenOffice document, and then saved as .doc, .pdf, or whatever. Zero software cost, and a pretty simple and gentle learning curve.

If text recognition is required (conversion to real, editable text), then xsane can work, but possibly commercial payware *may* do a better job.

Cheers

Accoreding to the datasheet:

"Fujitsu fi-5120C and fi-5220C workgroup scanners conveniently include everything inbox to get organizations up and scanning. With the full version of Kofax Advance Exchange Plus; VRS 4.x Professional, Adobe Acrobat Standard, and ScandAll 21 software, users get image enhance-ment functionality as well as the option of scanning directly to PDF, file, print or batch."

So, like most recent scanners, this thing was delivered with enough software to create PDFs.

Employer got CD?

Yes. Somewhere. But doesn't know where. And she's leaving the country for two weeks starting tomorrow morning.

However... this fills the bill admirably, and comes with a 30 day full function free trial. The price is just $89, which I find to be reasonable. I'll use it until she returns and finds the software that is supposed to go with the scanner, and if it turns out that the Adobe and ScandAll software is ugly, I'll go ahead and buy the Infothek software.

This software automatically skips blank pages, and is adjustable for brightness, threshold, and contrast so I have been playing with it, fine-tuning it to get the optimal compromise for legibility and image quality as about 20% of the pages are on a grey-shaded background.

The scanner itself is amazing. It has a 50-sheet document feeder, and when running duplex it scans both sides of the page simultaneously, or more likely nearly so as the two scan heads are probably offset to avoid interfering with each other due to shining through the paper. It will scan a 14 page stack full duplex, 10 pages one-sided, four pages double-sided for 18 page output in 47 seconds. If all 18 pages were double-sided it would take the same amount of time, but output would be 36 pages. That's pretty near 50 pages per minute at 300 dpi.

The first software I tried (the freeware that Robotic pointed me to) turned out to be less than satisfactory, it didn't let me access the TWAIN settings, it locked up if I tried to duplex, didn't give enough control over output quality. So, worst case is I spend $89 and buy this software; best case I get the Adobe and ScandAll from Ms. employer.

First estimate is somewhere around 15,000 pages to scan. If I had a 15,000 page document feeder and wanted to end up with a single 15,000 page PDF file it would only take me 5.2 hours. Since I have to create and name about 600-800 separate files, it may take a bit longer.

Oh, and the 15,000 pages are just the ones she had at home before she goes on vacation. The bulk of them are still at her office. Good news is... I'm being paid by the hour.

tanstaafl.

Of course my answer is "Try it on Ubuntu". I setup my HP all-in-one officejet scanner using a simple setup utility over ethernet. The FI-5120C is fully supported by the linux SANE project. (http://www.sane-project.org/sane-mfgs.html#Z-FUJITSU)

I wrote a simple script to scan many documents, do some simple processing and write them out as PNG. I suppose it wouldn't be difficult at all to convert the script to support output as PDF instead.

Ahh, here's that script I wrote: http://ben.nerp.net/scan_docs.sh

It's not complicated, and can be easily hacked up into doing lots of nifty automated things.

I know the "Alaskan mafia" reaches into every burg, but don't tell me. Troopergate?

He said "scanner", not "shredder".

Also, check his current "Loc:".

Ug! I'm sitting here with the software disk you need: Scansnap manager/organiser - I use the predecessor to the machine that you have for all my correspondence: Feed it in straight from the envelope, file it, back-up and shred, I wonder if it's available from a fujitsu site somewhere?

Get a (VERY) large hard disk -- looks like 15GB minimum for the finished product, perhaps 5X that space necessary for temporary shuffling of stuff.

Cheers

Right on the money. I'm averaging just about a megabyte per page at 300 dpi. Disk space is not a problem. I have more than a terabyte of space on my computer, one 300GB drive is just sitting there empty. Her ultimate plan is to store the files on a few 8GB flash drives, but I hope to talk her out of that -- that doesn't seem like permanent/secure storage to me. I'm thinking maybe when it's all done, burn the files onto DVD data disks. Don't they hold something like 2.2 GB?

Temporary shuffling is not a problem -- I'm doing one file at a time, rarely more than 50 pages per file.

tanstaafl.

Yes, I meant "into every burg" to include the bay area and silicon valley.

In 2003 I had about 30 job interviews, a good number of them were at motels out by the airport and arranged by anonymous women named "Cindy" and "Anna". So I get there and these guys wearing tinted glasses kept saying stuff like "So it's a given you know something about computers but what do you know about permafrost?" and asking "Why are manhole covers in Nome rectangular?".

I finally caught on. The Alaska Yakuza are *everywhere*.

Actually they hold about twice that. Four times that on dual-layer discs, but those have never (and I believe never will be) cost effective. If you can find them, they're at least four times the cost of regular DVD-Rs.

Anyone here know which of those two data storage methods is the most likely to last for the greatest number of years into the future?

Mmm.. for this amount of data, I would use a couple of RAID1 arrays, with 2-3 8GB USB keys in each array. And maybe a couple of duplicate DVD-DL discs as well.

For my own stuff here, a rapidly expanding collection of 140GB of photos and stuff, it's simply kept on a hard disk, with a full backup on a removeable hard disc.

Somewhat worrisomel, that, but nobody has put the effort into making a better backup media for the modern age yet.

Cheers

I don't know, but I wouldn't rely on either of them. I would probably keep them on a small, cheap hard drive, and transfer them to a new one every couple years. That's at the very least.

I'm starting to like the phrase "if the file doesn't exist in two places, it doesn't exist." It's a good backup mantra.

I think the modern backup media is online backup. Carbonite or Mozy could easily handle 140GB of data (it'll take forever to upload initially, but should be smooth after that). I certainly prefer it over ZIP disks. Or maybe you'd like a nice LS-120 drive?

In my experience Mozy does not easily handle that amount of data. I did in the end (after many, many months) get it to finish the initial backup of my 100GB of data.

However, it really struggle with the daily incremental backups. It can take hours for it to do the initial scan to work out what needs backing up. It then can take ages to backup a few megs of a multi gig file that has just changed a little (a great example of this is Virtual PC drive images). It often doesn't complete a days changes within the 10 hours or so I can let it run at full speed each day.

So great in theory, in practice my home made rsync backup still works much more reliably. Still, at least it works better than Carbonite...

I guess part of it is expectations. I recommend Mozy to home users, people who don't need to back up drive images. I think if Mark needs to back up 140GB of "photos and stuff," it should work as advertised, though the initial backup does take too long. The one area I'll agree on is backing up email. Naturally, a PST file is going to change daily, and depending on how much of a hoarder the person is (I'm a big one), that can get to be a huge file.

For me, I still think off-site storage (taking drives to relatives' houses or to work) is the best way to go.

If you're interested in storing your email long-term, I'd certainly advise against using PST files. They seem to get corrupted way too easily.

Well, not my email. I'm exclusively GMail. And the people who use Outlook that I care about are all obsessive about deleting emails, so their PST files are less than 100MB.

Regardless, I've never seen a PST file get corrupted unless it hit the 2GB limit on certain Exchange servers. Otherwise in the two businesses I've supported with over 750 people total, I've never seen one get corrupted.

Microsoft finally fixed the 2GB PST limit in Outlook 2003 and newer.

The PST files usually aren't actually corrupted when they hit 2GB. Its just that Outlook can't load them once they get past that point so you're kinda stuck. You have to truncate the file, use one of the many recovery tools or upgrade it to the newer can-cope-with-2GB-and-above PST format in Outlook 2003.

For long term mail storage, you should convert it to a text based format or something. I guess mbox would be okay if its just an archive.

Both will lose your data eventually. I'd guess the flash based one would be a bit better.

Flash memory is conservatively rated for about 10 years of data retention before your electrons get out of the gate.

Writeable optical media has issues with the dyes degrading or the metal layer oxidising. Originally the gold CD-Rs were alledgely rated for over 50 years life but I'm not sure I trust those ratings. All of the estimated lifetime tests were done via accelerated ageing in a lab. Its only now that we're getting proper old optical media.

The solution to this is to keep many backups of your important data and copy it onto whatever is mainstream every few years. Your pristine MFM HD from years ago may still work but as FireFox31 found, its a tad annoying to get the data off those things now.

I used to have an accountant who, I found out one day, posted his daily back up tape to himself: "If we have a fire or robbery in the night, we can get the syatem disks back from the bank and be up and running the next day, when the post comes".
I pointed out that if he left the tape on my desk each evening, I could just as easily take it home, saving my company around £800 a year!
I tried BT's digital vault, but found it cumbersome, sometimes it wouldn't recognise my password even when I pasted it, seemed to have a lot of downtime and missed files, without being able to list what they were, so I gave it up as a bad job.

One of the better long-term things you can do is use something like PAR (http://parchive.sourceforge.net/) to add Reed-Solomon data repair codes. This way you can lose/corrupt parts of the base archive and easily recover using the repair files.

Revised estimate: 25,000 pages. That's really about 17,000 sheets of paper, about half of which will be scanned duplex. And that is just the beginning, the files she had at her house. Then there are the files at her office. And let's not forget the storage unit full of files as well. Probably a quarter million page images by the time we're done.

Good news is that the *.pdf file format is quite compact -- the first 2200 images are taking just 240 MB. The whole thing ought to take no more than 30 GB.

It is taking longer to scan them in than I had anticipated, simply because the prep work takes longer than the scanning. About half the originals are stapled in booklet form, and I have to remove the staples and cut the booklets in half with scissors so they will go through the scanner. With the prep and the scanning, plus fixing the occasional paper jam, I'm averaging about 300 pages per hour. That's about 800 hours, I'm limited to about 20 hours per week (otherwise I get penalized by Social Security for too much income) so this little temp job will be good for about nine months. Perfect.

tanstaafl.

Get a guillotine?

The Scansnap comes with a transparent folder to avoid this, although I imagine that any transparent folder, or sheet, the size of your documents would do.
The Scansnap will cope with four thicknesses of paper in the folder, by which method you can make each two page spread a single scan and end up with a correlated document - at the same time being able to re-constitute the original booklet with staples.

I hope I never have so many documents that I *must* keep.

Not useful in my situation.

The booklets (when unfolded but before being cut in half) would give pages 11x17", which is bigger than the scanner can physically hold -- it uses a 50 page document feeder (it is not a flatbed scanner) through which the maximum paper size is 8.5" by 14" (Legal size). The original paper documents will all be shredded once the scanning is done, so re-constituting them is not an issue.

tanstaafl.

All I can say is, whatever data storage method you use, make sure there are multiple backups in geographically diverse locations. It would be a shame to lose so many months worth of work to a simple data loss.

Make daily backups of your work onto USB flash keys as you progress along with your job. At the end of the project, make your final master backups on more than one copy of more than one media. For example, two or three portable hard drives, two or three stacks of burned DVDs, and maybe even two or three sets of usb flash keys.

Then collate each group of diverse media (group=drive+DVDs+flash) and geographically separate them (1 at office 1 at home 1 in safe deposit).

Also, uuencode the whole thing, then print it out on acid-free paper and keep it in a nitrogen-filled vault.

You tease, but my description costs a pittance, and safely preserves all of the work represented by the scanning and by the information scanned from the paper. Protects it from media loss, disaster data loss, and slightly future-proofs it.

Oh. I was going to engrave it on titanium sheets and store it in a vacuum in total darkness at -273 degrees celsius.

Actually, I'm going to have her put the data on a USB hard drive to keep at her office, on another USB hard drive to keep at home, and on a set of DVD discs to keep in her safe deposit box.

The ironic thing is, the data is virtually valueless. The only reason it is being digitized is because of legal requirements. The chances are almost nonexistant that anybody will ever look at a single one of those digitized pages, but federal law requires them to be kept for at least five years after the parties involved reach the age of 21 years. Or something like that.

tanstaafl.

Ah. That should be good enough, then.

Sounds a bit overboard. Just make a Rosetta Disc out of a nickel alloy instead. Much more simple!

It does strike me that if it's never going to be read and is simply taking up space, the best back up would be to put all the paper in the cheapest lock up storage facility: Can't speak for your side of the pond, but if you do an annual rate without immediate access here, it's pretty cheap.
Sssh: That would do you out of a nice little earner!

I realize this is a good gig for you, but did anyone consider calling Iron Mountain (or equivalent) and asking them to ship the boxes off to some desert warehouse for x number of years before shredding them?

Matthew

Ah. I've been wondering throughout this entire thread why you don't seem to be too worried with how to effectively index the content that you are scanning for later search and retrieval.

I love Iron Mountain. Given, I don't work with them face to face, but when I need a report, our librarian takes care of it and I generally get it in my hand by 10am the next day. I think that is awesome, especially considering the amount of data my company alone stores there (most of the reports I look for are qualification test plans and reports from the late 70s to early 80s).

Iron Mountain isn't doing anything particularly special. They just store your big boxes of reports in hopefully a secure area. Show more love to your librarian IMO. Its up to them to tell Iron Mountain that they need box number 0324-233 because it contains the TPS reports from 1972 that you wanted.

We use Iron Mountain to store huge amounts of tape and paper documents along with a daily courier box. The previous incarnation of the offsite storage company before they got bought out by Iron Mountain had much better service. I can't remember what the last bill was like but it was several thousand GBP each month just on tape storage. Document storage is handled by another department and probably even worse due to physical size.

This is San Francisco Bay area, where real estate is preposterously expensive. This is the area where people routinely buy a very nice house for $800,000 and the first thing they do is bring in the bulldozer to knock it down and build their own $800,000 house. There are just a lot more people than land. That said, the cheapest storage would run about $1000--$1500 per year, and would have the drawback of not having the information readily available in the unlikely event that it might be needed. Imagine walking into a 5'x10' storage unit with boxes full of files stacked six feet tall, looking for the file for "John Smith, DOB 11-12-2002". Oh, yes, I forgot to mention -- the temperature outside the storage unit is 90 degrees F, just think what it's like inside!

And even though in all likelihood the files will never be read, nonetheless she is legally required to have them accessible.

The way we're doing it is better.

tanstaafl.

The data is all in discrete files, each ranging from as little as 15 pages to as many as (so far, at least) 500 pages. Each discrete file pertains to all services for a single individual, and should a search and retrieval ever be needed, that individual's name is the leading part of the filename. So, for example, a file might be named Smith, John DOB 01-15-2002 DOT 07-25-2007 making it easy to scroll down through a directory to find the file she needs. Probably by the time it's all done, there will be about 1000 files, a manageable number when scrolling through an alphabetized file list.

tanstaafl.

And you're not bothering with OCR? She'll never need to search for anything other than the patient(?)'s name?

My mother died last year, leaving a very active sole medical practice behind. Seven year record retention required and no one wanted to take her patients/practice on.

Her former office manager/secretary/bookkeeper/best friend got a very nice new metal shed on her farm, and a nice set of filing cabinets (under some well secured tarps against leaks).

On the first damp day after the seven years are up, we'll have a big bonfire. We expect not a single request from those files, but we gotta keep 'em at the ready. Of course, my mother took her notes cryptically, so no one else could even use them if they get them.

-jk

Tell them to make sure that metal shed is bolted strongly to the ground or concrete pad on which it rests.

A similar shed existed in the yard of a hilltop house I once owned. It was mounted on a very nicely-poured concrete slab. One year, there was a significant windstorm (not gale or hurricane, mind you, just a lot of wind) and the shed ended up at the bottom of the hill, while its contents stayed neatly stacked on the concrete slab.

Those metal sheds seem really sturdy and heavy when you're setting them up. The fact that they weigh a ton means nothing to the wind, when their shape and surface area turn them into what is essentially a kite without a string...

Correct. If there is ever a need to visit the records, it will be to retrieve the entire file for a single individual, and in that case all data in the file will be relevant.

tanstaafl.

Interesting point. I'll have to ask. Though knowing her and her husband, it's solid.

thanks,

-jk