#255107 - 29/04/2005 15:54
Removing a blank line from a text file
|
pooh-bah
Registered: 27/02/2004
Posts: 1900
Loc: London
|
How do I do this? It's on Linux, a friend suggested:
cat file | sed “/^.$/d” > newfile
but I've still got a LF at the end of the file (I think).
Attachments
254799-disp.csv (216 downloads)
|
|
Top
|
|
|
|
|
#255108 - 29/04/2005 17:28
Re: Removing a blank line from a text file
[Re: tahir]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
Most Unix text editors will put a LF at the end of the last line, sed probably included. You'll probably need a binary editor.
If you have it already, and you might, use vim. Load the file into vim with the -b option (vim -b filename), then, inside vim, unset the endofline option (:set noendofline), then save (:wq).
You can compare by running od on the file before and after. A good way would be "tail -1 filename | od -xc".
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#255109 - 29/04/2005 19:55
Re: Removing a blank line from a text file
[Re: tahir]
|
enthusiast
Registered: 11/06/2003
Posts: 384
|
Code:
# IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
sed 's/.$//' # assumes that all lines end with CR/LF
sed 's/^M$//' # in bash/tcsh, press Ctrl-V then Ctrl-M
sed 's/\x0D$//' # gsed 3.02.80, but top script is easier
# IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format
sed "s/$/`echo -e \\\r`/" # command line under ksh
sed 's/$'"/`echo \\\r`/" # command line under bash
sed "s/$/`echo \\\r`/" # command line under zsh
sed 's/$/\r/' # gsed 3.02.80
http://www.student.northpark.edu/pemente/sed/sed1line.txt http://www.google.com/search?q=sed+dos+unix
|
|
Top
|
|
|
|
|
#255110 - 29/04/2005 20:46
Re: Removing a blank line from a text file
[Re: tahir]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
So far, you've got three completely different answers. Let's figure out what it is you're trying to do.
The file you attached has no blank lines, and its last line ends with a newline, as is customary with a Unix text file, of which CSVs are a specific kind.
I assume you don't want to delete blank lines, as your sample contains none. If you do, that's what the command you stated will do, almost. It'll actually delete every line that contains only a single character.
Mataglap's answers will convert DOS/Windows text files to Unix text files and the other way around. (Although the nearly ubiquitous dos2unix and unix2dos commands will do the same thing.) That doesn't seem to match your description, though.
If you want to delete the final newline from a text file, what I suggested will work, although there are other ways, few of them one-liners, since Unix command line utilities are designed to work with text files and what you want to do would make it a non-standard text file.
On the other hand, maybe you're confused about the trailing newline and you think it's not supposed to be there. It is. Are you having a problem with the CSV file that makes it look like you have an extra blank line?
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#255111 - 30/04/2005 01:43
Re: Removing a blank line from a text file
[Re: wfaulk]
|
carpal tunnel
Registered: 17/12/2000
Posts: 2665
Loc: Manteca, California
|
When that file is opened with Wordpad, the insertion point can be moved to a blank line after the last line of data. A backspace will move the insertion point to the end of the last line of data.
I suspect he wishes to concatenate two such files together in a gapless form so there are no blank lines in the spreadsheet.
_________________________
Glenn
|
|
Top
|
|
|
|
|
#255112 - 30/04/2005 17:45
Re: Removing a blank line from a text file
[Re: gbeer]
|
carpal tunnel
Registered: 13/07/2000
Posts: 4174
Loc: Cambridge, England
|
Quote:
I suspect he wishes to concatenate two such files together in a gapless form so there are no blank lines in the spreadsheet.
But the file he posted is already in that form. If the final LF were deleted, concatenating two files would result in the first line of the second appearing alongside (concatenated to) the last line of the first, not after it as desired.
Peter
|
|
Top
|
|
|
|
|
#255113 - 30/04/2005 21:14
Re: Removing a blank line from a text file
[Re: peter]
|
enthusiast
Registered: 11/06/2003
Posts: 384
|
And if it's the EOF marker that's the issue under discussion, it really won't matter. If it's cat'ed with another file the OS will take care of it properly. In the unlikely case that it doesn't happen it's still a text file, so there will just be a weird line in the middle of the file that's easy to find or fix.
I don't understand what the issue is other than the classic DOS vs. UNIX EOL issue, which can easily be handled with any of the solutions above, tools like dos2unix/unix2dos or flip which are likely either installed or easily installed for any Linux distro. (If we're talking about another Unix os like IRIX or HP-UX or something sed might be the easiest choice.)
By hand it's just a matter of adding or removing a "^M" / "LF" / "\r" / "\d013" / "\o015" / "\00d" (pick your method of entering a representation of the character) from the end of each line, easily done with sed on Unix or on Windows with cygwin. Looking for "." is bad because that means "any single character" which will strip off the last character from the line whether it's a ^M or not, so just anchor (with the "$") a ^M at the end of the line.
I know of at least one free win32 text editor (PFE) that can easily save the file in either format via the bottom status bar indicator, and I'd really be surprised if some of the others like TextPad &c. can't do the same as well.
--Nathan
|
|
Top
|
|
|
|
|
#255114 - 01/05/2005 05:10
Re: Removing a blank line from a text file
[Re: wfaulk]
|
pooh-bah
Registered: 27/02/2004
Posts: 1900
Loc: London
|
Quote:
If you want to delete the final newline from a text file, what I suggested will work
On the other hand, maybe you're confused about the trailing newline and you think it's not supposed to be there. It is. Are you having a problem with the CSV file that makes it look like you have an extra blank line?
The file is submitted via an upload process and the last LF makes it unreadable for those purposes, the program that reads in the uploaded file can't deal with an extra LF at the end of the file (HM Customs & Excise)
Thanks so far
|
|
Top
|
|
|
|
|
#255115 - 01/05/2005 05:23
Re: Removing a blank line from a text file
[Re: wfaulk]
|
pooh-bah
Registered: 27/02/2004
Posts: 1900
Loc: London
|
Quote:
If you have it already, and you might, use vim. Load the file into vim with the -b option (vim -b filename), then, inside vim, unset the endofline option (:set noendofline), then save (:wq).
Can this be automated by a script file? The rest of the process is automatic and I need this to be as well if possible.
|
|
Top
|
|
|
|
|
#255116 - 01/05/2005 05:42
Re: Removing a blank line from a text file
[Re: tahir]
|
pooh-bah
Registered: 27/02/2004
Posts: 1900
Loc: London
|
BTW just so you know vim, cat, vi etc scare the crap out of me
|
|
Top
|
|
|
|
|
#255117 - 01/05/2005 05:48
Re: Removing a blank line from a text file
[Re: tahir]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
If that's really what you want to do, and, again, it's odd, and you want to automate it, I'd say a better way would be to write a program to do it. If you don't feel up to that, a fairly unoptimized one-liner might be: Code:
dd if=file of=newfile bs=1 count=`expr \`wc -c file | awk '{print $1}'\` - 1`
Ugh. Ugly. But it works.
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#255118 - 01/05/2005 05:52
Re: Removing a blank line from a text file
[Re: wfaulk]
|
pooh-bah
Registered: 27/02/2004
Posts: 1900
Loc: London
|
Many thanks Bitt
Knowing very little about file manipulation under Linux I'm going to try and attack this from the other end, I'll see if I can change the way the file is generated to remove the last LF.
|
|
Top
|
|
|
|
|
#255119 - 01/05/2005 06:10
Re: Removing a blank line from a text file
[Re: tahir]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
Well, like I said, that command will work. A more optimized one might be:
Code:
perl -pe 'BEGIN { $RS = undef; } chomp;' < file > newfile
or even:
Code:
perl -pie 'BEGIN { $RS = undef; } chomp;' file
(that one will overwrite the file in-place instead of having to start working with a new file as well as using built-in perl IO instead of relying on file descriptor redirection)
Anyway, they should be faster, assuming you have perl installed (you probably do) and assuming perl's read/write code is better optimized than the 1-byte-at-a-time dd-based one-liner I suggested above.
Edited by wfaulk (01/05/2005 06:12)
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
#255120 - 01/05/2005 06:39
Re: Removing a blank line from a text file
[Re: wfaulk]
|
veteran
Registered: 19/06/2000
Posts: 1495
Loc: US: CA
|
Mmmmm PIE!
_________________________
Donato MkII/080000565 MkIIa/010101253 ricin.us
|
|
Top
|
|
|
|
|
#255121 - 01/05/2005 09:05
Re: Removing a blank line from a text file
[Re: ricin]
|
pooh-bah
Registered: 27/02/2004
Posts: 1900
Loc: London
|
Many thanks again for all your comments, I hadn't realised but the application that generated the .csv was adding CR/LF as it's default EOR delimiter, once I worked out how to change it to CR it loaded fine.
One of these days I should really do a crash course in linux.
|
|
Top
|
|
|
|
|
#255122 - 01/05/2005 15:48
Re: Removing a blank line from a text file
[Re: Mataglap]
|
pooh-bah
Registered: 13/09/1999
Posts: 2401
Loc: Croatia
|
Quote:
"^M" / "LF" / "\r" / "\d013" / "\o015" / "\00d"
<nitpicking>
"^M" / "CR" / "\r" / "\d013" / "\o015" / "\x00d"
"^J" / "LF" / "\n" / "\d010" / "\o012" / "\x00a"
</nitpicking>
FWIW, my favourite free win32 editor is gvim (beware, www.gvim.org leads to a search site that tries to sneakily download and install some rubish!). It can save text files with DOS, Unix or MacOS EOLs, too.
_________________________
Dragi "Bonzi" Raos
Q#5196
MkII #080000376, 18GB green
MkIIa #040103247, 60GB blue
|
|
Top
|
|
|
|
|
#255123 - 01/05/2005 15:56
Re: Removing a blank line from a text file
[Re: bonzi]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14484
Loc: Canada
|
Quote:
sneakily download and install some rubish
What browser would permit that???
|
|
Top
|
|
|
|
|
#255125 - 01/05/2005 16:22
Re: Removing a blank line from a text file
[Re: bonzi]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14484
Loc: Canada
|
Quote:
configure myself a decent Linux machine...
Heck, no -- use OSX if it makes you happy! Linux isn't the only decent system out there!
Cheers
|
|
Top
|
|
|
|
|
#255127 - 02/05/2005 06:20
Re: Removing a blank line from a text file
[Re: wfaulk]
|
carpal tunnel
Registered: 13/02/2002
Posts: 3212
Loc: Portland, OR
|
Quote:
Well, like I said, that command will work. A more optimized one might be:
Code:
perl -pe 'BEGIN { $RS = undef; } chomp;' < file > newfile
or even:
Code:
perl -pie 'BEGIN { $RS = undef; } chomp;' file
IIRC, chomp() removes $RS, and, since you've set that to undef to slurp the whole file, chomp is a no-op -- see 'perldoc -f chomp' -- so I'm guessing this wouldn't modify the file at all. Not to mention, in the -pie version, the -i flag takes a parameter, which sucks up the 'e' flag. You need to use "perl -pi -e", instead. On the subject of flags, you'll also want "-MEnglish" -- $RS is in the English module.
Code:
perl -e'{local $/=undef; @a=<>;}chomp $a[-1];print @a' < file > newfile
or
Code:
perl -pi -e'BEGIN{$/=undef;}s/\n$//' file
One caveat to epwc -- both of these slurp the entire file into memory, so if the attached example file was only a snippet of a very large file, you may not want to use this method.
|
|
Top
|
|
|
|
|
#255128 - 02/05/2005 12:12
Re: Removing a blank line from a text file
[Re: canuckInOR]
|
carpal tunnel
Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
|
You're right. I tested the first one and then assumed the second would work. And I may have typoed chomp for chop in both cases. 
_________________________
Bitt Faulk
|
|
Top
|
|
|
|
|
|
(actually, a wrapper around it called 
) and Linux for everything else (now that support for Latin2 has matured).
Previous Topic
Index