Unoffical empeg BBS

Quick Links: Empeg FAQ | RioCar.Org | Hijack | BigDisk Builder | jEmplode | emphatic
Repairs: Repairs

Topic Options
#344507 - 26/04/2011 06:28 Secure memcpy question
frog51
pooh-bah

Registered: 09/08/2000
Posts: 2091
Loc: Edinburgh, Scotland
Morning, this question on secure memcpy for C over on security.stackexchange.com and originally on Stackoverflow has caused a bit of a religious war (which is why the stackoverflow one was pulled) but the original poster really wants to get a rational answer on safe and unsafe commands for non MS C.

So I thought I'd pop it up here as this is one of the few places I can think of that doesn't really do religious warring or irrational argument.

for those who don't use stackexchange, the original question is here:

Quote:
Buffer overflows are nothing new. And yet they still appear often, especially in native (i.e. not managed) code...

Part of the root cause, is usage of "unsafe" functions, including C++ staples such as memcpy, strcpy, strncpy, and more. These functions are considered unsafe since they directly handle unconstrained buffers, and without intensive, careful bounds checkings will typically directly overflow any target buffers.

Microsoft via SDL has banned use of those "unsafe" functions, and provides replacement functions for C++ - e.g. strcpy_s for strcpy, memcpy_s for memcpy, etc (depending on environment). Latest versions of Visual Studio will even let you do this automatically...

But what about "pure" C (i.e. not C++)?
And especially, what about non-MS platforms - including Linux and even non-VS compilers on windows...
Does anyone have safer replacement functions for these? Any recommended workarounds (besides simply doing more bounds checking...)?
Or are we all doomed to continue repeating our use of memcpy?
_________________________
Rory
MkIIa, blue lit buttons, memory upgrade, 1Tb in Subaru Forester STi
MkII, 240Gb in Mark Lord dock
MkII, 80Gb SSD in dock

Top
#344508 - 26/04/2011 12:43 Re: Secure memcpy question [Re: frog51]
tonyc
carpal tunnel

Registered: 27/06/1999
Posts: 7058
Loc: Pittsburgh, PA
I think the answer from ninefingers sums up my feelings on the issue. You can't ever make a really safe memcpy(), you just need to use it properly and test your code.

Using APIs like GLib and APR that do some memory management of data structures for you can be helpful, but ultimately, you're still free to memcpy() yourself into oblivion even when you use them exclusively, and the solution isn't to try to make memcpy safer, it's to use it properly, or not at all if you aren't disciplined enough for it (at which point, hopefully Clippy will pop up and recommend that you use Java or Ruby or something that keeps you from hanging yourself.)
_________________________
- Tony C
my empeg stuff

Top
#344509 - 26/04/2011 13:27 Re: Secure memcpy question [Re: tonyc]
DWallach
carpal tunnel

Registered: 30/04/2000
Posts: 3810
There are "safe" variants of all these functions available on every platform.

That said, I was recently looking at a bunch of reported security vulnerabilities in common applications (student final projects in my security class), and a common feature was that several of these real-world apps were doing string handling by hand. Example: say you want to do an global substitution (s/something-long/short/g), I saw this hand-coded as a for-loop with two variables, writing the shortened string on top of the long string. There was a boundary condition they missed, and thus an exploitable vulnerability.

This boils down to this deep need among C hackers to micro-optimize absolutely everything and to handle strings directly rather than through libraries. Now, if somebody could tell me how to address these dual tendencies...

Top
#344510 - 26/04/2011 13:43 Re: Secure memcpy question [Re: DWallach]
siberia37
old hand

Registered: 09/01/2002
Posts: 702
Loc: Tacoma,WA
Originally Posted By: DWallach
That said, I was recently looking at a bunch of reported security vulnerabilities in common applications (student final projects in my security class), and a common feature was that several of these real-world apps were doing string handling by hand. Example: say you want to do an global substitution (s/something-long/short/g), I saw this hand-coded as a for-loop with two variables, writing the shortened string on top of the long string. There was a boundary condition they missed, and thus an exploitable vulnerability.

This boils down to this deep need among C hackers to micro-optimize absolutely everything and to handle strings directly rather than through libraries. Now, if somebody could tell me how to address these dual tendencies...


The funny thing is I bet if you compared some of this micro-optimized code to code that used conventional libraries the library code would be basically the same speed or close enough to not really be a factor. There is a lot of value in knowning when to start doing memcpy (or even inline assembler) to optimize something and when to leave well enough alone. It's not always obvious when to do this with today's very smart compilers, and libraries that have been highly optimized over the years.



Edited by siberia37 (26/04/2011 13:44)

Top
#344511 - 26/04/2011 14:59 Re: Secure memcpy question [Re: siberia37]
canuckInOR
carpal tunnel

Registered: 13/02/2002
Posts: 3212
Loc: Portland, OR
Originally Posted By: siberia37
There is a lot of value in knowning when to start doing memcpy (or even inline assembler) to optimize something and when to leave well enough alone. It's not always obvious when to do this with today's very smart compilers, and libraries that have been highly optimized over the years.

I disagree. It's very obvious when to do this, and the answer is "after performance has been deemed unacceptable by the user, and profiling has demonstrated where the bottlenecks are."

Top
#344515 - 26/04/2011 17:53 Re: Secure memcpy question [Re: frog51]
peter
carpal tunnel

Registered: 13/07/2000
Posts: 4174
Loc: Cambridge, England
This is what encapsulation is for; i.e. this is (part of) what C++ is for. When you've got C code which needs to be written in a certain stylised way -- disable_irq/enable_irq pairs, memory handling -- C++ often lets you design an API in which only safe programs can be expressed. (RAII and std::string respectively, for those two examples.)

This part of C++ (unlike some of the others e.g. exceptions, RTTI) requires no runtime support and is in theory no less efficient than the equivalent C would be if written-out.

So the "pure C, not C++" questioner has it backwards, i.e. has needlessly restricted the question to not permitting the actual answer. (Unless they're using such a restricted embedded system that C++ is way out -- perhaps on a DSP like those squalid Sigmatel things that Rio Chiba/Carbon/etc were made from.)

And yes, IMO Linus Torvalds is dead wrong on this. The kernel is chock-full of C++ -- all those fops and vops structures are in fact just vtables -- it's just that he quixotically prohibits all this C++ from being written in the normal idiom.

Peter

Top
#344516 - 26/04/2011 18:24 Re: Secure memcpy question [Re: canuckInOR]
siberia37
old hand

Registered: 09/01/2002
Posts: 702
Loc: Tacoma,WA
Originally Posted By: canuckInOR
Originally Posted By: siberia37
There is a lot of value in knowning when to start doing memcpy (or even inline assembler) to optimize something and when to leave well enough alone. It's not always obvious when to do this with today's very smart compilers, and libraries that have been highly optimized over the years.

I disagree. It's very obvious when to do this, and the answer is "after performance has been deemed unacceptable by the user, and profiling has demonstrated where the bottlenecks are."


I think you need to add.. "and it can be proven that you could write something faster than the library". Unless your a wiz at counting clock cycles and understand modern assembler thoroughly (and how your compiler produces it) it really can be hard to beat a lot of library functions.

Top
#344519 - 26/04/2011 19:46 Re: Secure memcpy question [Re: siberia37]
canuckInOR
carpal tunnel

Registered: 13/02/2002
Posts: 3212
Loc: Portland, OR
Originally Posted By: siberia37
Originally Posted By: canuckInOR
Originally Posted By: siberia37
There is a lot of value in knowning when to start doing memcpy (or even inline assembler) to optimize something and when to leave well enough alone. It's not always obvious when to do this with today's very smart compilers, and libraries that have been highly optimized over the years.

I disagree. It's very obvious when to do this, and the answer is "after performance has been deemed unacceptable by the user, and profiling has demonstrated where the bottlenecks are."


I think you need to add.. "and it can be proven that you could write something faster than the library". Unless your a wiz at counting clock cycles and understand modern assembler thoroughly (and how your compiler produces it) it really can be hard to beat a lot of library functions.

I thought that part was implicit in the profiling bit, i.e. if you haven't profiled your replacement to whatever the bottleneck is, then you have no idea if you've actually made an improvement.

Top
#344646 - 02/05/2011 16:09 Re: Secure memcpy question [Re: canuckInOR]
frog51
pooh-bah

Registered: 09/08/2000
Posts: 2091
Loc: Edinburgh, Scotland
Thanks for the comments there guys - like I said, I knew it would be a rational conversation here :-)
_________________________
Rory
MkIIa, blue lit buttons, memory upgrade, 1Tb in Subaru Forester STi
MkII, 240Gb in Mark Lord dock
MkII, 80Gb SSD in dock

Top
#344647 - 02/05/2011 16:39 Re: Secure memcpy question [Re: frog51]
canuckInOR
carpal tunnel

Registered: 13/02/2002
Posts: 3212
Loc: Portland, OR
Originally Posted By: frog51
I knew it would be a rational conversation here :-)

We missed our opportunity for irrational conversation when Pi day went past. Perhaps we can try again on Tau day.

Top
#344652 - 02/05/2011 17:46 Re: Secure memcpy question [Re: peter]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14484
Loc: Canada
Originally Posted By: peter
And yes, IMO Linus Torvalds is dead wrong on this. The kernel is chock-full of C++ -- all those fops and vops structures are in fact just vtables -- it's just that he quixotically prohibits all this C++ from being written in the normal idiom.

The modern libata subsystem (SATA stuff) in Linux is an even better example of OOP: it uses object intheritence and the whole sheebang, elegantly done in C. smile

The choice of C for the kernel means it's easier to port to new platforms, requiring a less complex (aka. less buggy) compiler to get things up and running.

If one prefers to work with C++ in kernel space, then there are other kernels that do it that way.

Top
#344655 - 02/05/2011 21:05 Re: Secure memcpy question [Re: mlord]
siberia37
old hand

Registered: 09/01/2002
Posts: 702
Loc: Tacoma,WA
Originally Posted By: mlord

If one prefers to work with C++ in kernel space, then there are other kernels that do it that way.


Are you referring to the NT err I mean Windows kernel? It seems it's still mostly C for low-level stuff - with some C++ thrown in for higher level modules. At least that's what I remember reading. Of course MS gives developers a choice to use C or C++ for Kernel Mode Drivers with a nice long KB article about the Pros and Cons of each. Great light reading.

Top