You should read Secure Programming HOWTO. Here's the most common problems with non-suid/sgid programs:
malloc(sizeof(struct) * count) if count is very large,
it could actually allocate only a few bytes and it'd be easy to
exploit after that. These are quite difficult to get right,
especially since you can nowadays easily transfer the needed 2-4GB
of data over internet to be able to exploit some flaw. 64bit
architectures are also becoming more common so it could become
possible for a process to use more than 4GB of memory allowing
exploiting some bugs.
char *buf;
int i, len;
read(fd, &len, sizeof(len));
/* we forgot to check for < 0 */
if (len > 8000) { error("too large length"); return; }
buf = malloc(len);
read(fd, buf, len); /* len casted to unsigned and overflows */
For 64bit architectures:
void *mymalloc(unsigned int size) { return malloc(size); }
char *buf;
size_t len;
read(fd, &len, sizeof(len));
/* we forgot to check the maximum length */
/* 64bit size_t gets truncated to 32bit unsigned int */
buf = mymalloc(len);
read(fd, buf, len);
char *buf; size_t len; read(fd, &len, sizeof(len)); /* we forgot to check the maximum length */ buf = malloc(len+1); /* +1 can overflow to malloc(0) */ read(fd, buf, len); buf[len] = '\0';