HostedDB - Dedicated UNIX Servers

Building your firewall, Part 2

Building your firewall, Part 2

How to make sure your OS is ready to go

A good security architecture is one that is hardened throughout -- not just on the perimeter. A firewall is only as strong as the system that supports it. Before the firewall software is installed, the operating system must be hardened. (3,000 words)


By Carole Fennelly
Last month's column was devoted to planning the architecture of your firewall. While some administrators may consider such planning a management responsibility, it will become an administrator's responsibility if it's not properly addressed before the firewall is implemented. This article focuses on setting up the environment that supports your firewall software. Next month I'll conclude with advice on firewall deployment.

Operating system platform
Choosing a platform becomes almost a religious debate with some people, and it's not one I want to get into here. My choice is always a Unix system, and for this article I will describe a Solaris system (this is SunWorld,after all).

 Series title: Read the whole series! 

  • Part 1. Are you letting your firewall vendor decide your architecture?
  • Part 2. How to make sure your OS is ready to go

  • Be sure to check the requirements for your firewall software. My partner and I once went through the effort of installing Solaris 7 on a box and hardening the system, only to discover that the firewall software had not yet been ported to Solaris 7. We had to backtrack to Solaris 2.6. Read the firewall documentation before you start installing the platform.

    Hardening the operating system
    It would be foolish to rely on the firewall software alone to protect the system. Before installing the firewall software, the system should be hardened to limit potential exposure. If you purchase a "black box" firewall, the vendor should harden the system, though you have no way to be certain that everything is in order. If you're installing the firewall yourself, you can strip all unnecessary services from the OS. Peter Galvin and Hal Pomeranz offered guidelines for hardening your OS in two previous SunWorld columns (see Resources). Peter's tips in his Solaris Security FAQ are more up-to-date (and I hear he will be updating the FAQ page again soon).

    I've used Peter's guidelines quite often myself, with the following modifications:

    Installation options
    Some firewall applications will not work with just the core system support option, and it becomes very time-consuming to install libraries piecemeal. I install the developer system support and remove binaries I don't need.

    Filesystem mounts
    I used to make /tmp a separate filesystem from swap so that I could mount it nosuid and protect the system from denial-of-service attacks that would fill up /tmp and cause the system to run out of swap space. Thanks to input from Casper Dik, Sun's Solaris security architect, I no longer have to do so. The /tmp filesystem can be mounted nosuid even as part of swap. This procedure is not documented, though it does work. Following are the data from some testing I did (on Solaris 2.6). Irrelevant text has been omitted for clarity (as indicated by a period [.]):

    Entry from /etc/vfstab:

    #device         device          mount           FS      fsck    mount
    #to mount       to fsck         point           type    pass    at boot
    .....[other mount info here]
    swap    -       /tmp    tmpfs   -       yes     size=100m,nosuid
    #df -k
    swap                  102400     364  102036     1%    /tmp

    The line above shows that /tmp has 100 MB of available space. The following lines show there is 256 MB available to swap:

    # swap -l
    swapfile             dev  swaplo blocks   free
    /dev/dsk/c0t3d0s1   32,25      8 524392 524392

    From disk format:

    1       swap    wu     270 -  614      256.05MB    (345/0/0)   524400

    To test the nosuid option, I copied the sendmail binary into /tmp, retaining the set-user-id permissions:

    #cp -p /usr/lib/sendmail /tmp
    #ls -l /tmp/sendmail
    -r-sr-xr-x   1 root     bin       346996 Jun  9 04:31 /tmp/sendmail

    Running the binary as root will work because I am already root, not because of the suid permission:

    #/tmp/sendmail -d fennelly < /dev/null > /tmp/out
    # more /tmp/out
    Non-setuid binary: RunAsUid = RealUid = 0
    drop_privileges(0): Real[UG]id=0:1, RunAs[UG]id=0:1

    This is the message logged in /var/adm/messages:

    Sep 27 09:32:53 firebox unix: NOTICE: sendmail, uid 0: setuid execution not
    allowed, dev=1

    I get the following output when I run the same test as a nonprivileged user. Note that I am unable to queue the mail message and unable to create files in /tmp through the sendmail program, although I am able to do so through the standard shell:

    $/tmp/sendmail -d fennelly < /dev/null > /tmp/out
    ksh: /tmp/out: cannot create
    $/tmp/sendmail -d fennelly < /dev/null > /home/fennelly/out
    $ more /tmp/fennelly/out
    Non-setuid binary: RunAsUid = RealUid = 1002
    drop_privileges(0): Real[UG]id=1002:10, RunAs[UG]id=1002:10

    Output from /var/adm/messages:

    Sep 27 09:46:18 firebox unix: NOTICE: sendmail, uid 1002: setuid execution
    not allowed, dev=1
    Sep 27 09:47:18 firebox sendmail[328]: NOQUEUE: SYSERR(fennelly): queuename:
    Cannot create "qfJAA00328" in "/var/spool/mqueue" (euid=1002): Permission

    The following shows that I can create files in /tmp as a nonprivileged user:

    $ cat /etc/hosts > /tmp/hosts

    Aside from the lack of documentation of that feature, pre-Solaris 7 systems also have a bug in the /etc/mnttab entry. As far as I can see, it does not affect the operation:

    $ /usr/sbin/mount
    /export/home on /dev/dsk/c0t1d0s0 setuid/read/write/largefiles on Mon Sep 27
    09:00:00 1999
    /tmp on swap @ on Mon Sep 27 09:26:15 1999
    $ grep tmp /etc/mnttab
    swap    /tmp    tmpfs   @,dev=1 938438775

    For other filesystems, I use the following mount options:

    /usr read only, nosuid
    /opt read write, nosuid
    /var read write, nosuid

    If you need to have optional packages such as sudo to be suid, you can make a separate filesystem just for these.

    Strip down the OS
    Follow the guidelines in the Solaris Security FAQ for stripping out unnecessary services and binaries. There is more to remove if you install the developer system support.

    Prevent TCP sequence prediction attacks
    In the file /etc/default/inetinit, TCP_STRONG_ISS=2 is used to generate the initial sequence numbers suggested in RFC 1948. The manual page for TCP states that /etc/default/inetinit might not be the interface used in future releases, so be forewarned. There might be a small performance hit in connection setup time. Then again, performance always takes a hit when security is applied!

    Prevent buffer-overflow attacks
    Buffer-overflow attacks are becoming very popular (see Resources). You can protect the OS from some of them by adding the following lines to /etc/system:

    set noexec_user_stack=1
    set noexec_user_stack_log=1

    You will need to force a kernel rebuild by rebooting for the code to take effect. Theoretically, some programs that rely on an executable stack may not work at this point, though I do not know of any. Programs generated from objective C are potential candidates for problems (programs compiled from C or C++ source will not have a problem). Because objective C is not in widespread use, it's unlikely to be a problem for most applications.

    Newer versions of GNU C Compiler (later than 2.95) take a nonexecutable stack into account. As with any kernel modification, test your application before putting it into production, and please let me know if you come across problems. As noted in the following comments from Casper Dik, the noexec_user_stack option does not protect against all types of buffer overflows, nor does it work on all Sun hardware:

    A typical buffer overflow exploits work by doing two things to the stack: the return addresses are overwritten, and some code is added. The return addresses are made to point to the code, so the next return from function jumps into the hacker's code. noexec_user_stack protects against the simple form of attack, where the code to be executed lives on the stack; the OS will not allow the code to be executed. Since this requires hardware support, it's only effective on sun4u/sun4d/sun4m systems.

    Dr. Mudge of L0pht Heavy Industries adds that the noexec_user_stack option

    ... is a very useful flag that protects against many stack overflows. However, one must be cautious as this is not a panacea. Heap overflows and return to library functions are still quite exploitable. This flag also has no functionality on sun4c (or whatever the IPXs were) CPUs. In addition, this might cause problems with compilers that use trampolining in rare situations. One should also use the set noexec_user_stack_log=1. Everyone should thank Casper Dik for these cute little hacks.

    Padded cells
    I use the chroot system call for firewall applications, such as sendmail, that are susceptible to exploits. See my January 1999 column (in Resources) for specific details. Note: even if you are not using sendmail on the firewall and have the daemon disabled, turn off the suid bit on the binary. It is not needed on a firewall.

    For everything else, I follow the Solaris Security FAQ noted above.

    Network issues
    Just when you thought it was safe to start installing your firewall software, you need to address some networking issues before you go ahead.

    Routers are designed primarily to route packets. They also support access control lists (ACLs). You may be tempted to overcomplicate the ACLs to a point where the router acts like a firewall. Such complexity can be difficult to support and adds overhead to the router. Keep ACLs simple.

    Firewalls are designed primarily to enforce a security policy by inspecting the data stream. They can laos serve as a router. Using a firewall to route packets makes the firewall serve in a capacity that it was not designed for and adds complexity and overhead.

    Routers should just route, and firewalls should just perform packet inspection.

    Administration issues
    It doesn't matter how tight the firewall software is if an administrator installs a back door. Back doors are much more common than you might think. The reality is that most administrators work long hours, and it is much easier to log in from home to take care of problems than commute to work. There are secure methods for remote administration that provide strong authentication, access control, and encryption. A combination of those enables administrators do their job without compromising security.

    Authentication to the firewall
    On inbound connections, I definitely want some kind of token or one-time password authentication. I also believe in encrypting the pipe coming in (to avoid session hijacking).

    Token cards
    Quite a few companies offer devices to generate one-time passwords. The theory is to combine something you know, such as a personal identification number, with something you have, such as a card. The SecurID card from Security Dynamics is one example.

    S/Key is a one-time password program that allows for single-use passwords. It is incorporated in several manufacturers' software packages as a cheap token. You can also download the source and incorporate it into your own programs.

    Access control

    Final thoughts
    Next month, I'll conclude this series with a discussion of firewall software deployment.

    Much appreciation to Casper Dik of Sun Microsystems and Dr. Mudge of L0pht Heavy Industries for their contributions. Also to Jonathan Klein (Wizard's Keys), who contributed extensively throughout, and to Brian Martin ( for his review and suggestions.

    Disclaimer: The information and software in this article are provided as-is and should be used with caution. Each environment is unique and readers are cautioned to investigate, with their companies, the feasibility of using the information and software in the article. No warranties, implied or actual, are granted for any use of the information and software in this article, and neither author nor publisher is responsible for any damages, either consequential or incidental, with respect to the use of the information and software contained herein.

    About the author
    Carole Fennelly is a partner in Wizard's Keys Corporation, a company specializing in computer security consulting. She has been a Unix system administrator for more than 15 years on various platforms and has particularly focused on sendmail configurations of late. Carole provides security consultation to several financial institutions in the New York City area.

    Resources and Related Links
    Resources mentioned in this article Network address translation resources A listing of previous SunWorld articles on firewalls Firewall comparison tests Other useful firewall resources Other SunWorld resources