Each firewall was tested in a single-stream and multi-stream environment. In the single-stream environment, each benchmark simply pushed a single stream of data (either FTP or TTCP) through the firewall as a single connection. The multi-stream environment pushed five simultaneous streams through the firewall. The numbers shown for the multi-stream benchmark are sums of the individual runs. So, if a firewall had a multi-stream performance using TCP of 20 Mbps, that would mean that each of the five streams had an average performance of 4 Mbps.

Before testing the firewalls, Opus One had the two test systems talk to each other to see what the fastest possible performance would be. The FTP benchmark clocked in at a speedy 26 Mbps for the single stream and 31 Mbps for multiple streams. The limit in this case was probably disk speed. The TTCP performance of a single stream using the small 1 K buffers was 15 Mbps, while multiple streams hit 73 Mbps. In this case, the bottleneck was caused by the small TCP buffer sizes. With larger buffer sizes, multi-stream tests often turned in total performance numbers in excess of 90 Mbps.

Packet Filtering Performance Issues

Every firewall vendor likes to come up with some new terminology to describe why their product is different from the other firewalls. One may be a “stateful packet filter with proxy capabilities” while the other is a “third-generation hybrid security management tool.” What it really gets down to is that firewalls look at packets. The basic operation of a firewall is simple: a packet comes in one interface and the firewall either ships it out another interface or drops it. Of course, there are a hundred different variations on this operation, but the essence of network firewalls is forwarding packets.

Different firewalls do this in different ways. The traditional “packet filter” firewall does most of its decision-making based on the first 20 or 40 octets of an IP packet. It knows where the packet is going, where it came from, what the higher level protocol is (typically UDP or TCP), and what application is being called (such as FTP, Telnet, HTTP, or NNTP). Packet filters make most of their decisions based on what they see in the particular packet they’re looking at.

Opus One tested three of the firewalls that had packet filtering capabilities. The simplest packet filter is Global Internet’s Centri. Although the Centri is primarily a proxying firewall, it can also act as a packet filter—although a very simple one. The Centri lets the security manager make forwarding decisions based only on what’s in the current packet.

Both Check Point Software’s Firewall-1 and Network-1’s Firewall/Plus are primarily packet filtering routers, so they take packet filtering much further. They maintain “state” information about what packets have passed through the firewall recently. The firewall combines this information plus the contents of the packet in question to decide whether to drop it or pass it along.

Firewall/Plus is unique in its filtering because it doesn’t start with IP packets; it starts with Ethernet (or token ring) frames. Because the firewall gets all LAN packets at a very low level, it is the only one in this performance test that can handle non-IP protocols, such as IPX, NetBEUI, AppleTalk, and DECnet. Although Firewall/Plus’ user interface is a nightmare, it is the only product on the market that has such precise capability to see inside packets and make forwarding decisions at such a low level.

Another difference with Firewall/Plus is that it intercepts all LAN traffic. All other firewalls act as IP routers, connecting different IP networks or subnetworks. The Firewall/Plus isn’t a router—it’s a bridge. That means it puts the LAN interface into promiscuous mode, listening to all packets on the LAN. It can be truly invisible from any outsider, because you don’t even have to have TCP/IP on your NT system to run Firewall/Plus. The downside is that an NT system is not designed to act as a bridge, and Firewall/Plus’ performance was poor. Worse, when the LAN got busy, Firewall/Plus became so sluggish that the testers couldn’t control it with its own user interface and had to unplug it from the LAN to make any configuration changes.

Conventional wisdom holds that packet filtering firewalls will have superior performance to proxy products because, in theory, the processing is done at a lower level in the protocol stack (layer 3, the network layer). As usual, conventional wisdom is wrong. While Check Point’s Firewall-1 packet filtering product did very well when packet filtering FTP, Digital’s AltaVista beat everybody when proxying TTCP. The most significant factor in performance appears to be choice of operating system, with PORTUS’s proxy FTP performance beating out Firewall-1 by a small margin.