Performance and Security: Notes for System Administrators

The eLib Technical Concertation day last November brought together techies from many of the eLib projects. (See Clare McClean’s report in Ariadne issue 6 for more details [1]). A wide range of the technical issues associated with running electronic library services were discussed at the meeting but inevitably, given the time constraints, some of these were not covered in any great detail. However, two issues were clear from the meeting; that server performance and system security were both areas of concern to system administrators and that most eLib projects were using UNIX based machines to make their services available.

This brief article provides some hints and tips in these areas and also gives a set of pointers to further information.

Performance

What do you do if your users report slow access to your network based services? How can you identify the source of the problem? Is it your server hardware that is struggling with the load or are your users seeing underlying network problems? Can your server software be configured or re-written to make better use of the available machine resources?

The ‘uptime’ command will give a general indication of the load on your server. Typically this is shown in terms of the current number of users and the number of jobs in the ‘run queue’ averaged out over the last 1, 5 and 15 minutes. Run the command every so often to get a feel for the normal level of load. That way it should be fairly easy to spot if the load starts to creep up, showing that you may have a potential problem. Graphic tools (like ‘perfmeter’ on Suns for example) may give a better feel for how your server is doing and will allow you to view more specific information about CPU loading or network and disk traffic.

Consider your memory requirements. A machine that is short of memory will start swapping or paging as it gets busier. What this means is that, because there is not enough memory to hold all of the currently running processes in memory at the same time, the operating system will write some of the non-active processes out to disk in order to free up memory. When it needs to run those processes, the operating system must first read them back in from disk thus causing a delay. As a general rule of thumb, the more memory your server has the better.

To really see what is happening on your machine use a combination of ‘ps’, to show the processes that are running, ‘vmstat’ to give a summary of memory usage and any paging and swapping activity and, if necessary, ‘netstat’ and ‘iostat’ to show network and disk traffic.

The ‘top’ command [2] is a very useful ‘curses’ based tool for giving a detailed picture of what a machine is doing and will present much of the information available from the commands listed above in an easy to read manner.

While we’re on the subject of swapping and paging, don’t forget to configure enough swap space for your system. A typical figure is to have twice as much swap space as you have real memory. The combination of real memory and swap space is known as ‘virtual memory’ and is the limiting factor in the size and number of programs that your server can be running at the same time. If your machine runs out of virtual memory then no new processes will be started!

Consider the location of your server on the network. Is the route between your server and your users across a busy local area network? Is there a firewall between your site and the rest of the Internet and, if so, does it have any impact on network performance? As the traffic on the server increases, discuss your service with your Computer Centre or whoever looks after the network.

Consider the layout and usage of disks on your server. Accessing disks is slow in comparison with the CPU or the rate at which information can be moved to and from memory. On large systems with lots of disks the operating system will be able to pass requests out to lots of disks at the same time so it pays to share traffic across disks and disk controllers where possible.

If you want detailed information about tuning UNIX then ‘System Performance Tuning’ by Mike Loukides [3] is a good read. It covers both BSD and SysV based systems.

Web server software. For UNIX, Apache [4] seems to be the server of choice. Older server software, the CERN httpd version 3.0 for example, tends to perform badly, so avoid it. A detailed comparison of Web servers is available from WebCompare [5]. Apache offer some specific hints about tuning their server for better performance under various flavours of UNIX [6].

CGI scripting. If you’re using Perl then see the guidelines towards the back of the ‘Programming Perl’ book [7] on efficiency. Remember that there is an overhead associated with each process that the machine has to start so don’t start sub-shells unnecessarily. For security reasons be careful with the arguments that you pass to the sub-shells that you do start.

In some cases using server-side includes may be more efficient than using CGI scripts. Some servers offer alternative APIs which give better performance so this may be worth investigating. In the case of Apache, writing your own Apache module may be the most efficient way to get a job done.

Take care that your Web server doesn’t defeat Web caches unnecessarily. For example, dynamically generated pages are typically not cache-able and will therefore have to be regenerated by your server for each and every access. The same information held in static HTML pages may be cached much closer to your end users. The ‘Cache Now! campaign’ [8] pages have detailed instructions for how to make your Web pages more cache friendly.

Check your operating system and/or hardware supplier documentation. For example, if you (or your site) has a maintenance contract with Sun, then there will be various performance related white papers available from the SunSolve site [9] . Sun have a freely available, fairly detailed paper on World Wide Web Server Performance including a section on capacity planning [10] . If your service is run under Solaris 2.5.1 then Sun have some Web Server performance enhancements available in the form of the ‘Solaris Internet Service Supplement’ [11].

Don’t run daemons under ‘inetd’ - run them stand-alone. Also known as the daemon of daemons, ‘inetd’ listens for incoming connections for all the services listed in a configuration file and starts up the appropriate server daemon when it gets one. This gives a level of resilience because you can normally be sure that ‘inetd’ will be running but it is much more inefficient than running the daemons stand-alone. (If you do run daemons from ‘inetd’ under older version of the SunOS operating system then watch out for a bug on busy servers which causes inetd to hang every so often).

Very busy servers, particularly those serving requests to end-users across slow networks, may hit problems with the ‘TCP listen backlog’ under some operating systems. Basically this is a limitation that the operating system places on the number of connections in the TCP startup phase at any one time. See the detailed notes from Martin Hamilton’s Web caching pages [12] for more information.

Don’t forget dependencies that your service may have on external servers. For example, a service that restricts access based on the domain name of the connecting system will suffer if the performance of the DNS is poor.

Security

Security is inevitably something of a trade-off between protecting your service against attack and making your service available to a wide audience in an efficient way. Making a service publically available is bound to leave your servers more open to attack than they otherwise would be but with sensible configuration the risks can be minimised.

Assessing the potential threat to your service by hackers is non-trivial and well beyond the scope of this article. However, in offering a network based service it is well worth thinking through what are the things that you need to protect and what are the reasonable lengths to go to in order to protect them. Brent Chapman and Elizabeth Zwicky [13] identify three areas that need protection; your data (some of which may be publically available, some of which you may want to make available in a controlled way and some of which will be private), your resources (the CPU and disk resources on your machine) and your reputation (if an intruder gains control of your machine then they can send out information as if they were you). They also identify three basic categories of attack; intrusion (unauthorised users gaining access to your machine in order to use CPU time or disk space), denial of service (which may take the form of deleting files to prevent your service from running or flooding your service with requests thus preventing genuine users from gaining access) and information theft (stealing data from your server).

If you are running your service within the confines of a university campus network find out what your campus security policy is and what resources are available locally for making your service more secure. If your site runs a firewall, consider what level of access through the firewall is appropriate for your service. It is worth remembering that the security of your machine is somewhat dependent on the security of the site as a whole and that, in turn, is dependent on the security of all the machines at the site.

There are plenty of resources relating to security available on the Web. For information specifically related to security and the Web see the World Wide Web Consortium’s security pages [14] and those put up by the JANET-CERT [15].

For general UNIX security see the JANET-CERT Web pages [16]. Consider running COPS [17] or SATAN [18] which are both utilities that will automatically test your server for well known security loopholes. It is worth noting that SATAN can be used to monitor remote hosts and may therefore be used by hackers as the first step in attacking your site! The TCP Wrapper package [19] provides monitoring of incoming connections to various network services. It also provides access control to limit the address of machines that can connect to the system, remote username lookup (using RFC 931 protocol), and protection against machines that pretend to have someone else’s host name. JANET-CERT provide a list of other useful security related software [20].

Don’t forget to archive your system so that you can recover should things go seriously wrong.

If you are making your service available in a distributed way with machines at different sites and have system administrators logging in between those sites then be careful not to send system passwords across the network in clear. Use a system like ‘ssh’ [21] to provide strongly authenticated, encrypted sessions between your sites.

Every JANET site should have one or two JANET-CERT representatives who will receive timely notification of security alerts - it will probably be someone in the Computer Centre. It may be worth finding out who your site’s representatives are.

References

1. eLib Technical Issues Concertation Day,
   http://www.ukoln.ac.uk/ariadne/issue6/technical-day/
2. TOP, a top process display,
   ftp://ftp.groupsys.com/pub/top/
3. System Performance Tuning,
   http://www.ora.com/catalog/spt/noframes.html
4. Apache HTTP Server Project,
   http://sunsite.doc.ic.ac.uk/packages/apache/
5. WebCompare Web Site,
   http://www.webcompare.com/
6. Hints on Running a High-Performance Web Server,
   http://sunsite.doc.ic.ac.uk/packages/apache/docs/misc/perf.html
7. Programming Perl,
   http://www.ora.com/catalog/pperl2/noframes.html
8. The Cache Now! Web campaign,
   http://www.net.lut.ac.uk/caching/CacheNow/
9. SunService SunSolve Online Server,
   http://online.sunsolve.sun.co.uk/sunsolve/
10. World Wide Web Server Performance,
    http://www.sun.com/sun-on-net/Sun.Internet.Solutions/performance/
11. Solaris Internet Service Supplement
    http://www.sun.com/sunsoft/solaris/products/siss/prodspec.html
12. Other Frequently Asked Questions about WWW Caching - Unix TCP listen backlog
    http://egate.lut.ac.uk/caching/other/#listen1
13. Building Internet Firewalls,
    http://www.ora.com/catalog/fire/
14. W3C Security Resources,
    http://www.w3.org/pub/WWW/Security/Overview.html
15. Information about WWW security,
    http://www.ukerna.ac.uk/newsfiles/janinfo/cert/JANET-CERT/www.html
16. The JANET-CERT WWW Pages,
    http://www.ukerna.ac.uk/newsfiles/janinfo/cert/cert.html
17. COPS download site,
    ftp://cert.org/pub/tools/cops/
18. Information about SATAN,
    http://www.ukerna.ac.uk/newsfiles/janinfo/cert/JANET-CERT/SATAN.html
19. TCP Wrapper,
    ftp://ftp.win.tue.nl/pub/security/
20. Computer Security Software,
    http://www.ukerna.ac.uk/newsfiles/janinfo/cert/JANET-CERT/SOFTWARE.html
21. SSH (Secure Shell) information,
    ftp://ftp.cs.hut.fi/pub/ssh/

Author Details