SUSE Linux Enterprise Server

System Analysis and Tuning Guide

12/15/2010

All content is copyright © 2006– 2010 Novell, Inc. All rights reserved.

Legal Notice

This manual is protected under Novell intellectual property rights. By reproducing, duplicating or distributing this manual you explicitly agree to conform to the terms and conditions of this license agreement.

This manual may be freely reproduced, duplicated and distributed either as such or as part of a bundled package in electronic and/or printed format, provided however that the following conditions are fulfilled:

That this copyright notice and the names of authors and contributors appear clearly and distinctively on all reproduced, duplicated and distributed copies. That this manual, specifically for the printed format, is reproduced and/or distributed for noncommercial use only. The express authorization of Novell, Inc must be obtained prior to any other use of any manual or part thereof.

For Novell trademarks, see the Novell Trademark and Service Mark list http://www.novell.com/company/legal/trademarks/tmlist.html. Linux* is a registered trademark of Linus Torvalds. All other third party trademarks are the property of their respective owners. A trademark symbol (®, ™ etc.) denotes a Novell trademark; an asterisk (*) denotes a third party trademark.

All information found in this book has been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. Neither Novell, Inc., SUSE LINUX Products GmbH, the authors, nor the translators shall be held liable for possible errors or the consequences thereof.


Contents

About This Guide
1. Available Documentation
2. Feedback
3. Documentation Conventions
I. Basics
1. General Notes on System Tuning
1.1. Be Sure What Problem to Solve
1.2. Rule Out Common Problems
1.3. Finding the Bottleneck
1.4. Step-by-step Tuning
II. System Monitoring
2. System Monitoring Utilities
2.1. Multi-Purpose Tools
2.2. System Information
2.3. Processes
2.4. Memory
2.5. Networking
2.6. The /proc File System
2.7. Hardware Information
2.8. Files and File Systems
2.9. User Information
2.10. Time and Date
2.11. Graph Your Data: RRDtool
3. Monitoring with Nagios
3.1. Features of Nagios
3.2. Installing Nagios
3.3. Nagios Configuration Files
3.4. Configuring Nagios
3.5. Troubleshooting
3.6. For More Information
4. Analyzing and Managing System Log Files
4.1. System Log Files in /var/log/
4.2. Viewing and Parsing Log Files
4.3. Managing Log Files with logrotate
4.4. Monitoring Log Files with logwatch
III. Kernel Monitoring
5. SystemTap—Filtering and Analyzing System Data
5.1. Conceptual Overview
5.2. Installation and Setup
5.3. Script Syntax
5.4. Example Script
5.5. For More Information
6. Kernel Probes
6.1. Supported Architectures
6.2. Types of Kernel Probes
6.3. Kernel probes API
6.4. Debugfs Interface
6.5. For More Information
7. Perfmon2—Hardware-Based Performance Monitoring
7.1. Conceptual Overview
7.2. Installation
7.3. Using Perfmon
7.4. Retrieving Metrics From DebugFS
7.5. For More Information
8. OProfile—System-Wide Profiler
8.1. Conceptual Overview
8.2. Installation and Requirements
8.3. Available OProfile Utilities
8.4. Using OProfile
8.5. Using OProfile's GUI
8.6. Generating Reports
8.7. For More Information
IV. Resource Management
9. General System Resource Management
9.1. Planning the Installation
9.2. Disabling Unnecessary Services
9.3. File Systems and Disk Access
10. Kernel Control Groups
10.1. Technical Overview and Definitions
10.2. Scenario
10.3. Control Group Subsystems
10.4. Using Controller Groups
10.5. For More Information
11. Power Management
11.1. Power Management at CPU Level
11.2. The Linux Kernel CPUfreq Infrastructure
11.3. Tuning Options for P-states
11.4. Tuning Options for C-states
11.5. Creating and Using Power Management Profiles
11.6. Monitoring Power Consumption with powerTOP
11.7. Troubleshooting
11.8. For More Information
V. Kernel Tuning
12. Installing Multiple Kernel Versions
12.1. Enabling Multiversion Support
12.2. Installing/Removing Multiple Kernel Versions with YaST
12.3. Installing/Removing Multiple Kernel Versions with zypper
13. Tuning Per-Device I/O Performance
13.1. I/O Scheduler -- /sys/block/<device>/queue/scheduler
13.2. I/O Barrier Tuning
14. Tuning the Task Scheduler
14.1. Introduction
14.2. Process Classification
14.3. O(1) Scheduler
14.4. Completely Fair Scheduler
14.5. For More Information
15. Tuning the Memory Management Subsystem
15.1. Memory Usage
15.2. Reducing Memory Usage
15.3. Virtual Memory Manager (VM) Tunable Parameters
15.4. Non-Uniform Memory Access (NUMA)
15.5. Monitoring VM Behavior
16. Tuning the Network
16.1. Configurable Kernel Socket Buffers
16.2. Detecting Network Bottlenecks and Analyzing Network Traffic
16.3. Netfilter
16.4. For More Information
VI. Handling System Dumps
17. Tracing Tools
17.1. Tracing System Calls with strace
17.2. Tracing Library Calls with ltrace
17.3. Debugging and Profiling with Valgrind
17.4. For More Information
18. Kexec and Kdump
18.1. Introduction
18.2. Required Packages
18.3. Kexec Internals
18.4. Basic Kexec Usage
18.5. How to Configure Kexec for Routine Reboots
18.6. Basic Kdump Configuration
18.7. Analyzing the Crash Dump
18.8. Advanced Kdump Configuration
18.9. For More Information

List of Figures

2.1. iptraf Running in Interactive Mode
2.2. Example Graph Created with RRDtool
7.1. Architecture of perfmon2
8.1. GUI for OProfile
10.1. Resource Planning
12.1. The YaST Software Manager - Multiversion View
18.1. YaST2 Kdump Module - Start-Up Page

List of Tables

7.1. Supported Processors
7.2. Needed Packages
7.3. Read-Only Files in /sys/kernel/debug/perfmon/cpu*/
11.1. C-States
18.1. Recommended Values for Additional Kernel Command Line Parameters

List of Examples

2.1. vmstat Output on a Lightly Used Machine
2.2. vmstat Output on a Heavily Used Machine (CPU bound)
3.1. A Host Object Definition
3.2. A Service Object Definition
3.3. A Contact and Contactgroup Definition
4.1. Example for /etc/logrotate.conf
5.1. Simple SystemTap Script
5.2. Probe with Timer Event
5.3. printf Function with Format Specifiers
5.4. Using Global Variables
5.5. Monitoring Incoming TCP Connections with tcp_connections.stp
11.1. Example powerTOP Output