Sun Performance and Tuning ^{2nd Ed.}

Hailed in its first edition as an indispensable reference for system administrators, Sun Performance and Tuning has been revised and expanded to cover Solaris 2.6, the newest generation of SPARC hardware, and the latest Internet and Java server technologies.

Featuring a quick guide to get you started, as well as detailed reference information, this book is indispensable both for developers who need to design for speed and administrators who need to manage system and network performance.

Performance guru Adrian Cockcroft brings his unique expertise and structured approach to this complex and rapidly changing topic, providing detailed information on key aspects of performance management and system behavior that is not available anywhere else. Rich Pettit, author of the SE performance toolkit, describes the performance interfaces in Solaris and how to use this freely available toolkit to build your own customized performance-monitoring tools.

Key topics covered include:

• Performance Management and Measurement
• TCP and Internet Server Tuning
• JAVA Network Computer Server Sizing
• SPARC System Architectures
• Kernel Algorithms and Tuning
• How to Build Your Own Performance Tools
• Performance Rules and the virtual_adrian SE Tool

To get up to speed quickly on critical performance issues, this is the one book any Sun administrator, integrator or developer needs.

1. Quick Tips and Recipes
• Quick Reference for Common Tuning Tips
• The system will usually have a disk bottleneck
• Poor NFS response times may be hard to see
• Avoid the common vmstat misconceptions
• Don't panic when you see page-ins and page-outs in vmstat
• Use page scanner "sr" activity as your RAM shortage indicator
• Look for a long run queue (vmstat procs r)
• Look for processes blocked waiting for I/O (vmstat procs b)
• Check for CPU system time dominating user time
• Watch out for processes that hog the CPU
• Cold Start Procedure
• Configuration and Tuning Recipes
• Single-User Desktop Workstation Recipe
• Workgroup Server Recipe
• Database Server Recipe
• Multiuser Server with ASCII or X Terminals Recipe
• Internet and Java Server Recipes
2. Performance Management
• What Is Performance Management?
• Financial Management Scenario
• Computer System Performance Analogies
• Performance Management Cost Justification
• The Life Cycle of Performance Management
• Performance Tools Provided in Solaris 2
• Performance Management Products
• Performance Management Terminology and Functions
• Product Overview
• Product Evaluations
• Thoughts on the Future
3. Performance Measurement
• The Workload
• Fixed Workloads
• Random Workloads
• Managed Workloads
• Workload Characterization
• Configuration Changes
• Measurement Types and Relationships
• Fundamental Types
• Multiuser System Response Time
• Collecting Measurements
• Using Accounting to Monitor the Workload
• Collecting Long-Term System Utilization Data
• Percollator — A Customizable Data Collector
• Processing and Analyzing the Measurements
• Generating Averages with sar
• Using the Results Generated by ab Emulator System
• Obtaining Results on a Live System
• Analyzing the Results
• Producing a Sizing Model
• Looking for Trends
• Further Reading
4. Internet Servers
• Introduction to TCP
• Connections
• TCP/IP Patches and the Listen Queue Fix
• The TCP Slow Start Algorithm and Congestion Window Problems
• TCP Data Transfer
• Network Data Transfer
• TCP Flow Control
• TCP Parameter Tuning
• Features of the Solaris Internet Server Supplement
• TCP Monitoring with the SE Toolkit
• Further Reading
• TCP Transfer Characteristics for Web Servers
• Collecting a snoop Sequence
• Tiny HTTP Transfer Characteristics
• Medium-Sized HTTP Transfer with Retransmit Congestion
• A Clean Transfer to a Buggy TCP Client
• Small Window-Limited Transfers
• Tail End of Large Transfer with Window Close
• Large Transfer Retransmit Problems
• Increased Minimum Retransmit Time-out
• HTTP Persistent Connection Sequences
• Additional TCP Sequences
• Internet Web Server Sizing
• How Many Users?
• How Many Operations Per Day and Per Second?
• Network Capacity for HTTP
• HTTP-Only Server Performance
• Additional Web Server Workloads
• Disk Capacity and HTTP Access Patterns
• RAM Requirements
• RAM Usage — How Many Instances?
• HTTP Size Mixes and Benchmarks
• HTTP Performance  Measurement
• Web Server Performance Management
• The SE Toolkit Web Server Performance Collector, percollator.se
• Processing Collected Data
• Real-World HTTP Rate Changes over Time
• Load Curve for aa Week
• HTTP Load over a Month
• Proxy Web Caching Servers
• Cache Effectiveness
• Caching for Administrative Control
• Client Web Browser Configuration
• Clustered Proxy Cache Architectures
• Configuring and Monitoring the Netscape 2.5 Proxy Cache
• Observed Results in Practice
• Hardware Configuration Issues for a Proxy Cache
5. Java Application Servers
• Java Overview and Terminology
• What Is Java?
• Distributed Java Applications
• Client/Server Comparisons
• Client/Server Characteristics
• Java Computing Clients
• Dumb Terminal Clients
• Fat Clients: PCs and Workstations
• X-Terminal Clients
• Emulating NC-Based Users
• Java-Based Testing Tool
• NC Emulation Using JavaStar
• Problems and Restrictions
• Emulation System Loads
• Network Load Emulation
• Measuring and Predicting the Server Load with BGS Best/1
• Sizing Tests and Results
• Boot Sequence Analysis
• Booting STorms
• NFS Workstation Browser Startup Comparisons
• HotJava Views
• Simple HTML Web Browsing and Java Applets
• Applix Anywhere
• VT220 Terminal Emulation
• Corel Office for Java
• Custom RDBMS Clients
• Sun Sales Forecasting AApplicaation
• X-Terminal Emulation
• Windows PC Applications
• Overall Sizing Estimates
• Conclusion
6. Source Code Optimization
• Java Tuning
• The Metrognome Class
• Java Tuning and Java Virtual Machines
• GPercollator Performance and Size Issues
• The Java WorkShop Performance Analyzer
• Javaa WorkShop Profiler Display Examples
• When Does "64 Bits" Mean More Performance?
• SPARC and Solaris Versions
• 64-bit Floating-Point Arithmetic
• 64-bit Integer Arithmetic
• The Size of int and long
• Internal Buses or Datapaths
• External Buses and Caches
• 64-bit Addressing
• A Hundred Times Faster? Nothing to Scream About!
• How Does a 32-bit system Handle More Than 4 Gbytes of RAM?
• The Bottom Line
• Linker Options and Tuning
• Static Linking
• Dynamic Linking
• Solaris 1 Compatibility
• Mixed-Mode Linking
• Interposition and Profiling
• Dynamic Performance Improvements in Solaris 2.5
• SunSoft Official Position on Linking Options
• UltraSPARC Compiler Tuning
• An Incremental Plan for UltraSPARC Optimization
• Running the Existing Application on UltraSPARC Hardware
• Comparing Performance
• Processor-Specific Dynamic Libraries
• Migrating Your Application to at Least SPARCompilers 4.2
• Optimizing Code Scheduling for UltraSPARC
• The Implications of Nonportable Optimizations
• Building an UltraSPARC-Only Application
• Numerical Computation Issues
• Building a VIS Instruction Set-Enabled, Device-Specific Module
7. Applications
• Tools for Applications
• Tracing Applications
• Tracing System Calls With truss
• Tracing Shared Library Calls With sotruss
• Profiling Shared Libraries with LD_PROFILE
• Timing
• Process Monitoring Tools
• The Effect of Underlying Filesystem Type
• UNIX File System (UFS)
• UFS with Transaction Logging
• Tmpfs
• The NFS Distributed Computing File System
• Cachefs
• Veritas VxFS File System
• Direct I/O Access
• Customizing the Execution Environment
• File Descriptior Limits
• Databases and Configurable Applications
• Hire an Expert!
• Basic Tuning Ideas
• Increasing Buffer Sizes
• Using Raw Disk Rather Than File Systems
• Fast Raw Backups
• Balance the Load over All the Disks
• Which Disk Partition to Use
• The Effect on Indices
• How to Configure for a Large Number of Users
• Database Tuning Summary
8. Disks
• Disk Workloads
• Sequential versus Random Access
• Disk Reads and Read Caching
• Dirk Writes and the UFS Write Throttle
• Disk Write Caching
• Physical Disk Combinations
• RAID and Disk Arrays
• Disk Configuration Trade-off
• Disk Load Monitoring
• Output Formats and Options for iostat
• Idle Disks and Long Service Times
• The Solaris 2.5 Trace Capability
• I/O Trace: Commands and Features
• How iostat Used the Underlying Disk Measurements
• Filesystem Tuning
• Eagle Disk Pack
• Disk Specifications
• What the Disk Makers Specify
• What the System Vendors Specify
• Whaat You Can Work Out for Yourself
• Zoned Bit Rate (ZBR) Disk Rates
• IPI Disk Controllers
• SCSI Disk Controllers
• SCSI Interface Types
• Tagged Command Queuing Optimizations
• Setting the SCSI Options Variable
• Sun's SCSI Controller Products
• The SPARCstorage Disk Array
• The RSM2000, A1000 and A3000 Hardware RAID Subsystems
• The Enterprise Network Array 5000: A5000 FC-AL Subsystem
• The A7000 High End Disk Subsystem
9. Networks
• New NFS Metrics
• New Network Metrics
• Virtual IP Addresses
• Network Interface Types
• 10-Mbit SBus Interfaces — "le" and "ge"
• 100Mbit Interfaces — "be" and "hme"
• FDDI Interfaces
• SBus ATM 155-Mbit Asynchronous Transfer Mode Cards
• 622-Mbit ATM Interface
• Gigabit Ethernet Interfaces — vge
• Using NFS Effectively
• How Many NFS Server Threads?
• What Is a Typical NFS Operation Mix?
• The nfsstat Command
• NFS Clients
• NFS Server Not Responding
• The netstat Command
10. Processors
• Monitoring Processors
• CPU Control Commands — psrinfo, psradm, and psrset
• The Load Average
• The Data Behind vmstat and mpstat Output
• The Sources of vmstat Data
• Process Queues
• Virtual Memory Counters
• Paging Counters
• CPU Usage and Event Counters
• Use of mpstat to Monitor Interrupts and Mutexes
• Cache Affinity Algorithm
• Unix on Shared Memory Multiprocessors
• The Spin Lock or Mutexes
• Code Locking
• Data Locking and Solaris 2
• Monitoring Solaris 2.6  Lock Statistics
• Multiprocessor Hardware Configurations
• Distributed Memory Multiprocessors
• Shared Memory Multiprocessors
• Shared/Distributed Multiprocessors
• NUMA Multiprocessors
• CPU Caches
• CPU Cache History
• Cache Line and Size Effects
• A Problem with Linked Lists
11. System Architectures
• SPARC Architecture and Implementation
• Register Windows and Different SPARC CPUs
• The Effects of Context Switches and Interrupts
• Identifying Different SPARC CPUs
• SPARC CPU Cache Architectures
• Virtual Write-through Caches
• Virtual Write-back Caches
• Physical Write-back Caches: SuperSPARC and UltraSPARC
• On-Chip Caches
• The SuperSPARC with SuperCache Two-Level Cache Architecture
• UltraSPARC Two-Level Cache Architecture
• I/O Caches
• Block Copy Support
• Software Page Copies
• The Cache-Aligned Block Copy Problem
• Block Copy Acceleration Haardware
• Memory Management Unit Designs
• The Sun-4 MMU: sun4, sun4c, sun4e Kernel Architectures
• Contexts in the Sun-4 MMU
• Monitoring the Sun-4 MMU
• The SPARC Reference MMU: sun4m and sun4d Kernel Architecture
• The SPARC Reference MMU: Table-Walk Operation
• The UltraSPARC MMU Architecture
• Early SPARC System Architectures
• PSARC Uniprocessor VMEbus Systems
• The First Generation of SPARC Uniprocessor Desktop Systems
• Second-Generation Uniprocessor Desktop SPARC Systems
• Third-Generation Uniprocessor Desktop SPARC Systems
• Entry-Level Multiprocessor-Capable Desktop
• Multiprocessor Capable Desktop and Server
• Adding More Processors
• SuperSPARC-Based Multiprocessor Hardware
• Bus Architectures Overview
• MP Cache Issues
• Circuit-switched Bus Protocols
• Packet-switched Bus Protocols
• SuperSPARC XDBus Server Architecture
• SuperSPARC Server Architectural Overview
• SPARCserver 1000 System Implementation Overview
• SPARCserver 2000 System Implementation Overview
• Interrupt Distribution Tuning
• Console Terminal and Clock Interrupt  Issues
• Using prtdiag to Show the Configuration
• UltraSPARC Interconnect Architectures
• The UltraSPARC Port Architecture Switched Interconnect
• The Gigaplane Ultra Enterprise Server Backplane
• The Gigaplane XB Crossbar
• UltraSPARC III Interconnect Considerations
• UltraSPARC System Implementations
• The Ultra 5 and Ultraa 10
• The Ultra 1, Ultra 2 and Ultra 30, Ultra 60
• Larger UPA Based Systems: The Enterprise 450
• The Ultra Enterprise Server Architecture
• The E10000 Starfire Implementation
12. Caches
• Cache Principles
• Cache Reads That Succeed
• Cache Reads That Miss the Cache
• Cache Replacement Policies
• Cache Writes and Purges
• Cache Efficiency
• Generic Cache Behavior Summary
• File Access Caching with Local Disk
• Directory Name Lookup Table
• Inode Cache
• UFS Metadata Buffer Cache
• In-memory Page Cache
• Disk Array Write Cache
• The Standard I/O Buffer
• Read, Write, and Memory Mapping
• Networked File Access
• NFS Access Caching
• Rnode Cache
• In-memory Page Cache on NFS Client
• Disk Array Write Cache on Prestoserve
• NFS V3 and TCP/IP Improvements
• The Cache File System
13. RAM and Virtual Memory
• Memory Usage and Sizing Tools
• Understanding vmstat and sar Output
• Memory Measurements
• Page Fault Measurements
• Page-out Measurements
• Swapping Measurements
• Example vmstat Output Walk-through
• Virtual Memory Address Space Segments
• The Life Cycle of a Typical Physical Memory Page
• Free Memory — The Memory-Go-Round
• The Role of the Free List
• Kernel Memory
• Filesystem Cache
• Process Private Memory
• Free List Performance Problems and Deficit
• System V Shared Memory
• Filesystem Flush
• Scan Rate Treshold Indicating a Memory Shortage
• Kernel Values, Tunables, and Defaults
• Swap Space
• Swap Space Operations
• Swap Space Calculations
• System V Shared Memory, Semaphores, and Message Queues
• Shared Memory Tunables
• Semaphores
• Message Queues
14. Kernel Algorithms and Tuning
• Kernel Tuning
• Documented Configuration Tunables
• Tunable Parameters
• The Ones That Went Away
• Dynamic Kernel Tables in Solaris 2
• SunOS and Solaris Release Overview
• Solaris 2 Performance Tuning Manuals
• Using /etc/system to modify Kernel Variables in Solaris 2
• General Solaris 2 Performance Improvements
• Using Solaris 2 with Large Numbers of Active Users
• The Performance Implications of Patches
• Solaris 2.6 Performance Improvements
• Web Server Performance
• Database Server Peformance
• New and Improved Performance Measurements
• Parameters Derived from maxusers
• Changing maxusers and Pseudo-ttys in Solaris 2
• Autoconfiguration of maxusers in Solaris 2.3 Through Solaris 2.6
• Filesystem Name and Attribute Caching
• Vnodes, Inodes, and Rnodes
• Directory Name Lookup Cache
• The Inode Cache and File Data Caching
• Rnode Cache
• Buffer Cache
• Measuring the Kernel
• Using sar to Examine Table Sizes
• Kernel Memory Allocation
• Kernel Lock Profiling with lockstat in Solaris 2.6
• Kernel Clock Resolution
• Setting Default Limits
• Mapping Device Nicknames to Full Names in Solaris 2
• A Command Script to Do It for You
15. Metric Collection Interfaces
• Standards and Implementations
• The X/Open Universal Measurement Architecture Standard
• The Application Resource Measurement Standard
• Solaris 2 Native Metric Interfaces
• The Traditional Kernel Memory Interface
• Symbols
• Extracting the Values
• Using libkvm
• Procs and U-Areas
• Other Functions
• Caveats
• Summary
• The Solaris 2 "kstat" Interface
• Link Structure
• kstat Datat Types
• The kstat Library
• Reading the kstat Chain
• Writing the kstat Chain
• Caveats
• Summary
• Network Protocol (MIB) Statistics via Streams
• The Streams Mechanism
• Option Management Requests
• MIB Structures in Solaris
• The Network Device Driver Interface
• Process Data Sources
• Where Does the Data Come From?
• Process Information in Solaris 2.6
• Tracing process System Calls
• Process Memory Usage Information
• Who Ran What, When, and How Much Resource Was Used? Accounting!
• Process Data Structures
• The TNF Kernel Trace Probes
• The RPC Remote Statistics Daemon and perfmeter
• System Configuration Information
• System Identify and Basic Configuration
• The Openboot Device Tree — prtconf and prtdiag
• The Symon Configuration Daemon
• The SunVTS Diagnostic Test System
• SCSI Device Information
• Logical Volume Issues
• Solstice Disk Suite Configuration
• Veritas Volume Manager Configuration
• RSM2000 Hardware RAID Configuration
16. The SymbEL Example Tools
• Installing and Using SE3.0
• System Performance Monitoring with SE
• Monitoring Your System with SE
• Long-Term Monitoring with Calendar Manager
• Actions Explained for the Performance Rules
• Disk Rule
• Network Rule
• NFS Rule
• Swap Space Rule
• RAM Rule
• Kernel Memory Rule
• CPU Rule
• Kernel Mutex Rule
• DNLC Rule
• Inode Rule
• TCP/IP Rule
• Look Out, There's a Guru About
• The virtual_adrian.se Performance Tuner And Monitor
• Features of the SE Language
• iostat Written in SE
• Rule Construction
• The CPU Power Rule Explained in Detail
• Example Program Descriptions
• anasa.se
• aw.se
• collisions.se
• cpg.se
• cpu_meter.se
• cpus.se
• df_stats.se
• disks.se
• infotool.se
• iomonitor.se
• iost.se
• iostat.se
• kviewx.se
• kvmname.se
• live_test.se
• mon_cm.se
• monlog.se
• mpvmstat
• msacct.se
• multi_meter.se
• net.se
• net_example
• netmonitor.se
• netstatx.se
• nfsmonitor.se
• nfsstat-m.se
• nproc.se
• nx.se
• pea.se
• percopllator.se and /etc/rc3.d/S99percol
• GPercollator Java Browser
• ps-ax.se
• ps-p.se
• pure_test.se
• pwatch.se
• siostat.se
• syslog.se
• systune.se
• tcp_monitor.se
• uname.se
• uptime.se
• virtual_adrian.se and /etc/rc2.d/S90va_monitor
• virtual_adrian_lite.se
• vmmonitor.se
• vmstat.se
• watch.se
• xio.se
• xiostat.se
• xit.se
• zoom.se
17. The SymbEL Language
• Collection of Performance Data
• Tuning and Performance
• The kvm Library
• The kstat Library
• Management Information Base (MIB)
• The Network Device Driver (NDD) Interface
• Summaty
• Basic Structure
• First Program
• Simple Types
• Simple Program
• Embellishing the Simple Program
• Array Types
• Functions and Parameters
• Global Variables
• Operators and Expressions
• Variable Names
• Constants
• Integer
• Floating Point
• Character
• String
• Declarations
• Arithmetic Operators
• Logical Operators
• Bitwise Operators
• Increment and Decrement Operators
• Compressed Assignment Operators
• Address Operator
• Type Casting
• Control Flow
• Blocks
• If-Then-Else Statement
• Conditional Expression
• switch Statement
• Loops
• Breaak
• Continue
• Goto
• Functions, Procedures, and Programming Notes
• Function Return Types
• Scope
• Initializing Variables
• Notes About Arrays and Strings
• Recursion
• Built-in Functions
• Dynamic Constants
• Attachable Functions
• Built-in Variables
• Parameters to main and Its Return Value
• Structures
• Accessing Structure Members
• Arrays of Structures
• Structure Assignment
• Structure Comparison
• Structure as Parameters
• Structures as Return Values of Functions
• Language Classes
• The kvm Language Class
• The kstat Language Class
• The mib Language Class
• The ndd Language Class
• User-Defined Classes
• Pitfalls
• Tricks
• Returning an Arrays of Nonstructured Type from a Function
• Using Return Values from Attached Functions
• Using kvm Variables and Functions
• Using an attach Block to Call Interpreter Functions

1. Tunables Quick Reference
1. Tunable Kernel Parameters
• System V IPC Tunables
2. References
1. Internet Resources
• Performance Information
• Java Performance Information
• Freely Available Performance Tools
• Commercial Performance Tool Vendor Web Sites
• Sun Products
2. Document Descriptions

It's an OK book on tuning Sun/Solaris systems. Soe of it may be reused, as it is pretty generic, but the majority is nerdy Sun stuff, that could explain the fall of the Sun Empire.