Interactive System Activity Grapher - Concept and Configuration

Contents

• Summary

• Document history

• Typographic convention

• Data sources

• Internal architecture

• Window parts

• Internals

• Graphs

• Known Problems

• Usage

• Program Roadmap

Summary

ISAG is tool for graphical presentation of performance data which is collected using sar/sadc utilities.

Document history

Typographic convention

Data sources

Data sources are datafiles created by sar command in /var/log/sa directory. These files is examined using sar -f command and then is parsed into graphical representation. Default permissions for this directory are only for root's access.

Access to stored data files

If we need access to performance data, on their default location, for another users we have a problem. Good way is to make new user's group and add users, who needs this access, here.

How to do it:

The system have a isag user's group, and this this group has read access to this directory.

Default settings:

drwx------    2 root     root         1024 Nov 22 18:45 /var/log/sa

Needed permissions:

drwxr-x---    2 root     isag         1024 Dec 2  18:45 /var/log/sa

Users, who needs access to sar performance data, must be a members of isag group.

part of /etc/group file

dba:x:503:oracle,alice
isag:x:504:bob,charlie

And now bob and charlie has access to collected data.

Archiving old data files

If we have a old (for example) /var/log/sa/sa23 file, which was created in 23-rd of September, and now is 23-rd of October the new collected data will be appended to existent file. The result is: the file contains two pieces of unrelated data, and isag can make unexpected result/graph.

The best way is archiving performance data for some period, and rename data files to new name. This name can contain date and machine name. For example:

sa02.Dec.lenoch              sa10.Dec.lenoch  sa18.Dec.lenoch
sa04.Dec.brooks.seringas.fr  sa11.Dec.lenoch  sa19.Dec.lenoch
sa05.Dec.lenoch              sa12.Dec.lenoch  sa20.Dec.brooks.seringas.fr
sa07.Dec.lenoch              sa13.Dec.lenoch  sa20.Dec.lenoch
sa08.Dec.lenoch              sa16.Dec.lenoch  sa22.Dec.lenoch
sa09.Dec.lenoch              sa17.Dec.lenoch  sa23.Dec.lenoch

Sample of archiving script

Note: This is buggy. Consider what happens if subdirectory is contained there.

#!/bin/sh

CURRENT="sa`date +%d`"
SA_PATH=/var/log/sa
ARCH_PATH=/home/charlie/source/isag/sa
NEW_UID=isag
NEW_GID=isag

for F in `ls $SA_PATH` ; do
        if [ ! "$CURRENT" = "$F" ] ; then
                N="$F.`ls -l $SA_PATH/$F | awk '{print $6}'``date +%Y`.`uname -n`"
                mv $SA_PATH/$F $ARCH_PATH/$N
                chown $NEW_UID:$NEW_GID $ARCH_PATH/$N
        fi
done

Internal Architecture

The architecture is prepared for various data sources and various outputs.

Window Parts

Screen Shot

Note: Next two pictures aren't actual, this is only for illustration not as full functionality description.

Menu

This is a main program menu, View pull down menu contains all supported graphs.

Data (Source) Chooser

Data chooser contains a menu, where is list of all data files.

Chart

Chart is biggest part of program's window. Here is a graph from selected data source.

Scale

Scale is used to limit of maximal y-value in selected graph. This helps to eliminate local maximum.

Note: Graph is redrawn after moving mouse pointer outside scale. On-line redrawing is CPU hungry process.

Widgets hierarchy

Internals

How data are processed and how graph is created. This chapter describes program's internal architecture. Program is designed as fully parametrized. Design avoids code duplicity as much as possible: "generic functions". Some most specific parts are different from generic parts. And all parameters are program variables. This looks as best way to future extensibility, without rewriting/redesigning program, and variability, for example using for different data sources such as sar on other systems.

How are parameters organized

The parameters are arrays indexed by sar_switch (this term is often used in next text). For example: sar -u shows CPU utilization like this

Linux 2.2.16-22 (lenoch)        12/28/2000

12:09:23 PM       CPU     %user     %nice   %system     %idle
12:09:33 PM       all      3.20      0.00      1.20     95.60
12:09:43 PM       all      6.60      0.20      5.00     88.20
12:09:53 PM       all     45.80      0.00     33.20     21.00
12:10:03 PM       all     23.20      0.00     10.20     66.60
12:10:13 PM       all     92.20      0.00      7.80      0.00
Average:          all     35.44      0.02     10.92     53.62

and all array variables with index u are related to CPU utilization. These parameters (indexed variables) contains for example data processing script (sag_if_cmd(_)), Graph labels (ylabel(_), xlabel(_)) or current limits for a graph (val_max(_)).

How program works

I think that this is best way how I can describe program architecture.

First step: Data file must be selected. This is done through Data Chooser as is shown on screen shot. If data file is correctly choose, then program variable sag_if_archive is set to its name. On program beginning/start this variable contains empty string.

When data file is chosen then set_file procedure is called. This procedure has one parameter, data_file_name. The global variable sag_if_archive is set to this name. Next are refreshed sag_if_cmd array, which contains "data transformation script".

Note: sag_if_cmd contains data file names and must be changed after every data source change!

After this change graph is redrawn using new data file. But only in case if some graph was drawn for previous data file. When no graph or no file was chosen before the Chart part remains empty.

Next step: If view or graph is selected using menu or keyboard shortcut then new_chart() procedure is called. Parameter is a related sar_switch for selected graph.

sar sets a actual scale limits and last_graph is set to appropriate value. Next is called generic function draw() with sar_switch as parameter. The draw() function contains common actions for all graphs:

• Temporary data file, which contains textual data for graphing

• Creates temporary data transform script.

• Processes each output line from data transform script, using generic or specific function (sag_if_proc_line_xx()).

• Creates parameter file for gnuplot.

• Creates a graph using gnuplot.

• Removes temporary files.

Line processing: Because lines has a similar formats, then most graph data are created using generic function sag_if_proc_line_generic. Parameters are:

• l (sar_switch), which describes graph.

• fp (temporary data file descriptor).

• Line (from data transformation script).

During line processing are checked maximal graph values, data time stamp is converted from 12 hours to 24 hours (gnuplot can't care this kind of time representation) if needed, and selected columns are written into temporary data file. We can write down all data columns but it contains redundant information.

Special cases of line processing

If some metrics (or sar switch) requres special processing/care then:

• The sag_if_out_tokens($l) array variable must be empty

• The sag_if_proc_line_$l () function must be defined

• The sag_if__output_zero_metrics_$l () function must be defined

Actually only CPU has specific line processing, because this have instance name in data line. In future network activity will have specific function, there are also instance name (interface name).

List of program variables

Graphs

This chapter describes a graphs which are generated from sar data files. The screen shots are here. Each section contains sample from sar program related to switch and used data are in different colors.

I/O transfer rate

• screen shot

sar -b

01:02:28 PM tps rtps wtps bread/s bwrtn/s

...

Average: 57.77 51.64 6.13 119.17 32.71

Paging statistics

• screen shot

sar -B

01:02:28 PM pgpgin/s pgpgout/s

...

Average: 1676.61 24.78

Process creation

• screen shot

sar -c

01:02:28 PM proc/s

...

Average: 0.06

Memory and Swap usage

• screen shot

sar -r

01:02:28 PM kbmemfree kbmemused %memused kbmemshrd kbbuffers kbcached kbswpfree kbswpused %swpused

...

Average: 2904 141140 97.98 43890 14496 32080 526648 9600 1.79

Memory activities

This part needed a negative value feature. Metrics can be theoretically symeric.

• screen shot

sar -R

08:44:31 PM frmpg/s shmpg/s bufpg/s campg/s

08:44:35 PM -1.00 3.75 0.00 0.00

...

Average: 2.37 1.43 0.00 0.00

CPU utilization

• screen shot

sar -u

01:02:28 PM CPU %user %nice %system %idle

...

Average: all 31.61 0.03 20.73 47.63

inode status

• screen shot

sar -v

01:02:28 PM dentunusd file-sz %file-sz inode-sz %inode-sz super-sz %super-sz dquot-sz %dquot-sz rtsig-sz %rtsig-sz

...

Average: 16252 60 1.46 16362 99.86 10 3.90 25 2.44 0 0.00

System Switching

• screen shot

sar -w

01:02:28 PM cswch/s

...

Average: 352.83

System Swapping

• screen shot

sar -W

01:02:28 PM pswpin/s pswpout/s

...

Average: 1.85 6.05

Known Problems

• The graph aren't showed in correct size after window resizing.

• Locale is set on beginning of program to C because of wish and gnuplot. They have a problem with different number format in some locales like French or Czech (It can be a European problem).

Usage

isag [-p datafiles_path] [-c config_file] [-ght gr_height] [-gwd gr_width]
datafile path: path where are stored collected sar data files
config file: file where are stored some parameters, such as partial graphs scale, last showed graph and more (this depends on the version and list of values can be changed during development)
gr_height: Width of chart area, not whole window
gr_width: Height of chart area

Program Roadmap

Features and problems which will be solved in near future

• Network reports (switch -n DEV and -n SOCK)

• Memory statistics (switch -R), there is a problem with negative values. It requires redesigning (done 22.2.2001).

• Some features doesn't works well

Next part can be also named: Future Directions..

Planned features are.

• Choice which metric from one group will be showed. For example: Memory and Swap chart has these metrics:

• mem free

• mem used

• mem shrd

• buffers

• cached

• swp free

• swp used

And few metrics are not useful, such as memory and swap free.

The I/O transfer rate chart is currently without the tps metric, because this is a redundant information.

• Modification for another data sources, for instance textual outputs from iostat and others sources as sar on other unixes.

• 3D graph for historical data evaluation.

• The HP-UX textually archived data is processed

Small description: This is a system in pre-production phase. 16 Jan 2001 was working day, and peek before 3:00 o'clock is regular backup.

Last picture note: Sampling interval between 8:00 and 19:00 is different from other time and from weekend.

Porting notes

Generally: isag may vary in:

• List of sar switches and related commands
• Output data format and its processing
• Names of graphs, lines and labels

Final note: This list of planned features are not finished. Each useful idea can be added here. Please use the bug report form.

DD (C) 2000, 2001, 2002, 2003. Updated: 26 Jan 2003