Live Chat!

Plot your graphs with command line gnuplot

5 votos Vota!!

February 12th, 2009 mysurface

gnuplot is a command line driven graph plotter tools for us to generate graphs. The common graphs that we are looking forward to present the resource performance per seconds, hours, days, weeks or months are usually plot graphs, which it consist of lines and dots. gnuplot allows us to read the data from text files which contains values in tabular format.

For example, I have a module that extract raw values into logs based on the specified interval. The result will looks like this (test.dat):


##date time slot_1  slot_2  slot_3  slot_4  slot_5  slot_6  slot_7  slot_8  slot_9  slot_10
06/02/2009 09:16:49 16  6  15  5  14  4  17  3  14  4
06/02/2009 09:16:54 16  8  13  3  15  4  17  4  15  5
06/02/2009 09:16:59 18  8  11  5  15  6  14  3  16  5
06/02/2009 09:17:04 19  6  12  5  18  7  15  5  18  5
06/02/2009 09:17:09 19  7  14  6  16  6  16  6  18  4
06/02/2009 09:17:14 21  7  17  5  19  7  16  4  20  5
06/02/2009 09:17:19 20  5  19  6  17  5  15  3  17  3
06/02/2009 09:17:24 20  6  21  5  18  4  14  4  18  4
06/02/2009 09:17:29 17  5  19  4  16  4  15  4  17  2

gnuplot allows us to create a template describe how our graph outlook. To generate the graph, we just have to execute the template, just like executing bash script.

This is a sample of a gnuplot template that I use to construct a plot graph, (test1.pg)


#!/usr/bin/gnuplot
reset
set terminal png

set xdata time
set timefmt "%d/%m/%Y %H:%M:%S"
set format x "%H:%M"
set xlabel "time"

set ylabel "total actives"
set yrange [0:31]

set title "M7YC Performance per time"
set key reverse Left outside
set grid

set style data linespoints

plot "test.dat" using 1:3 title "slot 1", \
"" using 1:4 title "slot 2", \
"" using 1:5 title "slot 3", \
"" using 1:6 title "slot 4", \
"" using 1:7 title "slot 5", \
"" using 1:8 title "slot 6", \
"" using 1:9 title "slot 7", \
"" using 1:10 title "slot 8", \
"" using 1:11 title "slot 9", \
"" using 1:12 title "slot 10"
#

To allow the template to be executable, the first line of the template must add the SHABANG ‘#!/usr/bin/gnuplot’, similar to bash script. And also remember to perform chmod to your template.

chmod +x test1.pg

I would like to generate graph that shows the performance value over time, therefore I have to read datetime into my X axis and performance value into my Y axis. First two columns of my data file are datetime, I need to set the xdata as time, the define the time format in the raw data files and at last define the x format to appear in my graph.


set xdata time
set timefmt "%d/%m/%Y %H:%M:%S"
set format x "%H:%M"

I would like to display my keys outside the graphs as well as enable the grid to ease the reading.


set key reverse Left outside
set grid

The plot graph I intended to generate consist of lines and dots.

set style data linespoints

The important part is how to define the source of my raw data, how many items I wanted to plot into the graph, as well as giving each item a title. The plot format may vary based on different graph format, but for my case it is


plot [raw_data] using [x value's column in data file]:[y value's column in data files] title [item's name], ...

The plot portion can be shorten into this:


plot "test.dat" u 1:3 t "slot 1", \
"" u 1:4 t "slot 2", \
"" u 1:5 t "slot 3", \
"" u 1:6 t "slot 4", \
"" u 1:7 t "slot 5", \
"" u 1:8 t "slot 6", \
"" u 1:9 t "slot 7", \
"" u 1:10 t "slot 8", \
"" u 1:11 t "slot 9", \
"" u 1:12 t "slot 10"

Execute the template and redirect the binary stream to construct a png file.

./test1.pg > test1.png

test1.png

gnuplot is far more powerful than what I have illustrate here, you may want to checkout the official website for more info.

P.S. I am using gnuplot v4.2 while writing this.

Posted in Developer, gnuplot | Hits: 144321 | 5 Comments »

Python: Manipulate string or binary bytes with StringIO

2 votos Vota!!

November 25th, 2008 mysurface

Sometimes it is not convenient to construct string using equal (=) like this:

str = "Hello, "
...
str = str + "my name is "
...
str = str + Name
print str

In python, we have string stream (StringIO) that will behave like file stream, you can construct your string like this:

str=StringIO()
...
str.write("Hello, ")
...
str.write("my name is ")
...
str.write(Name)
print str.getvalue()

The same way, you can construct your binary bytes with StringIO and write it into file once you are done.


bin=StringIO()
bin.write("/x5F/x5F%c" % 0xFF)
...
file = open ("my.bin","wb")
file.write(bin.getvalue())
file.close()

Posted in Developer, python | Hits: 145176 | 12 Comments »

python: writing binary file

2 votos Vota!!

November 20th, 2008 mysurface

Python is the best scripting language which I found it out perform Bash script as well as Lua. I like python’s scripting syntax, which is make sense and very convenient in string manipulations. Its now come to binary format manipulation, how convenient is python?

I have carry out some research on python into writing binary files. Either writing byte by byte manually, construct a binary stream then write it one short, or convert it from hex string, they are easy to write in python.

To open a file for binary writing is easy, it is the same way you do for reading, just change the mode into “wb”.

file = open("test.bin","wb")

But, how to write the binary byte into the file?
You may write it straight away with hex code like this:

file.write("\x5F\x9D\x3E")
file.close()

Now, check it out with hexedit,

hexedit test.bin

You will see this:

00000000   5F 9D 3E                                                                                                _.>
00000020
00000040

Now, open the file to append more bytes:

file = open("test.bin","ab")

What if I want to store by bin value into a stream and write it one short?

s ="\x45\xF3"
s = s + "%c%c" % (0x45,0xF3)
file.write(s)
file.close()

Any convenient ways if I can obtained a hex string, and want to convert it back to binary format?
Yes, you just need to import binascii

import binascii
hs="5B7F888489FEDA"
hb=binascii.a2b_hex(hs)
file.write(hb)
file.close()

Isn’t it easy? I love python. You may want to discover more functions of binascii with ipython.

Posted in python | Hits: 112486 | 4 Comments »

Bash script that process input from pipeline or file redirection

2 votos Vota!!

November 15th, 2008 mysurface

I remember I wrote a post regarding how to makes python processing string from the pipeline stream. This time, I find the needs to do it in bash script.

I created a hex string splitter script for my friend who needs to decode the raw data that will be written in hex string such as “5F443D95FEA3D4787AEDC4″. Every time, he split them byte by byte manually into this form (“5F 44 3D 95 FE A3 D4 78 7A ED C4″) for better reading. He was complain that the manual process was always bothering him. I told him that can be done by writting a simple bash script.

My hexsplit.sh will do this:

./hexsplit.sh "5F443D95FEA3D4787AEDC4"

But what if he grep this hex string from other file? Or what if the hex string are stored in a file? I want my bash script to be able to process hex string from pipeline stream as well as file redirection too, like this:

 grep "hex" dummy.log | cut -d"]" -f2 | ./hexsplit.sh

or like this:

 ./hexsplit.sh < dummy.log

How to write this hexsplit.sh?

hexsplit.sh


#!/bin/bash

if [ -e $1 ] ;then read str; else str=$1;fi
len=`expr length $str`
for (( a=0; a<=$len; a=a+2 )); do echo -n ${str:a:2}" "; done
echo ""

The important line is the if statement. If there are no argument specified ( -e $1), I read from the stream, else i take it from $1 ( param 1). It is so simple isn’t it?

Have fun!

Posted in Bash, pipeline | Hits: 107224 | 3 Comments »

List command line history with timestamp

4 votos Vota!!

October 16th, 2008 mysurface

History is a common command for shell to list out all the executed commands. It is very useful when it comes to investigation on what commands was executed that tear down the server. With the help of last command, you be able to track the login time of particular user as well as the the duration of the time he/she stays login.

last
...
mysurface    tty7         :0               Mon Oct  6 20:07 - down   (00:00)
reboot   system boot  2.6.24.4-64.fc8  Mon Oct  6 20:06          (00:00)
mysurface    pts/8        10.168.28.44     Mon Oct  6 17:42 - down   (01:58)
mysurface    pts/7        :0.0             Mon Oct  6 17:41 - 19:40  (01:59)
mysurface    pts/6        :0.0             Mon Oct  6 17:27 - 19:40  (02:13)
mysurface    pts/5        :0.0             Mon Oct  6 17:27 - 19:40  (02:13)
mysurface    pts/5        :0.0             Mon Oct  6 15:52 - 15:59  (00:07)
...

If the command line history could provides the date time of the commands being executed, that may really narrow down the scope of the user actions that cause the server malfunction. By default, history do not append with timestamp, but it is easy to configure it to display timestamp, you just need to set one environment variable HISTTIMEFORMAT.

HISTTIMEFORMAT takes format string of strftime. Check out the strftime manual to choose and construct the timestamp that suit your taste. My favorite is “%F %T “.

export HISTTIMEFORMAT="%F %T "

Execute history again and you will see the effect on the spot, bare in mind that the timestamp for command lines that executed at previous sessions may not valid, as the time was not tracked.

...
  994  2008-10-16 02:27:40 exit
  995  2008-10-16 01:12:20 iptables -nL
  996  2008-10-16 01:47:46 vi .bash_profile
  997  2008-10-16 01:47:55 history
  998  2008-10-16 01:48:03 . .bash_profile
  999  2008-10-16 01:48:04 history
 1000  2008-10-16 01:48:09 exit
 1001  2008-10-16 02:27:43 history
...

I would suggest you to put the export into ~/.bash_profile as well as /root/.bash_profile. In case you do not have .bash_profile, you can choose to put into ~/.bashrc.

Don’t mess up my servers! Your actions will be track!

Posted in Common, history, last | Hits: 114043 | 10 Comments »

XSLT processor command line

3 votos Vota!!

September 7th, 2008 mysurface

XSLT stand for XSL transformation, and XSL is EXtensible Stylesheet Language, it is XML-based Stylesheet Language defined by W3C. XSLT is one of the famous XML technology, XML that uses XSL needs transformation before that make sense. Some application such as firefox browser do support XSLT, with xsl source specified, it will be transform automatically.

If you have XSL, and xml that uses XSL, you can also transform by using a command call xsltproc. xsltproc is a tool from libxslt. The usage is straight forward:

xsltproc mytemp.xsl mytarget.xml

The line above will dump the transformed xml to standard output, which you can redirect it to any files.

With -o, the transformed xml will be dumped into file.

xsltproc mytemp.xsl mytarget.xml -o myresult.xml

p.s. For xsltproc, the targeted xml do not need to specified the xsl inline such as:

<?xml-stylesheet href="mytemp.xsl" type="text/xsl" ?>

To know more about XSLT check out W3C XSLT tutorial, or check out ways of transform XML with XSLT. You can also get some samples from the W3C tutorial to try out xsltproc.

Posted in Misc | Hits: 79027 | No Comments »

KDE based Fluxbox open folder tips

2 votos Vota!!

August 30th, 2008 mysurface

I am a KDE based Fluxbox user, I am face an issue that some of my gnome applications that have feature of allowing me to locate and open folder doesn’t really work as expected. Those gnome applications are gnome-do and picasa. When I open a folder from gnome-do or picasa, it will be open in firefox. Instead, I want it to open at konqueror.

I didn’t find any solution for that, due to the fact there are not much KDE based Fluxbox users around. But, at last I figure out a work around.

I realized that those gnome applications will call xdg-open when it is asked to open the folders. xdg-open is a bash script that will checks for global environment variable $DE and decide what file browser to execute. I have extracted the important part for reference:


if [ x"$DE" = x"" ]; then
    # if BROWSER variable is not set, check some well known browsers instead
    if [ x"$BROWSER" = x"" ]; then
        BROWSER=htmlview:firefox:mozilla:netscape:links:lynx
    fi
    DE=generic
fi

case "$DE" in
    kde)
    open_kde "$url"
    ;;

    gnome)
    open_gnome "$url"
    ;;

    xfce)
    open_xfce "$url"
    ;;

    generic)
    open_generic "$url"
    ;;

    *)
    exit_failure_operation_impossible "no method available for opening '$url'"
    ;;
esac

The workaround is assign $DE with value “kde”, edit your ~/.bash_profile or ~/.bashrc with:

export DE=kde

Relogin again and it is done.

Posted in fluxbox, picasa | Hits: 72787 | No Comments »

Experiencing with vmware server and Sun xVM VirtualBox

1 votos Vota!!

August 17th, 2008 mysurface

Recently I have tried Sun xVM VirtualBox 1.6.4 , I have compare vbox with Vmware server 1.0.4. After couple days of testing, its time for me to share some personal findings towards them. I may not able to provide graphs, and accurate figures to shows their performance and specification, but what I do is to only share my point of view as a layman, and how I like or dislike them based on the user experience.

First of all, let me list down my host machine specs:
Dell Vostro 1400 with Intel Core 2 Duo CPU T5270 @ 1.40GHz, 2G Ram.
I use 512 Mb Ram to load my guest os, I tried loading up myrinix live CD, as well as installed Windows XP.
Uses Fedora 8, my WM uses Fluxbox 1.0.0.

Before looking into performance and features, let us first look at the screenshot! I am loading up myrinix live cd at both virtual tool. The reason I choose myrinix because this great Live CD pre install with vbox guest additions as well as vmware tools, so that I can enjoy the mouse integration to move my mouse IN and OUT the guest os.

Virtual Box have separate window for virtual os settings and guest OS itself.

Vmware Server makes every Guest OSes in MDI, separate each guest oses into tabs, and the guest os settings in sidebar, NEAT!

Read the rest of this entry »

Posted in Misc, VirtualBox, vmware | Hits: 73503 | 5 Comments »

Virtualize your operating system with qemu

2 votos Vota!!

August 2nd, 2008 mysurface

QEMU is a processor emulator, it allows you to run variety of unmodified guest operating systems such as Linux, Windows, Solaris, Dos etc just like Vmware and VirtualBox. In fact VirtualBox dynamic recompiler are based on QEMU. QEMU supports emulating several hardware platforms, including x86, AMD64, Alpha, MIPS, ARM, PPC, SPARC etc.

QEMU comes with command line interface, where you pass your configurations by parameters, but there are 3rd party GUI front end, such as Qemulator and Qemu-Launcher. To improves the performance of QEMU, we needs either KQEMU kernel module or KVM. Both of them are accelerator that to increase the execution speed of QEMU.

In this post, I covers installation of QEMU as well as KQEMU. As well as simple example of the use of QEMU command lines, how to create image files that allows to execute by both QEMU as well as Vmware.

Read the rest of this entry »

Posted in qemu, qemu-img, vmware | Hits: 76188 | 6 Comments »

Syslog: Sending log from remote servers to syslog daemon

2 votos Vota!!

July 23rd, 2008 mysurface

syslog is a standard for logging service in Linux, it usually run as daemon like syslogd or rsyslogd. Syslog daemon will be forward and store logs in /var/log directory, you may configure it to store at separate location if you want. (we will look into it later). And there is a major file that store majority of logs, which is messages. Therefore, you may want to monitor linux messages logs by tailing /var/log/message.

Logs entry may come from various services, applications, kernel as well as remote servers if you enabled your syslog daemon to accept remote logs submissions. Syslog protocol is now standardized within the Syslog working group of the IETF, and it is been defines in RFC 3164.

Rsyslog is an enhanced multi-threaded syslogd with a focus on security and reliability. I think newer linux distro already replace syslogd with rsyslogd. For more information you can check out the wikipedia.

In this post, I briefly explain the facility and log levels of syslog protocol, how to configure syslogd as well as rsyslogd to accept logs from remote and also how to send logs remotely.
Read the rest of this entry »

Posted in Admin, logger, nc, rsyslogd, syslogd, tail | Hits: 89247 | No Comments »