python: simple http server for file sharing

May 15th, 2009 mysurface

You do not need to setup apache server just to hosting your file for sharing. If you have python 2.5 and above, you can do this at the targeted directory to share.

python -m SimpleHTTPServer 9914

9914 is the port number I choose to host my web, assume my IP is 192.168.1.1, in firefox you can ask your friend to access this url: http://192.168.1.1:9914.

It is easy isn’t it? You do NOT even need to run it as root, just run it as normal user but make sure your firewall allow the chosen port number.

Posted in python | Hits: 212877 | 9 Comments »

sometimes, grep alone is enough to do the task

April 14th, 2009 mysurface

One thing that makes UNIXes command line so powerful is because of pipelines. Pipelines is a set of processes chained by their standard streams, so that the output of each process (stdout) feeds directly as input (stdin) of the next one. Simple words, pipelines pass the outcomes of a particular command as input to another command, and these chained allows you to do a command line combo!

A complex combo, I have constructed one last time in post what is your 10 common linux commands?

history | awk '{CMD[$2]++;count++;}END { for (a in CMD)print CMD[a] " " CMD[a]/count*100 "% " a;}' |  \
grep -v "./" | \
column -c3 -s " " -t | \
sort -nr | nl |  head -n10

But sometimes we tend to misuse them, I found someone likes to do this:

cat foobar.txt | grep "barfoo"

Actually grep alone is enough to do the search, we do not need cat or meow for that. As I show him this:

grep "barfoo" foobar.txt

Another example is list the names of all files under a given directory that match a particular string. Let say I wanna obtain a list of files that have string "foobar".

find /opt/src/kmess -type f | xargs grep -l "foobar"

Hey! find is cool but grep alone is enough to do the job.

grep -lr "foobar" /opt/src/kmess

Try to go through the grep manual, you will find out that grep is more powerful than what you expect.

Posted in grep, Text Manipulation | Hits: 194696 | 2 Comments »

Bit shifting can be done in python just like in c

April 1st, 2009 mysurface

It was amazing to discover that I can do bit shifting in python just like in c, the syntax makes no different at all. Let us look at how easy I can do a bit shifting. I write an example loop by shifting the bit leftwards.

for i in range(0,10):
    print "0x0001 << %d = 0x%04X" % ( i , 0x1 << i)

The result is

0x0001 << 0 = 0x0001
0x0001 << 1 = 0x0002
0x0001 << 2 = 0x0004
0x0001 << 3 = 0x0008
0x0001 << 4 = 0x0010
0x0001 << 5 = 0x0020
0x0001 << 6 = 0x0040
0x0001 << 7 = 0x0080
0x0001 << 8 = 0x0100
0x0001 << 9 = 0x0200

The important part in the example is this :

0x1 << i

Let say I wanna shift the bits of a value rightwards 3 times, I will do this:

value = 102
print value >> 3

The result will be 12.

To know more about what is bit shifting, check out wikipedia.

Have Fun!

Posted in Developer, python | Hits: 200666 | 4 Comments »

How can I avoid running a python script multiple times? Implement file locking.

March 13th, 2009 mysurface

Sometimes we just need a single instance of the python script to run at a time. Meaning, the python script itself should detects whether any instances of himself are still running and act accordingly.

Well, how to do it?

The idea is simple, the first instance of the python script will open a file, and lock it. Therefore when the consequent instances try to lock the file, it will fails.

The code is simple too.

import fcntl
def lockFile(lockfile):
    fp = open(lockfile, 'w')
    try:
        fcntl.lockf(fp, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except IOError:
        return False

    return True

The function will try to lock the file specified by me, if it success, return True, else return False.

From any point of my codes, I’ll do

if not lockFile(".lock.pod"):
        sys.exit(0)

Interesting?

Posted in Developer, python | Hits: 199968 | 7 Comments »

Plot your graphs with command line gnuplot

February 12th, 2009 mysurface

gnuplot is a command line driven graph plotter tools for us to generate graphs. The common graphs that we are looking forward to present the resource performance per seconds, hours, days, weeks or months are usually plot graphs, which it consist of lines and dots. gnuplot allows us to read the data from text files which contains values in tabular format.

For example, I have a module that extract raw values into logs based on the specified interval. The result will looks like this (test.dat):

##date time slot_1  slot_2  slot_3  slot_4  slot_5  slot_6  slot_7  slot_8  slot_9  slot_10
06/02/2009 09:16:49 16  6  15  5  14  4  17  3  14  4
06/02/2009 09:16:54 16  8  13  3  15  4  17  4  15  5
06/02/2009 09:16:59 18  8  11  5  15  6  14  3  16  5
06/02/2009 09:17:04 19  6  12  5  18  7  15  5  18  5
06/02/2009 09:17:09 19  7  14  6  16  6  16  6  18  4
06/02/2009 09:17:14 21  7  17  5  19  7  16  4  20  5
06/02/2009 09:17:19 20  5  19  6  17  5  15  3  17  3
06/02/2009 09:17:24 20  6  21  5  18  4  14  4  18  4
06/02/2009 09:17:29 17  5  19  4  16  4  15  4  17  2

gnuplot allows us to create a template describe how our graph outlook. To generate the graph, we just have to execute the template, just like executing bash script.

This is a sample of a gnuplot template that I use to construct a plot graph, (test1.pg)

#!/usr/bin/gnuplot
reset
set terminal png

set xdata time
set timefmt "%d/%m/%Y %H:%M:%S"
set format x "%H:%M"
set xlabel "time"

set ylabel "total actives"
set yrange [0:31]

set title "M7YC Performance per time"
set key reverse Left outside
set grid

set style data linespoints

plot "test.dat" using 1:3 title "slot 1", \
"" using 1:4 title "slot 2", \
"" using 1:5 title "slot 3", \
"" using 1:6 title "slot 4", \
"" using 1:7 title "slot 5", \
"" using 1:8 title "slot 6", \
"" using 1:9 title "slot 7", \
"" using 1:10 title "slot 8", \
"" using 1:11 title "slot 9", \
"" using 1:12 title "slot 10"
#

To allow the template to be executable, the first line of the template must add the SHABANG ‘#!/usr/bin/gnuplot’, similar to bash script. And also remember to perform chmod to your template.

chmod +x test1.pg

I would like to generate graph that shows the performance value over time, therefore I have to read datetime into my X axis and performance value into my Y axis. First two columns of my data file are datetime, I need to set the xdata as time, the define the time format in the raw data files and at last define the x format to appear in my graph.

set xdata time
set timefmt "%d/%m/%Y %H:%M:%S"
set format x "%H:%M"

I would like to display my keys outside the graphs as well as enable the grid to ease the reading.

set key reverse Left outside
set grid

The plot graph I intended to generate consist of lines and dots.

set style data linespoints

The important part is how to define the source of my raw data, how many items I wanted to plot into the graph, as well as giving each item a title. The plot format may vary based on different graph format, but for my case it is

plot [raw_data] using [x value's column in data file]:[y value's column in data files] title [item's name], ...

The plot portion can be shorten into this:

plot "test.dat" u 1:3 t "slot 1", \
"" u 1:4 t "slot 2", \
"" u 1:5 t "slot 3", \
"" u 1:6 t "slot 4", \
"" u 1:7 t "slot 5", \
"" u 1:8 t "slot 6", \
"" u 1:9 t "slot 7", \
"" u 1:10 t "slot 8", \
"" u 1:11 t "slot 9", \
"" u 1:12 t "slot 10"

Execute the template and redirect the binary stream to construct a png file.

./test1.pg > test1.png

test1.png

gnuplot is far more powerful than what I have illustrate here, you may want to checkout the official website for more info.

P.S. I am using gnuplot v4.2 while writing this.

Posted in Developer, gnuplot | Hits: 240637 | 14 Comments »

Python: Manipulate string or binary bytes with StringIO

November 25th, 2008 mysurface

Sometimes it is not convenient to construct string using equal (=) like this:

str = "Hello, "
...
str = str + "my name is "
...
str = str + Name
print str

In python, we have string stream (StringIO) that will behave like file stream, you can construct your string like this:

str=StringIO()
...
str.write("Hello, ")
...
str.write("my name is ")
...
str.write(Name)
print str.getvalue()

The same way, you can construct your binary bytes with StringIO and write it into file once you are done.

bin=StringIO()
bin.write("/x5F/x5F%c" % 0xFF)
...
file = open ("my.bin","wb")
file.write(bin.getvalue())
file.close()

Posted in Developer, python | Hits: 184498 | 5 Comments »

python: writing binary file

November 20th, 2008 mysurface

Python is the best scripting language which I found it out perform Bash script as well as Lua. I like python’s scripting syntax, which is make sense and very convenient in string manipulations. Its now come to binary format manipulation, how convenient is python?

I have carry out some research on python into writing binary files. Either writing byte by byte manually, construct a binary stream then write it one short, or convert it from hex string, they are easy to write in python.

To open a file for binary writing is easy, it is the same way you do for reading, just change the mode into “wb”.

file = open("test.bin","wb")

But, how to write the binary byte into the file?
You may write it straight away with hex code like this:

file.write("\x5F\x9D\x3E")
file.close()

Now, check it out with hexedit,

hexedit test.bin

You will see this:

00000000   5F 9D 3E                                                                                                _.>
00000020
00000040

Now, open the file to append more bytes:

file = open("test.bin","ab")

What if I want to store by bin value into a stream and write it one short?

s ="\x45\xF3"
s = s + "%c%c" % (0x45,0xF3)
file.write(s)
file.close()

Any convenient ways if I can obtained a hex string, and want to convert it back to binary format?
Yes, you just need to import binascii

import binascii
hs="5B7F888489FEDA"
hb=binascii.a2b_hex(hs)
file.write(hb)
file.close()

Isn’t it easy? I love python. You may want to discover more functions of binascii with ipython.

Posted in python | Hits: 211426 | 31 Comments »

Bash script that process input from pipeline or file redirection

November 15th, 2008 mysurface

I remember I wrote a post regarding how to makes python processing string from the pipeline stream. This time, I find the needs to do it in bash script.

I created a hex string splitter script for my friend who needs to decode the raw data that will be written in hex string such as “5F443D95FEA3D4787AEDC4″. Every time, he split them byte by byte manually into this form (“5F 44 3D 95 FE A3 D4 78 7A ED C4″) for better reading. He was complain that the manual process was always bothering him. I told him that can be done by writting a simple bash script.

My hexsplit.sh will do this:

./hexsplit.sh "5F443D95FEA3D4787AEDC4"

But what if he grep this hex string from other file? Or what if the hex string are stored in a file? I want my bash script to be able to process hex string from pipeline stream as well as file redirection too, like this:

 grep "hex" dummy.log | cut -d"]" -f2 | ./hexsplit.sh

or like this:

 ./hexsplit.sh < dummy.log

How to write this hexsplit.sh?

hexsplit.sh

#!/bin/bash

if [ -e $1 ] ;then read str; else str=$1;fi
len=`expr length $str`
for (( a=0; a<=$len; a=a+2 )); do echo -n ${str:a:2}" "; done
echo ""

The important line is the if statement. If there are no argument specified ( -e $1), I read from the stream, else i take it from $1 ( param 1). It is so simple isn't it?

Have fun!

Posted in Bash, pipeline | Hits: 153875 | 7 Comments »

List command line history with timestamp

October 16th, 2008 mysurface

History is a common command for shell to list out all the executed commands. It is very useful when it comes to investigation on what commands was executed that tear down the server. With the help of last command, you be able to track the login time of particular user as well as the the duration of the time he/she stays login.

last
...
mysurface    tty7         :0               Mon Oct  6 20:07 - down   (00:00)
reboot   system boot  2.6.24.4-64.fc8  Mon Oct  6 20:06          (00:00)
mysurface    pts/8        10.168.28.44     Mon Oct  6 17:42 - down   (01:58)
mysurface    pts/7        :0.0             Mon Oct  6 17:41 - 19:40  (01:59)
mysurface    pts/6        :0.0             Mon Oct  6 17:27 - 19:40  (02:13)
mysurface    pts/5        :0.0             Mon Oct  6 17:27 - 19:40  (02:13)
mysurface    pts/5        :0.0             Mon Oct  6 15:52 - 15:59  (00:07)
...

If the command line history could provides the date time of the commands being executed, that may really narrow down the scope of the user actions that cause the server malfunction. By default, history do not append with timestamp, but it is easy to configure it to display timestamp, you just need to set one environment variable HISTTIMEFORMAT.

HISTTIMEFORMAT takes format string of strftime. Check out the strftime manual to choose and construct the timestamp that suit your taste. My favorite is “%F %T “.

export HISTTIMEFORMAT="%F %T "

Execute history again and you will see the effect on the spot, bare in mind that the timestamp for command lines that executed at previous sessions may not valid, as the time was not tracked.

...
  994  2008-10-16 02:27:40 exit
  995  2008-10-16 01:12:20 iptables -nL
  996  2008-10-16 01:47:46 vi .bash_profile
  997  2008-10-16 01:47:55 history
  998  2008-10-16 01:48:03 . .bash_profile
  999  2008-10-16 01:48:04 history
 1000  2008-10-16 01:48:09 exit
 1001  2008-10-16 02:27:43 history
...

I would suggest you to put the export into ~/.bash_profile as well as /root/.bash_profile. In case you do not have .bash_profile, you can choose to put into ~/.bashrc.

Don’t mess up my servers! Your actions will be track!

Posted in Common, history, last | Hits: 220962 | 22 Comments »

XSLT processor command line

September 7th, 2008 mysurface

XSLT stand for XSL transformation, and XSL is EXtensible Stylesheet Language, it is XML-based Stylesheet Language defined by W3C. XSLT is one of the famous XML technology, XML that uses XSL needs transformation before that make sense. Some application such as firefox browser do support XSLT, with xsl source specified, it will be transform automatically.

If you have XSL, and xml that uses XSL, you can also transform by using a command call xsltproc. xsltproc is a tool from libxslt. The usage is straight forward:

xsltproc mytemp.xsl mytarget.xml

The line above will dump the transformed xml to standard output, which you can redirect it to any files.

With -o, the transformed xml will be dumped into file.

xsltproc mytemp.xsl mytarget.xml -o myresult.xml

p.s. For xsltproc, the targeted xml do not need to specified the xsl inline such as:

<?xml-stylesheet href="mytemp.xsl" type="text/xsl" ?>

To know more about XSLT check out W3C XSLT tutorial, or check out ways of transform XML with XSLT. You can also get some samples from the W3C tutorial to try out xsltproc.

Posted in Misc | Hits: 120080 | 2 Comments »