str = "Hello, "
...
str = str + "my name is "
...
str = str + Name
print str

In python, we have string stream (StringIO) that will behave like file stream, you can construct your string like this:

str=StringIO()
...
str.write("Hello, ")
...
str.write("my name is ")
...
str.write(Name)
print str.getvalue()

The same way, you can construct your binary bytes with StringIO and write it into file once you are done.

bin=StringIO()
bin.write("/x5F/x5F%c" % 0xFF)
...
file = open ("my.bin","wb")
file.write(bin.getvalue())
file.close()

I have carry out some research on python into writing binary files. Either writing byte by byte manually, construct a binary stream then write it one short, or convert it from hex string, they are easy to write in python.

To open a file for binary writing is easy, it is the same way you do for reading, just change the mode into “wb”.

file = open("test.bin","wb")

But, how to write the binary byte into the file?
You may write it straight away with hex code like this:

file.write("\x5F\x9D\x3E")
file.close()

Now, check it out with hexedit,

hexedit test.bin

You will see this:

00000000   5F 9D 3E                                                                                                _.>
00000020
00000040

Now, open the file to append more bytes:

file = open("test.bin","ab")

What if I want to store by bin value into a stream and write it one short?

s ="\x45\xF3"
s = s + "%c%c" % (0x45,0xF3)
file.write(s)
file.close()

Any convenient ways if I can obtained a hex string, and want to convert it back to binary format?
Yes, you just need to import binascii

import binascii
hs="5B7F888489FEDA"
hb=binascii.a2b_hex(hs)
file.write(hb)
file.close()

Isn’t it easy? I love python. You may want to discover more functions of binascii with ipython.

I created a hex string splitter script for my friend who needs to decode the raw data that will be written in hex string such as “5F443D95FEA3D4787AEDC4″. Every time, he split them byte by byte manually into this form (“5F 44 3D 95 FE A3 D4 78 7A ED C4″) for better reading. He was complain that the manual process was always bothering him. I told him that can be done by writting a simple bash script.

My hexsplit.sh will do this:

./hexsplit.sh "5F443D95FEA3D4787AEDC4"

But what if he grep this hex string from other file? Or what if the hex string are stored in a file? I want my bash script to be able to process hex string from pipeline stream as well as file redirection too, like this:

 grep "hex" dummy.log | cut -d"]" -f2 | ./hexsplit.sh

or like this:

 ./hexsplit.sh < dummy.log

How to write this hexsplit.sh?

hexsplit.sh

#!/bin/bash

if [ -e $1 ] ;then read str; else str=$1;fi
len=`expr length $str`
for (( a=0; a<=$len; a=a+2 )); do echo -n ${str:a:2}" "; done
echo ""

The important line is the if statement. If there are no argument specified ( -e $1), I read from the stream, else i take it from $1 ( param 1). It is so simple isn’t it?

Have fun!

last
...
mysurface    tty7         :0               Mon Oct  6 20:07 - down   (00:00)
reboot   system boot  2.6.24.4-64.fc8  Mon Oct  6 20:06          (00:00)
mysurface    pts/8        10.168.28.44     Mon Oct  6 17:42 - down   (01:58)
mysurface    pts/7        :0.0             Mon Oct  6 17:41 - 19:40  (01:59)
mysurface    pts/6        :0.0             Mon Oct  6 17:27 - 19:40  (02:13)
mysurface    pts/5        :0.0             Mon Oct  6 17:27 - 19:40  (02:13)
mysurface    pts/5        :0.0             Mon Oct  6 15:52 - 15:59  (00:07)
...

If the command line history could provides the date time of the commands being executed, that may really narrow down the scope of the user actions that cause the server malfunction. By default, history do not append with timestamp, but it is easy to configure it to display timestamp, you just need to set one environment variable HISTTIMEFORMAT.

HISTTIMEFORMAT takes format string of strftime. Check out the strftime manual to choose and construct the timestamp that suit your taste. My favorite is “%F %T “.

export HISTTIMEFORMAT="%F %T "

Execute history again and you will see the effect on the spot, bare in mind that the timestamp for command lines that executed at previous sessions may not valid, as the time was not tracked.

...
  994  2008-10-16 02:27:40 exit
  995  2008-10-16 01:12:20 iptables -nL
  996  2008-10-16 01:47:46 vi .bash_profile
  997  2008-10-16 01:47:55 history
  998  2008-10-16 01:48:03 . .bash_profile
  999  2008-10-16 01:48:04 history
 1000  2008-10-16 01:48:09 exit
 1001  2008-10-16 02:27:43 history
...

I would suggest you to put the export into ~/.bash_profile as well as /root/.bash_profile. In case you do not have .bash_profile, you can choose to put into ~/.bashrc.

Don’t mess up my servers! Your actions will be track!

If you have XSL, and xml that uses XSL, you can also transform by using a command call xsltproc. xsltproc is a tool from libxslt. The usage is straight forward:

xsltproc mytemp.xsl mytarget.xml

The line above will dump the transformed xml to standard output, which you can redirect it to any files.

With -o, the transformed xml will be dumped into file.

xsltproc mytemp.xsl mytarget.xml -o myresult.xml

p.s. For xsltproc, the targeted xml do not need to specified the xsl inline such as:

<?xml-stylesheet href="mytemp.xsl" type="text/xsl" ?>

To know more about XSLT check out W3C XSLT tutorial, or check out ways of transform XML with XSLT. You can also get some samples from the W3C tutorial to try out xsltproc.

I didn’t find any solution for that, due to the fact there are not much KDE based Fluxbox users around. But, at last I figure out a work around.

I realized that those gnome applications will call xdg-open when it is asked to open the folders. xdg-open is a bash script that will checks for global environment variable $DE and decide what file browser to execute. I have extracted the important part for reference:

if [ x"$DE" = x"" ]; then
    # if BROWSER variable is not set, check some well known browsers instead
    if [ x"$BROWSER" = x"" ]; then
        BROWSER=htmlview:firefox:mozilla:netscape:links:lynx
    fi
    DE=generic
fi

case "$DE" in
    kde)
    open_kde "$url"
    ;;

    gnome)
    open_gnome "$url"
    ;;

    xfce)
    open_xfce "$url"
    ;;

    generic)
    open_generic "$url"
    ;;

    *)
    exit_failure_operation_impossible "no method available for opening '$url'"
    ;;
esac

The workaround is assign $DE with value “kde”, edit your ~/.bash_profile or ~/.bashrc with:

export DE=kde

Relogin again and it is done.

First of all, let me list down my host machine specs:
Dell Vostro 1400 with Intel Core 2 Duo CPU T5270 @ 1.40GHz, 2G Ram.
I use 512 Mb Ram to load my guest os, I tried loading up myrinix live CD, as well as installed Windows XP.
Uses Fedora 8, my WM uses Fluxbox 1.0.0.

Before looking into performance and features, let us first look at the screenshot! I am loading up myrinix live cd at both virtual tool. The reason I choose myrinix because this great Live CD pre install with vbox guest additions as well as vmware tools, so that I can enjoy the mouse integration to move my mouse IN and OUT the guest os.

Virtual Box have separate window for virtual os settings and guest OS itself.

Vmware Server makes every Guest OSes in MDI, separate each guest oses into tabs, and the guest os settings in sidebar, NEAT!

Performance and Resources Usage
Seems both Vmware Server and Virtual Box loading speed are equally fast, and it speed close to native while I rambling in guest OS. I have such feeling of Vbox start up is a little bit faster than vmware, but uses more CPU resources than vmware. For saving and resuming guest os machine state, vbox really out performs vmware server, vbox takes less than 10 seconds to save a machines state and around 5 seconds to resume them, but vmware takes ages, even greater than shutdown and startup time, CRAZY!

Networking
Vmware support various of networking, and it allows us to switch between them in real time. I can switch NAT and Bridge anytime with loaded guest os. If I want to share internet with my host, I choose NAT. I do this when my host accessing internet through wireless LAN. I choose bridge when I want my guest OS to obtain a different IP from DHCP server. Meaning my host and guest os have separate IP address. With that, my guest OS now can be accessed from the real LAN!

Vbox also support various types of network mode too. By default, vbox uses NAT. Host Interface is equivalent to vmware Bridge mode. In order to allow Host Interface to be working, you need to setup manually, vbox does provides the guidelines for different host such as windows, solaris and Linux. But the steps is quite tedious and don’t bother trying out right now. Not to say you can’t switch network modes in real time, while your guest OS is running, the guest OS settings automatically disabled from configure. You can’t modify the settings even you save the guest os machine state, that is so ANNOYING!

But vbox support four virtual network adapter, if you have properly setup the Host Interface, you can manipulate those virtual Ethernets to bind to different network interface provided that your host machine have multiple network interface too. Vbox also provides different Network Adapter Hardware(Adapter Type) for you to choose. With some specific guest OS, you need to choose the correct Adapter Type, else it won’t works. For example freebsd guest os needs PCnet-PCI II instead of PCnet-PC III. Well, I don’t know what is the different between those options and the purpose of providing those options.

Disk Image format
Vmware supports only vmdk format, and that is its own disk image format. Vbox default disk image format is vdi, but vbox supports vmdk too!

cd-rom support
Both vmware and vbox support mounting and unmounting cd-rom from iso as well as host cd-rom devices. In vbox always remember to unmount first before mounting a new one.

Installing Windows XP
Both vmware and vbox gives a very good support for windows xp as guest os, such as Auto resize guest display ( Auto-fit Guest in vmware). Mouse integration allows moving mouse in and out the guest os. Those features will be activated after installing guest additions into windows xp guest os ( vmware tools for vmware).

Vbox provides Seamless Mode feature, which it blends the host and guest os together. Check out the screenshot!

I am running Firefox 3 from both Linux and Windows XP side by side

While moving my guest os’s apps around my Linux desktop, I experience flickery in display, probably because I run this in Fluxbox, I should try to run this in KDE or gnome.

Vbox support folder sharing between guest os and host os, this is a demanding feature that vmware server doesn’t provide, but I heard this feature is available in vmware workstation.

Some trivial Facts!
I am having dual boot for my laptop, meaning I have limited hardisk space for my ext3 partition, therefore I created vmdk format using qemu-img in my NTFS partition. After that, I load up my vbox and install Windows XP into the vmdk disk image. Next, I create a vmx file and try to load this vmdk at vmware server.
( For more information regarding qemu and vmx, check out this post.)

Ouch! vmware do not able to recognize this vmdk? I copy over the vmdk to ext3 and try again, it works! Meaning vmware for linux do not able to read the image in other filesystem except linux native filesystem? FUNNY!

Vmware virtualize different hardwares compare to vbox as well as qemu. There are risk of crashing windows if changes of hardwares, you will be throw into blue screen of death. But vmware handle it well, although I am installing windows in vbox, the windows is booting up nicely and detects the change of hardwares and depend me to just reactive windows. At this point, I shutdown my windows from vmware and try to boot it up from vbox, it crashed! I am in blue screen of death!

Summary
Both vmware and vbox have its pros and cons, I enjoy both of them.

Vmware Server:
1. Able to switch networks from NAT to Bridge mode in real time.
2. Organize guest oses in tabs and have their settings in sidebar.
3. Do not support File sharing. ( vmware workstation does but not vmware server)
4. Saving and resuming guest os machine state takes longer time than startup and shutdown.
5. Do not able to read disk image from NTFS and NAT filesystem.
6. Able to handle windows without crashing where windows installed using different virtual tool.

Virtual Box:
1. Provide 4 virtual network Adapter, but need manual setup Host Interface.
2. Support both vdi and vmdk format.
3. Seamless Mode for Windows XP guest OS is cool.
4. Support File sharing within guest os and host os.
5. Saving and resuming guest OS very fast.
6. Setting disabled while guest OS is running or saved.

QEMU is a processor emulator, it allows you to run variety of unmodified guest operating systems such as Linux, Windows, Solaris, Dos etc just like Vmware and VirtualBox. In fact VirtualBox dynamic recompiler are based on QEMU. QEMU supports emulating several hardware platforms, including x86, AMD64, Alpha, MIPS, ARM, PPC, SPARC etc.

QEMU comes with command line interface, where you pass your configurations by parameters, but there are 3rd party GUI front end, such as Qemulator and Qemu-Launcher. To improves the performance of QEMU, we needs either KQEMU kernel module or KVM. Both of them are accelerator that to increase the execution speed of QEMU.

In this post, I covers installation of QEMU as well as KQEMU. As well as simple example of the use of QEMU command lines, how to create image files that allows to execute by both QEMU as well as Vmware.

Installation
I believes that you may download QEMU from your repository, which it may already comes with kqemu. You may also manually compile both qemu and kqemu. My Linux distro repository contains older version of qemu, therefore I choose to compile qemu and kqemu myself.

You can download both qemu and kqemu from http://bellard.org/qemu/download.html.

Compilation and Installation for qemu and kqemu are straight forward. You just need to do the steps below:
1. untar the archive.

2. compile

./configure
make

3. install using root permission

make install

kqemu is a kernel object, it may not auto reboot every time you reboot your machines, check out the kqemu installation for Linux HERE.

To manually load kqemu, do this in root:

mknod /dev/kqemu c 250 0
chmod 666 /dev/kqemu
/sbin/modprobe kqemu

Boot up live cd using QEMU
To check out the Live CD iso without have to burn it to CD, i usually use qemu. For this I really appreciate the flexibility of command line interface, without much clicks here and there, I boot up my Live CD with a single line of command.

For this illustration, I choose Myrinix. Myrinix is a multi purpose Portable Linux based on Debian SID. it can be in Live CD format ,USB drive as well as install in windows xp ntfs partition without modify the Master Boot Record (MBR). Myrinix also supply a series of application module that can be easily added later. You may download the latest Myrinix Live CD iso from HERE.

Lets boot up the Live CD ISO now!

qemu -cdrom myrinix-kde-200808.iso -m 512 -boot d

Short and simple! Specified your iso file path at -cdrom, in case you want to load up the Live CD from the physical CD, you may change to -cdrom /dev/cdrom ( depends on your cdrom dev). -m 512 is to force qemu to use 512Mb on RAM, by default qemu uses only 128Mb. At last -boot d option indicates that I want qemu to boot up from cdrom.

Verify kqemu is loaded
During loading the Live CD, you can verify whether kqemu is it loaded and in use. Press Ctrl+Alt 2, and type:

info kqemu
kqemu support: enabled for user mode

As long as I see the positive message, kqemu is successfully loaded and in used. Press Ctrl+Alt 1 to goes back the emulator screen.

Connects to the Internet in guest OS
By default, qemu will act as dhcp server and support internet sharing from your Host machine. Therefore as long as my host have internet access, my guest OS in qemu may have internet access too.

Myrinix does not acquire IP from DHCP by default, I think a lots of Linux Live CD doesn’t do that too. Therefore I have to do it manually. Login to root and run dhclient on eth0.

su -
dhclient eth0

Connects to Host Machines
I can obtain virtual IP for internet sharing from 10.0.2.2 by default, the same way I may also connect to my host machine by this IP.

ssh mysurface@10.0.2.2

If my guest Linux have KDE installed, I may use fish in Konqueror for file transferring from my host machine to my current guest Linux. Check out how to do that at HERE.

Qemu provides more options, just type qemu, you will get a full list of options it support, check out the user document for more details information.

Install a guest OS
To install a guest OS using qemu, first thing I need to do is to create an image file using qemu-img. qemu-img support various of file format including vmdk. qemu support loading vmdk image files. Vmware image file is in vmdk format. Bare in mind, load up vmdk files created by vmware at YOUR OWN RISK. (I have a bad experience on loading up Red Hat ES4 vmdk created by vmware server, which it fails to boot up X after booting it up by qemu. My guess is both qemu and vmware emulates different set of hardware drivers. Well, at last I manage to fix it by reinstall vmware tools in Red Hat ES4.)

To create a 10G disk image in vmdk format using qemu-img like this: (Don’t worry, 10G disk image do not actually eats 10G of your physical Hard disk space)

qemu-img create -f vmdk myos.vmdk 10G

Now I may load up your installer CD with disk image with qemu like this:

qemu -hda myos.vmdk -cdrom myrinix-kde-200808.iso -m 512 -boot d

After I have done the installation, I will boot up my guest OS from disk image like this:
( You may want to point your cdrom to physical cdrom as well as enable USB.)

qemu -hda myos.vmdk -cdrom /dev/cdrom -m 512 -usb -boot c

qemu-img allows me to convert disk image too, in case I have downloaded image disk from somewhere, I can convert it to vmdk like this:

qemu-img convert -f qcow freebsd.qcow -O freebsd.vmdk

To load up image disk with vmware, I need another file, with extension of .vmx. Vmx file is a text file that contains configuration for your disk image. To generate one using http://www.easyvmx.com/. Remember to point your disk image to ide0:0.filename in vmx, for example my case:

ide0:0.fileName = "freebsd.vmdk"

Performance
It seems even with kqemu, qemu performance in terms of execution speed is still slower than vmware. I compare the same disk image in both vmware server as well as qemu. It will be obvious when I access the GUI menu in my guest OS.

There are still a lots of options and tips I fails to covers here, in case you have any tips, please feel no hesitate to leave me comments.

Logs entry may come from various services, applications, kernel as well as remote servers if you enabled your syslog daemon to accept remote logs submissions. Syslog protocol is now standardized within the Syslog working group of the IETF, and it is been defines in RFC 3164.

Rsyslog is an enhanced multi-threaded syslogd with a focus on security and reliability. I think newer linux distro already replace syslogd with rsyslogd. For more information you can check out the wikipedia.

In this post, I briefly explain the facility and log levels of syslog protocol, how to configure syslogd as well as rsyslogd to accept logs from remote and also how to send logs remotely.

Syslog categories logs into PRI which constructed by facility and severity/priority/log levels.

Facility defines the source of the log entries, what kind of services that send this logs. Lets look at Facility info that extracted from RFC 3164, each facility was been assign a numeric code.

    Numerical       Facility
          Code
           0             kernel messages
           1             user-level messages
           2             mail system
           3             system daemons
           4             security/authorization messages
           5             messages generated internally by syslogd
           6             line printer subsystem
           7             network news subsystem
           8             UUCP subsystem
           9             clock daemon
          10             security/authorization messages
          11             FTP daemon
          12             NTP subsystem
          13             log audit
          14             log alert
          15             clock daemon
          16             local use 0  (local0)
          17             local use 1  (local1)
          18             local use 2  (local2)
          19             local use 3  (local3)
          20             local use 4  (local4)
          21             local use 5  (local5)
          22             local use 6  (local6)
          23             local use 7  (local7)
 

Severity is the log levels that defines how critical of the log entries, from 0 - 7, 0 indicates the most critical and 7 is for debugging purpose.

    Numerical         Severity
          Code

           0       Emergency: system is unusable
           1       Alert: action must be taken immediately
           2       Critical: critical conditions
           3       Error: error conditions
           4       Warning: warning conditions
           5       Notice: normal but significant condition
           6       Informational: informational messages
           7       Debug: debug-level messages

PRI is a unique values constructed by facility and severity where severity takes 3 LSB (least significant bits) append with facility start from bit 4.

PRI = (facility << 3) + severity 

<< indicates left shift which I borrow it from c++ programming language, when I do left shift N its like multiply with 2^N(two to the power of N). In this context, PRI formula can be written as

PRI = (facility * 2^3) + severity 

PRI is important when you wanna send message to syslog, the default message PRI is user(1).notice(5) which the PRI value is 8 + 5 = 13. Meaning if you do not specified the PRI value, it will be treated as 13.

For common Linux distro, syslogd or rsyslogd should be started before you login to your system, you can verify that with ps.

ps aux| grep syslogd

Usually syslogd comes with distro does not configured to accept remote messages, unless -r is specified.

root      4232  0.0  0.0  13424   888 ?        Sl   Jul22   0:00 syslogd -m 0 -r

How to enable syslogd to accept remote message?
syslog will listening to UDP port 514 for messages sent remotely, but if your distro running rsyslogd, you can listen to TCP port and you may also need to specify the port number.

Different Linux distro may have different ways of configuration, let say if you are using Red hat based distro, your syslogd and rsyslogd configuration will be at /etc/sysconfig/syslog or /etc/sysconfig/rsyslog.

For the case of rsyslogd, change the SYSLOGD_OPTIONS in /etc/sysconfig/rsyslog from SYSLOGD_OPTIONS=”-m 0″ to SYSLOGD_OPTIONS=”-m 0 -r514″ for UDP and SYSLOGD_OPTIONS=”-m 0 -t514″ for TCP. For the case of syslogd, change it to SYSLOGD_OPTIONS=”-m 0 -r”.

After that, restart your syslog or rsyslog services, In Fedora or Red Hat, you may do this with root permission.

service syslog restart

Or you can just kill the syslogd process and start manually from console for testing.

rsyslogd -m 0 -r514

Syslog Configurations
syslog daemon includes a configuration files to specified which logs to keep and append it to which file based on the PRI stated above.

Below are the sample of /etc/rsyslog.conf

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none        /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                                    /var/log/secure

# Log all the mail messages in one place.
mail.*                                                         /var/log/maillog

# Log cron stuff
cron.*                                                       /var/log/cron

You defines what messages goes to what file like this:

facility.severity         log_files

For example:

user.notice              /var/log/user.notice

The line above indicates that, when I get syslog message with facility user and severity notice, it will be append to file /var/log/user.notice.

For more info, please check out here.

How to send message to syslogd?
For sending locally, we can use the logger command. Before you send the message to syslog, lets tail the message log.

tail -f /var/log/messages

Now send “hello world” with logger.

logger "hello world"

It will appear in /var/log/messages as well as /var/log/user.notice if you have configure syslog.conf to forward user.notice messages to /var/log/user.notice. This proves that the default message’s PRI is user.notice. You may assign different PRI value to logger. For example if I wanna send message with PRI = user.info:

logger -p user.info "testing 123"

How to send log message to remote server?
Unfortunately, you can’t send through logger. But you can manually send a plain text UDP package to remote servers that listening on UDP port 514. With the help of netcat(nc), we can send the message to remote syslogd as simple as logger command.

nc -w0 -u 192.168.1.1 514 <<< "logging from remote"

To assign your message a PRI, you need to specified PRI’s value in numeric.
User.Info’s PRI value is:
(1 << 3 ) + 6 = 8 + 6 = 14. ( refers back the numerical code of facility and severity )

nc -w0 -u 192.168.1.1 514 <<< "<14>User Info msg from remote"

If your rsyslogd are listening to TCP port, just ignore -w0 and -u:

nc 192.168.1.1 514 <<< "<14>User Info msg from remote through TCP."

Batch mode operation
top is a real-time application that keeps display processes info sorted by cpu usage, if you redirects the outputs to a file like command line below, you will having your results mixed with ugly symbols, which is not what you want.

top > top.txt

Well, top support batch mode. By specified top on batch mode, top will display the outputs without arrange the views properly, but if you redirects it to a file, you won’t get those messy symbols mixed up.

top -b > top.txt

I just want to monitor top ten processes every seconds!
You can limits your views of top ten using head and tail, but this will STOP top from continuing to monitor the processes.

top -d 1 | head -n 17 | tail -n 11

Fortunately, we have watch to help, but you needs to put top in batch mode, and also you can ask top to stop after done displaying the result for one time. This will increase the accuracy of the result.

watch -n 1 "top -b -n 1 | head -n 17 | tail -n 11"