copy files and create the directories if it doesn’t exist

June 3rd, 2011 mysurface

It always have a need for this simple operation.

I wanna copy certain files to a destination, but I wanna preserved the sub directories structure, any cp option to set? For examples, I wanna copy my_project/cpp/hello.cc to your_project folder, but I want it to appear as your_project/cpp/hello.cc automatically. Especially I wanna do it in a batch of file.

By just using cp command, until today, I do not see such option for it.

You need additional commands, dirname and mkdir.

You may read a list of file from a file like this


filelist="myfilelist.txt"
while read line
do
# do your copy over here
done < $filelist

First you extract the directory name of the file you wanna copy, and then create the directory, lastly copy over the file.


filelist="myfilelist.txt" # assume your filename with relative path.
target="your_project"
while read line
do
  target_dir=`dirname $line`
  mkdir -p $target_dir
  cp $line $target$target_dir
done < $filelist

P.S. The snapshot of the code above might not work for all case, it depends on your filename in your file list, but gives you and idea on how to copy files and create directories if it doesn’t exist.

Posted in dirname | Hits: 104929 | 15 Comments »

Your XML friend XPATH command line xmllint

March 5th, 2011 mysurface

Do you have a large XML to analyze? to query for info? Maybe you are using a XML viewer, a string search for that. But if your XML have a proper structure and you have understand enough for the structure, you may want to consider to use XPATH.

[Q] What is XPATH?
XPath is a “language” for finding information in an XML document. Something like SQL, it has its own syntax to help you to query for your info in an XML. To know more about XPATH, can check out this XPATH tutorial.

[Q] Is there any XPATH command that I can use to query my XML?
xmllint, which comes with libxml2.

xmllint provides you a shell where you can continuously to query your XML.

xmllint --shell myXML.xml

Shell mode is also a good way for you to learn up XPATH, type ‘help’ in the xmllint shell shows you a list of command it support.

Take books.xml sample as example.

To query all the books title, you can do this in the shell.

> cat //title
 -------
<title lang="en">Everyday Italian</title>
 -------
<title lang="en">Harry Potter</title>
 -------
<title lang="en">XQuery Kick Start</title>
 -------
<title lang="en">Learning XML</title>

xmllint shell support grep where you can search with string keywords

> grep XML
/comment() : c--       20
/bookstore/book[4]/title : t--       12 Learning XML

grep in shell is not XPATH syntax, for XPATH to do something similar to grep, you can use contains().

> cat //*[contains(*,"XML")]

XPATH supports many functions to enlighten your query.

[Q] I wanna use the XPATH result for further manipulation, can I get it with xmllint?
--xpath option.

You can do this.

xmllint --xpath //title books.xml

And.... xmllint can do more than that.

I hope you enjoy reading this, bye.

Posted in Text Manipulation, xmllint | Hits: 150795 | 10 Comments »

export environment variable from a bash script

February 1st, 2011 mysurface

You may want to create a bash script that export some environment variable for your shell, but infect every execution of bash script is a unique session, where the variable export within it can’t be carry backward back to the bash shell.

For example you have a env.sh which contains some environment variables

export HELLO=hello
export HELLO2=world

And you trying to execute the env.sh, then try to echo $HELLO $HELLO2 at your shell, you will get nothing.

The proper way to export environment variable from your script to your shell is using command source.

source env.sh
echo $HELLO $HELLO2
$ hello world

You can put your env.sh into /usr/bin, and you can do source env.sh at any location and it will still works. Just that you have to educate your users to use source instead of direct execute the script. But I realized that not much users aware of the source command.

There is always an alternative way to do the same thing in Linux world. You can create a new bash session from your bash script and make it stay there until user type ‘exit’. Bash support an interactive option for that.

bash -i

Ok, how to make use of it?

Let me gives you a case and explain why I wanna make use of bash -i instead of source.

As an embedded Linux programmer, sometimes I may need to compile some apps by using different set of toolchain based on the embedded device ‘s processor architecture. The common environment variable for toolchain are ARCH and CROSS_COMPILE. Besides that, I also want to remind the user regarding that particular shell is meant for building apps for embedded device, therefore I would like to change the color and format of existing bash prompt, which i can make use of PS1 variable.

build_arm.sh

#!/bin/bash
export ARCH=arm
export CROSS_COMPILE=arm-none-linux-gnueabi-
export PS1="\e[31mBUILD_ARM \w \e[m\n\$"

bash -i

When execute build_arm.sh, the bash prompt will turn red, and environment variable for ARCH and CROSS_COMPILE will be set, until user type ‘exit’, everything will resume back to normal. Why? because bash -i interactively wait for user inputs.

./build_arm.sh
BUILD_ARM ~
$

More PS1 tips? checkout this Angelina Jolie.

Posted in Bash, source | Hits: 125416 | 5 Comments »

Why ‘export’ if i could just direct assign the variables in bash?

January 25th, 2011 mysurface

I can direct assign value to a variable in my bash shell why I still need the command ‘export’ ? What is the different between them?

If you have doubts about this, please continue to read on it.

Lets do an experiment in a bash shell:

$ HELLO=world
$ export HELLO2=world2
$ echo $HELLO $HELLO2
world world2

Hey, it works exactly the same!

Not really exactly the same if you use it in your bash script. Let me show you an example. Let say I have a bash script main.sh , which it calls child.sh.

child.sh

#!/bin/bash
echo $HELLO $HELLO2

main.sh

#!/bin/bash
HELLO=world
export HELLO2=world2
./child.sh

Now lets execute ./main.sh

$ ./main.sh
world2

Hey! the ‘world’ has gone!

When we assign the variable directly, we make the variable local, when we call a child script, the local variable will not available for the child script. In contrast, if we export the variable, we make it appear as an environment variable, which it will be available globally within that bash session. That is the reason, child.sh reads $HELLO2 but not $HELLO.

Posted in Bash, export | Hits: 136163 | 8 Comments »

config file for your bash script

January 22nd, 2011 mysurface

Sometime you may wish to create a bash script to do something based on a config file, what actually was in your mind? How to implement the config file for your bash script?

First I think of using command ‘cut’, where I can create a list of tag=value, then uses ‘cut‘ to read out the tag and value. But that is so wrong, because I could just using command ‘source’ to do it instead.

Let say I have a config.ini

PATTERN="$hello*"
FILENAME=hello.txt
OUTPUT=result.txt

And I want my bash_script to read these variables and perform grep.


#!/bin/bash
source config.ini

grep ${PATTERN} ${FILENAME} > ${OUTPUT}

The keyword ‘source’ will import the variable from config.ini into my script, made them available for my use. The source line can be also replace it into this:

. config.ini

DOT(.) is equivalent to command ‘source’.

I can also source another bash script which contains if-else logics, the same way I source my config.ini.

But bare in mind, if your target source file contains ‘exit’, it will exit your parent script.

For example, I have a parent.sh sourcing config.sh, where my config.sh contain exit.

config.sh

#!/bin/bash
HELLO=world
exit

parent.sh

#!/bin/bash
source config.sh
echo ${HELLO}

When I try to execute the ./parent.sh, my script immediately exit before it execute the ‘echo’.

I hope that this post will gives you an idea of using command ‘source’ to implement your config file for your bash script.

Posted in Bash, source | Hits: 120165 | 5 Comments »

BTTB: looping for shell script under embedded linux

November 17th, 2010 mysurface

You may already realized Linux happened to appear at many places, such as web server, storage server, desktop, kiosk machine, mobile devices. Yes, more and more devices running embedded Linux. Yeah, Android is a modified version of Linux kernel too!

Scarcity is still an issue, embedded Linux can be very different to Linux that hosted on desktop or high-end servers. You may be provided with simple shell (sh but not bash, not ksh and not csh) and limited commands, it bundles into a BusyBox, The Swiss Army Knife of Embedded Linux.

I am a bit upset when I holding this so called Swiss Army Knife ( probably a customized one, super tiny one), why this command also don’t have, that command also not available. Well, I am still quite new to embedded Linux perhaps.

My busybox does not have bash, therefore a lots of bash syntax can’t be applied to my script, ouch! I do wrote some post about bash. Such as if … then … fi and download multiple files from a site using for loop like c in bash. The if…then…fi examples may still valid for shell script, because that is the fundamental one.

The very basic for loop for shell script is like this


for p in 1 2 3 4 5
do
     echo $p
done

You can do simplified it by using command seq ( if seq is available in your busybox)


for p in `seq 1 5`
do
     echo $p
done

Well, my swiss army knife doesn’t contain me seq command, it doesn’t have bash, no top command, giving me a fake ps with limited info. I can’t check my how busy is my processors and thread easily (because fake ps do not tells you that), I need to access it through /proc/[PID] and thank God, it does provide me the cut command.

Yes I able to cut by using my swiss army knife.

Due to the scarcity, I learn more about /proc. /proc is a pseudo-filesystem that provides tones of process information. Check out the proc 5 manual for detail information.

man 5 proc

What I am interested is the file /proc/[PID]/stat, the status information about the process. stat provides a long array of values, I am interested on column 1,2,14 and 15 which is PID, Command, utime and stime.

Ok, list me all the process’s utime and stime. I wrote a simple script call probe.sh.


#!/bin/sh
for p in /proc/[0-9]*;
do
    cat $p/stat | cut -d" " -f1,2,14,15
done

The output looks decent, time takes less then 1 second.


...
6 (cpuset) 0 0
608 (scsi_eh_0) 0 1
612 (scsi_eh_1) 0 0
615 (scsi_eh_2) 0 1
661 (kpsmoused) 0 0
671 (kstriped) 0 0
677 (kondemand/0) 0 0
7 (khelper) 0 8
707 (usbhid_resumer) 0 0
854 (udevd) 0 11

real    0m0.965s
user    0m0.138s
sys     0m0.752s

But the list is long and overwhelming. What if I have targeted pid range?


#!/bin/sh
START=2000
END=3000
until [ $START -gt $END ]
do
    p="/proc/$START/stat"
    if [ -f $p ] ;
    then
        cat $p | cut -d" " -f1,2,14,15
    fi
    START=`expr $START + 1`
done

The output is promising, but it took too long to produce that.


2391 (syslog-ng) 0 0
2392 (syslog-ng) 1 11
2451 (gpm) 0 0
2882 (dhcpcd) 0 0

real    0m7.145s
user    0m0.959s
sys     0m5.640s

The expr for arithmetic function seems to consume a lots of processing time.

Another approach, which amazingly gives correct result and fast.


START=2000
END=3000
for p in /proc/[0-9]*;
do
    pid=`basename $p`
    # omg no basename? replace with the line below
    # pid=`cat $p | cut -d"/" -f3`
    if [ $pid -ge $START ]
    then
        if [ $pid -le $END ]
        then
            cat $p/stat | cut -d" " -f1,2,14,15
        fi
    fi
done

Check out the result. Outstanding isn’t it?


2391 (syslog-ng) 0 0
2392 (syslog-ng) 1 11
2451 (gpm) 0 0
2882 (dhcpcd) 0 0

real    0m0.550s
user    0m0.069s
sys     0m0.447s

I had briefly illustrate the looping mechanism on shell script, I hope that gives you some ideas while you working under embedded Linux.

Posted in cut, for, if, seq, Text Manipulation | Hits: 170793 | 10 Comments »

VirtualBox: startup guest machine in background

October 31st, 2010 mysurface

Usually we start our virtual machine by clicking through the GUI. If your guest machine is a server, it would be making more sense to be run as background process.

Well, my case, I have ‘gentoo’ installed in virtual box, which expose to outside world through ssh. I would like it to start as background process, how can I do that?

Virtual Box allows you to start your virtual machine through command line VBoxHeadless, by adding ‘&’, you can make the process run as background.

VBoxHeadless -s [guestname] &

I wrote a bash script to help myself perform the background startup, reboot, poweroff and status query. Assume my guest machine name as ‘gentoo’, I wrote a script name as ‘gentoo’. Now I can do ‘gentoo start’ to start it at background.

#!/bin/bash

if [[ -z "$1" ]]
then
    echo "Usage:"
    echo "      gentoo [status|start|reboot|poweroff]"
    exit
fi

if [[ $1 == "status" ]]
then
    # print out the machine state
    VBoxManage showvminfo gentoo | grep State
elif [[ $1 == "start" ]]
then
    # start without vrdp
    VBoxHeadless -s gentoo --vrdp=off &
elif [[ $1 == "reboot" ]]
then
    # send ctrl+alt+del to guest machine
    VBoxManage.exe controlvm gentoo keyboardputscancode 1d 38 53 b8 9d
elif [[ $1 == "poweroff" ]]
then
    # simulate pressing power off button, will power cut immediately
    VBoxManage controlvm gentoo poweroff
fi

With the capabilities of command line to manipulates the guest machine, it helps to support for more customization and automation. This is the main reason I like about VirtualBox.

Have fun :)

Posted in VBoxHeadless, VBoxManage, VirtualBox | Hits: 154360 | 4 Comments »

VirtualBox: Port Forwarding for NAT network

October 29th, 2010 mysurface

If you came across the solution of setting up port forwarding for ssh port like this:


VBoxManage setextradata [guestname] "VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/HostPort" 2222
VBoxManage setextradata [guestname] "VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/GuestPort" 22
VBoxManage setextradata [guestname] "VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/Protocol" TCP

You may want to try out this alternative approach. This approach is using the same command ‘VBoxManage’, but it is more flexible and simple.

1. First of all, I power off my guest machine (virtual machine), this approach refuse to work if my guest machine is running.

2. Next, I query my guest machine’s vminfo.

Let say my guest machine ‘s name is ‘gentoo’ (I am gonna use this as example for the entire post) , I will do this:

VBoxManage showvminfo gentoo

It shows a long list of very informative results. I am only interested on my guest machine’s NIC info. Therefore I do this instead.

VBoxManage showvminfo gentoo | grep NIC
NIC 1:           MAC: 0800272DC585, Attachment: NAT, Cable connected: on, Trace: off (file: none), Type: virtio, Reported speed: 0 Mbps, Boot priority: 0
NIC 1 Settings:  MTU: 0, Socket( send: 64, receive: 64), TCP Window( send:64, receive: 64)
NIC 2:           disabled
NIC 3:           disabled
NIC 4:           disabled
NIC 5:           disabled
NIC 6:           disabled
NIC 7:           disabled
NIC 8:           disabled

Based on the result of vminfo, indicates that my guest machine configured a NIC which is NIC1.

Ok, now lets do a port forwarding. Take ssh port as an example. I would like to forward my guest ssh port (22) to my host machine at port (2222). I will do this:

VBoxManage modifyvm gentoo --natpf1 "ssh,tcp,,2222,,22"

The syntax for modifyvm –natpf shows as below:

VBoxManage [guestname] --natpf[1-N] "[function-name],tcp|udp,[hostip],[hostport],[guestip], [guestport]" 

‘function-name’ can be any name which well describe the port forwarding rule.

Lets querry the vminfo again.

VBoxManage showvminfo gentoo | grep "NIC 1"
NIC 1:           MAC: 0800272DC585, Attachment: NAT, Cable connected: on, Trace: off (file: none), Type: virtio, Reported speed: 0 Mbps, Boot priority: 0
NIC 1 Settings:  MTU: 0, Socket( send: 64, receive: 64), TCP Window( send:64, receive: 64)
NIC 1 Rule(0):   name = ssh, protocol = tcp, host ip = , host port = 2222, guest ip = , guest port = 22

Observed that I ignore the IP address settings because I am using DHCP on my guest machine NIC.

To delete the port forward rule, I do this:

VBoxManage [guestname] --natpf[1-N] delete [function-name]: 
VBoxManage gentoo --natpf1 delete ssh

At last, I boot up my guest machine to verify the port forwarding setting.

VBoxManage startvm gentoo

VBoxManage can do more than just port forwarding, check it out the manual if you want to know more.

Hope you enjoy.

Posted in VBoxManage, VirtualBox | Hits: 151030 | 3 Comments »

Install Gentoo for kernel hacking

October 24th, 2010 mysurface

I am curious about what are Linux kernel do? Many years ago, due to the tedious steps to compile the kernel, I give up dive into it at the very beginning stage, which is compiling kernel. But recently this urge come back to me. I started to wonder whether is it exist a Linux distro which provides kernel build environment.

I came across a forum (lost track of the link), someone mention in fact Gentoo is a good candidate for the kernel build. As simple as emerge the kernel source, and you will have the latest kernel source code under /usr/src/linux.

emerge gentoo-sources

Installing Gentoo is not like installing Ubuntu/Fedora/Suse, where you can clicks the way and get a fancy desktop. I don’t want a fancy Gnome/Kde, what I want is a text console, which allows me to learn and hack the kernels, therefore Gentoo came as a charm for me.

Gentoo Linux is a distro that gives you the best optimized system, best performance. It sounds like the best distro for you isn’t it? The reason why Gentoo can be so optimized and performance wise competent because you have to compile everything including kernel. emerge is a command that helps you download the source codes, resolve dependencies, compile and install it right for you.

The challenging build steps comes with a very details installation guide, you can start with it by reading Gentoo handbook.

I create an image on VirtualBox, network setting attach to NAT. under Advanced option Adapter type set as virtio-net. The reason I choose virtio-net as the adapter type, because that is the only Ethernet kernel module available on the Gentoo minimum live CD which VirtualBox (version 3.2.2) could provide.

I need to enable virtio as kernel module in kernel config, refers to the live cd, there are few kernel modules to enable. virtio, virtio_ring, virtio_net and virtio_pci.

In order to have a broaden view on console, I enable uvesafb. You can follow the guide here. It giving me problems that, no matter how I set my video option, I can only see half of the screen> Fortunately, I found some guides for Gentoo under VirtualBox.

Remove consolefont from boot runlevel

rc-update del consolefont boot

Replace video option:

video=uvesafb:800x600-16,mtrr:4,redraw

For the fully customized OS like Gentoo, it is important to know the various configuration files. Here I wanna highlight some of the important one.

Mount Mapping table: /etc/fstab
This is the common one, to configure the mounting table, you have to edit this file.

Grub Bootloader config: /boot/grub/grub.conf
This is where you adding your kernel entries as your boot option. Video option is done here too.

Make config: /etc/make.conf
Installing package means compiling, to optimizing your build, defines your setting here. Portage related variables can define here too, such as GENTOO_MIRRORS and USE.

Kernel Modules Autoload: /etc/modules.autoload.d/kernel-2.6
You can define your extra kernel modules to autoload while booting up.

List of available kernel module installed.

find /lib/modules// -type f -iname '*.o' -or -iname '*.ko' | less

Network config: /etc/conf.d/net
Define your static IP or dhcp over here.

There are a lots more yet to discover for me, at least for now I am happy with Gentoo ;)

Hope you have fun too.

Posted in emerge, gentoo | Hits: 150694 | 7 Comments »

Gnu Global, the programmer’s friend

May 25th, 2010 mysurface

GNU Global (gtags) is a source code tagging system similar to ctags but more than that. I always use vim as my c/c++ source code editor, and using ctags to helps me travels from function call to function definition. I do write a post indicates how does ctags works with vim.

The reason I shifted to gtags is because ctags is no longer sufficient to me. I have to trace the codes from large volume than last time, and they scatter into many different folders.

Gtags provides me not only travels to definition of the function calls, but also allows me to locate the function calls from the definition, the reverse way. To be precise, it capable of locate object definitions, object references, both of them based on patterns I specified.

Gtags works with vim too, by doing some keymaping tweaks, it works the same like ctags do, and more.

To get gnu global install to your system, check out the official website’s download page.

After the installation, goto your root path of your c/c++ project, and trigger this and go make a coffee.

gtags

Gtags will pickup whatever needed and create FOUR tagging db file, refers here for more info.

In order to allow Gtags working in vim, you have to copy over the gtags.vim into vim plugin’s folder. ( Refers to here on how to install vim’s plugin script.)

Vim with gtags
For vim with ctags, hitting Ctrl+] will jump you to object definition, I gonna do the same for vim with gtags. Besides that, I gonna map Ctrl+\ to list all the object references.

Now, lets us edit vimrc, copy and paste those lines into your vimrc.

"gtags
:map <C-]> :Gtags<CR><CR>
:map <C-\> :Gtags -r<CR><CR>
:map <F10> :cclose<CR>
:map <F11> :cp<CR>
:map <F12> :cn<CR>

*nix, vimrc is at ~/.vimrc
windows gvim, Edit > Startup Settings

When Gtags list all the object references, it will open a split windows, to travels from one to another, use :cn and :cp, to close the split windows use :cclose, all are mapped to F11, F12 and F10.

You can find your codes in your shell too, by using command global.

To list all definitions of “GetExecute”, you do this:

global GetExecute

For more details listing, you can add in -x option, which it will print out the line as well as the line number.

global -x GetExecute

To list all the references of “GetExamples”, you do this:

global -rx GetExamples

You can add in your regex in your search such as *, see this:

global -rx GetLinuxBy*

You can search by symbol as well as grep too.

global -sx GetLinuxBy*
global -gx GetLinuxBy*

And more.

Thanks to Shigio YAMAGUCHI, Hideki IWAMOTO et al. for such wonderful tools and hope you enjoy coding with global.

Posted in Developer, global, gtags, gvim, vi, vim | Hits: 243201 | 7 Comments »