Gnu Global, the programmer’s friend

May 25th, 2010 mysurface

GNU Global (gtags) is a source code tagging system similar to ctags but more than that. I always use vim as my c/c++ source code editor, and using ctags to helps me travels from function call to function definition. I do write a post indicates how does ctags works with vim.

The reason I shifted to gtags is because ctags is no longer sufficient to me. I have to trace the codes from large volume than last time, and they scatter into many different folders.

Gtags provides me not only travels to definition of the function calls, but also allows me to locate the function calls from the definition, the reverse way. To be precise, it capable of locate object definitions, object references, both of them based on patterns I specified.

Gtags works with vim too, by doing some keymaping tweaks, it works the same like ctags do, and more.

To get gnu global install to your system, check out the official website’s download page.

After the installation, goto your root path of your c/c++ project, and trigger this and go make a coffee.

gtags

Gtags will pickup whatever needed and create FOUR tagging db file, refers here for more info.

In order to allow Gtags working in vim, you have to copy over the gtags.vim into vim plugin’s folder. ( Refers to here on how to install vim’s plugin script.)

Vim with gtags
For vim with ctags, hitting Ctrl+] will jump you to object definition, I gonna do the same for vim with gtags. Besides that, I gonna map Ctrl+\ to list all the object references.

Now, lets us edit vimrc, copy and paste those lines into your vimrc.

"gtags
:map <C-]> :Gtags<CR><CR>
:map <C-\> :Gtags -r<CR><CR>
:map <F10> :cclose<CR>
:map <F11> :cp<CR>
:map <F12> :cn<CR>

*nix, vimrc is at ~/.vimrc
windows gvim, Edit > Startup Settings

When Gtags list all the object references, it will open a split windows, to travels from one to another, use :cn and :cp, to close the split windows use :cclose, all are mapped to F11, F12 and F10.

You can find your codes in your shell too, by using command global.

To list all definitions of “GetExecute”, you do this:

global GetExecute

For more details listing, you can add in -x option, which it will print out the line as well as the line number.

global -x GetExecute

To list all the references of “GetExamples”, you do this:

global -rx GetExamples

You can add in your regex in your search such as *, see this:

global -rx GetLinuxBy*

You can search by symbol as well as grep too.

global -sx GetLinuxBy*
global -gx GetLinuxBy*

And more.

Thanks to Shigio YAMAGUCHI, Hideki IWAMOTO et al. for such wonderful tools and hope you enjoy coding with global.

Posted in Developer, global, gtags, gvim, vi, vim | Hits: 46757 | 3 Comments »

vim: Jump to the last known cursor when file open

May 21st, 2010 mysurface

When dealing with source codes by using vim, I usually open and close the files repeatably. It will be good if vim can automatically jump me to the last known position when I reopen a file.

To have vim do so, you must have viminfo, which usually exist by default.

Add in the setting into your ~/.vimrc

if has("autocmd")
  autocmd BufReadPost *
    \ if line("'\"") > 1 && line("'\"") <= line("$") |
    \   exe "normal! g`\"" |
    \ endif
endif

Ok, I have no idea what how it does the magic, but I know if your vim editor have autocmd capability, it will do wonders.

P.S. For gvim users, you can edit the vimrc by accessing the menu Edit > Startup Settings from the gvim.

Posted in gvim | Hits: 45820 | 2 Comments »

Why my written shell script doesn’t work in cygwin?

May 20th, 2010 mysurface

You may facing the problem like this. The workable shell script in Linux, after you copy and paste into windows editor and save as .sh, it doesn’t work under Cygwin shell. Why?

Bear in mind that, Cygwin shell giving you a *nix environment (either Unix or Linux), which is actually a bash environment. Everything task carry on outside the bash environment will be under the windows environment.

Line terminator between windows and Linux environment are different. When you hit enter, in windows it actually inject code CR and LF into the end of the line; where in *nix only CR is inject.

Therefore if your script was written by using windows text editor (even w32 gvim), it will gives you line terminator with CR and LF.

How to verify?
Check your written shell script with cat -v, if your script display something ^M at the end of the line, that indicates the text file is created under windows environment. And CR LF hurts the execution of bash script, ouch!

cat -v myscript.sh

How to fix it?
There is a command for you to convert windows text format into unix text format.

To convert windows text format into unix text format, do this:

dos2unix myscript.sh

Now you are ready to run the script, but remember to chmod to the script to make it executable before executing it.

chmod +x myscript.sh

Posted in Common, cygwin, dos2unix | Hits: 47210 | No Comments »

Makes your Microsoft Windows Incredible with Cygwin

May 15th, 2010 mysurface

I need to use windows OS, it is my job requirements; there are certain software only run at windows. I wish I could have Linux environment so that I can have commands which helps me on doing repetitive task, command such as grep, find, less, tail are my favorites.

In order to have Linux environment, there are three ways.

1. Install Linux for dual boot.
2. Install Linux into virtual machine by using vmware, virtual box.
3. Install Linux emulator which is Cygwin.

My requirement is to use Linux commands to ease my works under Windows, therefore what I actually needs is Cygwin.

What is Cygwin?
Based on the definition from the official website, Cygwin is a Linux-like environment for Windows. Basically Cygwin helps you download a series of exe and dll, and keep under C:\Cygwin\bin, you can run almost all exe under this directory at your windows command prompt.

So what is the different between Cygwin console and windows command prompt? Cygwin initiate the bash login using command prompt, open C:\Cygwin\Cygwin.bat with notepad will reveal the secret.

This does the bash login:

bash –login -i

Installation:
It is too easy that not even worth mentioning here, download a Setup.exe from cygwin official website and run, hit series of ‘Next’ button and you are done with the default setup. One thing to bear in mind is if you need to download extra package; you may need to run the setup.exe again, so don’t delete the setup.exe after the installation.

To ease the triggering setup.exe from cygwin console, you create a symlink at /usr/bin point to the setup.exe.

Let say I download my setup.exe and store at C:\

ln –s /usr/bin/cygsetup /cygdrive/c/setup.exe

Therefore when I trigger cygsetup, it will run setup.exe.

Cygwin maintain all drives in /cygdrive, /cygdrive/c indicates C: and you can change directory to C: by doing this:

cd c:

Adding new package:
Same as installation, adding new package by clicking setup.exe, at the last GUI, you may type the package name you wanna install into a search text box or manually choose from the list. Just click under ‘New’ column if you choose to install.

Bash Environment

Maybe you may ask, besides that there are a lots of Linux commands for us to use, what else is different compare to windows default dos command prompt?

Cygwin giving you are bash environment.

You have a home directory, which is /home/ or ~, and the physical location is at C:\Cygwin\home\.

You have your bashrc and bash_profile like bash in Linux.

You may execute your windows programs.

You may run the bash script; within your bash script you may execute the windows exe such as explorer, calc, regedit etc.

The path methodology is different from dos, it start with / which is a root directory. Therefore all the drives will be map under /cygdrive.

Cygpath tells you the path information.
Sometimes we need to obtain a windows path format on the current directory we are in at cygwin, can we obtain the windows path or dos path?

The answer is yes, because cygwin provides cygpath.

Let say I have traverse to ‘Program Files’, in bash the unix path is /cygdrive/c/Program Files, in order to obtain the windows path format, we can do this:

cygpath -aw . 

To obtain the dos path format ( short string upto 8 characters for directories and filename)

cygpath -ad . 

To convert back the windows path to unix path, you can do this:

cygpath -au “C:\Program Files” 

There are more to share about how cygwin helps me in the windows environment, but for now, I rest my case.

Hope you enjoy Cygwin!

Posted in cygwin | Hits: 49325 | No Comments »

tshark: perform filters to rip out a pcap from a large pcap

April 13th, 2010 mysurface

One time I been given numbers of very large pcap files, ask me to do some analysis on http traffics. The given pcap is raw traffics pcap capture from servers, most of the packets in the pcap for me is redundant, because I am only interested in http traffics.

Opening a large pcap with wireshark is killing me, as it takes time to arrage the packet’s info into the GUI, And I have numbers of large pcap for me to analyse, also I required to open multiple instance of wireshark to compare them, that will definitely slow down my system because it will be eating too much resources.

So how? I am so desperate on this.

First, I fire up my command line wireshark - tshark. I pass my ‘Display filters’ to tshark and ask it to rip only the http traffics for me into an output pcap file.

tshark -r server_10_0_0_17_20100401.pcap -R "tcp.srcport==80" -w http_10_0_0_17_20100401.pcap

tshark will read the pcap by using -r. I specify DISPLAY FILTER by using -R. At last I define my output pcap by using -w.

As simple as that, now I am happy with it.

There are more to discover what tshark capable of, feel free to check out the manuals.

Posted in Network, tshark | Hits: 65667 | No Comments »

Install RPM with yum

January 31st, 2010 mysurface

I am a Linux user for couple of years, and I am using fedora for quite sometimes too. To install the packages that I need, first I will check them out at my repository using yum.

yum search something

p.s Replace ’something’ with your keywords.

If I manage to obtain the package list, I can install them like this:

yum install something

But that is not always the case. Sometimes the packages I need is just not exist under my repository, I can only obtain the rpm files from the external site. By installing rpm file using rpm command introduces a problem. I have to resolves the dependencies of the particular rpm myself.

Then, I have to search more rpm files to resolve the dependencies.

But today, I realized that I could install rpm files using yum, and yum resolves the dependencies for me. The command line is quite straight forward too. Let say I wanna install ’something.i586.rpm’, I can install it with yum like this:

yum install something.i586.rpm

Thanks to yum developers, it save a lots of my time.

Posted in yum | Hits: 103473 | 2 Comments »

Split screen console terminal ~ Terminator

January 3rd, 2010 mysurface

Sometimes, you may have many logs to trace at once, you may need to copy and paste text from files to files, and tabbed terminal is just not convenient enough.

So, you hope for a terminal that allows you to split your console screen into either horizontal or vertical flexibly. If you have such requirements like I do, then you are looking for console terminal software just like Terminator.

Terminator implements those flexibility of arranging terminals based on Gnome terminal. It allows you to split your screen putting the terminals as a single view. You can switch the terminals by just drag and drops. You can still adjust the size of each terminals. The tabbed features remains, therefore you can still have multiple tabs and each tab you may have multiple terminals.

Terminator re-implements the shortcut keys capabilities, you can create and edit .config/terminator/config to define your shortcut keys there. Check out the manual page for more info.

man terminator_config

At last please check out this ridiculous screenshot:

Have Fun!

Posted in Misc, terminator | Hits: 121051 | 1 Comment »

Compile and execute 32 bits application in 64 bits operating system

October 1st, 2009 mysurface

The world is going towards 64 bits machines and operating system for personal computer, but there are still some libs and software only support 32 bits. Therefore in this transition period, there is a needs to support both 64 bits and 32 bits applications.

Lately I been using x86_64 Fedora Linux. There are requirements for me execute some 32 bits binaries as well as compiling to produce 32 bits binaries.

First of all, I do not aware of any great different for me based on user experience while migrating from 32 bits to 64 bits Fedora. Maybe this is because the fedora repository is in the good shape, where everything that I had downloaded by using yum, will works perfectly.

Until one day I downloaded some legacy projects from the 3rd party website, that piece of software comes with 32 bits libs which needs 32 bits compilation. I started to realized that the binaries that I had compiled in my 64 bits environment is not working at 32 bits OS even though I am using the legacy version of gcc 3.4 compiler.

Fedora x86_64 allows you to run 32 bits software given that the dynamic libs is installed properly. I realized that my fedora does have two directories for lib, /lib and another one is /lib64. When I am searching for a particular lib using yum, let say libxml2, I realized that yum is actually returns me both 32 bits and 64 bits result.

yum search libxml2
...
libxml2.i386 : Library providing XML and HTML support
libxml2.x86_64 : Library providing XML and HTML support
...

By default if I execute “yum install libxml2″, yum will install the 64 bits one for me. In my case I want the 32 bits glibc, what I have to do is to specify the exact package name.

yum install libxml2.i386

After the installation, now I can run the 32 bits application that requires libxml2.

How about compiling 32 bits application in 64 bits operating system?

First of all, I need to make sure I had installed all the development packages of the 32 bits libs required. Let takes libxml2 as the example, I need to yum ‘libxml2-devel.i386′.

Next tell gcc that I am going to compile the codes into 32 bit binaries by specify the -m32 CFLAGS. Assume I wrote a simple hello.c.

gcc -m32 -o hello{,.c} -lxml2

Now I need to produce the binary that can be run under Red Hat es4. I know the default version of gcc for RHES4 is 3.4 which I need gcc34 in my fedora. ( I need to yum another package ‘compat-gcc-34.x86_64′ for that) . With that now I can compile the codes at my x86_64 fedora to produce binary that can run at RHES4.

gcc34 -m32 -o hello{,.c} -lxml2

Posted in Developer, Misc, gcc | Hits: 157581 | 1 Comment »

How to mount a ssh server?

September 18th, 2009 mysurface

After FUSE (Filesystem in Userspace) was invented, it opens up a lots of possibility to mount a remote filesystem on various protocol, such as FTP and SSH. FUSE is an API to implement a fully functional filesystem in a userspace program.

I had wrote a post shares the idea how we can mount a ftp host, the same way we can mount a ssh point using sshfs.

With sshfs, we can mount a remote directory through ssh, the command line works like this:

sshfs user-name@host-ip:remote-dir mount-point

Mount-point is simply a directory which we can trigger mkdir to create one. If we want to mount the user’s home directory at the remote server, we can leave the remote-dir as blank. For examples I want to connect to 192.168.1.1, I can do this: ( Assume I created a folder call ssh_mount )

sshfs surface@192.168.1.1: ssh_mount

As simple as that, we had done mounting the ssh server. But bear in mind, it is not the same like you ssh to the server. We are just mounting a remote filesystem, we are still working under the local system environment.

Posted in Admin, sshfs | Hits: 161506 | No Comments »

Serialize INI configuration to python dictionary

August 30th, 2009 mysurface

A lots of configuration file is in the format of INI, if we manage to serialize INI configuration to python dictionary, we can write python scripts to analyse and generate back to INI files.

Below is the sample of INI configuration: config.ini

#
# GENERAL section contains generic configuration
#
[GENERAL]
IP=192.168.1.4
Port=2143

#
# LOG section contains logging settings
#
[LOG]
LogLevel=5
LogPath=/var/log
LogType=Info Trace Error Debug

I created few python function use to serialize the INI configuration into dictionary. But there are some assumption to make. Assume that the comment of INI is start with char ‘#’. python dictionary do not takes in comments, because comment is not configuration. But what if after I change the configurations in dictionary and want to serialize back to INI file with the original comments stayed in as well? Yes you can remain the comments with the set of python functions I wrote.

Lets look at what function I have.
grep (str_list,pattern,neg=False): is my internal function to filter element in string list (first param).

ParseConfig (name=None): put the INI config filename as param, dictionary object will be returned.

ConfigTemplate (name=None): The construct the INI structure in another dictionary, storing the Section Name as well as your comments.

ConfigWrite (template=[],config=[],filename=”"): This function is to serialize back the python dictionary back to INI file. First param takes in the template dictionary object, second takes in configuration dictionary object.

Lets us look at all functions definition as well as the examples: ParseConfig.py

#/usr/bin/env python
import sys,os,datetime,time,re,string
from StringIO import StringIO

GREP_NEGATIVE=True

def grep (str_list,pattern,neg=False):
    """grep() will return list of match case.
Param: str_list - must be a list of string, each element in list is a new line
                  can use .split('\n') to obtain list from mutiple line of string
Param: pattern - re pattern can be composed by re.compile()"""
    sres=[] #empty list
    for s in str_list:
        r=re.search(pattern,s,re.VERBOSE)
        if r and neg==False:
            sres.append(s)
        elif (not r) and neg==True:
            sres.append(s)

    return sres

def ParseConfig (name=None):
    global GREP_NEGATIVE
    """ Parsing config file to this function, will serialized to dictionary
@name  = filename
@return = dictionary"""
    configdict={}
    loglist=[]

    while True:
        try:
            f = open(name,"r")
        except IOError:
            print "Err ",name,"not exist"
            return None
            pass
        else:
            break

    try:
        for line in f:
            #strip all the whitespaces and write all lines to list.
            loglist.append(line.strip(string.whitespace))
    finally:
        f.close()

    # filter out the comment
    loglist=grep(loglist,"^\#",GREP_NEGATIVE)

    # make a dictionary
    xL=[] #current L
    itemdict={}
    for item in loglist:
        L=grep([item],"^\[.*\]$") # section
        if L:
            xL=[L[0].lstrip("[").rstrip("]")]
            itemdict={}
            #print xL
        else: # items?
            if len(xL)>0:
               if item:
                    #split index and values.
                    iL=item.split("=")
                    #replace all '\t' to " " in values
                    iL[1]=string.replace(iL[1],"\t"," ")
                    # split the values into list
                    iiL=iL[1].strip(string.whitespace).split(" ")
                    real_list=[]
                    for item in iiL:
                        # read each value in the list, if got something append to the real_list
                        if len(item.strip(string.whitespace)) is not 0:
                            real_list.append(item.strip(string.whitespace))
                    # create another dict into the section
                    itemdict[iL[0].strip(string.whitespace)]=real_list
                    configdict[xL[0]]=itemdict
    return configdict
pass

def ConfigTemplate (name=None):
    try:
        f=open(name,"r")
    except IOError:
        return None

    loglist=[]
    try:
        for line in f:
            #strip all the whitespaces and write all lines to list.
            theline = line.strip(string.whitespace)
            loglist.append(theline)
    finally:
        f.close()

    loglist=grep(loglist,"^\#|^\[")
    if len(loglist) is 0:
        loglist = None
    return loglist

def ConfigWrite (template=[],config=[],filename=""):
    global DebugDupConfigWrite

    if len(template) is 0 or len(config) is 0 or filename is "":
        return False

    try:
        f = open(filename,"w")
        for line in template:
            r=re.search("^\#",line,re.VERBOSE)
            if r:
                f.write(line + "\n")
            j=re.search("^\[",line,re.VERBOSE)
            if j:
                f.write(line + "\n")
                # get all section line up
                section_name = line.strip(string.whitespace).lstrip("[").rstrip("]")
                if config.has_key(section_name):
                    sec_dict = config[section_name]
                    #
                    element_list=[]
                    for element in sec_dict:
                        element_list.append(element)
                    element_list.sort()
                    for element in element_list:
                        # index = values....
                        #print element, sec_dict[element]
                        f.write(element + " = " + "\t".join(sec_dict[element]) + "\n")
                    f.write("\n")
                else:
                    pass
                    #f.write("CAN'T FIND THE ELEMENTS in section[%s]! DATA CORRUPTED.\n" % section_name)
                pass # end if j

            pass # end for line
        f.close()
        return True

    except:
        return False

# sample calling
myDict = ParseConfig ("config.ini")
template = ConfigTemplate ("config.ini")
ConfigWrite(template,myDict,"test.ini")

print "COMPLETE DICTIONARY"
print myDict

print "ALL TAGS"
for c in myDict:
    print " ",c

print ""
print "ALL ITEM IN 'GENERAL'"
for d in myDict["GENERAL"]:
    print " ",d

print ""
print "ALL ITEM IN 'LOG'"
for d in myDict["LOG"]:
    print " ",d

print ""
print "DIRECT ACCESS 'LOG's LOGTYPE'"
print myDict["LOG"]["LogType"]

print "SO, YOU CAN SEE THAT IS THE LIST, IT ITERATE the LIST"
for e in myDict["LOG"]["LogType"]:
    print " ",e

print "DIRECT ACCESS 'LOG's LOGTYPE 2nd value'"
print "second value is ",myDict["LOG"]["LogType"][1]

I hope my example is straight forward for you to understand how to use it, wish you enjoy python scripting :)

Posted in Developer, python | Hits: 168030 | 2 Comments »