sed, awk (gawk), and Perl

As a programmer, I like the Unix commands “find” and “grep”. They are too good. “find” to find the files recursively based on file name, file type, etc, then execute command towards the files.

“grep” is even more better, when I want to find some words in the source code, and I can use regular expression. Combine “grep” with “sed”, one can perform search and replace.

However, when I want to search and replace for multiple patterns, “sed” will be difficult to be used. One can write a shell script to perform the task. But “awk” (or GNU awk) will be a better solution for me. Write an awk script, can search and replace easier, and awk script allows to use variables.

But, “awk” is not good enough. “awk” is difficult to perform search and replace with regular expression that needs backreferences and lazy quantifier. To do these tasks, Perl is the best!

Yes! I never thought that Perl is that good. Because for me, using Perl to do web development is more difficult than using PHP. Using Perl to write an application is not better than using C/C++, because it is an interpreted language, not compiled language. But, if using Perl as a programmer’s tool, it is good for everything. Recursive file renaming with regular expression and recursive search and replace with regular expression, I can only solve these two tasks with Perl script.

(Though I was taught to use zsh to perform recursive file renaming with regular expression, but now I think Perl script is powerful enough.)

Chromium is really good in netbook

Compare to the Firefox, I prefer to run Chromium in my netbook. This is because the screen of the netbook is small, and the resolution is low. Using Firefox, the toolbars and file menu already occupies half of the screen. I think Firefox 4.0 will solve my problem.

Then, using Firefox on the netbook with Ubuntu, playing Castle Age will make Firefox slow down. May be the reason is the netbook memory is smaller than normal laptop.

However, I am still using Firefox on Ubuntu. Because of the powerful extensions on Firefox, such as DownThemAll, FlashGot, etc.

Huawei E1552 on Ubuntu

I was using Huawei E220, a very old USB Modem. Yet, running on Ubuntu does not have any problem.

Then I tried the new Huawei E1552 on Windows. Nicer interface. Newer is better. But when I tried it on Ubuntu, yes, Huawei Mobile Partner icon is on the desktop. Can open it. But I cannot connect to internet.

So, in order to solve this problem, I tried to search from Internet. Right, I need USB_ModeSwitch to solve this problem. Reading the page, you will find a lot of steps need to follow.

Whatever they are, I just use Synaptic Package Manager, and install “usb-modeswitch”. Then plug in the Huawei USB Modem, yeah! Now can connect to Internet even using Ubuntu. No command-line needed. Easy.

Regular expression, find and replace using Unix tools

I need to add some text to a lot of text files. Previously, I used jEdit to perform regular expression for several files. But currently, I need to add the text for about 120 files. In order to do this, I tried using Unix tools, “sed”, “grep”, and “xargs”.

Firstly, I tried MSYS, yet it fails. Because the “sed” version was too old. Then, I used Cygwin.

My problem was to add the following text to the text files:

Background { skyColor 1 1 1 }

Since the text files are computer generated, all the text files have following text:

DEF Camera

And the text only occurred once for each file.

Therefore, the best solution is to use regular expression for search and replace to produce the following result:

Background { skyColor 1 1 1 }

DEF Camera

Therefore, I used the following command to solve the problem. (Do not copy paste, because the user must type the “newline”).

$ grep -l "DEF Camera" *.wrl | xargs -l sed -i -e 's/\(DEF Camera\)/Background { skyColor 1 1 1 } \
> \
> \1/g'

First of all, all the files are in .wrl format. So, I used

grep -l "DEF Camera" *.wrl

to get all the file names where the files contain “DEF Camera”. The option “-l” indicates the result is the file name. Then I used xargs and sed to search and replace the text (actually I am not familiar with “xargs” command). Then,

sed -i -e 's/\(DEF Camera\)/Background { skyColor 1 1 1 } \
\
\1/g'

“sed” command cannot write newline like C, such as ‘\n’. Thus, I need to type ‘\’ followed by ‘enter’ key. This will produce exact newline to the I/O stream. Unlike jEdit or PHP regular expression, one needs to use ‘\(‘ and ‘\)’ instead of ‘(‘ and ‘)’ for replacement by reference. And the ‘\1’ is used instead of ‘$1’ for the reference.

Finally, using this command helps me to add the text to about 120 files with several seconds. Good!

MinGW and Cygwin

I am a software developer. I like cross-platform development. And I prefer lower level programming language like C. My OS is Windows. Thus, I normally use Visual Studio Express Edition to compile the source code. But using Microsoft Visual C++ compiler is not a good idea for cross-platform development, because it highly depending on the Microsoft platform. Besides that, it doesn’t have C99 standard.

So, the best cross-platform C and C++ compiler is GCC (GNU Compiler Collection). However, it is used in Unix-like OS. Yet, there are two projects porting GCC to Windows: MinGW and Cygwin. These two projects provide us GCC to compile cross-platform source code such as libjpeg, libpng, zlib, etc. This indicates that, we can also write our own cross-platform source code and compiled by using GCC in Windows and other Unix-like OSes.

However, there are limitations and advantages for both projects. Cygwin is more powerful to develop the cross-platform applications. This is because Cygwin provides a POSIX layer on Windows. Therefore, it is able to compile the source code which depends on X11, pthreads, etc. As a result, we can use Cygwin to compile some of the libraries that Visual C++ cannot compile.

On the other hand, MinGW is highly depends on Win32 API. Therefore, MinGW cannot compile the source code depends on X11. And because of this limitation, several cross-platform libraries downloaded from Internet are difficult to be compiled under MinGW.

Therefore, Cygwin can be considered better than MinGW in the sense of software development. However, there is licensing issue about Cygwin. Using Cygwin to develop an application will link against cygwin1.dll. And, this cygwin1.dll is released under GPL license. Consequently, the applications or libraries linking against cygwin1.dll will be affected by GPL license. Cygwin licensing terms mention:

In accordance with section 10 of the GPL, Red Hat permits programs whose sources are distributed under a license that complies with the Open Source Definition [See http://www.opensource.org/docs/osd/  for the precise Open Source Definition and a list of the licenses certified by OSI as conforming to that definition] to be linked with libcygwin.a/cygwin1.dll without libcygwin.a/cygwin1.dll itself causing the resulting program to be covered by the GNU GPL. [Cygwin API Licensing Terms]

As a result, the applications and libraries need not be GPLed, but must be open source and conform to Open Source Definition. MinGW which doesn’t depend on cygwin1.dll allows permissive license. That means, if we develop an application using MinGW, we can commercialise the application and make the application close source. Therefore, in the sense of commercialisation, MinGW is more suitable to be used.

Actually, Cygwin has a feature allows us to build the application without linking against cygwin1.dll. We can do this by using “-mno-cygwin” during the linking of the objects and libraries. This is because Cygwin provides MinGW runtime libraries. That means we can use Cygwin as it is MinGW, but giving up the POSIX related functions.

Because of the licensing term, I am very confused. And I also emailed and asked for the information about the Cygwin licensing terms. But nobody can a definitive answer. Since Cygwin license allows linking to cygwin1.dll with the Open Source Definition, that means if library A is zlib/libpng license, it will not be GPLed. Then, the question is, if another library, B, linked against library A, which is zlib/libpng, yet A linked against cygwin1.dll, could we commercialise the library B without open source?

Still very doubtful.

Success booting PC-BSD through USB flash drive

Hahaha. When I wrote my previous post, I found that I possibly made a mistake when copying the IMG image to USB pendrive. Yes, I used “dd” incorrectly because I didn’t know how to use it exactly.

This was how I used “dd” in Linux:

dd if=PC-BSD-7.1.1-x86.img of=/dev/sdb1 bs=1024

Now, the problem was the “/dev/sdb1”. We should use “/dev/sdb” instead of “/dev/sdb1”. This is because “sdb1” indicates the partition, yet “sdb” indicates the drive. And I should copy (dd) the whole file system and data of IMG to the drive, not to the partition. That was why, after I used the statement above, I still found that my pendrive was in FAT32 format.

So, just now I tried the following statement:

dd if=PC-BSD-7.1.1-x86.img of=/dev/sdb bs=1024

Then, I successfully booting PC-BSD. Besides that, the file system from “fdisk -l” shows that the drive is in FreeBSD format. So, I realised that, the booting of an OS also depends on the file system. If I set the boot sector correctly, and the boot loader also set correctly, but the file system is different, then the booting will fail.

That is why, syslinux targets on FAT32 and isolinux targets on ISO filesystem. Though Linux kernel for the booting is already in the drive, but file system is different will cause the booting failed. So, syslinux project is targeted to solve the file system problem for booting the OS.

After I successfully booting in PC-BSD. But I decided not to install it. Because the screen resolution is too big for my netbook. @_@

Note: Remember backup USB pendrive data before “dd”, cause data will be removed.

Difficulties for installing PC-BSD

I like Unix philosophy. I like open source. I want to install PC-BSD to my netbook, but finally I realised that I failed to do so. I have tried several ways, then I give up now.

Actually, I preferred to install FreeBSD. But FreeBSD is targeted for server, so I choose PC-BSD. And PC-BSD has GUI installation, which makes everything easier (I tried it on VMware). So, I downloaded DVD ISO image. Then, I tried to use UNetbootin, since the site mentioned that it supports to create LiveUSB for FreeBSD 6.3 and 7.0. I assumed that it can also support for PC-BSD which is based on FreeBSD. However, I failed.

I found that UNetbootin uses syslinux. It is used for creating LiveUSB for Linux. Syslinux project also includes isolinux for creating LiveCD for Linux. So, I finally realised the reason why there is two different applications, syslinux and isolinux. This is because isolinux is for ISO image and it has its own file system, yet syslinux is for FAT file system. That is why I failed to use UNetbootin to boot PC-BSD in pendrive.

Then, I assumed that syslinux is for Linux installation. So, I tried to use “dd” command as following the instruction from PC-BSD Users Handbook:

To write the USB ISO file to a Flash Card or USB pen drive you can do this with the Unix command ‘dd’:
dd if= of=/dev/da0 bs=1m
Just substitute da0 with the device name of your USB stick.

Since the instruction mention the ISO file, I tried to use the ISO image I downloaded. But I still failed to boot into the PC-BSD in pendrive. Then, I assumed that ISO file system for the pendrive which is formatted as FAT32 (if I am wrong, please tell me, because I am a newbie in FreeBSD and Linux).

Next, I think the possible solution is to use the IMG image provided from the site. Downloaded, “dd”, and I still failed. So, I give up now. And continue using Linux.

(I think may be I used “dd” for the “of” argument incorrectly, I will try out again tomorrow.)

I personally still have an idea to boot the PC-BSD through USB pendrive using syslinux. But I didn’t tried out. Since it is possible to boot the USB pendrive with syslinux, and it is possible to boot FreeBSD through GRUB (or GAG, or LILO). So, if it is possible to use syslinux to boot to GRUB as a boot manager, then from GRUB to boot into PC-BSD, then installation is possible.

So, the conclusion based on my experience is:
PC-BSD is still not yet matured enough for LiveUSB installation and netbook. Unlike Linux distributions, there are several Linux distributions targeted on netbook, which allow users to install easily.

Note: Remember backup USB pendrive data before “dd”, cause data will be removed.

gparted from Parted Magic

The story continues from my previous post.

I used Parted Magic to perform the partition tasks. However, I found that Kingston DataTraveler 2.0 USB Device cannot boot as LiveUSB. It cannot find pmagic-4.5.sqfs. And even Parted Magic 4.8, there is something like dev_list cannot be found. But, there is no problem for Kingston DT 101 II USB Device. Weird!

Okay, after I delete E: F: and resize D: to occupy the space. But in Windows, I found that the D: properties show that the space is still the same. But in Disk Management utility, the space is already used for D:. I tried CHKDSK, the problem still there. Then I remember a phrase from a movie:

成也风云，败也风云
(Success with “wind cloud”, fail with “wind cloud”)

So, again, I use gparted to check the partition (D:). And yes, it is fixed now, with the actual size.

GRUB problem

Playing around my HP Mini Netbook with different OSes really troublesome.

Today, I am trying to install PC-BSD on my netbook. But before I install PC-BSD, I need to make some space for it. And according what I read,

Be aware that BSD operating systems, and hence PC-BSD, only recognise primary partitions and consider any logical partitions as a whole primary partition. Trying to install on a logical partition will convert your extended partition into a primary partition and erase all logical partitions of your system. PC-BSD can be installed on any primary partition; it doesn’t necessarily have to be on the first one. [PC-BSD guide]

However, the following is my partitions:

Firstly, since I accept the concept that “data” is the most important thing. So, I just want to merge D:, E:, F: drives. So using Parted Magic, I delete E: and F:. Then, resize D: and left about 15G for unallocated space. The unallocated space I want to use for PC-BSD. However, since my harddisk is initially formatted with primary partition and Ubuntu in the logical partition, the unallocated space between D: and Ubuntu cannot be create as a primary partition. So, I move Ubuntu partition next to D: so that the unallocated space will be at the end of the harddisk.

Now, the problem comes. When I restart the netbook, GRUB cannot run properly, it goes to “grub rescue”. And I don’t know how to use GRUB. I tried to study about GRUB, grub-install is “definitely less safe, because there are several ways in which your computer can become unbootable” (GRUB manual). I don’t want to take risk. So, the only solution is to reinstall Ubuntu. Haiz….

Ubuntu Netbook Remix USB detection

As a newbie to Linux, using Ubuntu Netbook Remix gave me some trouble. Because I found that, when I plug in the USB pendrive, there is no response. So I cannot mount it manually, and there is no information in /dev drive.

Search for a long time for the solution, study a lot, read the forum. I found some solution like “udev”. However, I already installed udev, but there is no detection found. Finally I join Ubuntu IRC to ask for information, but there is no solution yet.

The problem I face is like this:
After I login to Ubuntu, then I plug in the USB pendrive. There is no response. I read the /var/log/messages, there is the detection of the USB as sdb. But I tried to look for the /dev/sdb1, but it is not existed. So, I can not mount the partition. So, I tried to use “fdisk -l”, right, there is no partition in sdb. And I tried to use “lsusb”, there is the information about my pendrive. This is annoyance. If I reboot Ubuntu without removing the pendrive, then the pendrive is automatically mounted. The udev is already installed.

After joining the IRC, one of the friends ask me to logout then login. I haven’t tried it. But then, when I tried to do so, after inserting the pendrive, after a while, the pendrive is automatically mounted!(???!!) I also don’t know why.

Then, I tried to plug in the USB mouse, but the mouse cannot detected automatically. So, though the USB pendrive is solved, USB mouse still not yet be solved.