Arch Linux manual recovery

Though SystemRescueCD is powerful, if it is system specific, then it cannot work. With the recent changes of the Arch Linux filesystem, I made a mistake that I cannot start Arch Linux after a careless reboot.

The instruction from the official site mentions,

1) Fix any non-official packages with files in /bin, /sbin or /usr/sbin to put those files in /usr/bin.

The term “fix” does not give me too much solution about the packages. These packages mostly come from AUR. What can I do to “fix” them? There is no much hint for me. Thus, I ignore this part, though I can list those packages.

So, I followed the instructions until the step “pacman -S filesystem”. This shows the conflict because of the /usr/sbin. Therefore, I assumed that reboot may “solve” the problem. But this is the cause of the problem.

As a result, I cannot boot into the system anymore. And I know that, this is the package installation error. Even if I use SystemRescueCD, it cannot help for package installation, because it does not have pacman. Luckily, I have a bootable pendrive with Arch Linux installation image. So, I can only recover with this pendrive.

Fortunately, Arch Linux now has wifi-menu (from netctl package), which is so useful that I can access WiFi with text-based interface. WiFi connection by commands is exhaustive. After connection, then I use the elinks to browse the related pages. I am so happy that the pages like Wiki is working gracefully with the text-based web browser.

I tried chroot, but it does not work. Finally, I mount the partitions, move the /usr/sbin to another location, then use the pacstrap as in the Wiki to install the base packages. Now, reboot, everything work fine. Then I moved the sbin just now to the /usr/sbin again.

So, if you don’t have desktop environment, don’t worry. Firstly, we need to know how to switch the TTY with Alt+F1 (or F2, F3, F4, …). This is helpful just like we switch the windows with Alt+Tab. By using this, we can have a screen to read the webpages with elinks, a screen with manpage of a command, and a screen with the commands that we are going to perform.

Secondly, we need to know how to access internet from command-line, especially the WiFi. The wifi-menu from Arch Linux simplifies a lot of work. Else we need to know the commands like “ip”, “iw”, “dhcpcd”, and so on.

Thirdly, we need to know how to use some text-based applications, especially “elinks” or “links” as web browser. The text editor such as “vim” or “nano” is useful. To view the file, “cat” with “less” is useful. Furthermore, “man” or “info” is helpful to understand how to use certain commands.

With these knowledge, then system recovery should have no much problem.

Portable OS with Qemu (partially success)

LiveUSB is too nice, especially for recovery. KNOPPIX or Puppy Linux can also be used as OS temporary to do the work and save the file. However, LiveUSB is too distro dependent. Thus, I am wondering how to prepare a portable OS by virtualisation through Qemu.

The advantages of this idea is to

• allow the user to install any OS, meaning that the user can install any packages to do any work.
• have an OS in an FAT32 formatted pendrive where the OS can have the size exceeds 4G.
• be portable as a portable application.

However, there are some disadvantages.

• Qemu virtualisation without KVM in Windows is slow.
• Pre-compiled Qemu may not up-to-date and some DLLs may be missing.
• Compilation of Qemu may needed.
• Only the OS which has LVM support can work.

Since Qemu is open source and cross platform, assuming that Qemu can work on any OS, then this method is theoretically worked. However, there are several things need to be considered when preparing the OS image(s).

Firstly, because FAT32 does not allow file size exceeds 4G. Therefore, we need to use qemu-img to create several disk images less than 4G. However, Qemu can only load 4 images as the harddisk at most, i.e., -hda, -hdb, -hdc, -hdd. Thus, it will have the difficulty to prepare an OS more than 16G (4 x 4G). Yet, based on this page, it is possible to solve this problem. As a result, I solved this problem by options such as

qemu-system-i386 -m 1024 -hda hd001.img -usb -usbdevice disk:hd002.img \
-usbdevice disk:hd003.img \
-usbdevice disk:hd004.img \
-usbdevice disk:hd005.img \
-usbdevice disk:hd006.img

By using -usb and -usbdevice, Qemu can have more than 4 disk images.

Secondly, and luckily, Linux supports LVM. By using LVM, we can just combine these virtual hard disks together and use as a single large hard disk virtually in LVM. Then we can partition this LVM hard drive as any way we want. Before preparing LVM hard drive (physical volume), please see the next important point.

Thirdly, the partition for boot. I tried to create /boot in the LVM physical volume, it failed to boot from GRUB2. Therefore, my solution is separate the /boot as one standard partition, then the others remain in LVM physical volume.

Fourthly, GRUB2 is needed.

As a result, I can boot into the portable OS using Qemu in a FAT32 from a Linux OS. Everything works fine in Linux. However, I still not yet figure out booting this portable OS from Windows using Qemu. Thus, this method is partially success.

Change a computer but preserve the OS (Arch Linux) and data

This is interesting. Previously, if I want to change a computer, I will (re-)install the OS and other software on the target computer. Then only move the data.

Since Arch Linux is a minimalistic distro, and installation only available with network connection. This will be exhaustive to install the OS on the target computer. Not only wasting time downloading, but also setup the configurations, such as web server, FTP server, hibernation, power save, etc. User’s configurations are easier, because I can just copy all the files in /home/$USER directory to the target computer. The only problem may be the differences of the owner ID and group ID. But that is not really a big deal. So, the main problem is the root (/).

Cloning the partition is not a good solution. Firstly, the target computer is different, I wish to preserve the original OS. If cloning the partition, the target computer needs to have extra space for the cloned partition. Secondly, configuration such as GRUB, fstab, and other partition related configuration will not work. To fix them, I need to find all these configurations, but how do I get them?

Luckily, Arch Linux is a minimalist distro, which has its great advantage. So, this is what I did.

Firstly, I get the latest installation ISO. Then, partition the target computer with SystemRescueCD. (I tried with LVM, but at the end failed, because the first partition is started at sector 63. Installing GRUB where the /boot is in the LVM requires larger partition space.)

Then, boot into the Arch Linux live media, and follow all the installation steps as in the official installation guide. As expected, the installation of the “base” requires network connection. It is around 100+ MB. (I really hope that, if it is possible, Arch Linux official installation ISO will contain at least “base” group, so that it is possible to install a minimal OS without network connection.)

After installing everything, I made sure that the original OS and Arch Linux are bootable through GRUB.

Then the essential part is transferring the whole data from old computer to new computer. Firstly, I need to transfer the data from old to an external harddisk. Thus, I used the following command in old computer (based on this page),

tar -czpvf backup.tar.gz --one-file-system --exclude=/backup.tar.gz \
--exclude=/dev --exclude=/tmp --exclude=/sys --exclude=/proc \
--exclude=/mnt --exclude=/media --exclude=/boot \
--exclude=/etc/{hostname,fstab,localtime,locale.conf,vconsole.conf} /
#exclude any folder that we don't want

Tar is the best solution (for me) to preserve all the file permission and ownership. Besides that, tar also allows to compress the data (with gz). If the size doesn’t matter, we can omit the “-z” flag and the file can be named as “backup.tar” without “.gz” suffix. So, several directories (folders) are excluded, because these may be computer dependent. The /boot must be excluded, because the partitions of the computers are different. And finally some files in /etc needs to be excluded. These files are actually the files which we need to edit during the installation of the Arch Linux, especially fstab. (That is why I said luckily Arch Linux is minimalistic.) Then, use the same method to backup the /home and any other data.

Next, to extract the data from the tar, I didn’t use the Arch Linux in the new computer itself. This is because I afraid that overwriting some of the files (especially the running process) are not allowed or causing damage of the system. Thus, I use SystemRescueCD and mount the root partition of Arch Linux, Then,

tar -xzpvf backup.tar.gz -C /mnt/arch --numeric-owner

Then, I also use the same way to extract other data.

Finally, restart. Everything work fine. The only thing I need to reconfigure is the /etc/suspend.conf from uswsusp-git because of the partition differences. Besides that, I also reset the pacman-key as mentioned here.

Extracting files from zip which contains non-UTF8 filename in Linux

Previously, I have made a post about extracting SHIFT_JIS filename encoding in zip file.

However, this method does not work when the filenames contain GBK (simplified Chinese) encoding. As a result, I found a general solution for the non-UTF8 encoding.

The method is almost the same, but more generic way.

Firstly, the problem we face is after extracting the files, the filenames are unreadable. Not only that, we cannot convert the filename even we are using “convmv”, “iconv”, or even “uconv”. This is normally caused by our OS locale setting. To make our OS (Linux) more generic to read almost any languages (East Asia languages, right-to-left langues, etc), our OS is normally has the UTF8 locale. It may be en_US.UTF8, ja_JP.UTF8, zh_CN.UTF8, zh_TW.UTF8, en_GB.UTF8, etc.

The problem we extract the non-UTF8 filenames from zip in UTF8 environment will cause our filenames irrecoverable. This is because the non-UTF8 is write as UTF8 without any conversion. That is why, when we want to convert the filenames with these mojibake, it is always fail, no matter we are in UTF8 or non-UTF8 environment. Because we are not using the correct encoding when extracting the file.

But (in my opinion) there is no extraction tool allows conversion of the filename. As a result, we need to preserve the default encoding from the files, to write in our UTF8 environment.

In my old post, I use a non-UTF8 language, that is ja_JP for extracting SHIFT_JIS. Thus, a more generic way is extract our filename with LANG=”C”. That is ANSI C language without any UTF8 encoding.

env LANG=C 7z x file.zip

As a result, you will see the filenames has a lot of question marks.

Then now, we can convert the file with our real encoding.

convmv -f gbk -t utf8 --notest -r * #for filename which is GBK coded

Stable release, cutting edge scheduled release, or rolling release?

Again, choosing the Linux distro, based on different philosophies, we may choose a different distro, though they are all using Linux kernel.

I would like to use Debian to represent stable release Linux distros, Fedora to represent cutting edge Linux distros with scheduled release, and Arch Linux to represent the rolling release. If you visit DistroWatch.com, you can see the difference of the packages version (Debian, Fedora, Arch Linux), where Debian has very few green colour packages, Fedora has a lot of green colour with some red colour packages, and Arch Linux has a lot of green colour but no red colour packages. The green colour indicates the latest (stable) release version of the package; red indicates the beta version (development or testing).

The stable software is normally what an end user always wants. If you are looking for the performance, stable Linux distros is always what you need. Especially if you are choosing a Linux distro for server use. The more stable a software, the less bugs it has, because they are fixed. However, the drawback is that the user may not experience the latest features of the packages. For example, a new web technology is introduced which can be run by the modern web browser, as a result the users cannot try the technology immediately.

The Linux distro like Fedora, which contains latest stable release and also the beta release, is also important. Because we need to know that, the software needs to be tested to discover the bugs, and then fix the bugs. If there is nobody test the software, the bugs will not be found, and if the developers release it, it is still buggy as the beta version. That is why, Linux distro like Fedora is very important to experience the latest packages.

The rolling release becomes more and more popular recently, such as PC-BSD adopting the rolling release. The rolling release like Arch Linux, allows the users to upgrade the system to the latest stable release, without a full installation like Windows, Ubuntu, and else. That is why rolling release is also cutting edge. Unlike Fedora, it does not have the beta versions as in the DistroWatch.com. However, since it is cutting edge, some users may claim that it is not suitable for server. Yes, we need to know that, the bugs are everywhere, the only difference is the criticality of the bugs, whether it affects the whole system to function or not. Because it is a rolling release, most of the bugs are come from the upstream (original developers). As a results, this can help in the bugs report for the upstream to fix them.

Even you are not using rolling release, most of the Linux distros allow the users to access the repositories for testing. This also allows the users to have a look on the latest packages.

So, if you are looking for the latest packages, you may try rolling release distros, if you are afraid of bugs, you can  try stable distros, and if you are a developers and not afraid of bugs, you can try to use the beta version or testing version from the repositories.

In my opinion, if you are targeting a server, stable and non-rolling release is more preferable.

Choosing Linux distros

Linux is something wonderful, which until today I am still learning. Not only the command-line, but also filesystem, distributions, philosophies, open source, packaging, Linux kernel, desktop environment, etc…

It is very interesting to know why there are so many Linux distros. As I was a newbie in Linux, I was thinking on how to choose the distros and what are the differences of these distros. Some users state that Linux is the kernel, not a Linux distribution. And even some users say that whatever Linux distribution, it can be customised until identical to the other distribution (interface only).

There are a lot of reviews and comparisons on the Linux distros on the internet, I will not go through those things.

Philosophy

Firstly to choose Linux, besides we need to try the distro itself, we also need to know the distro’s philosophy, or the purpose of the Linux distro project started. There are so many distros, the main reason is the project leaders have different ideas. That is why they start the project to build their ideal OS using Linux kernel. Some distros adopt open source software and format only, as a result no proprietary software is supported. Then the users may need to get the software through other repository or download the binary files and install. The most prominent issue is the multimedia codecs. For example, OpenSUSE and Ubuntu CDs do not contain the restricted multimedia formats. However, the users can still install these codecs after OS installation. Some of the distros are more lenient with the license, such as Linux Mint or even Arch Linux.

Packaging (distribution)

The philosophy decides how the Linux and the other packages are distributed. Some distros target for the server, some for desktop computer, some for live media only (USB flash drive), some for old computer, some for gaming. Because of the target audience, the developers will package the related software to distribute to the users.

Based on the philosophy, some of the distros will distribute latest packages, or cutting edge packages (such as beta version), or old and stable packages. For example, Debian is well-known for the stable packages; Fedora is well-known for the cutting edge packages.

Package manager

In order to distribute the packages, a package manager is a must. That is why, there are Debian-based, RPM-based, pacman-based, or others. The purpose of the package manager is to manage the packages. It can get the packages from the repository. We can use it to install and uninstall the packages.

Each Linux distro should only use one package manager. It is possible to install several package managers in an OS. It is even possible to run the other package manger and install the package to the OS. For example, let say I am using Arch Linux, I get the source for aptitude (Debian’s package manager) and compile it. Then I can install the packages from Debian’s repository. However, the effect is not good. This is because the packages are built for target distro (for instance, Debian). Even the binary executable file may be compatible, the filesystem hierarchy may be different. In practical, using multiple package managers normally do not work, this is because the package manager will check for the dependencies. Therefore, do not think that you can have an OS which you can just get any packages from any repository.

(But please remember that, you can always build your own package from the scratch. However, some of the application really depending on the distros.)

Patch

Linux is an open source kernel. As a result, some developers may patch the Linux kernel in order to do something special. For example, linux-ice from Arch Linux AUR, it is a patched kernel to support TuxOnIce (suspend and hibernation features). So, no need to surprise why some distros can work like this, some distros cannot work like this.

Similar to the Linux kernel, the kernel modules (normally drivers) can also be patched for special purpose.

Community

Though the community is not part of the Linux distro, it has some influences on the distro. When we have the problems dealing with the distro, we can get some help from the forum. Each distro has different community. We should not ask the questions about other distros in an unrelated forum. Nobody can help. And no one likes to help. Because you are in a wrong channel.

Documentation

Similarly, documentation is not part of the Linux distro, but it is part of the project. Well documented information is very important for the self-learning users.

 

Therefore, to choose the Linux distro, we need to know the philosophy of the distro. Find a distro which fits to our appetite. If there is any problem, read the documentation and ask the correct community. Always bear in mind that, bugs are always there. Even we are using Windows, there are also bugs. The only problem is the severity of the bugs and how long will the bugs be fixed. Use the channel such is IRC, forum, bugs report, or mailing list to help us to utilise the OS better.

And always bear in mind, we can change to another Linux distro any time when we want.

Why not Internet Explorer?

Firstly, I want to state that I am totally anti (dislike) Internet Explorer. There are several reasons that I don’t like Internet Explorer.

Internet Explorer version less than 9 has a lot of problems with CSS. As a result, as a developer, developing the webpages for among IE and other web browsers is exhaustive. In order to make the compatibilities, the CSS is almost web browser dependent. Besides that, CSS3 which is supported by most of the modern web browsers, does not work well in Internet Explorer. Luckily there is a trick using PIE. But this is not the best solution.

The next problem of the IE is the upgrading problem. To upgrade the IE, it is depending on the OS. To use IE8, the Windows must be at least Windows XP. This is not a great problem, because Windows XP is not yet extinct. But to use IE9, the Windows must be at least Windows Vista with SP2 (Service Pack 2). That means, whoever uses the Windows XP cannot upgrade IE to version 9. Next, to use the latest IE10, the Windows must be at least Windows 7 with SP1. That means, whoever uses Windows Vista cannot use IE10. As a result, the users can only choose to upgrade their OS or computer, or remain using old and good for nothing web browser which cannot cope the current web technologies, or lastly change to another web browser.

As a developer, to develop the webpages for web browser independent should be easy, because of the standardisation of HTML, CSS, and XML. However, it will be difficult because of IE. Moreover, to develop the webpages for different IE versions is even sillier. If using IE8 and turn on the Developer Tools, we can choose IE8 standards or IE7 standards. This is something interesting, one web browser with two different standards for development. This indicates that webpages for IE10 may different from IE9 or IE8 or IE7 or more.

The best reason not to use IE is because there are much more better alternatives: Firefox, Google Chrome, Opera, or Safari. These web browsers are free. Besides that, they are platform independent except Safari.

Why do I need to spend extra money to upgrade the computer in order to get a latest OS because of the desire to use the latest web browser? Changing a web browser is cheaper than changing an OS and even more cheaper than change a computer. So, unless IE is cross platform, following W3C standards, and remain free, I will not use IE or develop the webpages targeting on IE.