Real hackweek, protected hackweek and long hackweek

I have been using KVM a lot, but never took time to understand how kvm works. I used some time from this hackweek to get rid of that regret.

Virtual Machine eXtensions instructions allow trap-and-emulate virtualization. And KVM exposes VMX in a convenient way to userspace in Linux. Virtual Machine Monitors(VMM) like qemu-kvm use the KVM API exposed by linux to emulate virtualize software.

x86_64 processors boot in real-mode. In this mode it can use only 16-bit addresses, ie., upto 1MByte RAM. The execution would begin at physical address 0xFFFFFFF0. Then the software has to switch to protected mode where protection and paging is possible. Paging is optional, but almost all OSes use demand paging extensively. Now 4 GBytes of linear address space is used. And then CPU can be switched to long mode i.e., 64-bit mode. Paging should be disabled in 32-bit mode, before switching to long mode. There are also other modes of operation like virtual-8086 mode to allow executing legacy real-mode software from protected mode, SMM for OS transparent execution of OEM specific code.

I had limited time and very very limited skill at hand. So aiming for the sky was not an option. Hence wrote a very simple VMM that directly starts the guest at address 0H, in 32-bit protected mode with paging disabled. And supports only insb and outsb as the only form of interaction possible for the guest. The guest is a simple static linked 32-bit program that doesn't use any library, and linked to start from 0x0. The guest simply reads a byte using insb and sends byte+1 back via outsb. The guest would halt, when it gets the, "Answer to the Ultimate Question of Life, the Universe, and Everything". The VMM reads the byte value to pass to the guest from stdin and prints its response in stdout.

The KVM API is really very easy to understand and use. But some knowledge of the processor was required to make use of it. Intel manuals helped there. I don't have good understanding of things yet, but something is better than nothing.

I was occupied by quite a lot of things in life and work in the recent past. So I wasn't really planning to participate and make this a real hackweek. Also a National holiday for Republic day of India, bang in the middle of the week prevents this hackweek from being a long hackweek! But seeing videos of my colleagues from various parts of world having fun, I couldn't resist and decided to go for the virtual hackweek. I thank my employer for giving me this protected hackweek, and let me learn/do things protected from everyday work.

Hackweek V

I had been toying with the idea that, to split a file on my hard-disk, I have to read and write to a new file and then truncate the old file. But isn't it lots of unnecessary I/O. I already have all the data on disk. I should be able to change the meta-data alone and mark the file contents after some length as a different file.

So during this hackweek, I implemented 2 system calls on Linux, sys_split and sys_join. And added support for these calls to the FAT file-system. http://lkml.org/lkml/2010/6/9/200 With this patch one can split a file into 2 or join 2 files without doing much I/O!

Why FAT?
Normally creating a new file and then truncating a file approach, needs temporary free space as well. I once had a need to split files when there was no free space available. On advanced filesystems, sparse-file support(i.e., files with holes) helped. But FAT does not support sparse files. Also I wanted these to work on thumb drives which are mostly formatted as FAT32.

SLE11 SP1 release
This Hackweek was scheduled to follow the release of SLE11 SP1. And all the Bangalore employees, who worked for SLE11 SP1 were given a portable 160 hard-drive as a gift in the middle of hackweek. The transcend disk has a one-touch back up button, which works to sync selected folders, using a proprietary software available only on Windows. No support in Linux or Mac.

gnireenignE one-touch button

I thought, if the button press could be detected, I could use it to unmount the disk! So I set out to reverse engineer it. Usbmon + KVM + windows XP. Got the usbmon traces. I was expecting to see a simple Interrupt endpoint. But it was a bulk end-point only interface. And the software was continuously polling to get the button status. It was quite interesting to decipher the USBS and USBC's, but the SCSI/ATAPI payload had an unknown command DFh. And I assumed that by sending the command, if a button had been pressed in between, I would get a different value. And was trying hard for hours. But failed. Then took few more traces using Windows. Tried hard. When I almost gave up, I found couple of return bytes slightly different! Voila. The return value is different, only if the button is held down during the command. Now, I have a script to sync/unmount with a one-touch button! If you have this device, you can modify the script to do what ever you want it to do. Let your creativity flow freely.

Who is Linux?

Yesterday, I was bored in the afternoon and had a digital camera with me. I thought of making my own steadicam, but then when I saw the tux over my TV, this contest came to my mind. And the result is an entry to the competition.

I spent only few minutes. I think it is quiet OK, given its very low production value. I am not trying to win, but just like I send crappy patches to Linux and waste kernel developers time, I uploaded it to waste 43 seconds of your time.

Back from National Free Software Conference - 2010

I, along with Chen, the Maintainer of Gnome Evolution project gave a talk in National Free Software Conference - 2010, Bangalore. Our talk was titled "Developing a Free software, Inside story"

I talked about who all are developing Linux Kernel, and how one can get started etc... Then Chen talked from his perspective. It was an inspiring speech, even I felt greatly motivated by him. He started with how he was excited, when he made changes to the software he used and showed it to his friends when he was new to open-source. Later he talked about Gnome and Evolution community. In the end he had lot of fans around him, and it took him nearly an hour to leave the hall after the talk!

Before our talk, we attended couple of talks. "Free Software Business Model – Software as a service" by someone from CGI. It was the usual corporate talk explaining CGI's business and then what is SaaS, PaaS, IaaS,... There was "Debate on Open Standards draft 2.4" by Venkatesh Hariharan, Policy Director, Red Hat and Prabir Purkayastha, Knowledge Commons. It was about Indian government's policies regarding mandating open standards and formats.

I came to know about this conference only 2 days ago for the first time. It was hosted by the Free Software Movement of Karnataka(FSMK), co-sponsored by Dept of IT Karnataka, Kerala IT Mission and Govt of West Bangal. The punch-line of the conference was "Free Software is the Future... The Future is ours..." I predict in future, Conferences won't provide network as everyone would have mobile Internet themselves. And this conference was ready for tomorrow already!

ullae-veliyae: Live graphs for I/O, CPU...

Wrote a small utility to print live graphs of I/O per-process. Find more information here. As a bonus added graphs for CPU as well.

RPM can be found here: http://download.opensuse.org/repositories/home://nikanth/openSUSE_11.1/noarch/

Cloning multiple git repos

Many of the maintainers of linux-kernel maintain a git repository. I usually have clones of various such repositories.

For example I have clones of
Linus Torvalds's repo:
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
Block Maintainer, Jens Axboe's repo:
git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-2.6-block.git
...

If I do individual clones of all these repositories, it downloads and maintains duplicate copies of same objects wasting disk space, and network bandwidth.
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
git clone git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-2.6-block.git

So I was looking for a way to share the common objects so that duplicate objects wont waste disk and network. And no surprise, git has a way to do that. Just that I was unaware of a simple option, "--reference".
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
git clone --reference linux-2.6/ git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-2.6-block.git

---

Difference between cloning Jens' git with and without --reference to Linus's git.

# git clone git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-2.6-block.git
Initialized empty Git repository in /home/knikanth/labs-sw/linus/linux-2.6-block/.git/
remote: Counting objects: 1180249, done.
remote: Compressing objects: 100% (295444/295444), done.
remote: Total 1180249 (delta 984716), reused 1073684 (delta 878311)
Receiving objects: 100% (1180249/1180249), 289.32 MiB | 496 KiB/s, done.
Resolving deltas: 100% (984716/984716), done.
Checking out files: 100% (27842/27842), done.
# du -sh linux-2.6-block/
714M linux-2.6-block/


# git clone --reference linux-2.6/ git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-2.6-block.git
Initialized empty Git repository in /home/knikanth/labs-sw/linus/linux-2.6-block/.git/
remote: Counting objects: 111061, done.
remote: Compressing objects: 100% (19021/19021), done.
remote: Total 100463 (delta 84138), reused 95679 (delta 79959)
Receiving objects: 100% (100463/100463), 23.21 MiB | 1209 KiB/s, done.
Resolving deltas: 100% (84138/84138), completed with 8189 local objects.
Checking out files: 100% (27842/27842), done.
# du -sh linux-2.6-block/
468M linux-2.6-block/

---

--reference automatically sets up .git/objects/info/alternates to obtain objects from the reference repository. Now I wonder whether it is possible to have circular references, multiple references, etc.. The plural file name, "alternates" suggests it should be possible, but "git clone" ignores multiple --reference on the command line!

BTW git uses SHA-1 digests to identify objects. I wonder what is the chance of a SHA-1 collision and how git handles it? The SHA-1 digest has 40 Hex-digits == 160 bits.. So at most, only 2¹⁶⁰ objects are possible. :-)

Type checking macros

#define to_cpumask(bitmap)                                              \
     ((struct cpumask *)(1 ? (bitmap)                                \
                         : (void *)sizeof(__check_is_bitmap(bitmap))))
static inline int __check_is_bitmap(const unsigned long *bitmap)
{
     return 1;
}


The above macro is nothing but just
#define to_cpumask(bitmap) (struct cpumask *)(bitmap)

But it also does a compile-time checking that the parameter passed is of type
(const unsigned long *). If it is defined as an function all these ugly magic is not
needed. But a function cannot be used as a static initializer. Try declaring a variable,
static int a = printf("ok");
and one would get an error, "Initializer element is not constant".

I really wonder whether gcc might one day optimize out your type-checking call to the unused
function in tha above macro?! A solution could be providing an official gcc extension to assert types?!
Also the above code compiles fine without sizeof constification as well i.e., just return a void * from the
__check_is_bitmap() and remove the sizeof and casting.

I found the above code in the linux kernel. See http://lkml.org/lkml/2009/3/25/22
for the discussion. Even the ubiquitous printf, printk or any vararg code does not check for types. So
I wonder whether kernel developers need such tricky^Wugly code just for type checking, given that
the callers would likely read the definition of the macro as well. It is not a closed api.

Any other project uses such type-checking macros?

Leanux - Running Linux on FAT just for fun

Most of the flash devices assume that fat is the only file-system. And there are plenty of electronic devices that run linux on fat. So just to know what it is to run linux on vfat, I tried it myself.

The openSUSE 11.1 installer wont allow you to partition the root as vfat. So I made 2 partitions, one ext3 and one vfat. Installed on ext3. Copied the whole partition to fat. I copied original files in place of symbolic links. Added vfat filesystem module to initrd . Modified grub boot from the fat partition with init=/bin/bash.

FAT's lack of support for posix file-permissions. was easy to overcome by mounting with all permissions to everyone, without users or noexec. It doesn't support symlinks, and special files. mknod will fail. But /dev is a tmpfs and it just copies the persistent files from /dev to the tmpfs on boot, instead of copying I created fresh device files in tmpfs. `mount` failed trying lock the file /etc/mtab. Mounting with -n worked. I guess plenty of things would fail if I try to run a proper desktop on FAT as it is.

I booted only to /bin/bash. When I tried doing a "exec init 1", it complained something like cannot remove /var/run/do_confirm. I didn't proceed further. This was a fun way to kill time while getting some insight of the booting process.

Linux Nikanth News (2)

After I posted Linux Nikanth News, the next LWN weekly edition's kernel page has Andrew Mortans reply to me as part of the Quotes of the week.

I was at the receiving end from the master for not writing good patch description. And I do not seem to have learned from it! :(

p.s: Earlier by mistake all of my posts were syndicated in planetsuse. Now it has been changed to aggregate posts with a specific label. Hopefully this post wont show up anywhere else other than in my blog. If you are seeing this elsewhere, please leave a comment.

Latest and greatest(untested)?!

Linus released

• 2.6.24 on January 24th 2008,
• 2.6.26 on July 13th, 2008, and
• 2.6.27 on October 9 2008
  

Here is when it reached the users

• Gentoo 2008.0 released on July 6, 2008 has linux 2.6.24 which was ~5.5 months old.

• Ubuntu 8.10 released on October 30th, 2008 has 2.6.27 which was ~0.6 months old.

• Fedora 10 released on 25 November 2008 has 2.6.27 which was ~1.5 months old.

• Opensuse 11.1 released on December 18th, 2008, has 2.6.27 wihch was roughly ~2.3 months old.

• Debian 5.0 released on February 14th, 2009 has 2.6.26 which was roughly ~7 months old.
  

Distros used by power-users seems to be always running slightly older version compared to the distros aimed at the layman! Yeah, but power users^Wdevelopers use the unstable/development/factory/head version of their favourite distro and not the released stable. But Ubuntu stable(!) is based on debian unstable! And debian stable is so outdated. opensuse seems to be quiet the latest but not straight from the unstable development snapshot.

p.s: I use only opensuse regularly among these distros. So any mis-information and bias is likely. ;)

Linux Nikanth News

I do not post many things related to computer science or what I do. So just to update some of my friends here are couple of LWN articles where I was mentioned.

I was mentioned for kreplace - hotpatching running kernel in Nov 24. And recently for the cgroup based oom killer controller Also note that both the projects were not merged in the Linus tree, at least not yet. ksplice is superior to kreplace and it seems to be on its way for inclusion, and user-space notification seems to be the preferred approach of oom killer control.

/me plans to blog more tech stuff soon