Below is an email conversation I had with Mario Smit about the design rationale for sanos. It is a discussion of JavaOS architecture in general and sanos in particular. It also presents and explains some of the ideas which were the foundation for the design approach in sanos.
From: "Mario Smit" <mario dot smit at hccnet dot nl>
To: "'Michael Ringgaard'" <mri at jbox dot dk>
Sent: Thursday, June 26, 2003 9:24 PM
Michael,
[...]
Already some years ago I came to the conclusion that a lot of effort was going into writing layers and layers on top of each other in modern programming. Mostly trying to bring object-based abstraction layers on top of C-style interfaces. Like .NET on top of COM+, COM, Windows API, Kernel32, etc.
It would be nice to strip away all that layers and let a byte-code language run on top of the hardware. Which, if carried out properly, would be even pretty quick, recognize the hardware and generate fast machine-code. The JVM would control process-boundaries and security.
One approach to this was JavaOS of SUN, but somehow SUN stopped this project. Another approach was JOS with a mini-kernel and a JVM, they stopped too. Then the JX-project got my attention where they try to run byte-code already in the kernel, where even device-drivers are programmed in Java. Then I came across your project consisting of a mini-kernel and the normal Hotspot-VM of Sun.
The major problem with all the other approaches was that they try to build their own VM, which is incredibly complex. The Kaffe project is already busy from '97 and still they cannot run all Java programs. Another thing that I like about your approach is that you use the MS-environment to edit and build the project. In my own attempt to do something like this, I used DevStudio to edit and Cygwin with a cross-compiler to create ELF-libraries which could be started by the GRUB-bootloader in VM-Ware. But this was horrible because all this switching back and forth.
Recently Sun released the source-code of the JDK, including the Hotspot-VM. I downloaded it under the research-license of their community effort. I would like to tinker with it a bit to let it run immediately in kernel-mode, like in the JX project. I think sanos would be a nice kernel to let it run in. Then you would not have to emulate Kernel32.dll, MSVCRT.dll, etc. In the end I would like to let the kernel also recognize other bytecodes, like the .NET bytecodes.
Of course my approach has some pitfalls, one will be the licensing problem of Sun. As long as you do research you can do whatever with the JDK source-code. When you want to share your work with others in source or binary form you have to ask Sun before being able to do so. Maybe this will change in the future.
Anyway, if you have any thoughts about this, I am interested.
Best regards,
Mario
From: "'Michael Ringgaard'" <mri at jbox dot dk>
To: "Mario Smit" <mario dot smit at hccnet dot nl>
Sent: Saturday, June 28, 2003 8:56 PM
Subject: Sanos design rationale
Hi Mario,
I totally agree with your comments about the unneeded complexity introduced by the multiple layers of software that does not contribute to the functionality of the system. These layers also often introduce performance bottlenecks, security holes as well as stability and scalability problems. Also the impedance mismatch between the layers often results in lost functionality, i.e. features of the underlying layer not being exposed through the next layer. As you stack layer upon layer this problem multiplies, resulting in suboptimal solutions.
My experience as a system architect during the last ten year is mainly from designing business application server platforms. During this period I have come to the conclusion that the main (technical) obstacle to developing robust, secure, and agile systems is coping with complexity. It therefore seems natural to try to remove some of the complexity introduced by the unneeded layers.
As each layer introduce a new level of abstraction you could be led to believe that by removing layers you will be confronted with an "abstraction gap". In my experience this problem is often not as severe as you might expect. These layers are often the result of historic evolution rather than by a coherent design choice. As an example you might take the C runtime (CRT) library in Microsoft Visual C. The CRT is built upon the Win32 API, but originally this API was part of the native Unix API (actually the MSVCRT is derived from Xenix). This means that the low level I/O routines (open(), close(), etc.) has to be built/emulated on top of the Win32 API. The rest of the I/O routines (e.g. fopen(), fread(), etc.) are then built on top op this emulation layer, and it gets even worse when you add various object oriented abstractions on top of this.
This leads me to the HotSpot JVM. It is built using the MSVCRT for basic ANSI C runtime library support and Win32 functions for virtual memory management, thread control, exception handling and socket communication. It uses an internal layer (HPI) for platform dependent functions. Low level I/O functions in msvcrt are used for file I/O. The stream I/O functions are only used for debugging output. The strategy for making HotSpot run under sanos was to make a native sanos API which directly supported the MSVCRT as well as the Win32 functions. The MSVCRT functions are implemented directly on top of the native sanos API and do not use the Win32 wrappers in sanos. If you take a look at the kernel32 and msvcrt wrappers in sanos you will find that most functions maps directly to one of the underlying native API functions. I hope this is an indication of that the layer impedance mismatch to a large degree has been avoided.
One option that I haven't exploited yet is to write a native sanos hpi.dll, which uses the native sanos API. This could eliminate many calls though some of the layers. It could be done by implementing the HPI functions in jinit.exe and install this HPI when the VM is created. As you mention, the source code for the HPI for Windows can be obtained from Sun. I think it would be rather straight forward to implement a sanos HPI.
You mentioned that you would want to run the JVM in kernel mode like they do in the JX project. From a theoretical point of view this is a very appealing idea and the JX project has shown that it is a viable approach. Also from a minimalism point of view this could simplify the overall architecture. This said, I think it would be helpful to try to analyze what could be practically gained from this approach and which problems it might introduce.
One reason for running directly in kernel mode might be to gain better performance. I am reluctant to believe this is necessarily true. First of all, the CPU does not run faster in kernel mode than in user mode. There is no "magic" gained on this account. It is worth to remember that it is not the operating system that executes you application, it is the processor. The overhead of running in user mode is primarily due to:
Sanos is designed using a traditional model with user mode (ring 3) and kernel mode (ring 0). Each system call must transition the processor from ring 3 to ring 0. This has traditionally been done using a software interrupt (in sanos int 48). These software interrupts cost a lot of cycles. For each interrupt the CPU must save all the registers after switching mode. This overhead has increased relative as the processors has becoming faster and on a modern processors each interrupt now takes more than thousand cpu cycles. To counter this the Pentium processors supports sysenter/sysleave. These specialized instructions are used in sanos to make syscalls up to 10 times faster. Anyway, ring transitions are still an overhead that one might want to avoid by running solely in ring 0.
Another source of overhead is the parameter validation of all syscalls. All buffers are check for validity to protect the kernel. If the only application running on the computer is the JVM one might argue that there is nothing gained in protecting the kernel at the expense of the application. If the application crashes it doesn't matter that the kernel survives. For all practical purposes the computer has crashed. Most of the parameter validation in sanos can be turned off by compile time options. As part of the parameter validation handles are use by the kernel to avoid returning pointers to internal structures in the kernel. Handle usage is tracked by locking and releasing handles. This also constitutes an overhead, although I would guess that it is very low. All user buffers passed as arguments to kernel routines should also be locked to ensure these remain valid for at least the duration of the call. This feature has not yet been implemented fully in sanos.
The user/kernel mode design is traditionally also used to provide each process with its own virtual address space. This is normally done by dividing the virtual address space in a reserved kernel space (in sanos address 0x80000000 and above) and a user space, and by having separate page tables for each process. This requires the kernel to switch page tables when switching context from one process to another. This is not as cheap as one might expect because this will flush the TLB and sometimes also the internal CPU cache. This overhead is avoided in sanos because it uses a single process design.
One option that I have not yet tied out is to let the user code run in ring 0. This means that there is no protection of the kernel, but as I described above, this might not be such a big sacrifice for better performance. This would avoid the syscall overhead and replace it with a simple indirect call. I have not thought through all the consequences of this, but I think it would be doable, and does not require a lot of code changes.
One might want to run the JVM in kernel mode to be able to implement os functionality like schedulers, memory managers, network stacks and drivers in Java. While this is technically feasible (as shown in the JX project) I'm not sure what the practical benefits are. First of all, when using the HotSpot VM, you need to get the kernel up and running before you can start the JVM, but if some of the kernel has been written in Java, this present you with a chicken and egg problem. This might be fixable by changing the source code for the HotSpot VM. Another approach might be to use a micro kernel architecture. In micro kernels many of the subsystems mentioned above is executing in ordinary "user mode" code. Micro kernels are also appealing because of their simple minimalistic designs, although this often leads to less than optimal performance.
The sanos kernel is also written entirely in C for practical reasons. I believe that C is actually a good system programming language. There is a lot of kernel source available written in C. Many parts of sanos has been ported from other systems (see the contributions list for some). For instance, there is many drivers for Linux, which can easily be ported to sanos. There isn't yet a lot of os code in Java. The C programming language also allows you to have very precise control, which is needed for interfacing with the hardware and at the same time allows you to program in a relatively high-level language. I think it would be difficult to implement some of the non-copying buffer management schemes in the I/O systems in Java. Although processors are fast today the speed of memory access is lacking behind. Therefore efficient buffer management is still important. This will be difficult to achieve in a garbage collected environment. The tightly coupled buffer memory management in the kernel combined with the advanced garbage collected memory management for user data seems to be a good division of labor.
Java on the other hand is an excellent application programming language by far superior to C for most applications. The many J2EE servers also demonstrates Java as a platform of choice for implementing middleware services, especially due to its superiority in component based architectures and ease of network programming.
Although the kernel has been implemented in C, I think it is fair to call sanos with HotSpot VM a JavaOS. Taken together sanos takes up less than 2% of the combined code base, and the JDK (written in java) is 2/3 of the code base.
I hope the above comments explain some of the design rationale behind sanos, but I'm open to discussion on any of these topics. Regardless of my comments above I still think it would be interesting to try to explore the possibilities for simplifying the design and trying different models for running the JVM. Until now I have only used the source code for the HotSpot VM as a reference to look up the usage of the different API calls used. I have never tried to modify the code myself and built my own VM. This sounds like an interesting idea. Let me know about the results of your experiments.
You also mentioned running other kinds of byte code. I have looked at two open source implementation of the CLR for .NET. Mono (http://www.go-mono.com) is a free implementation of the CLR and .NET class libraries. Rotor (aka Shared Source Common Language Infrastructure) (http://msdn.microsoft.com/net/sscli) is an "shared source" implementation of CLR from Microsoft. I have not yet tried to port any of these to sanos, but it would be interesting to give it a shot.
-- Michael