New vfs for Jari OS (in general)

Hi all,
After spending some time with testing, prototypes implementation, investigation and so on, new vfs layer was born. Actually it's going from my last prototype, but there are many stuff to implement (a huge set of small things). But from now it's possible to tell - it's completely new and completely different from old vfs.
The question is why the project needed a new vfs layer, it might be explained.
First reason is architecture design changes killing separate vfs service (it moves to domain holder), actually vfs service has a search tree and task resource storage and it was a dirty job to keep it sync with name service and procmgr service events. It was a really small, but design was not unclear and each extension was a headache. Also, the old architecture design required an additional fslink service and special tool to have a deal with it, instead of running filesystem service itself. As the last argument - it was too hard to collect stats about vfs resources.
Second reason is a moving from unirpc to rpcv7 object model that changes all concept of rpc layer design and implementation i.e. to adopt old vfs layer you will need to completely rewrite all rpc related parts.
Third reason is a bulky design of old implementation, it was growing from very limited design and wasn't ready to extend in a normal way. I mean implementation of extended things such as async i/o, sockets and other IPC objects was difficult. Also, old vfs layer was not optimized by design, that's because vfs layer has a low priority while project began. There are many problems with it - really slow cache, memory mapping was screwed ugly after general parts was implemented, name resolving was made quickly and it's a potential hole to make a DDoS for vfs service. Well, there are many bad things might be enlisted here.
Conclusion - I didn't find any reason to adopt old vfs layer instead I decided to design and implement a completely new one.

Design was going around the principles explained below.
Optimization via architecture design and extensive use of the all advanced features provided by the microkernel. µString has a good IPC feature - message forwarding. It was implemented while old vfs architecture design was complete and using this feature was able to create a dramatically changes in old vfs implementation. VFSv2 (I will call old vfs layer implementation so) use a generic IPC scheme i.e. receive - resolve - reply where resolve creates new IPC cycle send - wait for reply for each filesystem (sure, in case of symlinks only), that made vfs service thread to work with one request while resolving, opening and so on - simply said vfs thread was blocked by the client request, while this request will be done. VFSx (I will call new vfs layer as VFSx from here) is built on the transaction model, that mean each client request is a transaction with unique ID, vfsx thread will create a transaction, resolve the name, append a special header to the message and forward it to the file system, in case of file system return vfsx thread will update special header, resolve again and forward again. File system take care about transactions too, if request is processed filesystem will commit or decline the transaction and return data to the client, if request resolve should continue (in case of symlink for example) filesystem will modify special header with a symlink data and forward it to vfsx. This approach keeps vfsx threads from blocking, because transaction operations, resolving is a very fast, especially if we will compare it to vfsv2.  Another point is a double buffering of client i/o metadata to communicate with opened resources and huge syncing operations set. VFSx was included to the domain holder service, this service is a task manager as well as name resolver, domain holder contains all special entries for each task as well as task tree (µString provide very basic functionality and doesn't know anything about terminals and other service side OS implementations), one of those information is the task iolinks. iolink is an abstraction make client connection to the resolved resource, to keep it in a required way domain holder must know all client connections i.e. iolinks to submit an events to services holds client iolinks (for example while client forking you need to clone all of its own iolinks). To avoid additional copying and IPC cycles, each task has a shared pages with domain holder (it made in security way), this area is served by the client and by domain holder. Finally, this modification allows to ease implement a failsafe feature. VFSx is designed to be tolerate for file system crash, in case of this event domain holder marks iolinks dead, when the filesystem was rebooted the old file system entry on domain holder marks restored, and if client will try to interact with iolink it will be restored and pointed to a recently rebooted filesystem instead of old crashed one. Going more deeper - vfsx allows async i/o mode and provide events on vfs objects, that was totally absent in vfsv2 (yep, excluding network socket implementation that was ugly screwed to the vfsv2 layer).

Tolerate to the µString(it's a microkernel if you don't know) modifications: yes, I decide to implement a new IPC type avoiding copy_from/copy_to and double/triple buffering. This feature moving its own roots to the vfsx block backend and memory mapping implementation. Briefly, filesystem didn't send some buffer with data copied from page cache, instead of it filesystem just send an event with information about page location to the block device service, block device service just "map" it and do all read/write operations. Later (in newer releases) Jari OS will provide an additional read/write functions works as mentioned for the libc to extend POSIX read/write functions. What about mappings, well µString now will send all required events on mmap()/msync()/munmap()/etc.

Provide clean and easy customizing API - it will allows to deploy filesystem porting really faster than on vfsv2. This also affects old known vnode cache - in vfsv2 it was vnode and dentry cache united together, vfsx has the dir_entry cache and fnode cache in such way as it implemented in many systems (em, Linux like example).

Overall, vfsx is more faster, cleaner and feature rich than vfsv2. I cannot describe all changes and all implementation specifics. However, I will try to explain more with the next posts, and documentation.

See you,

Jari OS: fork() in microkernel based OS

Well, this day makes huge parts of my brains works.
Designing microkernel systems you have a good skills base, designing microkernel and multiservice system you have an excellent skills base.
Today was a day of fork() call - it has many things to discuss, think, imaginate and so on.
At the end of a day, after all problems was detected - I found a solution to solve fork() call.
Let's begin and I will explain fork() nature as is...
Fork is a POSIX call that creates a clone of a calling process - with one thread, but the clone should have the same resources - i.e. memory (not at all), opened file descriptors, IPC stuff.
In Jari OS resources are isolated - VFS know about files, kernel knowns about IPC and so on - you will have a headache about syncronization, performance - always while architecture design - welcome to microkernel-based OS.
In this case you need to develop non-trivial trick with things that monolithic kernels made easly, but ... microkernel-based OSes is clever and better at all.
Well, you have a task, you have a resources and you need to fork() POSIX call - the first solution is simple - just delegate this job to process server and deal is made, within server timeslice - and this is bad (but QNX does it in this way). Other solution is more sophiscated - and you need to make a stages for fork():

• Request to delegate a fork() for myself
• Make a fork() skel within my timeslice
  
• Send a request about fork() call made by me

In this big deal you will affect microkernel, process service, virtual file system service and your task.
Going deeper I will explain each stage of fork() and after those I will explain how it works on different sides.
Your first request is a request to the process service where you tell that you wants to make a fork(). Process service will create a mould on VFS of your resources, toggle bit in kernel that allow you to make a sys_fork() syscall, and if all is ok it will reply with ok message :)
After this you are going to make a syscall to fork - be warn! kernel toggle off bit that allows you to make this call, and you cannot do it yourself - only trusted process service does.
Call is done - you have a skel - you are making a second call to process service - i.e. - "fork is done - please make my child runnable" - process service tells VFS to make mould active and working - and changes status of the child - and replies you with "OK all is done".
This scheme allows task to make a one fork() call at a time (i.e. you are able to make other child after old child is created) - I don't think that this is a bad, other architecture going to make other applications design.
On this I want to explain some details.
"Forkable bit" can be toggled on only by trusted process, and it's switched off while fork called.
Task has a timestamp - and both kernel and process service known it at the start of all.
VFS's mould has a parent and while parent is exist - mould will not be deleted, orphaned mould will be deleted with timeout.
Checking for a valid child pid makes not only on parent pid - also on timestamp.

Uff, I can explain other details in comments.

Scheme VM like a microkernel service

Yes, it's an idea. I'm thinking about benefits in the microkernel and multiservice OS on desktop or some kind of development environment. Well, I know all the things that going in microkernel while you calling fork() after those exec() - it takes a lot of system time.
Usually within shell you makes fork()/exec() every time when you starts anything - ls,cat ... or you can wrong with some arguments for utility and this will increase system job time. Also, it's a waste of a time to implement small utils on C, there are few POSIX things, few useful things, few OS specific things and so on. Better one - run login/shell environment within VM context - like a thread, all utils will runs like a thread (thread creation in Jari OS isn't so wasteful) and writing simple things on scheme is a simple and fast process. It doesn't mean that Jari OS reject exec/fork and other POSIX calls, it's mean that we can take a benefits on desktop I think.
Many people told me that microkernel is sucks and microkernel is bad for desktop - well, most of them doesn't know anything about OS design, but others are right in some case. But I want to make some note - there are no existing microkernel (and multiservice) OSes with a pure microkernel and multiservers design (I don't take early, prototype-like, featureless systems, like Jari OS and some line of other interest projects, HelenOS for example) - yes it's a first thing to think that microkernel sucks, second it's a performance - yes context switching, calls partially going outside of the caller time slice and so on, it's not the problem as well, for most I/O operations context switch and related thing is very little compare to disk I/O or other I/O (no, I don't mean highband special devices), also most of technical microkernel critics based on existing solution - that can has ugly design (I don't want to make a shit showing examples here).
My thesis - Well designed and implemented microkernel and multiservice OS can works at the same speed as monolithic OS works, but ugly designed monolithic OS will be ran slowly then microkernel based OS with good design. Also, you should remember that some abstractions in monolith made within user space - like example GUIs, and I don't think that some operations will be slowly on microkernel.
And ... the main idea - for microkernel world you should keep in mind microkernel oriented solutions, instead of porting old and ugly tricks from monolithic - schemevm running as microkernel service is such solution.

Jari Operating System: reburned

I don't have a time to write more.
This is just a copy-paste from news feed:
After Jari was frozen for a many years, now Jari is wake up and preparing to work, for a hard work.
Jari Operating System will be a not microkernel/hybrid based system, otherwise it will use an exokernel, that developing separately. It's mean that Jari Operating System is a set of user-space drivers, network libs, libOS, libc and so on. It's a top level of a exokernel.
While exokernel under extreme development stage and not all syscalls are well defined, this project will wait for this and will temproary suspended (for several weeks or one month, it cannot be determined now). After those userspace wrappers will be implemented and user-space development begins.
After core libs are well tested and exokernel will be done this all will be ported to it.
In case of exokernel Jari changing default architecture to amd64 (not x86_64, just amd64 in case of its specific hardware features), but Jari portable, and all depends on exokernel ports.
So, I will present an exokernel -> http://mstring.jarios.org , it's called MuiString (greek mui at the begging) that shows it's small size, and string just a mode for kernel flexibility and atomicity (you know about string theory).
Generally it mean that Jari found a kernel to base on, ... it was a
long search ...

Also I want to introduce a new Jari OS homesite - http://jarios.org, you can find a lot of
information there.

Thanks.

MuiString: Introduction to the architecture #1

Today we're have a many implementations of the microkernel architectures, but there are too similar and have many problems.
The first problem is a speed, the second one is a security.
Generally microkernel more secure than monolithic-based operating system, but with some restrictions, if you grow up speed, you are loosing in security.
But it's a general problems - more complicated thing - we're using C/C++/ObjC to implement architecture, and there are language takes a many time and limits in the implementing.
My general idea is to use a functional programming language - and I've choose scheme - it's better, simply.
Look for the block diagram:
MuiString separated to the three logical parts, the first low two layers implemented on C (or on Cext in future), the third layer implemented on C/Cext and on scheme itself.
All hardware specific things are hidden in the MuiString.
All services should be implemented on scheme, yes, we're can use C-written servers and libs (it should be done for liblinux/libposix), but in general case all other will be implemented on scheme and ran on scheme VM.
All IPC will be operate via scheme forms - in this case all security models will be implemented without IPC calls restrictions, and it can be made on the VM (or language) level.
There are several security models and several objects on the higher level:

• Trusted servers
• Key signed
• Group politics

Trusted servers can allow secure transmit via determined servers, for example - you trust to the TCP/IP server, LAN server also trust it, but there are no directions from you to the LAN server, but LAN server and you can get access to execute scheme forms from/on LAN server, because there are trusted via TCP/IP server.

I will write more researching results in the next post.

Research: new ideas to system development

Doing some research on operating systems design you can choose the kernel model and in this case you can expect that microkernel is more stable, powerful and flexible compairing with monolithic design, exokernels are too platform specific and more complicable for application level platforms (i.e. embedded systems with too limited functionality). But like a bonus you will got the new area of research in this area - choosing and developing an instruments for implementation.
On this area, in my opinion, are old and good instrument is a pure C language for microkernel, for low level. But in case of many reasons C development has a long time for something bigger than microkernel, for microkernel you can extend C with macros or write your own preproccessor to extend C (like my project extC). This is good and right, but for microkernel servers and user level is better to use something else, more powerful with paradigms and more easy to develop.
It's not C++, not only in case of my hate to pluses, there are no reason to use C++ in any kind of development, C++ contain many ugly stuff and it has an ugly design overally. There are ObjectiveC, but this language will not decide our wishes.
I'm looking to functional programming languages - this way is more powerfull, well-designed lisp-like language is really simple to implement, they all has a simple syntax - and learning didn't takes a long time - it's intended to be meta-level development.
With its powerful I'm thinking about crazy things - what about to design a several DSL that can be simply implemented on Lisp/Scheme/Haskell , for different types of microkernel servers.
In example, you can research for often used language features and constructions that required for implement a filesystem, or block device driver, or network device driver etc...
I'm thinking to go deeper in this area after MuiString will ran. I promise to make a deep research, implement expiremental DSLs for this purposes and test it for perfomance and other things.
Be updated, take a look to functional programming of your research - it's a wonderful world of new features and paradigms.

MuiString higher memory management

I've made some researching and emulation (not so serios, but representative) about memory management in microkernel OS.
Many kernels does memory management fully in kernel space, others separate it and offer non-trivial API for doing this in userspace.
We're have many restrictions - security and performance. If you will targeting to one of this properties you will loose on other. If you want to have a very strong and secure mechanism you will make a many context switching within kernel and user spaces - it will slow down your system perfomance, if you will don't care about security you will make it faster ...
I decide to make a basics virtual memory manager operations within kernel space, and offer simple API (just 4 calls) - address_space_get(), address_space_set(), address_space_alloc(), address_space_release(). And ... bind page fault exception to userspace area virtual mm server.
In this scheme microkernel just mark/unmark address space, it doesn't makes a check (and it will works fast) , and microkernel doesn't care about page fault exception - it's a server headache.
The security implemented on userspace - with ACL server that cannot be exchanged - like other varios servers (you must load its via multiboot modules way), all requests to ACL cached to the IPC cache server (it cannot be exchanged too) to avoid premiary access violation checks and grow up speed of requests. On the figure below I've tried to show it graphically (I like this, but I don't like to make a graphics ...):

User task just trying to take some virtual memory and extend it's address space - it's a deal of libs to make all job with calling of vmm server. Vmm server asks about access rights/limits/etc ... on ACL server, if all granted - vmm will allocate space for user task, if will be need - it will calls MuiString microkernel via simple API.

This structure allows to make secure and fast (compairing with some highly secured or highly fast microkernel systems) - it's a good design for implementing different models of memory allocations (real-time, preemption, so on ...)

MuiString introduction

Ok, I've told many information about Jari and so on ... But it was just a buzz words.
Now I want to present MuiString (mstring) microkernel project.
What is done?
- base structure
- low level stuff
How much time it will takes ? How long will I told about it without microkernel on legs ? Hm, so I think that this times ends in case of microkernel progress.

I promise to write more technical things here and ... look for links below -
MuiString homepage - http://mstring.berlios.de
MuiString wiki - http://mstring.berlios.de/wiki
I have a memory manager and core of scheduler in my sandbox, it's under testing and preparing to be committed to the general CVS repository.
Look for my messages here soon.

Thanks.

New directions, new profile, new ideas

Jari development has a new targets in development. The first of all we're decided to move our dev/hours to the exiting GNU mach microkernel and implement our ideas based on it. This decision based on the real time needs via the following points :
- there are many trivial task thats simply takes the time on its
- there are already exiting code base that can be used for Jari ideas
- there are no time to create a newly microkernel from scratch
The possible ways may be different and depends on the development flow in the time.
I don't know yet if we will takes the GNU Hurd servers to modify and extend, but you can be sure we will split Jari ikernel and Jari EZA code base with the GNU Mach code base.
With this objectivies I hope that we will ran in the early dates.
Now, I've planned to get Jari run on the third decade of the 2009 with the minimal drivers set, libc, CLI and varios utilities.
Look out this blog for further details in the future.