New vfs for Jari OS (in general)

Hi all,
After spending some time with testing, prototypes implementation, investigation and so on, new vfs layer was born. Actually it's going from my last prototype, but there are many stuff to implement (a huge set of small things). But from now it's possible to tell - it's completely new and completely different from old vfs.
The question is why the project needed a new vfs layer, it might be explained.
First reason is architecture design changes killing separate vfs service (it moves to domain holder), actually vfs service has a search tree and task resource storage and it was a dirty job to keep it sync with name service and procmgr service events. It was a really small, but design was not unclear and each extension was a headache. Also, the old architecture design required an additional fslink service and special tool to have a deal with it, instead of running filesystem service itself. As the last argument - it was too hard to collect stats about vfs resources.
Second reason is a moving from unirpc to rpcv7 object model that changes all concept of rpc layer design and implementation i.e. to adopt old vfs layer you will need to completely rewrite all rpc related parts.
Third reason is a bulky design of old implementation, it was growing from very limited design and wasn't ready to extend in a normal way. I mean implementation of extended things such as async i/o, sockets and other IPC objects was difficult. Also, old vfs layer was not optimized by design, that's because vfs layer has a low priority while project began. There are many problems with it - really slow cache, memory mapping was screwed ugly after general parts was implemented, name resolving was made quickly and it's a potential hole to make a DDoS for vfs service. Well, there are many bad things might be enlisted here.
Conclusion - I didn't find any reason to adopt old vfs layer instead I decided to design and implement a completely new one.

Design was going around the principles explained below.
Optimization via architecture design and extensive use of the all advanced features provided by the microkernel. µString has a good IPC feature - message forwarding. It was implemented while old vfs architecture design was complete and using this feature was able to create a dramatically changes in old vfs implementation. VFSv2 (I will call old vfs layer implementation so) use a generic IPC scheme i.e. receive - resolve - reply where resolve creates new IPC cycle send - wait for reply for each filesystem (sure, in case of symlinks only), that made vfs service thread to work with one request while resolving, opening and so on - simply said vfs thread was blocked by the client request, while this request will be done. VFSx (I will call new vfs layer as VFSx from here) is built on the transaction model, that mean each client request is a transaction with unique ID, vfsx thread will create a transaction, resolve the name, append a special header to the message and forward it to the file system, in case of file system return vfsx thread will update special header, resolve again and forward again. File system take care about transactions too, if request is processed filesystem will commit or decline the transaction and return data to the client, if request resolve should continue (in case of symlink for example) filesystem will modify special header with a symlink data and forward it to vfsx. This approach keeps vfsx threads from blocking, because transaction operations, resolving is a very fast, especially if we will compare it to vfsv2.  Another point is a double buffering of client i/o metadata to communicate with opened resources and huge syncing operations set. VFSx was included to the domain holder service, this service is a task manager as well as name resolver, domain holder contains all special entries for each task as well as task tree (µString provide very basic functionality and doesn't know anything about terminals and other service side OS implementations), one of those information is the task iolinks. iolink is an abstraction make client connection to the resolved resource, to keep it in a required way domain holder must know all client connections i.e. iolinks to submit an events to services holds client iolinks (for example while client forking you need to clone all of its own iolinks). To avoid additional copying and IPC cycles, each task has a shared pages with domain holder (it made in security way), this area is served by the client and by domain holder. Finally, this modification allows to ease implement a failsafe feature. VFSx is designed to be tolerate for file system crash, in case of this event domain holder marks iolinks dead, when the filesystem was rebooted the old file system entry on domain holder marks restored, and if client will try to interact with iolink it will be restored and pointed to a recently rebooted filesystem instead of old crashed one. Going more deeper - vfsx allows async i/o mode and provide events on vfs objects, that was totally absent in vfsv2 (yep, excluding network socket implementation that was ugly screwed to the vfsv2 layer).

Tolerate to the µString(it's a microkernel if you don't know) modifications: yes, I decide to implement a new IPC type avoiding copy_from/copy_to and double/triple buffering. This feature moving its own roots to the vfsx block backend and memory mapping implementation. Briefly, filesystem didn't send some buffer with data copied from page cache, instead of it filesystem just send an event with information about page location to the block device service, block device service just "map" it and do all read/write operations. Later (in newer releases) Jari OS will provide an additional read/write functions works as mentioned for the libc to extend POSIX read/write functions. What about mappings, well µString now will send all required events on mmap()/msync()/munmap()/etc.

Provide clean and easy customizing API - it will allows to deploy filesystem porting really faster than on vfsv2. This also affects old known vnode cache - in vfsv2 it was vnode and dentry cache united together, vfsx has the dir_entry cache and fnode cache in such way as it implemented in many systems (em, Linux like example).

Overall, vfsx is more faster, cleaner and feature rich than vfsv2. I cannot describe all changes and all implementation specifics. However, I will try to explain more with the next posts, and documentation.

See you,

Approach to the file system design

Within microkernel operating system you may do different kind of interesting things.

Today is a day of file systems, I'm currently working on the file layer general design and there are some feature to implement, well, I can told that it's a several features implemented may.

The first is a copy-on-write feature, well it might be implemented over the file system implementation.

while you have an vnode on top of the implementation you may create a virtual remote link to other file system vnode, and it will goes transparently for implementation. When you will got a miss on the block device of the current filesystem you may take this data from another one, remotely.

Another case of this feature is a creating a new function that will clone files in the fast way, just creating a shadow vnode on the destination file system.

I will research this cases later and write it here, it's not so good to write it in bus.

MuiString: Introduction to the architecture #1

Today we're have a many implementations of the microkernel architectures, but there are too similar and have many problems.
The first problem is a speed, the second one is a security.
Generally microkernel more secure than monolithic-based operating system, but with some restrictions, if you grow up speed, you are loosing in security.
But it's a general problems - more complicated thing - we're using C/C++/ObjC to implement architecture, and there are language takes a many time and limits in the implementing.
My general idea is to use a functional programming language - and I've choose scheme - it's better, simply.
Look for the block diagram:
MuiString separated to the three logical parts, the first low two layers implemented on C (or on Cext in future), the third layer implemented on C/Cext and on scheme itself.
All hardware specific things are hidden in the MuiString.
All services should be implemented on scheme, yes, we're can use C-written servers and libs (it should be done for liblinux/libposix), but in general case all other will be implemented on scheme and ran on scheme VM.
All IPC will be operate via scheme forms - in this case all security models will be implemented without IPC calls restrictions, and it can be made on the VM (or language) level.
There are several security models and several objects on the higher level:

• Trusted servers
• Key signed
• Group politics

Trusted servers can allow secure transmit via determined servers, for example - you trust to the TCP/IP server, LAN server also trust it, but there are no directions from you to the LAN server, but LAN server and you can get access to execute scheme forms from/on LAN server, because there are trusted via TCP/IP server.

I will write more researching results in the next post.

Research: new ideas to system development

Doing some research on operating systems design you can choose the kernel model and in this case you can expect that microkernel is more stable, powerful and flexible compairing with monolithic design, exokernels are too platform specific and more complicable for application level platforms (i.e. embedded systems with too limited functionality). But like a bonus you will got the new area of research in this area - choosing and developing an instruments for implementation.
On this area, in my opinion, are old and good instrument is a pure C language for microkernel, for low level. But in case of many reasons C development has a long time for something bigger than microkernel, for microkernel you can extend C with macros or write your own preproccessor to extend C (like my project extC). This is good and right, but for microkernel servers and user level is better to use something else, more powerful with paradigms and more easy to develop.
It's not C++, not only in case of my hate to pluses, there are no reason to use C++ in any kind of development, C++ contain many ugly stuff and it has an ugly design overally. There are ObjectiveC, but this language will not decide our wishes.
I'm looking to functional programming languages - this way is more powerfull, well-designed lisp-like language is really simple to implement, they all has a simple syntax - and learning didn't takes a long time - it's intended to be meta-level development.
With its powerful I'm thinking about crazy things - what about to design a several DSL that can be simply implemented on Lisp/Scheme/Haskell , for different types of microkernel servers.
In example, you can research for often used language features and constructions that required for implement a filesystem, or block device driver, or network device driver etc...
I'm thinking to go deeper in this area after MuiString will ran. I promise to make a deep research, implement expiremental DSLs for this purposes and test it for perfomance and other things.
Be updated, take a look to functional programming of your research - it's a wonderful world of new features and paradigms.

MuiString higher memory management

I've made some researching and emulation (not so serios, but representative) about memory management in microkernel OS.
Many kernels does memory management fully in kernel space, others separate it and offer non-trivial API for doing this in userspace.
We're have many restrictions - security and performance. If you will targeting to one of this properties you will loose on other. If you want to have a very strong and secure mechanism you will make a many context switching within kernel and user spaces - it will slow down your system perfomance, if you will don't care about security you will make it faster ...
I decide to make a basics virtual memory manager operations within kernel space, and offer simple API (just 4 calls) - address_space_get(), address_space_set(), address_space_alloc(), address_space_release(). And ... bind page fault exception to userspace area virtual mm server.
In this scheme microkernel just mark/unmark address space, it doesn't makes a check (and it will works fast) , and microkernel doesn't care about page fault exception - it's a server headache.
The security implemented on userspace - with ACL server that cannot be exchanged - like other varios servers (you must load its via multiboot modules way), all requests to ACL cached to the IPC cache server (it cannot be exchanged too) to avoid premiary access violation checks and grow up speed of requests. On the figure below I've tried to show it graphically (I like this, but I don't like to make a graphics ...):

User task just trying to take some virtual memory and extend it's address space - it's a deal of libs to make all job with calling of vmm server. Vmm server asks about access rights/limits/etc ... on ACL server, if all granted - vmm will allocate space for user task, if will be need - it will calls MuiString microkernel via simple API.

This structure allows to make secure and fast (compairing with some highly secured or highly fast microkernel systems) - it's a good design for implementing different models of memory allocations (real-time, preemption, so on ...)