SDK tools for OS development

New ideas came to my head, all about operating system features and development.

On last friday I decide to make some modifications on autotools stuff in GNU sed and try to compile it with Jari OS libc headers and functions. Yep - it's works! This means that our libc is ready for most of cases, many of them working now. This event gave me some new ideas about development tools for Jari OS: firstly I will port GNU gcc to Jari OS (I mean, that gcc will handle our ld scripts and libraries right, without many keys and other things), secondly I will construct SDK template (already done, but not publicied yet).
Also, to have a better support and big community additional feature needed - IDL integrated to SDK. Well, it's mean - that you have set of the templates for each type of service i.e. block device driver library, char device driver, file system and so on ...
In most cases you shouldn't known about specific internal things of system libraries, also you shouldn't care about interface changes (headers renaming, new functions, new order of calls and so on). This feature will generate libraries specific code in compile time and link your specific implementation. I.e. what will be written one time - will live all time ;)
For this feature it's better to use scheme (for templates, configs and other stuff) and some core of this will be written traditionally on C.

Jari OS going to be a completely OS

My last task in Jari OS project was symlinks supports, now I'm working on postponed logic in libv2 (general library for file system services).
I cannot be on track in changes ;) Now project has a POSIX, fork(), execve(), file systems, initial networking support.
On background I'm porting GNU coreutils to Jari OS, progress going fast, I think that alpha release will has many interesting stuff.
To be a fully featured OS not so many features need, our ext2fs support and IDE drivers are 80% completed.
I think that year of active development that will finished on 1-sep-2009 will made a first beta version with few drivers and networking support.
Now prject has one network device support (e1000) and Jari OS host (network service) replies on ICMP requests.
What about ELF support, now we're working on dynamic libraries support - it coming soon, currently only static ones supported.
Files i/o is ready, but maybe some functions (not often used ones) doesn't works properly. To make a fully featured shell pipes requered, and I'm working on it now.
I'm planning to implement shared memory fs and POSIX functions for it, in addition I will add Jari OS specific API for this, - for some internal purposes, for GIX(GUI service of Jari) like example.
After alpha release (1-jun-2009) I will take time for GIX implementation. Also - there are many porting tasks.
I hope community of open-source developers will be interested in new microkernel OS - it's a good field for research, development and it's good to take skills up-to-date.
So,

Jari OS: fork() in microkernel based OS

Well, this day makes huge parts of my brains works.
Designing microkernel systems you have a good skills base, designing microkernel and multiservice system you have an excellent skills base.
Today was a day of fork() call - it has many things to discuss, think, imaginate and so on.
At the end of a day, after all problems was detected - I found a solution to solve fork() call.
Let's begin and I will explain fork() nature as is...
Fork is a POSIX call that creates a clone of a calling process - with one thread, but the clone should have the same resources - i.e. memory (not at all), opened file descriptors, IPC stuff.
In Jari OS resources are isolated - VFS know about files, kernel knowns about IPC and so on - you will have a headache about syncronization, performance - always while architecture design - welcome to microkernel-based OS.
In this case you need to develop non-trivial trick with things that monolithic kernels made easly, but ... microkernel-based OSes is clever and better at all.
Well, you have a task, you have a resources and you need to fork() POSIX call - the first solution is simple - just delegate this job to process server and deal is made, within server timeslice - and this is bad (but QNX does it in this way). Other solution is more sophiscated - and you need to make a stages for fork():

• Request to delegate a fork() for myself
• Make a fork() skel within my timeslice
  
• Send a request about fork() call made by me

In this big deal you will affect microkernel, process service, virtual file system service and your task.
Going deeper I will explain each stage of fork() and after those I will explain how it works on different sides.
Your first request is a request to the process service where you tell that you wants to make a fork(). Process service will create a mould on VFS of your resources, toggle bit in kernel that allow you to make a sys_fork() syscall, and if all is ok it will reply with ok message :)
After this you are going to make a syscall to fork - be warn! kernel toggle off bit that allows you to make this call, and you cannot do it yourself - only trusted process service does.
Call is done - you have a skel - you are making a second call to process service - i.e. - "fork is done - please make my child runnable" - process service tells VFS to make mould active and working - and changes status of the child - and replies you with "OK all is done".
This scheme allows task to make a one fork() call at a time (i.e. you are able to make other child after old child is created) - I don't think that this is a bad, other architecture going to make other applications design.
On this I want to explain some details.
"Forkable bit" can be toggled on only by trusted process, and it's switched off while fork called.
Task has a timestamp - and both kernel and process service known it at the start of all.
VFS's mould has a parent and while parent is exist - mould will not be deleted, orphaned mould will be deleted with timeout.
Checking for a valid child pid makes not only on parent pid - also on timestamp.

Uff, I can explain other details in comments.

File systems on Jari OS

There was a great job on last two weeks. File subsystem are designed in Jari OS - since we have a system with separated services - libraries is a good approach to avoid a huge amount of code and errors with changes it.
Actually file subsystem is divided by several general parts:

• VFSv2 service
• set of libraries for each filesystem

I don't want to include all stuff for filesystem in one library, because it's a bad idea, for example tmpfs doesn't has backend to block device, pagecache, postponed calls and so on. But anyway this set of libraries can be completed by one, generic library that should be included to each file system - libv2.
In this case we're have the following list:

• libv2 (general library)
• libv2backend (backend to the block device layer)
  
• libv2pgcache (page cache library)
  
• libv2ppcall (library serves postponed calls)

Each file system architecture should determine what will be used, generally , regular file system will always use all set of libraries - i.e. it works with block device and use page cache in this case, and it will not replies immediately on all cases - it will have postponed calls list to reply.
For libv2 - there are nothing difference - it will always support everything on logic layer.
I can't see other solutions - our VFS service should be overloaded - and many calls going directly to file system service.
So, while implementation tmpfs and initfs - implementation hasn't any complexities.

Scheme VM like a microkernel service

Yes, it's an idea. I'm thinking about benefits in the microkernel and multiservice OS on desktop or some kind of development environment. Well, I know all the things that going in microkernel while you calling fork() after those exec() - it takes a lot of system time.
Usually within shell you makes fork()/exec() every time when you starts anything - ls,cat ... or you can wrong with some arguments for utility and this will increase system job time. Also, it's a waste of a time to implement small utils on C, there are few POSIX things, few useful things, few OS specific things and so on. Better one - run login/shell environment within VM context - like a thread, all utils will runs like a thread (thread creation in Jari OS isn't so wasteful) and writing simple things on scheme is a simple and fast process. It doesn't mean that Jari OS reject exec/fork and other POSIX calls, it's mean that we can take a benefits on desktop I think.
Many people told me that microkernel is sucks and microkernel is bad for desktop - well, most of them doesn't know anything about OS design, but others are right in some case. But I want to make some note - there are no existing microkernel (and multiservice) OSes with a pure microkernel and multiservers design (I don't take early, prototype-like, featureless systems, like Jari OS and some line of other interest projects, HelenOS for example) - yes it's a first thing to think that microkernel sucks, second it's a performance - yes context switching, calls partially going outside of the caller time slice and so on, it's not the problem as well, for most I/O operations context switch and related thing is very little compare to disk I/O or other I/O (no, I don't mean highband special devices), also most of technical microkernel critics based on existing solution - that can has ugly design (I don't want to make a shit showing examples here).
My thesis - Well designed and implemented microkernel and multiservice OS can works at the same speed as monolithic OS works, but ugly designed monolithic OS will be ran slowly then microkernel based OS with good design. Also, you should remember that some abstractions in monolith made within user space - like example GUIs, and I don't think that some operations will be slowly on microkernel.
And ... the main idea - for microkernel world you should keep in mind microkernel oriented solutions, instead of porting old and ugly tricks from monolithic - schemevm running as microkernel service is such solution.

Jari Operating System: small steps

I want to make an announcement of development steps.
First of all is a really progress on microkernel (http://mstring.jarios.org) - there are many features implemented:
- memory management (O(1) allocator)
- interrupts handling for uspace
- memory i/o for uspace
- ioports for uspace
- realtime scheduler
- running static ELF binaries that loaded from GNU grub
- Fast IPC (sync ports)
Actually muistring allow to write userspace drivers and subsystems now, but some things not done yet.
Uspace (Jari) is developing now and some things works:
- nameserver
- terminals server
- text VGA driver
- keyboard driver
- syslibs (libos, libc, liballoc)
Some later I will post link to showcase.
More information here - http://jarios.org

Jari Operating System: reburned

I don't have a time to write more.
This is just a copy-paste from news feed:
After Jari was frozen for a many years, now Jari is wake up and preparing to work, for a hard work.
Jari Operating System will be a not microkernel/hybrid based system, otherwise it will use an exokernel, that developing separately. It's mean that Jari Operating System is a set of user-space drivers, network libs, libOS, libc and so on. It's a top level of a exokernel.
While exokernel under extreme development stage and not all syscalls are well defined, this project will wait for this and will temproary suspended (for several weeks or one month, it cannot be determined now). After those userspace wrappers will be implemented and user-space development begins.
After core libs are well tested and exokernel will be done this all will be ported to it.
In case of exokernel Jari changing default architecture to amd64 (not x86_64, just amd64 in case of its specific hardware features), but Jari portable, and all depends on exokernel ports.
So, I will present an exokernel -> http://mstring.jarios.org , it's called MuiString (greek mui at the begging) that shows it's small size, and string just a mode for kernel flexibility and atomicity (you know about string theory).
Generally it mean that Jari found a kernel to base on, ... it was a
long search ...

Also I want to introduce a new Jari OS homesite - http://jarios.org, you can find a lot of
information there.

Thanks.

Decomposing big project, separating is better

You can find more information on the link in the post title.
Overally, following to the general idea - "separate all works and job to smaller ones" I decide to run microkernel separatly - like independed project.
This intended to be more effective, for example - currently I have EZA layer working for 60% (not so stable) and this was made after reorganization of the microkernel structure - and this just for less then 10 hours of normal works on the problem.
MString (MuiString) - is a name for Jari microkernel - and it's a new tag on this blog ;)
Now I'm thinking on diferent than C/Cext language for server space - it's a big flame, but I'm thinking that fully functional languages or partly functional is better than typical imperativies ones. Anyway, microkernel must be implemented on plain C - it's more comfortable language.

New directions, new profile, new ideas

Jari development has a new targets in development. The first of all we're decided to move our dev/hours to the exiting GNU mach microkernel and implement our ideas based on it. This decision based on the real time needs via the following points :
- there are many trivial task thats simply takes the time on its
- there are already exiting code base that can be used for Jari ideas
- there are no time to create a newly microkernel from scratch
The possible ways may be different and depends on the development flow in the time.
I don't know yet if we will takes the GNU Hurd servers to modify and extend, but you can be sure we will split Jari ikernel and Jari EZA code base with the GNU Mach code base.
With this objectivies I hope that we will ran in the early dates.
Now, I've planned to get Jari run on the third decade of the 2009 with the minimal drivers set, libc, CLI and varios utilities.
Look out this blog for further details in the future.

Jari: small new things

While the low level under the heavy development stage, I'm returned to the compiler and some software. But about it by the way ...
C compiler:
We're developing the new C dialect, eC - stands for extended C.
No, it doesn't mean the next C++ or something else. Our dialect targeted to functional programming. I will extend the C macros, add the lambda functionality, and optimize all works with the types.
Also eC compiler will have a mechanism that will checks the works with pointers and variables to avoid the double malloc/free, it will be like an option.
Backing to C, it will backward compatible i.e. you will be able to compile C sources with eC compiler.
About this and other new features I will blog separately some later.
LibC/LibeC:
We will include the several memory allocation models, we will provide the standart alloc() and free(), but with it we're can offer the wrappers and callbacks to make the internal memory allocator simply and faster. You will be able to create struct mem_alloc_t and use it with your own way, with the libc/libec standart functions/politics or assign your function that preffered for your application. Also the libs will have several new functions for allocating memory - rt_alloc(), local_alloc(), ext_alloc(), func_alloc(), there functions will provide the more flexible parallel programming issues.
In addition, libs will be extended for string operating functions, the libec will have the standart high customizable parser.
coreutils:
Core utils will includes tools for parallel task management, via this tools you can get ran processes via your cluster by the your custom way, run nonlocal architecture binaries on the needed node and other features.

See for next blog messages, I will give here more examples and will describe all things written above.