Tuesday, December 08, 2009

The venture should always have a logical end.

Recent months have become a complete disappointment in my life, but my life is not so interesting, by the way my life has a feedback to my projects.
I'm a little upset, to be honest, I'm defeated with project development, and mostly with it goals.
I'm afraid that it will be not so possible to complete the Jari OS as a platform, but the hope remains.
I have a look at the project from the other side, from the side of engineer need to choose a base platform for the final solution. Jari OS isn't ready, and will not ready for several years (I don't think that good news happens) and it's very sad. Anyway, I'm learn again and I have a couple of real problems in the design architecture, management and so on, I know - there are many stuff written and works now. I cannot told that the project failed in case of it's basics concepts or other more deeper reasons, and I cannot told that project is succeful - this venture project hasn't a logical end, but it should has.
Well, I decide reorganize my works on the project and try to give a new life to Jari OS as soon as it possible.
First, I will write a documentation about platform, fix a design and so on.
Second, I will try to have a wide view of the future requirements.

When Jari OS will finished (it might be fail, might be not) - this will be the logic end of the venture, if it will has a continue - it will became a serious project.

Friday, November 06, 2009

OS choice

I have a direct bearing on the development of operating systems, so I can do some analysis of existing solutions on the market. The market of operating systems is full, it is filled to the brim. What? There are many solutions for different domains. Now, I'm faced with a choice of OS for the next project and I can't say that among the existing (Jari OS is a good platform, but aren't ready, and as a woman requires a lot of time) I can find a suitable, I considered the decision on free software, or are not expensive ones - I don't found suitable one. I will not describe my criteria, and I don't want to dilute the flame of useless discussions on an eternal themes.
My experience in the architecture design and development allows me to say - the development of OS is interesting at the beginning and in the middle is a boring job, it's one of the causes of OS market saturation product is covered only part of the requirements and products that are not so flexible.
I came to the conclusion that I can easily take the pieces of the finished OS components and quickly make a monolith in which the applications operate in the kernel mode - for the first time to write applications I need is a situable solution, and then choose right solution(if Jari OS is ready, I will take it).
I'm talking about the fact that I can't understand the meaning of many projects. These are two extremes as ever. The first - the development of general purpose operating systems, I believe that it is a useless idea. Second - the development of narrow-purpose operating system, the appointment is so narrow that except of the author of the project, no one can use it. I don't understand why nobody wants to create a platform for the OSes, flexible so that the possession of the editor (emacs for example), knowledge of the OS design and implementation you can get what you want.
The development of such platform, will not take a lot of time, if we have the necessary number of developers (1-3 years for production use).
I started to do this (since I've understood that multi-purpose OS isn't required now), and continues as the forces I have, but I do not have so much time and resources (just a few people).
I'm disappointed completely and forever, no - not in my projects and local fails, but in humans and OS market.

Tuesday, November 03, 2009

Interest and finance always diverge.

I always tried to combine the work on interesting (and even better on my own) project, and to capitalize on this.
I was never interested in large amounts of money.
I can say that it turned out, several times and with varying degrees of success. And I can do what the findings of these cases, which may be useful in the future.
First, if you managed to establish your own small company for money sponsoring organization, then you should not relax. Financing institution may devour you with your project. In such cases it makes sense to have several organizations concerned with various equity participation.
In general, your own company is a separate conversation.You will not be able to form a small company and develop further in most cases. I'm not a businessman, and therefore can't discuss other ways of creating a company. From the viewpoint of an engineer is better to go a different way. For example, you already have a project that is developed. It's not a secret that the development of the project rather resource-intensive exercise, and intensive development without funding is unlikely. In this case it's necessary to find company, which is interested in such kind of a project. If such company was found, then you hire to work and will assemble a team of specialists. But you shouldn't think then that life was a success, you will have a short time and insatiable wishes of management, and these suggestions will tend to rise.
The deal with the closed project makes no sense in this case - your exciting project will turn into another ugly products, and work on it will be the usual routine. Make your project licensed under free software license (GNU GPL/GNU LGPL), companies offer a product, not a software in most cases. Also, take an existing open and free source code to your project - no one is interested with development all things from scratch.
But don't forget - this symbiosis is not eternal. Ultimately, when your design will come from the door of the research department, the company will not expand the work on your project, instead, your work will be directed to the specific requirements of company management. At this stage it is necessary to separate the project from the company and develop it in an open community of developers. It follows from this - to work on the project is open, and in the broader framework than require your employers.
Well, Jari OS is on this stage now, and I hope that all works on this project will be helpful for community of free software. Also, I will continue works on project with smaller team.
In case of Jari OS, I had a several mistakes: a small open community, not documented internals.

Wednesday, September 16, 2009

Life?! Humppa!

Life can be difficult, bad, but it is life and it is a beautiful thing.
I've designed a new io layer for file system layer, also I've made refactoring in this layer - and ... all is going on !

Hi all.

Wednesday, September 02, 2009

My adventures.

Have you ever been in the adventure?
No, not in what is funny and stupid, simple and easy, but exciting, interesting and complex,which you would like to repeat, remember the adrenaline rush and the battle with elements.
I have a one type of this adventures: is a kayak travels, not on wild water, not for one day.
Imagine, close your eyes and imagine.
Wind is a storm wind and rain goes to your face, waves is a large long and high waves, noise of the breaking waves on the rocks is loud, splashes is around you.
You sit in a 5-meters narrow vessel less than one meter at its widest point, you have a paddle in your hands. Wave when turning ridge lean on you, it doused your kayak, it broken and you apply bank, sometimes when you pass to proceed from the ridge. Waves put you and fell on the stones.
Rudder pedal is turned completely to the end is the only way get to keep the right direction, paddle and your hands are ready for very quick movements.
Periodically nose kayak sinks a wave, you enter between the waves from times to times and you watch as you moves two meters of water column. Adrenaline are released in the blood, and you don't feel a cold around the cold water and wind, you amused danger. You can suddenly capsized and drowned in the waves, if you not alienate from the crest of a wave when time is not consistent, or if you do not stand up under the right degree of wave.
There is a chance that you will inflict on the rocks and broken on them, if you lost control and come close to a coast. Yes, this is an adventure, and it amused you. There is a situation and there is a possibility.
After the storm you will come out of the kayak and drink some alcohol with your friends, who also received a storm with you. And ... you will laugh, and you will have the desire to once again pass a storm.
These impressions for a long time will stay in your memory, you will remember your struggle with the waves and wind.
When you will sit in a warm house, thinking about current problems, you will want to come back again, to drop everything and go back to adventure again.
Fighting in the bustle of the city should not stay in the brain, it kills creativity, it don't makes a sense.
However, you choose between the warm beaches with sun loungers and cocktails, and real adventure.
I sometimes think that is more important, the first or second hobby, and always understand that my hobbies is all that I have, and theirs all important.

Tuesday, September 01, 2009

Second stage of Jari OS development is one year old!

Hi readers, I cannot keep silent about it. Here the full story - http://jarios.org/node/35.

Thanks.

Wednesday, August 26, 2009

Flat in memory file systems

Few days ago I've implemented vnode cache - it works and file system works faster. But now I'm working on file mappings.
I found an interesting subject due to mappings design: if you have a page aligned in memory file system - you can directly maps it's storage to address space of a process.
This will avoid extra memory loss - you don't need to copy/paste data from file system storage to memory.
For example, init.fs is a flat (but not page aligned) file system - and if we will provide a page aligned file system we can avoid memory loss.
This feature already implemented in some monolithic kernels, but I don't seen that in microkernel multiservice-based OSes.

Wednesday, August 19, 2009

Vnode cache design

I found a development quest in Jari OS: all begins from device drivers services design, some drivers are actually is a library that linking to service that support some hardware part, in this case driver developer folks need to have a dlopen()-family functions to link this libraries at runtime, for example if block service find a specific host it should be linked runtime, not statically. Well, OS hasn't dlopen-family support functions, saying more - dynamically linked ELF support are very early (it works, but it works slowly and takes many service time slice) - the reason is a poor file mapping support, you can map file, but when you will close file descriptor you will got a segfault on page fault in this mapped area. This is a complex problem, to support dlopen() , ELF support should be improved i.e. work with libraries and binaries should be going outside of process manager service and, not be done via read/write calls - one of the benefits will be a dlopen() support.
I know, that one of the weak place of the file subsystem layer is a libv2(it's a VFS-like layer, but implemented as a library that linked to each file system(don't confuse - VFS service is a filesystem manager, resource storage, redirection mechanism and so on)). Libv2 supports many POSIX features already, and operates via abstract vnode_t (in linux it's inode called) and there are too poor vnode cache, so every mapping is connected to vnode and if cache decide to destroy it - mapping can be lost. Like a solution is a map count handling, but it's a very simple - more complex solution is a creating more sophiscated cache.
Assume a file system with a high load, you are always performs lookups, maps, read, write operations - there are possible situation when vnode might not have positive map and open counters for a short amount of time, and there are possible rarely used vnode i.e. some vnode may be mapped sometime, but not accessible for a day - and it will waste you data structure and memory - and you cannot delete it at all. On other side - you cannot use vnode metadata that connected to real file system node, because lookup calls don't modify accessed time of this. Well, what the solution could be - first I determined several vnode stages with it's own features, second - I designed a table for vnodes that mapped, but doesn't accessed - it will not take resources as much as regular vnode takes, but virtually will present.
The next vnode stages might be:
  • strong vnode: not a candidate to move to the virtual vnode list, has a positive open and mapped count, and was recently used
  • potential oldies vnodes: has only mapped count or opened count positive and wasn't recently used
  • oldies vnode: vnodes in the virtual vnodes list
  • died vnodes: vnodes that was removed via unlink call
At defined period some thread will lookup the cache and decide what to do with vnode, this rate will be calculated by the file system activity (calls per last period). But what about died vnodes? Assume that you mapped file, somebody remove it, vnode id was taken by root only file ... well, died vnodes list as a low cost list that storage vnode of this types and when you will try to read or write to it, or you will unmap it, or got a page fault - information will be updated - you will got an error (or seg fault), file system will decrement the counters, and when this counters will zeroed - died vnode will go to heaven, and free it's id ;)
I'm not sure that I will implement all features at near time, but anyway this functionality will be added to libv2 - and I will continue my quest - I will improve page cache, improve memory events protocol, design and implement linker with dlopen ;)

Tuesday, July 07, 2009

My new kayak - seayak

On june I've bought a new kayak, manufacturer wrote in his list of products that this is a seayak type.
Well, like my double seat kayak it's a folding boat. I like folding kayaks in case of their mobility and storing place.
At current moment I've already made a two tests - is a two cycles of build, and yes - it's a on-water test.
Firstly I will give a more information about seayak:
Length: 4.9 meters
Weight: 22 kg
Frames: 4
Shipped with steering.
First built was taken up to 3 hours! And it's typically for new folding kayaks from triton ltd.
On first contruction I've broken one of the details and I was need to make an ugly hack that I've showed on image below:
Manufacturer has changed this detail for free via guarantee.
All construction set are typically , excluding head profile - it looks like on seayaks.
Second construction goes well and takes up to 1 hour and 10 minutes (via kayak manual it should take 62 minutes).
Letters with name of kayak model was removed and I think about new, original name for this boat.
Like a good design it has on-top stuff for small things, gps and other useful things on water (pack for cigarets or tobacco pipe ;) )
Generally it's a good model - it has a good on water performance, not so hard to build (not on first construction process;) ), good profile (I've tested it on big open water with big waves).
Now I'm thinking about new name ... and some modification - I want to add new on top fixtures for other things.
Other photos you can look here and here.

Saturday, May 09, 2009

SDK tools for OS development

New ideas came to my head, all about operating system features and development.

On last friday I decide to make some modifications on autotools stuff in GNU sed and try to compile it with Jari OS libc headers and functions. Yep - it's works! This means that our libc is ready for most of cases, many of them working now. This event gave me some new ideas about development tools for Jari OS: firstly I will port GNU gcc to Jari OS (I mean, that gcc will handle our ld scripts and libraries right, without many keys and other things), secondly I will construct SDK template (already done, but not publicied yet).
Also, to have a better support and big community additional feature needed - IDL integrated to SDK. Well, it's mean - that you have set of the templates for each type of service i.e. block device driver library, char device driver, file system and so on ...
In most cases you shouldn't known about specific internal things of system libraries, also you shouldn't care about interface changes (headers renaming, new functions, new order of calls and so on). This feature will generate libraries specific code in compile time and link your specific implementation. I.e. what will be written one time - will live all time ;)
For this feature it's better to use scheme (for templates, configs and other stuff) and some core of this will be written traditionally on C.

Wednesday, April 22, 2009

Jari OS going to be a completely OS

My last task in Jari OS project was symlinks supports, now I'm working on postponed logic in libv2 (general library for file system services).
I cannot be on track in changes ;) Now project has a POSIX, fork(), execve(), file systems, initial networking support.
On background I'm porting GNU coreutils to Jari OS, progress going fast, I think that alpha release will has many interesting stuff.
To be a fully featured OS not so many features need, our ext2fs support and IDE drivers are 80% completed.
I think that year of active development that will finished on 1-sep-2009 will made a first beta version with few drivers and networking support.
Now prject has one network device support (e1000) and Jari OS host (network service) replies on ICMP requests.
What about ELF support, now we're working on dynamic libraries support - it coming soon, currently only static ones supported.
Files i/o is ready, but maybe some functions (not often used ones) doesn't works properly. To make a fully featured shell pipes requered, and I'm working on it now.
I'm planning to implement shared memory fs and POSIX functions for it, in addition I will add Jari OS specific API for this, - for some internal purposes, for GIX(GUI service of Jari) like example.
After alpha release (1-jun-2009) I will take time for GIX implementation. Also - there are many porting tasks.
I hope community of open-source developers will be interested in new microkernel OS - it's a good field for research, development and it's good to take skills up-to-date.
So,

Thursday, March 12, 2009

Jari OS: fork() in microkernel based OS

Well, this day makes huge parts of my brains works.
Designing microkernel systems you have a good skills base, designing microkernel and multiservice system you have an excellent skills base.
Today was a day of fork() call - it has many things to discuss, think, imaginate and so on.
At the end of a day, after all problems was detected - I found a solution to solve fork() call.
Let's begin and I will explain fork() nature as is...
Fork is a POSIX call that creates a clone of a calling process - with one thread, but the clone should have the same resources - i.e. memory (not at all), opened file descriptors, IPC stuff.
In Jari OS resources are isolated - VFS know about files, kernel knowns about IPC and so on - you will have a headache about syncronization, performance - always while architecture design - welcome to microkernel-based OS.
In this case you need to develop non-trivial trick with things that monolithic kernels made easly, but ... microkernel-based OSes is clever and better at all.
Well, you have a task, you have a resources and you need to fork() POSIX call - the first solution is simple - just delegate this job to process server and deal is made, within server timeslice - and this is bad (but QNX does it in this way). Other solution is more sophiscated - and you need to make a stages for fork():
  • Request to delegate a fork() for myself
  • Make a fork() skel within my timeslice
  • Send a request about fork() call made by me
In this big deal you will affect microkernel, process service, virtual file system service and your task.
Going deeper I will explain each stage of fork() and after those I will explain how it works on different sides.
Your first request is a request to the process service where you tell that you wants to make a fork(). Process service will create a mould on VFS of your resources, toggle bit in kernel that allow you to make a sys_fork() syscall, and if all is ok it will reply with ok message :)
After this you are going to make a syscall to fork - be warn! kernel toggle off bit that allows you to make this call, and you cannot do it yourself - only trusted process service does.
Call is done - you have a skel - you are making a second call to process service - i.e. - "fork is done - please make my child runnable" - process service tells VFS to make mould active and working - and changes status of the child - and replies you with "OK all is done".
This scheme allows task to make a one fork() call at a time (i.e. you are able to make other child after old child is created) - I don't think that this is a bad, other architecture going to make other applications design.
On this I want to explain some details.
"Forkable bit" can be toggled on only by trusted process, and it's switched off while fork called.
Task has a timestamp - and both kernel and process service known it at the start of all.
VFS's mould has a parent and while parent is exist - mould will not be deleted, orphaned mould will be deleted with timeout.
Checking for a valid child pid makes not only on parent pid - also on timestamp.

Uff, I can explain other details in comments.

Monday, March 09, 2009

File systems on Jari OS

There was a great job on last two weeks. File subsystem are designed in Jari OS - since we have a system with separated services - libraries is a good approach to avoid a huge amount of code and errors with changes it.
Actually file subsystem is divided by several general parts:
  • VFSv2 service
  • set of libraries for each filesystem
I don't want to include all stuff for filesystem in one library, because it's a bad idea, for example tmpfs doesn't has backend to block device, pagecache, postponed calls and so on. But anyway this set of libraries can be completed by one, generic library that should be included to each file system - libv2.
In this case we're have the following list:
  • libv2 (general library)
  • libv2backend (backend to the block device layer)
  • libv2pgcache (page cache library)
  • libv2ppcall (library serves postponed calls)
Each file system architecture should determine what will be used, generally , regular file system will always use all set of libraries - i.e. it works with block device and use page cache in this case, and it will not replies immediately on all cases - it will have postponed calls list to reply.
For libv2 - there are nothing difference - it will always support everything on logic layer.
I can't see other solutions - our VFS service should be overloaded - and many calls going directly to file system service.
So, while implementation tmpfs and initfs - implementation hasn't any complexities.

Wednesday, February 25, 2009

Scheme VM like a microkernel service

Yes, it's an idea. I'm thinking about benefits in the microkernel and multiservice OS on desktop or some kind of development environment. Well, I know all the things that going in microkernel while you calling fork() after those exec() - it takes a lot of system time.
Usually within shell you makes fork()/exec() every time when you starts anything - ls,cat ... or you can wrong with some arguments for utility and this will increase system job time. Also, it's a waste of a time to implement small utils on C, there are few POSIX things, few useful things, few OS specific things and so on. Better one - run login/shell environment within VM context - like a thread, all utils will runs like a thread (thread creation in Jari OS isn't so wasteful) and writing simple things on scheme is a simple and fast process. It doesn't mean that Jari OS reject exec/fork and other POSIX calls, it's mean that we can take a benefits on desktop I think.
Many people told me that microkernel is sucks and microkernel is bad for desktop - well, most of them doesn't know anything about OS design, but others are right in some case. But I want to make some note - there are no existing microkernel (and multiservice) OSes with a pure microkernel and multiservers design (I don't take early, prototype-like, featureless systems, like Jari OS and some line of other interest projects, HelenOS for example) - yes it's a first thing to think that microkernel sucks, second it's a performance - yes context switching, calls partially going outside of the caller time slice and so on, it's not the problem as well, for most I/O operations context switch and related thing is very little compare to disk I/O or other I/O (no, I don't mean highband special devices), also most of technical microkernel critics based on existing solution - that can has ugly design (I don't want to make a shit showing examples here).
My thesis - Well designed and implemented microkernel and multiservice OS can works at the same speed as monolithic OS works, but ugly designed monolithic OS will be ran slowly then microkernel based OS with good design. Also, you should remember that some abstractions in monolith made within user space - like example GUIs, and I don't think that some operations will be slowly on microkernel.
And ... the main idea - for microkernel world you should keep in mind microkernel oriented solutions, instead of porting old and ugly tricks from monolithic - schemevm running as microkernel service is such solution.