Deeper than deep

Something more: about general system high-level design

2011-10-18T10:41:00.003+03:00

Well, I want to repeat something again, but with more words.

Today I want to introduce intercommunication design within the
whole system.
Firstly, I want to tell - IDL is a good idea in theory and is a bad idea
on practice.
The main reason - it's implementation. IDL usually describe interfaces and,
in addition, creates some code, that packs the request, get it, call it, and, finally,
reply. I.e. receive, call and reply to the message, nothing more.
But, in our system we have something more complex, let's enlist it:

IPC message forwarding
Postponed calls (blocking operations)
IPC message modification operations

Well, on that point you need something more featured than receive - call - reply
cycle, you need to determine when you need to forward a message, on which
point you need to do it, what is immutable message parts and what parts
should be modified and/or cut off.

The other "funny" thing is postponed messages, in this case IDL generated code
must contain many stuff like quick allocation of memory and other stuff.
Finally, generated code is coming to be huge, and in addition you need to modify
it in some different cases.

Another problem is a problem with interfaces, i will try to describe it briefly.
For each instance you have an interface with a predefined set of functions,
many functions are similar, or identical: for example, device has a read() function,
and file has a read() function, but you have a one set of interfaces for file system
and other set of interfaces for devices. That mean that you need to try develop some
generic interface, and extend it every time with your own functions again and again.
On practice you will have a big set of libraries with interfaces and it will be a real headache.

From other point of view, you can resolve the problem with it on the libc client side,
just determine the resource and call specific function i.e. POSIX read() in our case
will call read_file() or read_device() functions depends on the opened resource.
I think - the last idea is very very ugly, I don't think that it's a good idea to change libc
every time when you adding something new to the system, also, we should be modular,
in some cases I can turn off support for pipes of sockets - and I don't want to recompile
libc, all the other system parts for it.

That's why IDL sucks, I spent many time to solve the problems with nontrivial things like
message advanced control and I don't see the sense. But, the old one - uni_rpc_t is sucks too,
on practice uni_rpc_t creates more problems that it's should solve.

And, I faced that pretty solution going from other way to solve this problem.
Each time while you design RPC/IDL/something_else_... you should think more closely
to the system and it's objects, my error was in the way to solve this problem, I think
about RPC/IDL/etc ... only, but you need to think about system also.
What I mean: system might be presented like a set of objects hosted on different servers and
interacting each one with other, and in this case there are no difference between
file and task, i.e. file, task, device, ipc object - it's node, with predefined set of functions.
Well, yes, task and file has a different operations, but all possible operation might
be represented via one set.
Anyway you will have a very similar operations, in example, changing file owner uid and changing
task effective uid is very similar, on RPC level there are no difference between it.
What I decide: I decide to represent all objects (or it's high-level representation) as a node,
each node has a 2 groups of operations, first one for control the node and manage it attributes,
second one for data i/o (yep, task will doesn't have data i/o interface, but we can do it, if needed).
In that case RPC is going to be simple, you have RPC signature in every message,
this signature points to the following things:

Function group
Function within group
IDs to select right node for operation

Other, after RPC signature, is going to the implementation.

How are ipcbox and sbuf abstractions used ?
It's a good question with a brief and simple answer, ipcbox is used by the
rpc library routines to operate with IPC, sbuf is used for data, because implementation
gets the sbuf like a data, rpc code don't allocating anything for implementation,
just sbuf.
Why is it pretty ?
Because it's simple and this solution doesn't require to implement a huge set of
API for each task, i.e. good old getvfspid() getsomeothercrap() bla bla bla.
Each task, has the special system reserved iolink, depends on task role (file system,
regular task, translator, resource carrier, etc ...) its node has a set of operations.
For example, to make a fork() you just need to make a control request via system
iolink to your node with fork() related data, to change your effective uid you just
send a stat request to your node representation, but if you are a regular task (i.e. doesn't have a rights
to link file system onto namespace tree) your node don't have the operation of such kind,
otherwise you will able to make a special call to your system iolink to do it.
Is it simple ? I guess it's very simple solution.

Jari OS RPCv7 and Vnode

2011-10-07T13:13:00.002+03:00

I wrote before why IDL failed. And I know that old uni_rpc_t RPC is failed too, it's not flexible, ugly and slow.

I'm working on this system about 5-6 years, well, and there are 6 attempts to create good IPC/RPC chain:

- v1 create a lot of structures and calls and implement it on the microkernel side

- v2 create one generic RPC in microkernel side

- v3 remove it from microkernel

- v4/v5 uni_rpc_t and related, try to assign all operation to the standard POSIX calls

- v6 IDL

None of this approaches are applicable.

I guess it's all about a way to resolve the problem. Each time I was try to solve one problem, not the set of those.

But, let's back to my favorite method of problem solving: divide and solve.

When you have a one big problem, it's better to divide it to many little ones, and solve it - one by one, and finally your big problem will be solved. Simple, isn't it ?

And now, we have a system, with many objects, with many problems, but ... make a guess, all of operations within this complex system is requests to the objects.

In that case, you just need to create the one generic object that may represent anything you want and provide generic interface to this object.

Well, you are welcome to the RPCv7.

Usually you can divide all your operations between several groups, to take a data, to control object and so on. Also, in Jari OS you need to have session with this object. That's all.

All your requests might be described with one generic header, not the data of the requests, not the some specific flags like it was done in v4/v5 versions, the header will describe request only.

Checkout this http://jarios.org/gitweb/?p=jari_os.git;a=blob;f=libs/uclibc/libc/sysdeps/jarios/common/sys/rpcv7.h;h=578b978d69fd2d2b6d0f0b990b231114d1058dc2;hb=HEAD

struct rpc_signature.

Other stuff, are not RPCv7 stuff, it might be your data, headers and so on.

It was a first small problem solved. The second one is object representation.

And I got it, vnode, all things might be represented as vnode, it might have a memory mapping, set of many specific operations, many IDs, relations ... and RPC callbacks, the standard set

of it.

Here you can look at this ones: http://jarios.org/gitweb/?p=jari_os.git;a=blob;f=libs/uclibc/libc/sysdeps/jarios/common/sys/vnode.h;h=f466d9b1e8542ab7e4e54dff8ada5a24890c0761;hb=HEAD

All you need - just implement this functions and forget about IPC/RPC headache.

That's an example of the fork() - http://jarios.org/gitweb/?p=jari_os.git;a=blob;f=libs/uclibc/libc/sysdeps/jarios/common/fork.c;h=b18fd40e95f8a1016736cf83aada8689e9432375;hb=HEAD

And carrier function:

http://jarios.org/gitweb/?p=jari_os.git;a=blob;f=services/nscarrier/vnode_task.c;h=006d0f7573b7771b7c140e1262c2f39d16c5a671;hb=HEAD

All data going via sbuf, just forget about IPC/RPC chain.

Well, I will write later about this implementation more, but you can always checkout the fresh sources.

Why IDL fail, or, sometimes you need to fail first, to rise next

2011-10-07T12:16:00.003+03:00

IDL for microkernel multiservice system is fail. No, not just because it wasn't implemented, going deeper - it's implemented.

Okay , let's try to understand why, but first - I spent a very lot of time to design and implement IDL for this purposes without any applicable results - that's my fail first, and that's an idea fail second.

Usually you have a simple IDL case: take the message, parse it, call the server function, serialize it and reply. It's common for client-server architecture widely used, and yes, there are many already implemented IDLs for this simple case. Most of IDLs covers this needs.

You can told me about L4 IDL - you are welcome, I didn't see any serios developed with L4 IDL, and I can told you - it will suck.

My approach was more complex.

In case of Jari OS services you need more than receive - call - reply cycle. Request might be postponed, or changed and forwarded.

In real system (not like L4 ping-pong demos), to resolve resource by the name (symlinks and files) you always need to forward it, because symlink and file might be located on different servers. Also, on real system, blocking of server threads is a quite poor idea.

IDL concept was very simple before I've added postponed and forwarded messages, but it wan't so fatally, to get away with this idea.

The next problem was faced later, like time-bomb, it was message modification.

Finally I've got something huge, and not flexible and scalable.

IDL implementation with support of this features are not simply than some RPC interface.

Well, at this point I decide to stop it. I spent something about 1 year to design, research, test and try it - and got the ugly solution.

However, in Jari OS you will be able to use IDL with this features, but it will not used in the base system.

IDL research implementation here - http://jarios.org/gitweb/?p=jari_os.git;a=summary , checkout idlc branch.

Approach to the file system design

2010-05-12T19:59:00.002+03:00

Within microkernel operating system you may do different kind of interesting things.

Today is a day of file systems, I'm currently working on the file layer general design and there are some feature to implement, well, I can told that it's a several features implemented may.

The first is a copy-on-write feature, well it might be implemented over the file system implementation.

while you have an vnode on top of the implementation you may create a virtual remote link to other file system vnode, and it will goes transparently for implementation. When you will got a miss on the block device of the current filesystem you may take this data from another one, remotely.

Another case of this feature is a creating a new function that will clone files in the fast way, just creating a shadow vnode on the destination file system.

I will research this cases later and write it here, it's not so good to write it in bus.

Protocol points: is IDL a mixture for headache?

2010-04-14T09:26:00.003+03:00

There are a problem with a distributed message passing based system - it's a protocol.

It's a very bad to implement all protocol stuff manually, you will spent a many time on the

very simple, but annoying job, and you will create s new bugs, a lot of ugly code and so on ...

In the real distributed systems is many kind of intercommunication exist, I mean that you need some protocol for system-wide bus, some other protocol for services of the group A and some other for the other groups. Usually you will has a tons of different protocols and it will be a problem to create new one or update/modify existing.

I found two ways to fix this headache and one of this way is widely used in the systems of such kind, it's IDL. It may be ORB/CORBA like interface description language and usually is it.

Mainly, you need to pass some typed message and get some typed response to this message, and this problem is finely solved in CORBA-like IDLs, in contrast if you are working in the complex distributed system you need to have many indirect message passing system that will impact to your protocol.

In the system of such kind you also need to determine the message sender, his priority, and his level of the access or, more widely, what the access rights has this message. While you are going deeper you will understand that for some complex systems (like distributed system for transport queue management) that should works with determined time range you will need something like QoS (yep, I mean here quality of service). There impacts are going to your protocol.

Now, imagine that you have a CORBA-like IDL and you need extended option described above ...

You will describe some proto on IDL, and you will implement QoS, protocol for access inheritance manually ... In this case it's a good point to think where is headache located ?

The other way - it's a runtime solution for passing arguments via messages it will has the same disadvantages, but will be simply in implementation.

Well, the question is open - where is a mixture for headache with distributed message passing ?

The venture should always have a logical end.

2009-12-08T21:14:00.003+02:00

Recent months have become a complete disappointment in my life, but my life is not so interesting, by the way my life has a feedback to my projects.
I'm a little upset, to be honest, I'm defeated with project development, and mostly with it goals.
I'm afraid that it will be not so possible to complete the Jari OS as a platform, but the hope remains.
I have a look at the project from the other side, from the side of engineer need to choose a base platform for the final solution. Jari OS isn't ready, and will not ready for several years (I don't think that good news happens) and it's very sad. Anyway, I'm learn again and I have a couple of real problems in the design architecture, management and so on, I know - there are many stuff written and works now. I cannot told that the project failed in case of it's basics concepts or other more deeper reasons, and I cannot told that project is succeful - this venture project hasn't a logical end, but it should has.
Well, I decide reorganize my works on the project and try to give a new life to Jari OS as soon as it possible.
First, I will write a documentation about platform, fix a design and so on.
Second, I will try to have a wide view of the future requirements.

When Jari OS will finished (it might be fail, might be not) - this will be the logic end of the venture, if it will has a continue - it will became a serious project.

OS choice

2009-11-06T01:37:00.002+02:00

I have a direct bearing on the development of operating systems, so I can do some analysis of existing solutions on the market. The market of operating systems is full, it is filled to the brim. What? There are many solutions for different domains. Now, I'm faced with a choice of OS for the next project and I can't say that among the existing (Jari OS is a good platform, but aren't ready, and as a woman requires a lot of time) I can find a suitable, I considered the decision on free software, or are not expensive ones - I don't found suitable one. I will not describe my criteria, and I don't want to dilute the flame of useless discussions on an eternal themes.
My experience in the architecture design and development allows me to say - the development of OS is interesting at the beginning and in the middle is a boring job, it's one of the causes of OS market saturation product is covered only part of the requirements and products that are not so flexible.
I came to the conclusion that I can easily take the pieces of the finished OS components and quickly make a monolith in which the applications operate in the kernel mode - for the first time to write applications I need is a situable solution, and then choose right solution(if Jari OS is ready, I will take it).
I'm talking about the fact that I can't understand the meaning of many projects. These are two extremes as ever. The first - the development of general purpose operating systems, I believe that it is a useless idea. Second - the development of narrow-purpose operating system, the appointment is so narrow that except of the author of the project, no one can use it. I don't understand why nobody wants to create a platform for the OSes, flexible so that the possession of the editor (emacs for example), knowledge of the OS design and implementation you can get what you want.
The development of such platform, will not take a lot of time, if we have the necessary number of developers (1-3 years for production use).
I started to do this (since I've understood that multi-purpose OS isn't required now), and continues as the forces I have, but I do not have so much time and resources (just a few people).
I'm disappointed completely and forever, no - not in my projects and local fails, but in humans and OS market.

Interest and finance always diverge.

2009-11-03T11:16:00.004+02:00

I always tried to combine the work on interesting (and even better on my own) project, and to capitalize on this.
I was never interested in large amounts of money.
I can say that it turned out, several times and with varying degrees of success. And I can do what the findings of these cases, which may be useful in the future.
First, if you managed to establish your own small company for money sponsoring organization, then you should not relax. Financing institution may devour you with your project. In such cases it makes sense to have several organizations concerned with various equity participation.
In general, your own company is a separate conversation.You will not be able to form a small company and develop further in most cases. I'm not a businessman, and therefore can't discuss other ways of creating a company. From the viewpoint of an engineer is better to go a different way. For example, you already have a project that is developed. It's not a secret that the development of the project rather resource-intensive exercise, and intensive development without funding is unlikely. In this case it's necessary to find company, which is interested in such kind of a project. If such company was found, then you hire to work and will assemble a team of specialists. But you shouldn't think then that life was a success, you will have a short time and insatiable wishes of management, and these suggestions will tend to rise.
The deal with the closed project makes no sense in this case - your exciting project will turn into another ugly products, and work on it will be the usual routine. Make your project licensed under free software license (GNU GPL/GNU LGPL), companies offer a product, not a software in most cases. Also, take an existing open and free source code to your project - no one is interested with development all things from scratch.
But don't forget - this symbiosis is not eternal. Ultimately, when your design will come from the door of the research department, the company will not expand the work on your project, instead, your work will be directed to the specific requirements of company management. At this stage it is necessary to separate the project from the company and develop it in an open community of developers. It follows from this - to work on the project is open, and in the broader framework than require your employers.
Well, Jari OS is on this stage now, and I hope that all works on this project will be helpful for community of free software. Also, I will continue works on project with smaller team.
In case of Jari OS, I had a several mistakes: a small open community, not documented internals.

Life?! Humppa!

2009-09-16T01:29:00.002+03:00

Life can be difficult, bad, but it is life and it is a beautiful thing.
I've designed a new io layer for file system layer, also I've made refactoring in this layer - and ... all is going on !

Hi all.

My adventures.

2009-09-02T01:12:00.003+03:00

Have you ever been in the adventure?
No, not in what is funny and stupid, simple and easy, but exciting, interesting and complex,which you would like to repeat, remember the adrenaline rush and the battle with elements.
I have a one type of this adventures: is a kayak travels, not on wild water, not for one day.
Imagine, close your eyes and imagine.
Wind is a storm wind and rain goes to your face, waves is a large long and high waves, noise of the breaking waves on the rocks is loud, splashes is around you.
You sit in a 5-meters narrow vessel less than one meter at its widest point, you have a paddle in your hands. Wave when turning ridge lean on you, it doused your kayak, it broken and you apply bank, sometimes when you pass to proceed from the ridge. Waves put you and fell on the stones.
Rudder pedal is turned completely to the end is the only way get to keep the right direction, paddle and your hands are ready for very quick movements.
Periodically nose kayak sinks a wave, you enter between the waves from times to times and you watch as you moves two meters of water column. Adrenaline are released in the blood, and you don't feel a cold around the cold water and wind, you amused danger. You can suddenly capsized and drowned in the waves, if you not alienate from the crest of a wave when time is not consistent, or if you do not stand up under the right degree of wave.
There is a chance that you will inflict on the rocks and broken on them, if you lost control and come close to a coast. Yes, this is an adventure, and it amused you. There is a situation and there is a possibility.
After the storm you will come out of the kayak and drink some alcohol with your friends, who also received a storm with you. And ... you will laugh, and you will have the desire to once again pass a storm.
These impressions for a long time will stay in your memory, you will remember your struggle with the waves and wind.
When you will sit in a warm house, thinking about current problems, you will want to come back again, to drop everything and go back to adventure again.
Fighting in the bustle of the city should not stay in the brain, it kills creativity, it don't makes a sense.
However, you choose between the warm beaches with sun loungers and cocktails, and real adventure.
I sometimes think that is more important, the first or second hobby, and always understand that my hobbies is all that I have, and theirs all important.

Second stage of Jari OS development is one year old!

2009-09-01T22:29:00.001+03:00

Hi readers, I cannot keep silent about it. Here the full story - http://jarios.org/node/35.

Thanks.

Flat in memory file systems

2009-08-26T01:34:00.002+03:00

Few days ago I've implemented vnode cache - it works and file system works faster. But now I'm working on file mappings.
I found an interesting subject due to mappings design: if you have a page aligned in memory file system - you can directly maps it's storage to address space of a process.
This will avoid extra memory loss - you don't need to copy/paste data from file system storage to memory.
For example, init.fs is a flat (but not page aligned) file system - and if we will provide a page aligned file system we can avoid memory loss.
This feature already implemented in some monolithic kernels, but I don't seen that in microkernel multiservice-based OSes.

Vnode cache design

2009-08-19T23:39:00.003+03:00

I found a development quest in Jari OS: all begins from device drivers services design, some drivers are actually is a library that linking to service that support some hardware part, in this case driver developer folks need to have a dlopen()-family functions to link this libraries at runtime, for example if block service find a specific host it should be linked runtime, not statically. Well, OS hasn't dlopen-family support functions, saying more - dynamically linked ELF support are very early (it works, but it works slowly and takes many service time slice) - the reason is a poor file mapping support, you can map file, but when you will close file descriptor you will got a segfault on page fault in this mapped area. This is a complex problem, to support dlopen() , ELF support should be improved i.e. work with libraries and binaries should be going outside of process manager service and, not be done via read/write calls - one of the benefits will be a dlopen() support.
I know, that one of the weak place of the file subsystem layer is a libv2(it's a VFS-like layer, but implemented as a library that linked to each file system(don't confuse - VFS service is a filesystem manager, resource storage, redirection mechanism and so on)). Libv2 supports many POSIX features already, and operates via abstract vnode_t (in linux it's inode called) and there are too poor vnode cache, so every mapping is connected to vnode and if cache decide to destroy it - mapping can be lost. Like a solution is a map count handling, but it's a very simple - more complex solution is a creating more sophiscated cache.
Assume a file system with a high load, you are always performs lookups, maps, read, write operations - there are possible situation when vnode might not have positive map and open counters for a short amount of time, and there are possible rarely used vnode i.e. some vnode may be mapped sometime, but not accessible for a day - and it will waste you data structure and memory - and you cannot delete it at all. On other side - you cannot use vnode metadata that connected to real file system node, because lookup calls don't modify accessed time of this. Well, what the solution could be - first I determined several vnode stages with it's own features, second - I designed a table for vnodes that mapped, but doesn't accessed - it will not take resources as much as regular vnode takes, but virtually will present.
The next vnode stages might be:

strong vnode: not a candidate to move to the virtual vnode list, has a positive open and mapped count, and was recently used
potential oldies vnodes: has only mapped count or opened count positive and wasn't recently used
oldies vnode: vnodes in the virtual vnodes list
died vnodes: vnodes that was removed via unlink call

At defined period some thread will lookup the cache and decide what to do with vnode, this rate will be calculated by the file system activity (calls per last period). But what about died vnodes? Assume that you mapped file, somebody remove it, vnode id was taken by root only file ... well, died vnodes list as a low cost list that storage vnode of this types and when you will try to read or write to it, or you will unmap it, or got a page fault - information will be updated - you will got an error (or seg fault), file system will decrement the counters, and when this counters will zeroed - died vnode will go to heaven, and free it's id ;)
I'm not sure that I will implement all features at near time, but anyway this functionality will be added to libv2 - and I will continue my quest - I will improve page cache, improve memory events protocol, design and implement linker with dlopen ;)

My new kayak - seayak

2009-07-07T23:15:00.006+03:00

On june I've bought a new kayak, manufacturer wrote in his list of products that this is a seayak type.
Well, like my double seat kayak it's a folding boat. I like folding kayaks in case of their mobility and storing place.
At current moment I've already made a two tests - is a two cycles of build, and yes - it's a on-water test.
Firstly I will give a more information about seayak:
Length: 4.9 meters
Weight: 22 kg
Frames: 4
Shipped with steering.
First built was taken up to 3 hours! And it's typically for new folding kayaks from triton ltd.
On first contruction I've broken one of the details and I was need to make an ugly hack that I've showed on image below:
Manufacturer has changed this detail for free via guarantee.
All construction set are typically , excluding head profile - it looks like on seayaks.
Second construction goes well and takes up to 1 hour and 10 minutes (via kayak manual it should take 62 minutes).
Letters with name of kayak model was removed and I think about new, original name for this boat.
Like a good design it has on-top stuff for small things, gps and other useful things on water (pack for cigarets or tobacco pipe ;) )
Generally it's a good model - it has a good on water performance, not so hard to build (not on first construction process;) ), good profile (I've tested it on big open water with big waves).
Now I'm thinking about new name ... and some modification - I want to add new on top fixtures for other things.
Other photos you can look here and here.

SDK tools for OS development

2009-05-09T19:31:00.003+03:00

New ideas came to my head, all about operating system features and development.

On last friday I decide to make some modifications on autotools stuff in GNU sed and try to compile it with Jari OS libc headers and functions. Yep - it's works! This means that our libc is ready for most of cases, many of them working now. This event gave me some new ideas about development tools for Jari OS: firstly I will port GNU gcc to Jari OS (I mean, that gcc will handle our ld scripts and libraries right, without many keys and other things), secondly I will construct SDK template (already done, but not publicied yet).
Also, to have a better support and big community additional feature needed - IDL integrated to SDK. Well, it's mean - that you have set of the templates for each type of service i.e. block device driver library, char device driver, file system and so on ...
In most cases you shouldn't known about specific internal things of system libraries, also you shouldn't care about interface changes (headers renaming, new functions, new order of calls and so on). This feature will generate libraries specific code in compile time and link your specific implementation. I.e. what will be written one time - will live all time ;)
For this feature it's better to use scheme (for templates, configs and other stuff) and some core of this will be written traditionally on C.

Jari OS going to be a completely OS

2009-04-22T14:02:00.002+03:00

My last task in Jari OS project was symlinks supports, now I'm working on postponed logic in libv2 (general library for file system services).
I cannot be on track in changes ;) Now project has a POSIX, fork(), execve(), file systems, initial networking support.
On background I'm porting GNU coreutils to Jari OS, progress going fast, I think that alpha release will has many interesting stuff.
To be a fully featured OS not so many features need, our ext2fs support and IDE drivers are 80% completed.
I think that year of active development that will finished on 1-sep-2009 will made a first beta version with few drivers and networking support.
Now prject has one network device support (e1000) and Jari OS host (network service) replies on ICMP requests.
What about ELF support, now we're working on dynamic libraries support - it coming soon, currently only static ones supported.
Files i/o is ready, but maybe some functions (not often used ones) doesn't works properly. To make a fully featured shell pipes requered, and I'm working on it now.
I'm planning to implement shared memory fs and POSIX functions for it, in addition I will add Jari OS specific API for this, - for some internal purposes, for GIX(GUI service of Jari) like example.
After alpha release (1-jun-2009) I will take time for GIX implementation. Also - there are many porting tasks.
I hope community of open-source developers will be interested in new microkernel OS - it's a good field for research, development and it's good to take skills up-to-date.
So,

Jari OS: fork() in microkernel based OS

2009-03-12T00:54:00.003+02:00

Well, this day makes huge parts of my brains works.
Designing microkernel systems you have a good skills base, designing microkernel and multiservice system you have an excellent skills base.
Today was a day of fork() call - it has many things to discuss, think, imaginate and so on.
At the end of a day, after all problems was detected - I found a solution to solve fork() call.
Let's begin and I will explain fork() nature as is...
Fork is a POSIX call that creates a clone of a calling process - with one thread, but the clone should have the same resources - i.e. memory (not at all), opened file descriptors, IPC stuff.
In Jari OS resources are isolated - VFS know about files, kernel knowns about IPC and so on - you will have a headache about syncronization, performance - always while architecture design - welcome to microkernel-based OS.
In this case you need to develop non-trivial trick with things that monolithic kernels made easly, but ... microkernel-based OSes is clever and better at all.
Well, you have a task, you have a resources and you need to fork() POSIX call - the first solution is simple - just delegate this job to process server and deal is made, within server timeslice - and this is bad (but QNX does it in this way). Other solution is more sophiscated - and you need to make a stages for fork():

Request to delegate a fork() for myself
Make a fork() skel within my timeslice
Send a request about fork() call made by me

In this big deal you will affect microkernel, process service, virtual file system service and your task.
Going deeper I will explain each stage of fork() and after those I will explain how it works on different sides.
Your first request is a request to the process service where you tell that you wants to make a fork(). Process service will create a mould on VFS of your resources, toggle bit in kernel that allow you to make a sys_fork() syscall, and if all is ok it will reply with ok message :)
After this you are going to make a syscall to fork - be warn! kernel toggle off bit that allows you to make this call, and you cannot do it yourself - only trusted process service does.
Call is done - you have a skel - you are making a second call to process service - i.e. - "fork is done - please make my child runnable" - process service tells VFS to make mould active and working - and changes status of the child - and replies you with "OK all is done".
This scheme allows task to make a one fork() call at a time (i.e. you are able to make other child after old child is created) - I don't think that this is a bad, other architecture going to make other applications design.
On this I want to explain some details.
"Forkable bit" can be toggled on only by trusted process, and it's switched off while fork called.
Task has a timestamp - and both kernel and process service known it at the start of all.
VFS's mould has a parent and while parent is exist - mould will not be deleted, orphaned mould will be deleted with timeout.
Checking for a valid child pid makes not only on parent pid - also on timestamp.

Uff, I can explain other details in comments.

File systems on Jari OS

2009-03-09T16:27:00.003+02:00

There was a great job on last two weeks. File subsystem are designed in Jari OS - since we have a system with separated services - libraries is a good approach to avoid a huge amount of code and errors with changes it.
Actually file subsystem is divided by several general parts:

VFSv2 service
set of libraries for each filesystem

I don't want to include all stuff for filesystem in one library, because it's a bad idea, for example tmpfs doesn't has backend to block device, pagecache, postponed calls and so on. But anyway this set of libraries can be completed by one, generic library that should be included to each file system - libv2.
In this case we're have the following list:

libv2 (general library)
libv2backend (backend to the block device layer)
libv2pgcache (page cache library)
libv2ppcall (library serves postponed calls)

Each file system architecture should determine what will be used, generally , regular file system will always use all set of libraries - i.e. it works with block device and use page cache in this case, and it will not replies immediately on all cases - it will have postponed calls list to reply.
For libv2 - there are nothing difference - it will always support everything on logic layer.
I can't see other solutions - our VFS service should be overloaded - and many calls going directly to file system service.
So, while implementation tmpfs and initfs - implementation hasn't any complexities.

Scheme VM like a microkernel service

2009-02-25T00:48:00.003+02:00

Yes, it's an idea. I'm thinking about benefits in the microkernel and multiservice OS on desktop or some kind of development environment. Well, I know all the things that going in microkernel while you calling fork() after those exec() - it takes a lot of system time.
Usually within shell you makes fork()/exec() every time when you starts anything - ls,cat ... or you can wrong with some arguments for utility and this will increase system job time. Also, it's a waste of a time to implement small utils on C, there are few POSIX things, few useful things, few OS specific things and so on. Better one - run login/shell environment within VM context - like a thread, all utils will runs like a thread (thread creation in Jari OS isn't so wasteful) and writing simple things on scheme is a simple and fast process. It doesn't mean that Jari OS reject exec/fork and other POSIX calls, it's mean that we can take a benefits on desktop I think.
Many people told me that microkernel is sucks and microkernel is bad for desktop - well, most of them doesn't know anything about OS design, but others are right in some case. But I want to make some note - there are no existing microkernel (and multiservice) OSes with a pure microkernel and multiservers design (I don't take early, prototype-like, featureless systems, like Jari OS and some line of other interest projects, HelenOS for example) - yes it's a first thing to think that microkernel sucks, second it's a performance - yes context switching, calls partially going outside of the caller time slice and so on, it's not the problem as well, for most I/O operations context switch and related thing is very little compare to disk I/O or other I/O (no, I don't mean highband special devices), also most of technical microkernel critics based on existing solution - that can has ugly design (I don't want to make a shit showing examples here).
My thesis - Well designed and implemented microkernel and multiservice OS can works at the same speed as monolithic OS works, but ugly designed monolithic OS will be ran slowly then microkernel based OS with good design. Also, you should remember that some abstractions in monolith made within user space - like example GUIs, and I don't think that some operations will be slowly on microkernel.
And ... the main idea - for microkernel world you should keep in mind microkernel oriented solutions, instead of porting old and ugly tricks from monolithic - schemevm running as microkernel service is such solution.

Jari OS: device drivers development wellness

2008-11-21T17:20:00.004+02:00

I have a lot of time spent with drivers development in different OSes/kernels. Generally it was microkernel or hybrid kernels and I cannot told that this process takes me into pleasure.
Jari OS on other side is a new microkernel OS that can takes in advance all my skills with system design.
Actually every Jari driver is a microkernel service working in userspace like a trusted task. All IO mechanics has a protocol and every time you need to support it with all changes. It's a bad, but for this case I've designed a generic device server layer that will hide all semantics within several library functions.
All of this located in system libraries - in LibOS part.
Include and you will have server implemented. Really is a generic layer - on top of this I will design and implement other layers - for console drivers, input drivers, block drivers and so on.
Like example of this I will show a code snippet of the generic VGA console driver uses this generic layer:


#include <sys/types.h>
#include <sys/mman.h>
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <string.h>
#include <wchar.h>
#include <errno.h>
#include <time.h>
#include <datalib/usrtc.h>
#include <ns/ns.h>
#include <scheduler.h>
#include <proto/rpc.h>
#include <devio/devio.h>
#include <devio/server.h>
#include <termio/termio.h>

#include <termio/term.h>

#include "vga.h"
#include "hwops.h"

#define DRVNAME  "vgacons.d"
#define DEVNAME  "/console"

static size_t __vga_write(void *buf,uni_rpc_t rpc,size_t size,size_t offset);
static size_t __vga_read(void *buf,uni_rpc_t rpc,size_t size,size_t offset);
static int __vga_ioctl(void *buf,uni_rpc_t *rpc,size_t size);

struct device_ops dev_op = {
  __vga_write,
  __vga_read,
  NULL,
  __vga_ioctl,
  __vga_write,
  __vga_read,
  NULL,
  NULL,
};

int main(int argc,char **argv)
{
  device_server_t *sys;
  int r=0;
  void *iobuf=mmap(NULL,4096,MMAP_RW|MMAP_PHYS,NOFD,(void*)VGA_TM_BASE);
  char buff[256];

  task_scheduler_control(getpid(),SYS_SCHED_CTL_SET_PRIORITY,30);

  vga_init(iobuf);

  vga_print("[vgacons] i've started\n");

  sys=malloc(sizeof(*sys));
  if(!sys)
    abort();

  r=init_device_server(sys,DIRECTIO_READ | DIRECTIO_WRITE | HW_READ | HW_WRITE,
                       2048,4096,512,1,DRVNAME,DEVNAME,&dev_op,4);

  sprintf(buff,"sys->allowed = %p\n",sys->allowed);
  vga_print(buff);

  if(r!=0) {
    sprintf(buff,"r=%d\n",r);
    vga_print(buff);
    abort();
  }

  vga_print("[vgacons] device server run\n");

  device_server_start(sys);

  vga_print("[vgacons] :((\n");

  return 0;
}

static size_t __vga_write(void *buf,uni_rpc_t rpc,size_t size,size_t offset)
{
  char *msg=buf;
  size_t rsize=size;

  if(size<=sizeof(rpc)) {
    return -EINVAL;
  }
  msg+=sizeof(rpc);
  size-=sizeof(rpc);

  switch(rpc.arg3) {
  case PHYT_WRITE:
    uni_print((cschar_t*)msg,size);
    break;
  case PHYT_WRITEPOS:
    uni_printxy((cschar_t*)msg,size,rpc.arg4,rpc.arg5);
    break;
  case PHYT_SCROLL:
    if(rpc.arg4==SCROLL_UP) {
      uni_scrollup(rpc.arg5);
      uni_printxy((cschar_t*)msg,size,0,0);
    } else if(rpc.arg3==SCROLL_DOWN) {
      uni_scrolldown(rpc.arg5);
      uni_printxy((cschar_t*)msg,size,0,uni_height()-rpc.arg5);
    } else
      return -EINVAL;
    break;
  default:
    return -EINVAL;
    break;
  }

  return rsize;
}

void ascii_puts(const char *msg)
{
  return;
}

static size_t __vga_read(void *buf,uni_rpc_t rpc,size_t size,size_t offset)
{

  vga_print("ridden\n");

  return 0;
}

static int __vga_ioctl(void *buf,uni_rpc_t *rpc,size_t size)
{
  uni_rpc_t rrpc;

  if(size<sizeof(rrpc))
    return EINVAL;

  memcpy(&rrpc,buf,sizeof(rrpc));
  switch(rrpc.arg3) {
  case PHYT_DISABLEAUTOSCROLL:
    uni_scrolldisable();
    rpc->rpc_call=0;
    break;
  case PHYT_ENABLESCROLL:
    uni_scrollenable();
    rpc->rpc_call=0;
    break;
  case PHYT_GETSIZE:
    rpc->rpc_call=0;
    rpc->arg0=uni_width();
    rpc->arg1=uni_height();
    break;
  case PHYT_CLEAR:
    uni_clear();
    break;
  default:
    return EINVAL;
    break;
  }

  return 0;
}

Jari Operating System: LibAlloc updates

2008-11-12T19:28:00.002+02:00

Some of today's deal within Jari OS project was dlmalloc (Douglas Lee memory allocator) porting.
So, it's done and now I've rejected using of ugly smalloc (simple memory allocator).
Dlmalloc is a great thing - it can be ported anywhere, for example - Jari OS doesn't have sbrk() function, and will not have this oldish shit, otherwise - all is done via mmap() and dlmalloc allow to use it without sbrk().
Porting to the Jari OS mmap() semantics taken few minutes only.
Like usually you can look out source code - git clone git://rep.jarios.org/syslibs.git - LibAlloc/lib/dlmalloc.c

thanks.

Jari Operating System: small steps

2008-10-28T12:59:00.003+02:00

I want to make an announcement of development steps.
First of all is a really progress on microkernel (http://mstring.jarios.org) - there are many features implemented:
- memory management (O(1) allocator)
- interrupts handling for uspace
- memory i/o for uspace
- ioports for uspace
- realtime scheduler
- running static ELF binaries that loaded from GNU grub
- Fast IPC (sync ports)
Actually muistring allow to write userspace drivers and subsystems now, but some things not done yet.
Uspace (Jari) is developing now and some things works:
- nameserver
- terminals server
- text VGA driver
- keyboard driver
- syslibs (libos, libc, liballoc)
Some later I will post link to showcase.
More information here - http://jarios.org

Jari Operating System: reburned

2008-09-16T23:27:00.001+03:00

I don't have a time to write more.
This is just a copy-paste from news feed:
After Jari was frozen for a many years, now Jari is wake up and preparing to work, for a hard work.
Jari Operating System will be a not microkernel/hybrid based system, otherwise it will use an exokernel, that developing separately. It's mean that Jari Operating System is a set of user-space drivers, network libs, libOS, libc and so on. It's a top level of a exokernel.
While exokernel under extreme development stage and not all syscalls are well defined, this project will wait for this and will temproary suspended (for several weeks or one month, it cannot be determined now). After those userspace wrappers will be implemented and user-space development begins.
After core libs are well tested and exokernel will be done this all will be ported to it.
In case of exokernel Jari changing default architecture to amd64 (not x86_64, just amd64 in case of its specific hardware features), but Jari portable, and all depends on exokernel ports.
So, I will present an exokernel -> http://mstring.jarios.org , it's called MuiString (greek mui at the begging) that shows it's small size, and string just a mode for kernel flexibility and atomicity (you know about string theory).
Generally it mean that Jari found a kernel to base on, ... it was a
long search ...

Also I want to introduce a new Jari OS homesite - http://jarios.org, you can find a lot of
information there.

Thanks.

ReadLeaf: updates

2008-09-16T23:23:00.001+03:00

I want to make announcment.
ReadLeaf 0.2alpha is out with many new features.
You can look out for more info here - http://www.readleaf.org

Xmonad: optimizing working space

2008-08-05T11:29:00.002+03:00

Some later I wrote about Gnome and how to change default metacity wm (that very good and cannot be in use if you want to make your works faster) with windowmaker.
I've decide reject Gnome and use pure wm without desktop environment, but also I've decided to try tiling wm written on Haskell - XMonad.
It's really usable, for indication I'm using a dzen2 and some set of handmade icons.
Here my ~/.xmonad/xmonad.hs


 import XMonad hiding ((|||))
 import XMonad.Operations
 import System.Exit
 import Graphics.X11
 import System.IO

 import qualified XMonad.StackSet as W
 import qualified Data.Map        as M

 import XMonad.Hooks.DynamicLog
 import XMonad.Hooks.UrgencyHook
 import XMonad.Layout.Tabbed
 import XMonad.Layout.NoBorders
 import XMonad.Hooks.ManageDocks
 import XMonad.Layout.Grid
 import XMonad.Layout.ToggleLayouts


 import XMonad.Layout.LayoutCombinators ((|||))
 import XMonad.Util.Run (spawnPipe)


 ------------------------------------------------------------------------
 -- Common variables
 --
 myTerminal      = "urxvt"
 myBorderWidth   = 1
 myModMask       = mod4Mask
 myNumlockMask   = mod2Mask
 myWorkspaces    = ["Shells","Emacs","WWW","IM", "Docs", "Media" ] ++ map show [6..9]
 myNormalBorderColor  = "#3e7464"
 myFocusedBorderColor = "#0b610b"
 myNormalBg =  "#0b0b0b"
 myNormalFg =  "#385c64"
 myFocusedBg = "#598867"
 myFocusedFg = "#760202"
 myFont = "-misc-*-*-*-*-*-10-*-*-*-*-*-*-*"
 myBitmapsDir = "/home/tirra/Icons/"

 myDefaultGaps :: [(Int,Int,Int,Int)]
 myDefaultGaps = [(14,0,0,0)] -- 15 for default dzen

 menuCmd = "dmenu -fn '" ++ myFont ++ "' -nb '" ++ myNormalBg ++ "' -nf '" ++ myNormalFg ++ "' -sb '" ++ myFocusedBg ++ "' -sf '" ++ myFocusedFg ++ "'"
 dzen2 = "dzen2 -p -h 14 -ta l -bg '" ++ myNormalBg ++ "' -fg '" ++ myNormalFg ++ "' -w 600 -sa c -fn '" ++ myFont ++ "'"

 ------------------------------------------------------------------------
 -- Key bindings. Add, modify or remove key bindings here.
 --
 myKeys conf@(XConfig {XMonad.modMask = modMask}) = M.fromList $
     [ ((modMask,               xK_r     ), spawn ("exec `dmenu_path | " ++ menuCmd ++ "`"))
     , ((modMask,               xK_q     ), kill)
     , ((modMask .|. shiftMask, xK_a     ), spawn "urxvt")
     , ((modMask .|. shiftMask, xK_e     ), spawn "emacs")
     , ((modMask .|. shiftMask, xK_m     ), spawn "urxvt -e mutt")
     , ((modMask,               xK_space ), sendMessage NextLayout)
     , ((modMask .|. shiftMask, xK_space ), setLayout $ XMonad.layoutHook conf)
     , ((modMask,               xK_n     ), refresh)
     , ((modMask,               xK_Tab   ), windows W.focusDown)
     , ((modMask,               xK_j     ), windows W.focusDown)
     , ((modMask,               xK_k     ), windows W.focusUp  )
     , ((modMask,               xK_m     ), windows W.focusMaster  )
     , ((modMask,               xK_Return), windows W.swapMaster)
     , ((modMask .|. shiftMask, xK_j     ), windows W.swapDown  )
     , ((modMask .|. shiftMask, xK_k     ), windows W.swapUp    )
     , ((modMask,               xK_h     ), sendMessage Shrink)
     , ((modMask,               xK_l     ), sendMessage Expand)
     , ((modMask,               xK_t     ), withFocused $ windows . W.sink)
     , ((modMask              , xK_comma ), sendMessage (IncMasterN 1))
     , ((modMask              , xK_period), sendMessage (IncMasterN (-1)))
     , ((modMask .|. shiftMask, xK_q     ), io (exitWith ExitSuccess))
     , ((modMask              , xK_f), (sendMessage (Toggle "Full")) >> sendMessage (Toggle "Grid"))
     , ((modMask .|. shiftMask, xK_r     ),
           broadcastMessage ReleaseResources >> restart "xmonad" True)
     ]
  
     ++

     --
     -- mod-[1..9], Switch to workspace N
     -- mod-shift-[1..9], Move client to workspace N
     --
     [((m .|. modMask, k), windows $ f i)
         | (i, k) <- zip (XMonad.workspaces conf) [xK_1 .. xK_9]
         , (f, m) <- [(W.greedyView, 0), (W.shift, shiftMask)]]


 ------------------------------------------------------------------------
 -- Mouse bindings: default actions bound to mouse events
 --
 myMouseBindings (XConfig {XMonad.modMask = modMask}) = M.fromList $

     -- mod-button1, Set the window to floating mode and move by dragging
     [ ((modMask, button1), (\w -> focus w >> mouseMoveWindow w))

     -- mod-button2, Raise the window to the top of the stack
     , ((modMask, button2), (\w -> focus w >> windows W.swapMaster))

     -- mod-button3, Set the window to floating mode and resize by dragging
     , ((modMask, button3), (\w -> focus w >> mouseResizeWindow w))

     -- you may also bind events to the mouse scroll wheel (button4 and button5)
     ]

 ------------------------------------------------------------------------
 -- Layouts:

 -- You can specify and transform your layouts by modifying these values.
 -- If you change layout bindings be sure to use 'mod-shift-space' after
 -- restarting (with 'mod-q') to reset your layout state to the new
 -- defaults, as xmonad preserves your old layout settings by default.
 --
 -- The available layouts.  Note that each layout is separated by |||,
 -- which denotes layout choice.
 --


 myTabConfig = defaultTheme {
   activeColor         = "#6666cc"
  , activeBorderColor   = "#000000"
  , inactiveColor       = "#666666"
  , inactiveBorderColor = "#000000"
  , decoHeight          = 10
 }


 ------------------------------------------------------------------------
 -- Window rules:

 myManageHook = composeAll [ className   =? "Firefox-bin"            --> doF(W.shift "WWW")
               , className   =? "Iceweasel"       --> doF(W.shift "WWW")
               , className   =? "Gajim.py"         --> doF(W.shift "IM")
               , className   =? "Evince"           --> doF(W.shift "Docs")
               , className   =? "Emacs"            --> doF(W.shift "Emacs")
               , title       =? "mutt"             --> doF(W.shift "WWW")
               , className   =? "URxvt"            --> doF(W.shift "Shell")
               , className   =? "Stardict"         --> doF(W.shift "Docs")
               ]



 ------------------------------------------------------------------------
 -- dzen2 configuration
 --

 myPP h = defaultPP {
                     ppOutput = hPutStrLn h
                   , ppCurrent = dzenColor myFocusedFg  myFocusedBg . pad
                   , ppVisible = dzenColor myNormalFg myNormalBg . pad
                   , ppTitle = dzenColor "#117418" "" . wrap "{ "  " }"
                   , ppHiddenNoWindows = const ""
                   , ppWsSep =  " ^fg(#113d41)^r(1x12)^fg() "
                   , ppSep      =  " ^fg(#113d41)^r(2x2)^fg() " -- "|"
                  -- , ppLayout  = dzenColor "#946144" "" .
                  --    (\x -> case x of
                  --       "Tall" -> "^i(/home/dk/.icons/dzen2/tall.xbm)"
                  --        "Mirror Tall" -> "^i(/home/dk/.icons/dzen2/mtall.xbm)"
                  --        "Full" -> "^i(/home/dk/.icons/dzen2/full.xbm)"
                  --    )
  }

 myUrgencyHook = withUrgencyHook dzenUrgencyHook
   { args = ["-bg", myNormalBg, "-fg", "#117418"] }


 ------------------------------------------------------------------------
 -- Run xmonad with the settings you specify. No need to modify this.
 --
 main = do din <- spawnPipe dzen2
           xmonad $ myUrgencyHook $
            defaultConfig {
       -- simple stuff
         terminal           = myTerminal,
       --  focusFollowsMouse  = myFocusFollowsMouse,
         borderWidth        = myBorderWidth,
         modMask            = myModMask,
         numlockMask        = myNumlockMask,
         workspaces         = myWorkspaces,
         normalBorderColor  = myNormalBorderColor,
         focusedBorderColor = myFocusedBorderColor,
         defaultGaps        = myDefaultGaps,

       -- key bindings
         keys               = myKeys,
         mouseBindings      = myMouseBindings,

       -- hooks, layouts
         layoutHook         = toggleLayouts (noBorders Full) $
                              smartBorders $ tiled ||| Mirror tiled ||| Full ||| Grid ||| tabbed shrinkText myTabConfig,
         manageHook         = (manageHook defaultConfig <+> myManageHook) <+> manageDocks,
         logHook            = dynamicLogWithPP $ myPP din
     }
    where
            tiled = Tall 1 0.03 0.68

There are many tuning for yourself can be made. For dzen2 I wrote two files - one of this just runs it, other is a daemon that takes information and provide it to dzen2 panel.
Main file - ~/.xmonad/panel.sh


#!/bin/bash

export LANG=en_US.UTF-8
FONT='-misc-*-*-*-*-*-10-*-*-*-*-*-*-*'
DZEN="dzen2 -p -h 14 -fn $FONT"
BG="#0b0b0b"
FG="#385c64"
MONITOR=/home/tirra/.xmonad/monitor.pl

SW=1280
Y=0
X=600
$MONITOR | $DZEN -e '' -x $X -y $Y  -ta r -bg $BG -w $(($SW-$X)) -fg $FG -sa c

And ~/.xmonad/monitor.pl where you need to made some modifications:


#!/usr/bin/perl

use strict;

###
# Configurable constants
#
## Xbm icons directory
use constant XBM_IDIR => '/home/tirra/Icons';
## Colors, separators and other stuff of this kind
use constant FG_COLOR    => '#385c64';
use constant BG_COLOR    => '#010101';
use constant TITLE_COLOR => '#946144';
use constant ELEM_COLOR  => '#5c0000';
use constant ICON_COLOR  => '#00baff';
use constant VSEP        => '^fg(#113d41)^r(1x12)^fg()';
use constant ISEP        => '^p(6)^fg(#113d41)^r(2x2)^fg()^p(6)';
## IDLE time
use constant IDLE  => 1;
## Constants for CPU load measurment
use constant FSTAT    => '/proc/stat';
use constant NCPUS    => 2; # number of CPUs in system
## Battery constants
use constant BAT_STATUS      => '/sys/class/power_supply/BAT1/status';
use constant BAT_CHARGE_NOW     => '/sys/class/power_supply/BAT1/charge_now';
use constant BAT_CHARGE_FULL => '/sys/class/power_supply/BAT1/charge_full';
use constant BAT_FULL_ICON  => XBM_IDIR . '/b_full.xbm';
use constant BAT_CRITICAL_ICON   => XBM_IDIR . '/b_critical.xbm';
use constant BAT_EMPTY_ICON  => XBM_IDIR . '/b_empty.xbm';
use constant BAT_25_ICON  => XBM_IDIR . '/b_25.xbm';
use constant BAT_50_ICON  => XBM_IDIR . '/b_50.xbm';
use constant BAT_75_ICON  => XBM_IDIR . '/b_75.xbm';
use constant BAT_CHRG_ICON  => XBM_IDIR . '/ac_01.xbm';
## Constants for memory monitoring
use constant MEMINFO => '/proc/meminfo';
## Net interface constants
use constant ROUTE         => '/sbin/route';
use constant NETPATH       => '/sys/class/net/';
use constant NET_UP_ICON   => XBM_IDIR . '/arr_up.xbm';
use constant NET_DOWN_ICON => XBM_IDIR . '/arr_down.xbm';
use constant NET_PPPDOWN_ICON => XBM_IDIR . '/ppp_down.xbm';
use constant NET_PPPUP_ICON => XBM_IDIR . '/ppp_up.xbm';
use constant NET_ETHDOWN_ICON => XBM_IDIR . '/eth_down.xbm';
use constant NET_ETHUP_ICON => XBM_IDIR . '/eth_up.xbm';
use constant NET_WLANDOWN_ICON => XBM_IDIR . '/wlan_down.xbm';
use constant NET_WLANUP_ICON => XBM_IDIR . '/wlan_up.xbm';
## Jabber constatns
use constant JNAME         => 'IM';
use constant ACOLOR        => '#117418';
## Weather constants
use constant WTITLE        => 'Out';
use constant WID           => 'yourweatheridhere';
use constant WUPD_INT       => 30 * 60;

######
# Cpu load measurment in percents.
#
my @cpus = undef;
my $stat_fh = undef;

sub init_cpus()
{
    for (my $i = 0; $i < NCPUS; $i++) {
        $cpus[$i] = { 'old_idle' => 1, 'old_load' => 1 };
    }

    open($stat_fh, '<', FSTAT) or die "Can't open file " . FSTAT . " for reading $!";   
}

sub get_cpus_load_perc()
{
    my @cpus_load = undef;

    for (my $i = 0; $i < NCPUS; $i++) {
        while (<$stat_fh>) {
            next if (!/^cpu$i.+/);
            # /proc/stat should contain all data about CPU in the following
            # format(all data about particular CPU is written in one line):
            # cpu:      
            # 
            my @load = split /\s/;
            my $idle_time = $load[4];
            my $load_time = $load[1] + $load[2] + $load[3] + $load[5] + $load[6] + $load[7];
            $cpus_load[$i] = int(($load_time - $cpus[$i]->{'old_load'}) /
                                 ($idle_time + $load_time - $cpus[$i]->{'old_idle'} -
                                  $cpus[$i]->{'old_load'}) * 100);
            $cpus[$i]->{'old_idle'} = $idle_time;
            $cpus[$i]->{'old_load'} = $load_time;
            last if ($i++ == NCPUS);
        }
    }

    die "Can't make seek on " . FSTAT . "$!" if (seek($stat_fh, 0, 0) < 0);
    return @cpus_load;
}

sub show_cpu_info()
{
    my @cpu_info = get_cpus_load_perc();
    my $tmp = "";

    print "^fg(" . TITLE_COLOR . ")CPU:^fg()^p(6)";   
    for (my $i = 0; $i < NCPUS; $i++) {
        $tmp .= "$cpu_info[$i]%";
        $tmp .= " " . VSEP . " " if ($i + 1 != NCPUS);
    }

    print $tmp;
}

######
# Laptop's battery state
#

# returns an array of two elements
# where first element is a current power of battery in percens
# and the second one is a boolean value showing if the battry is
# charged or not.
sub get_battery_state()
{
    my @bat_state = undef;
    my $bat_max = 0;
    my $bat_cur = 0;

    open(BFC, '<', BAT_CHARGE_FULL) or die "Can't open file " . BAT_CHARGE_FULL . "for reading $!";
    while () {
        $bat_max = $_;
        last;
    }

    close(BFC);

    open(BFC, '<', BAT_CHARGE_NOW) or die "Can't open file " . BAT_CHARGE_NOW . "for reading $!";
    while () {
        $bat_cur = $_;
        last;
    }

    close(BFC);
    $bat_state[0] = int($bat_cur / $bat_max * 100);

    open(BFC, '<', BAT_STATUS) or die "Can't open file " . BAT_STATUS . "for reading $!";
    while () {
        $bat_state[1] = $_;
        last;
    }

    close(BFC);


    return @bat_state;
}

sub show_battery_info()
{
    my @binfo = get_battery_state();
    my $icon = BAT_EMPTY_ICON;
   
    if($binfo[0] <= 1) {
    $icon = BAT_EMPTY_ICON;
    }
    elsif($binfo[0] <= 10) {
    $icon = BAT_CRITICAL_ICON;
    }
    elsif($binfo[0] <= 25) {
    $icon = BAT_25_ICON;
    }
    elsif($binfo[0] <=50) {
    $icon = BAT_50_ICON;
    }
    elsif($binfo[0] < 99) {
    $icon = BAT_75_ICON;
    }
    elsif($binfo[0] >= 99) {
    $icon = BAT_FULL_ICON;
    }


    print "^fg(" . ICON_COLOR . ")^i($icon)^fg()";
}

######
# Memory usage monitor.
#

sub get_meminfo()
{
    my ($mtotal, $mactive, $stotal, $sfree, $cached) = (0, 0, 0, 0, 0);
    my %minfo = undef;

    open(MIF, '<', MEMINFO) or die "Can't open file " . MEMINFO . " for reading $!";
    while () {
        if (/^MemTotal:\s+(\d+).*$/) {
            $mtotal = $1;
        }
        elsif (/^Active:\s+(\d+).*$/) {
            $mactive = $1;
        }
        elsif (/^SwapTotal:\s+(\d+).*$/) {
            $stotal = $1;
        }
        elsif (/^SwapFree:\s+(\d+).*$/) {
            $sfree = $1;
        }
        elsif (/^Buffers:\s+(\d+).*$/) {
            $cached = $1;
        }
    }
   
    close(MIF);
    $minfo{'in_use'} = int($mactive / $mtotal * 100);
    $minfo{'cached'} = int($cached / $mtotal * 100);
    $minfo{'swap'} = int(($stotal - $sfree) / $stotal * 100);
   
    return %minfo;
}

sub show_mem_info()
{
    my %minfo = get_meminfo();

    print "^fg(" . ICON_COLOR . ")MEM:^fg() ";
    print $minfo{'in_use'} . "% ";
}

######
# Network monitor.
#

my @old_netstat = (0, 0);
my $network_avail = 0;

# Get current speed of the interface and
# quantoty of received and transmitted trough it bytes
sub get_net_info(@_)
{
    my $ifname = shift;
    my $file = NETPATH . "/$ifname/statistics/rx_bytes";
    my @stat = undef;

    open(RXF, '<', $file) or die "Can't open file $file for reading $!";
    my $rx_bytes = ;
    close(RXF);

    $file = NETPATH . "/$ifname/statistics/tx_bytes";       
    open(TXF, '<', $file) or die "Can't open file $file for reading $!";
    my $tx_bytes = ;
    close(TXF);

    $stat[0] = int(($rx_bytes - $old_netstat[0]) / 1024);
    $stat[1] = int(($tx_bytes - $old_netstat[1]) / 1024);
    $old_netstat[0] = $rx_bytes;
    $old_netstat[1] = $tx_bytes;
   
    return @stat;
}

sub show_net_info()
{
    my $cmd = ROUTE . " -n | grep -e '^0.0.0.0' | awk '{print \$8}'";
    my $iface = `$cmd`;
    my $iconppp = NET_PPPDOWN_ICON;
    my $iconwlan = NET_WLANDOWN_ICON;
    my $iconeth = NET_ETHDOWN_ICON;

    chomp($iface);

    if ($iface eq 'ppp0') {
    $iconppp = NET_PPPUP_ICON;
    }

    if ($iface eq 'wlan00') {
    $iconwlan = NET_WLANUP_ICON;
    }

    if ($iface eq 'eth0') {
    $iconeth = NET_ETHUP_ICON;
    }


    if ($iface eq '') {
        $network_avail = 0;
        @old_netstat = (0, 0);
        print " ^fg(" . ICON_COLOR . ")^i($iconeth)^fg()";
        print " ^fg(" . ICON_COLOR . ")^i($iconwlan)^fg()";
        print " ^fg(" . ICON_COLOR . ")^i($iconppp)^fg()";
        return;
    }

    my @stat = get_net_info($iface);
   
    $network_avail = 1;
        print " ^fg(" . ICON_COLOR . ")^i($iconeth)^fg()";
        print " ^fg(" . ICON_COLOR . ")^i($iconwlan)^fg()";
        print " ^fg(" . ICON_COLOR . ")^i($iconppp)^fg()";

#    print "^fg(" . TITLE_COLOR . ")$iface:^fg()^p(6)$stat[0]";
#    print "^fg(" . ELEM_COLOR . ")^i(" . NET_DOWN_ICON . ")^fg() ";
#    print VSEP . " $stat[1]^fg(" . ELEM_COLOR . ")^i(" . NET_UP_ICON . ")^fg()";
}

####
# Jabber monitor.
#

sub get_nmessages()
{
    my @procs = split('\n', `ps aux | grep gajim`);
    my $found = 0;

    foreach (@procs) {
        next if (/.*?\s+e?grep.*$/);
        $found = 1;
        last;
    }
    if ($found) {
        my $num = `gajim-remote get_unread_msgs_number`;
        chomp($num);
        return $num;
    }

    return '-';
}

sub show_jabberinfo()
{
    my $nmsg = get_nmessages();
    print "^fg(" . ICON_COLOR . ")" . JNAME . ":^fg()^p(6)";
    if (($nmsg ne '-') && ($nmsg > 0)) {
        print "^fg(" . ACOLOR . ")$nmsg^fg()";
    }
    else {
        print "$nmsg";
    }
}

###
# Weather monitor
#

my $last_wupd_time = 0;
my $last_temp = undef;

sub get_temperature()
{
    my $cmd = 'weather -i ' . WID;
    my @data = split(/\n/, `$cmd`);
    my $temp = undef;

    foreach (@data) {
        next until(/\s+Temperature:.+?\((-?\d+)\s+C\).*$/);
        $temp = "$1";
        $temp = "+$temp" if ($1 > 0);
        $temp .= "C";
        last;
    }
   
    return $temp;
}

sub show_temperature()
{
    my $time = time();
    my $temp = undef;

    if (!$network_avail) {
        $temp = '-';
    }
    elsif (($time - $last_wupd_time) >= WUPD_INT) {
        $temp = get_temperature();
        $last_temp = $temp;
        $last_wupd_time = $time;
    }
    else {
        $temp = $last_temp;
    }

    print "^fg(" . ICON_COLOR . ")" . WTITLE . ":^fg()^p(6)$temp";
}

####
# Time
#

sub show_time()
{
    my $time = `date +'%H:%M'`;
   
    chomp($time);
    print $time;
}

my $old_fh = select(STDOUT);
$| = 1;
select($old_fh);
init_cpus();
for (;;) {
    print ISEP;
    show_time();
    print ISEP;
    show_jabberinfo();
    print ISEP;
    show_temperature();
    print ISEP;
    show_mem_info();
    print ISEP;
    show_battery_info();
    show_net_info();
    print "\n";
    sleep(IDLE);
}

There are my Icons : Icons.tar.gz

And this is a snap of my workspace:

ReadLeaf: research pane

2008-07-31T13:52:00.002+03:00

Today, how I've wrote later, readleaf 0.1 was released and finished lazy stage - now it's a platform for research and implementation.
Both, MuiString and ReadLeaf has a one general idea, but MuiString is more deeper in implementation and some decisions need to be tested on other layer - quick and fast before this innovation will be implemented within MuiString and Jari overally.
ReadLeaf is a good start point for this type of research, I will use new way in development primary here - in area of the project.
I'm too lazy for repeations - you can see the general overview here - http://readleaf.berlios.de/concept.html
Currently already implemented:

redleafd
internal mm
basic research

Version 0.2 will be already with new features, now it's on the testing and development stage, but not so long from finish.

MuiString: Introduction to the architecture #1

2008-07-21T00:21:00.007+03:00

Today we're have a many implementations of the microkernel architectures, but there are too similar and have many problems.
The first problem is a speed, the second one is a security.
Generally microkernel more secure than monolithic-based operating system, but with some restrictions, if you grow up speed, you are loosing in security.
But it's a general problems - more complicated thing - we're using C/C++/ObjC to implement architecture, and there are language takes a many time and limits in the implementing.
My general idea is to use a functional programming language - and I've choose scheme - it's better, simply.
Look for the block diagram:
MuiString separated to the three logical parts, the first low two layers implemented on C (or on Cext in future), the third layer implemented on C/Cext and on scheme itself.
All hardware specific things are hidden in the MuiString.
All services should be implemented on scheme, yes, we're can use C-written servers and libs (it should be done for liblinux/libposix), but in general case all other will be implemented on scheme and ran on scheme VM.
All IPC will be operate via scheme forms - in this case all security models will be implemented without IPC calls restrictions, and it can be made on the VM (or language) level.
There are several security models and several objects on the higher level:

Trusted servers
Key signed
Group politics

Trusted servers can allow secure transmit via determined servers, for example - you trust to the TCP/IP server, LAN server also trust it, but there are no directions from you to the LAN server, but LAN server and you can get access to execute scheme forms from/on LAN server, because there are trusted via TCP/IP server.

I will write more researching results in the next post.

Redleaf: new plans and ways

2008-07-18T02:47:00.004+03:00

As of redleafd 0.1beta release (now it can be used for some purposes with sure) I'm take an idea, I want to implement some features for my work - it's a time sharing, job statistics, projects states etc ...
I've decided to make it via web, yes, in case of public (but restricted use) using.
What the deal I'm talking about ?
There are a complete service for each one who cares and working on open-source project, scheduling community events, working time and want to make a proof of participating and real developer state.
For example - I have a many mails from people who wants me to take they like a developers to open source project, in the past I've including everybody, but there are no effect on this - nobody works, people was enlisted and nothing else.
On real practice - I want to take a view to each of candidate to participate in project via this service. There are no service for it yet.
Also, there are a good thing for employer to avoid time loose with non right mans.
eh, you can told me about linkedin - linkedin is not really make a proof of your skills and jobs - it's suck.
So, it was a some non-technical introduction ...
I'm reserving 4-5 hours per week for redleaf, and I want to introduce technologies will used for this project:

- httpd - redleafd (small and fast)
- scheme_modula (module for the redleaf that suppose to run scheme code as quick as it possible)
- web ui software written on scheme
- parsing/collecting server side daemons on python || scheme (it depends on my research with libs)

In this case I will release in near time redleaf 0.1 (beta version can be found here - http://prdownload.berlios.de/readleaf/redleaf-0.1beta.tar.gz ), that will supports basic http stuff only.
After those I'm planning to reorganize architecture and rewrite some code, to support dynamic modules and so on for flexibility, the next stage will be a set of modules. After this stage I will implement scheme_modula and starts to write web ui and server side daemons.
In this case I'm need to anybody who knows any of the following - or all of this stuff:

- C and UNIX IPC
- Data structures and memory management techniques
- Scheme
- any of - cvs/git/svn/arch/etc ... on really good level
- Python
- html/css and optionally javascript
- Gimp and some creative techniques for image/icons creating

Benefits you will get working on this project: you can become famous (maybe, who knows?), you can update your skills on programming and especially on scheme programming - on real practice, you will have a short and cool name for imap account - and invitation to the newly created service ... hmm like authors.
I'm looking for you - contact me via jabber - tirra at njs.netlab.cz

Thank you.

Tips'n'Tricks: GCC useful macros for debugging

2008-07-15T23:40:00.003+03:00

Continuing my series for newbies/beginners I want to told about gcc macros useful for debugging.
Everytime on C you have deal with memory, and very big part of bugs are memory managing related.
You must know differents ways to decide question with it, wrappers to general mmap/malloc/munmap/free functions, implement your own allocator (if you really know what you're doing).
Anyway, it's better to know without debugger (just use stdout to make this deal) when you are allocate and freed memory. GCC has a several really useful macros for this.
For example let's write a simple example with free() function:


void *__my_free(char *file,int line,char *function,void *p)
{
 if(p)    free(p);
 else     {
   fprintf(stderr,"Trying to free nil pointer.\n");
   fprintf(stderr,"At `%s:%d' via '%s'\n",file,line,function);
 }

 return NULL;
}

It's really a big shit deal to point everywhere info about function, line and file ;)
Just use this macro:


#define my_free(p)  __my_free(__FILE__,__LINE__,(char *)__FUNCTION__,p)

This is all, for more detailed documentation see gcc manual.

Research: new ideas to system development

2008-07-02T22:50:00.003+03:00

Doing some research on operating systems design you can choose the kernel model and in this case you can expect that microkernel is more stable, powerful and flexible compairing with monolithic design, exokernels are too platform specific and more complicable for application level platforms (i.e. embedded systems with too limited functionality). But like a bonus you will got the new area of research in this area - choosing and developing an instruments for implementation.
On this area, in my opinion, are old and good instrument is a pure C language for microkernel, for low level. But in case of many reasons C development has a long time for something bigger than microkernel, for microkernel you can extend C with macros or write your own preproccessor to extend C (like my project extC). This is good and right, but for microkernel servers and user level is better to use something else, more powerful with paradigms and more easy to develop.
It's not C++, not only in case of my hate to pluses, there are no reason to use C++ in any kind of development, C++ contain many ugly stuff and it has an ugly design overally. There are ObjectiveC, but this language will not decide our wishes.
I'm looking to functional programming languages - this way is more powerfull, well-designed lisp-like language is really simple to implement, they all has a simple syntax - and learning didn't takes a long time - it's intended to be meta-level development.
With its powerful I'm thinking about crazy things - what about to design a several DSL that can be simply implemented on Lisp/Scheme/Haskell , for different types of microkernel servers.
In example, you can research for often used language features and constructions that required for implement a filesystem, or block device driver, or network device driver etc...
I'm thinking to go deeper in this area after MuiString will ran. I promise to make a deep research, implement expiremental DSLs for this purposes and test it for perfomance and other things.
Be updated, take a look to functional programming of your research - it's a wonderful world of new features and paradigms.

Memory management - buddy system trick.

2008-07-02T12:46:00.003+03:00

How I wrote before, on low level we're need a simple and fast allocator with minimal external fragmentation. It was buddy system. I've used a binary buddy in case of simple and clean code and perfomance.
But we're need to store a free buddies list, but we haven't any allocators lower and we're cannot allocate something for it, and we're cannot create linked list of pointers to the blocks and chunks - and it's not so clever - the really big overdraft on it.
Theory:
I've decide that I can use a bitmap in this case, for grow up speed I will use a bitmap for each buddy list level in this case for first level we're need just 1 bit per attribute, for second 2 bits per attribute ... and so on. Also we're need to make a simple initialization and keep information about separating. In this case we're using 2 bits per buddy block, I've called them - the first one is negative bitmap, the second one is positive bitmap.
What are they means? The first bit attribute (negative) points that if the bit is set it's a free block, if not it's used (or unmapped) block. Second bit attribute (positive) points that if the bit set it's not separated block, otherwise is. In this model we're simply full negative bitmap with 0x0and positive bitmap with 0xffffffff, and how you can understand we will use unsigned integer 32bit sized ones for bitmap.
Make a simple calculations and you will get a good news - for map buddy with 16-parts you need 64bits or 8 bytes, isn't pretty?
Implementation:
So, the first I will declare the general structure that will describe buddy system:


typedef struct __bbuddy_type {
  uint32_t *pbmp;
  uint32_t *nbmp;
  uint32_t pn;
} bbuddy_t;

How you can see there are two pointers and what the neck ?
So, I'll explain it - on low level you need to take a pointer (determining the size you need) and use it for your first buddy - it's simply than something else. Also, I store buddy_t first and after it I'm storing uint32_t data.
Also we're need an macro that will calculate the overall size to store the buddy system, let's take it:


#define bbuddy_size(n)  (((((n/32)==0) ? 1 : ((n/32>1) ? (n/32)+2 : n/32+1))*(2*sizeof(uint32_t)))+\
                         sizeof(bbuddy_t))

Explaining it, for parts that bigger than 1/32 of buddy mapped area we're need just one uint32_t , for 1/32 yet another one, and for smaller parts working a general rule to calculate size. Like in squares roots calculation ;)
Also you will need for several useful macros that will calculates index of pointer and bit position within bitmap area:


#define bbuddy_indexp(p,n)  ((p/32 > 1) ? ((p/32)+(n/32)) : p/32)
#define bbuddy_indexbn(n)   ((n/32>0) ? n%32 : n)

The first macro will calculate index to choose correct uint32_t data, and second one to choose a bit position, be careful using this macros as is not a good practice, I will show examples of using its.
In the implementation explanation I don't want to show you all the source code, it's a simple - you can implement it yourself without any kind of problems.
But there are some useful code snippets I'll show.
The first point on implementation is initialization of buddy system, assign a correct pointers for bitmaps, init first block free. Here be careful with pointers shifting deals, I will show you the simple one, that works correctly, but don't forgot that C operator '+' on pointers shifts pointers with the size of pointer type.
I've used char* - it's a clean and transparent to understand:


uint8_t bbuddy_init(bbuddy_t *b,uint32_t max_part)
{
  int yy=(max_part/32)*2,i;
  char *ptr=(char*)b;

  if(b && max_part) {
    ptr+=sizeof(bbuddy_t);
    b->pbmp=(uint32_t*)ptr;
    ptr+=(sizeof(uint32_t)*((max_part/32 > 1) ? (max_part/32)+2 : (max_part/32)+1));
    b->nbmp=(uint32_t*)ptr;
  } else
    return 1;

  b->pn=max_part;

  max_part/=32;

  for(i=0;i<yy;i++) {
    b->nbmp[i]=nil;
    b->pbmp[i]=fil;
  }

  /* init first */
  b->nbmp[0] |= (1 << 0);
  b->nbmp[0] |= (1 << 1);

  return 0;
}

NOTE: nil it's a macro defined to 0x0 and fil is a macro defined to 0xffffffff.
Here we go ... we're a correctly init buddy system structure.
Now, I want to wrote about allocations/splitting/freeing specifics of this implementation.
I've implement an allocation function with mind that I've giving a part number of buddy system and it's trying to return me a number of avialable and allocated part. Look below:


static uint32_t __bbuddy_block_alloc(bbuddy_t *b,uint32_t align,uint8_t m_flag)
{
  uint32_t p_indx=align/32;
  uint32_t n_indx=0,i=0,o=0;

  if(p_indx) {
    for(i=0;i<p_indx;i++) {
      if(b->nbmp[p_indx+i]!=0x0)
        goto __is_free_long;
    }
    m_flag++;
    return __bbuddy_block_alloc(b,align/2,m_flag);
  }

  if(!p_indx) { /* checking small layers, big blocks */
    o=align*2;
    for(i=align;i<o;i++) 
      if((b->nbmp[p_indx] & (1 << i)) && !m_flag) {
        b->nbmp[p_indx] &= ~(1 << i); /* used */
        b->pbmp[p_indx] |= (1 << i); /* not separated */
        return i-align;
      } else if((b->nbmp[p_indx] & (1 << i)) && m_flag) {
        b->nbmp[p_indx] &= ~(1 << i); /* used */
        b->pbmp[p_indx] &= ~(1 << i); /* separated */
        /* mark childs free */
        align*=2;         i*=2;
        p_indx=align/32;
        n_indx=bbuddy_indexbn(i);
        b->nbmp[p_indx] |= (1 << n_indx);
        b->nbmp[p_indx] |= (1 << (n_indx+1));
        b->pbmp[p_indx] |= (1 << n_indx);
        b->pbmp[p_indx] |= (1 << (n_indx+1));

        m_flag--;
        return __bbuddy_block_alloc(b,align,m_flag);
      }
    if(i<2)
      return ENOBLOCK;
    else {
      m_flag++;
      return __bbuddy_block_alloc(b,align/2,m_flag);
    }
  }

 __is_free_long: 
  while(o<32) {
    if((b->nbmp[p_indx+i] & (1 << o)) && !m_flag) { /* yep, found */
      b->nbmp[p_indx+i] &= ~(1 << o); /* used */
      b->pbmp[p_indx+i] |= (1 << o); /* not separated */
      return (p_indx > 1) ? (((p_indx+i)-2)*32)+o : o;
    }
    else if((b->nbmp[p_indx+i] & (1 << o)) && m_flag) { /* make sep */
      b->nbmp[p_indx+i] &= ~(1 << o); /* used */
      b->pbmp[p_indx+i] &= ~(1 << o); /* separated */
      /* mark childs free */
      align*=2;        p_indx=align/32; o*=2;
      p_indx=bbuddy_indexp(align,o);
      n_indx=bbuddy_indexbn(o);
      b->nbmp[p_indx] |= (1 << n_indx);
      b->nbmp[p_indx] |= (1 << (n_indx+1));
      b->pbmp[p_indx] |= (1 << n_indx);
      b->pbmp[p_indx] |= (1 << (n_indx+1));

      m_flag--;
      return __bbuddy_block_alloc(b,align,m_flag);
    }
    o++;
  }


  return 0;
}

How you can see there are difference between 1/16 and bigger parts - I've showed it, it's separated via source code implementation in case of making additional checking and different calculation in both situations, you can split there parts into one to make code lines counter smaller , but it doesn't give any perfomance effect (tested).
Freeing buddy block must be assigned with splitting buddy into bigger block if possible, I've implement this via recursive separate function to make a lightweight to read release function, also I will point to indexes calculations here.
And I want you to understand, I'm giving to release function number of smaller blocks like offset to release, after those I'm calculating possible variants of blocks can be and if found I'm releasing it, and only after it I'm calls my splitter function. Also, I don't include additonal checking code, make it yourself ;), look below:


static uint32_t __bbuddy_block_release(bbuddy_t *b,uint32_t num,uint32_t i)
{
  uint32_t p_indx,n_indx;
  uint32_t layer=b->pn,ls=layer,a=0;

  /* first look up on higher possible layer
   * if there are now, look deeper
   */
  if(num>b->pn)
    return EINVALIDINDX; /* error encount */

  while(ls) {    ls/=2; a++;  } 
  if(i==0) {
    for(i=(a-1);i>0;i--) {
      if(num%2) break;
      else num/=2;
    }
  } 

  layer=(1 << i);

  n_indx=(layer<32) ? (num+layer) : bbuddy_indexbn(num);
  p_indx=bbuddy_indexp(layer,num);
  if(!(b->nbmp[p_indx] & (1 << n_indx)) && (b->pbmp[p_indx] & (1 << n_indx))) { 
    b->nbmp[p_indx] |= (1 << n_indx); /* mark free */
    return __bbuddy_split_up(b,layer,num);
  } else if(!(b->nbmp[p_indx] & (1 << n_indx)) && !(b->pbmp[p_indx] & (1 << n_indx))) { /* going deep */
    i++;
    if(i>=a)
      return EBUDDYCORRUPED;
    else
      return __bbuddy_block_release(b,num*2,i);

  }

  return 0;
}

In case of my initialization and implementation of bitmap, I'm starting to look up for the block from the biggest blocks, because in othercase there are will be errors on cleanly initied bitmap and buddy will corruped.
Also, I don't want to describe my splitter function and I will not show it's source code - it's a trivial too like others.
Be patient, on my code (I wrote it and tested within 1-2 hours) some checks is absent, and some code I've specially removed - take your brains on working way ;)
Like API functions I've made a wrapper to hide all recursion in static functions, and with assign of initial counters values.
On next post I will try to explain how we're can use this abstraction to allocate real pointers, or you can think yourself and assume this method - it's a too trivial ;)

Memory management - low allocator base - binary buddy.

2008-06-30T12:55:00.003+03:00

As I intended to memory managers implementing, I decide to write here about several techniques about this.
I don't interested on allocators in uspace now. I want to describe some tips and tricks on memory management while you haven't anything - just raw gdt/tss/ so on and raw addresses.
On low level of memory management and allocation system we're need to have a simple and fast memory allocator, it must avoid external fragmentation as strong as it possible.
There are one good allocator that doesn't sick with big external fragmentation - buddy system. You can find many it's types, but I'd like binary type - it's a very simple in calculation (that really needs).
Some words about buddy system... Buddy system is a such system that based on buddy relations for splitting memory areas. This system has several limitation - you must know deep of separation, I mean how much blocks maximally can contents buddy itself, and overally it can maps fixed memory area size (but it can be depends on implementation). The goal of this technique is to find free block aligned to requested size, when you are looking for some block size - you will separate block with bigger size and so on, while you didn't finf more situable block size. Look for illustrations:

Fig. 1
The first bar shows clean buddy system without any allocated block (4096 bytes). When we're requesting to allocate block with size 512 byte we're separate zone to two blocks, and one of this zone separate we're separate too and use one of them - like it shown on the second bar. If we're request to allocate three more 512-sized block we will got the picture displayed on third bar.
It's a good and relativetly fast methodics, but has many restrictions - for example assume that your buddy system has 16byte-sized minimal blocks - and you request 10 bytes - so, you will get a wasted space - and so on - it's an inernal fragmentation. Be sure - you can align block sizes to other numbers , like you want and depend on your target, but anyway it will not safe to use buddy for all allocations.
Another point is a free/busy/separated blocks list storing - on low level you cannot use binary trees in case that they need to have some memory allocation technique already, usually for this used simple pointers lists, but it's a worse practice.
On the next post about memory management I will describe my solution of this problem with some useful code snippets.

Protest award domain name nativization from ICANN

2008-06-27T20:19:00.004+03:00

So, there are bad news, no ICANN supports 'internalization' of domain names, now it's a possible to register a domain with your native language.
Due to many reasons - I decide to apply a protest for it - it's nothing, but maybe you will be participate on this.
And so on - I'll not post anything on languages differs from english, no jabber/blog/boards/sms/emails messages on non english from tomorrow !
I don't want to make a freedom to spooffing and idiotic things.

Participate if it possible.

I can feel your skeptics looks, yes - I'm just man and I don't have a voice in ICANN, but assume if there are many people like me? You know - world is small ;) And there are just time need ... and this protest will be effective. But anyway, if you will do nothing with it, in future with idiotic domains you will told yourself - "fuck'em up, I don't participate in protest, maybe my voice can be useful at there times ... shit shit shit" - feeling difference - take a voice, participate in this protest, control your life!

If you will participate, please comment here - thank you!

MuiString higher memory management

2008-06-24T01:48:00.003+03:00

I've made some researching and emulation (not so serios, but representative) about memory management in microkernel OS.
Many kernels does memory management fully in kernel space, others separate it and offer non-trivial API for doing this in userspace.
We're have many restrictions - security and performance. If you will targeting to one of this properties you will loose on other. If you want to have a very strong and secure mechanism you will make a many context switching within kernel and user spaces - it will slow down your system perfomance, if you will don't care about security you will make it faster ...
I decide to make a basics virtual memory manager operations within kernel space, and offer simple API (just 4 calls) - address_space_get(), address_space_set(), address_space_alloc(), address_space_release(). And ... bind page fault exception to userspace area virtual mm server.
In this scheme microkernel just mark/unmark address space, it doesn't makes a check (and it will works fast) , and microkernel doesn't care about page fault exception - it's a server headache.
The security implemented on userspace - with ACL server that cannot be exchanged - like other varios servers (you must load its via multiboot modules way), all requests to ACL cached to the IPC cache server (it cannot be exchanged too) to avoid premiary access violation checks and grow up speed of requests. On the figure below I've tried to show it graphically (I like this, but I don't like to make a graphics ...):

User task just trying to take some virtual memory and extend it's address space - it's a deal of libs to make all job with calling of vmm server. Vmm server asks about access rights/limits/etc ... on ACL server, if all granted - vmm will allocate space for user task, if will be need - it will calls MuiString microkernel via simple API.

This structure allows to make secure and fast (compairing with some highly secured or highly fast microkernel systems) - it's a good design for implementing different models of memory allocations (real-time, preemption, so on ...)

Emacs #1 - saving your state after exit , and tabbar use

2008-06-23T13:51:00.004+03:00

Often you need to restore your emacs session, but it's not a good way to load all buffer per start, just keep it.
Also, there are useful addition to emacs called tabbar.el it can keeps your buffers without its creating.
I don't want to write where you can find this file - if you are using debian you can find it out in emacs-goodies-el package.
So, the first step is to enable tabbar mode by default:


(tabbar-mode t)

After this I recommend to bind a keys for switch between tabs:


(global-set-key [(control shift w)] 'tabbar-forward)
(global-set-key [(control shift q)] 'tabbar-backward)
(global-set-key [(control shift e)] 'tabbar-forward-group)
(global-set-key [(control shift d)] 'tabbar-backward-group)

You can use your own key bindings depending on your choice and feel.
The second step is to use a desktop addition (that comes with emacs22):


(when (fboundp 'desktop-load-default)
  (desktop-load-default)
  (mapcar
   (lambda (symbol)
     (add-to-list 'desktop-globals-to-save symbol))
   '((buffer-name-history      . 100)
     (dired-regexp-history     . 20)
     (extended-command-history . 100)
     (file-name-history        . 500)
     (grep-history             . 50)
     (minibuffer-history       . 100)
     (query-replace-history    . 60)
     (read-expression-history  . 60)
     (regexp-history           . 60)
     (regexp-search-ring       . 20)
     (search-ring              . 20)
     (shell-command-history    . 50)))
;;;    (desktop-read)
  )

(add-to-list 'desktop-locals-to-save 'buffer-file-coding-system)
(add-to-list 'desktop-locals-to-save 'tab-width)

(setq-default desktop-missing-file-warning nil)
(setq-default desktop-path (quote ("~")))
(setq-default desktop-save t)
(setq-default desktop-save-mode t)

(setq-default save-place t)

And this all, this code snippets from my ~/.emacs file.

Using Gnome ?! really with modifications ...

2008-06-23T00:45:00.003+03:00

I'm using Gnome 2.xx in case of some features that I need.
The first, if you know, network-manager knows about many specifics of many Wi-Fi modules tricks, for example for my RaLink card (that cannot be changed in my notebook) - it's needed to be downed and after this up with new network settings. So, I'm using many wireless networks (office, pubs, train stations, passenger terminals, ...) - but other network managers can't operates with my card, doing many stuff manually isn't sounds good for me - I'm lazy...
If you are using Gnome you know the base minus of it - it's a default window manager - metacity, this wm cannot remember anything about windows position, size and other varios attributes.
Also, on other side there are tiled window managers, but its takes many time to make a sense on configs and features.
The better way is to use my old friend - Window Maker wm, there are simple setting up, and it has many look and features that can be offered in tiled managers.
It can be looks like on my screenshot -

So, but Gnome feature - changing window manager is deprecated and all settings made over a gconf was failed. Solution was found - I've create ~/.gnomerc file and add the following line -

export WINDOW_MANAGER=wmaker

This line helps for it.
Also you can change some things like abort nautilus loading etc ... It will safe your memory and will turn off features for idiots.

AMD64 #1 - Long Mode

2008-06-20T14:04:00.004+03:00

Backing to old days the first tree of microkernel was targeted to the x86 (32bit), going deeper the parent of all post projects (ilix) was targeted to embedded hardware i.e. arm and one internal architecture that was 24 bit.
Nowadays we're have a public accessible and cheep 64bit long architecture, x86 continuing but with 64bit long extension, but on real - it's looks different.
AMD64 allows many good extended features compairing with basic x86 32bit long structure. To use AMD64 features on full filled hand we're must operate in long mode (amd64 specific mode).
On truth, long mode is a mixed mode i.e. it operates on 64-bit mode and compatibility mode at one time. It has self minuses - it's a flat memory model. But it has 64-bit addressing and we're don't need for tricks with extension from intel to use more than 4Gb address space addressing, that relatively ugly and looks like an ugly hack.
Like a microkernel developer and low developer at one time - the first problem is initialization.
I've read AMD64 documentation directly from AMD, but there are no anything to really help with it, not at all, but if you want to make a sense quickly it's not usable.
I've designed a trick with it, saying simply it consist from following steps :
32bit code:

- initialize stack pointer (regarding your boot method and loader, I'm using grub and multiboot)

- init bootstrap GDT

- jump to 'meet point' within existing GDT

- save parameters from grub (will be need on if you using grub)

- we're falled on 32bit protected mode (legacy mode on AMD64)

- check for varios CPU features (it must be made if you want to made all like it must be in good kernels)

- check for AMD64 long mode support is a really needed (you should do it, otherwise you can except some bug - I don't know why - check it if I correct)

- all is ok, - enabling 64bit page translations (regarding to documentation set - cr4.pae=1)

- setup pages tables

- enable long mode (via EFER reg - setting LME to 1)

- enable paging in long mode (it will activate long mode and we're falling to compatibility mode)

- just jump to your 64bit code

64bit code:

- do your stuff, have a lot of fun ... ;)

I'm sure that there are direct long mode switching, but for me it was more quickly to make it like I've describe.

MuiString introduction

2008-06-17T19:01:00.002+03:00

Ok, I've told many information about Jari and so on ... But it was just a buzz words.
Now I want to present MuiString (mstring) microkernel project.
What is done?
- base structure
- low level stuff
How much time it will takes ? How long will I told about it without microkernel on legs ? Hm, so I think that this times ends in case of microkernel progress.

I promise to write more technical things here and ... look for links below -
MuiString homepage - http://mstring.berlios.de
MuiString wiki - http://mstring.berlios.de/wiki
I have a memory manager and core of scheduler in my sandbox, it's under testing and preparing to be committed to the general CVS repository.
Look for my messages here soon.

Thanks.

Web fun - redleaf

2008-06-04T19:22:00.002+03:00

How I wrote before - I opened a new project just for fun - and now it has releases and it's working - http://redleaf.berlios.de
But there are no time for prolongate any active development - I've spent less than 10 hours for this, and in future this will so.
If you want to take my flash point on this project - contact me, I'll be glad for it.
Anyway, this project will be like my testing platform - and some kinds of tasks will be implemented within this and tested.

Decomposing big project, separating is better

2008-06-04T19:06:00.002+03:00

You can find more information on the link in the post title.
Overally, following to the general idea - "separate all works and job to smaller ones" I decide to run microkernel separatly - like independed project.
This intended to be more effective, for example - currently I have EZA layer working for 60% (not so stable) and this was made after reorganization of the microkernel structure - and this just for less then 10 hours of normal works on the problem.
MString (MuiString) - is a name for Jari microkernel - and it's a new tag on this blog ;)
Now I'm thinking on diferent than C/Cext language for server space - it's a big flame, but I'm thinking that fully functional languages or partly functional is better than typical imperativies ones. Anyway, microkernel must be implemented on plain C - it's more comfortable language.

Following to the Web 2.0 idiotism or more fun with web

2007-12-28T18:27:00.000+02:00

Do you need web ? Speaking more - do you need web 2.0 ?
No, I don't, but anyway in any cases of your open source development you need in the web, because it's a communicate point - where you can exchange some ideas/source code or other related stuff for your project.
Exiting web servers are too feature overloaded or doesn't contain any features in your need. Also, you need CMS for your stuff - and most of them request from you apache, php, mysql, set of the many many libs ... And in the final phase you have an overloaded web server with there things, but you don't get that you want.
The new try_to_run project is a redleaf - is a simple and small web server (already implemented) and a big set of modules targeted to the developers need. It includes the configurable CMS, small toys set - like C source code to html with highlighting converting and many other.
You know already about monsters like sourceforge.net - yeah, now I'm working on the concept of the solution to make responsible for developers install and run the web software to get done.
Currently, this project on the development stage, and httpd works already with all needed stuff and features.
Why I need to do this ? So, it's just my fun like my old projects that was run, some of theirs working in the present days - like xneur - it was fun too ;)

New directions, new profile, new ideas

2007-10-01T12:48:00.000+03:00

Jari development has a new targets in development. The first of all we're decided to move our dev/hours to the exiting GNU mach microkernel and implement our ideas based on it. This decision based on the real time needs via the following points :
- there are many trivial task thats simply takes the time on its
- there are already exiting code base that can be used for Jari ideas
- there are no time to create a newly microkernel from scratch
The possible ways may be different and depends on the development flow in the time.
I don't know yet if we will takes the GNU Hurd servers to modify and extend, but you can be sure we will split Jari ikernel and Jari EZA code base with the GNU Mach code base.
With this objectivies I hope that we will ran in the early dates.
Now, I've planned to get Jari run on the third decade of the 2009 with the minimal drivers set, libc, CLI and varios utilities.
Look out this blog for further details in the future.

Jari: small new things

2007-04-11T15:01:00.000+03:00

While the low level under the heavy development stage, I'm returned to the compiler and some software. But about it by the way ...
C compiler:
We're developing the new C dialect, eC - stands for extended C.
No, it doesn't mean the next C++ or something else. Our dialect targeted to functional programming. I will extend the C macros, add the lambda functionality, and optimize all works with the types.
Also eC compiler will have a mechanism that will checks the works with pointers and variables to avoid the double malloc/free, it will be like an option.
Backing to C, it will backward compatible i.e. you will be able to compile C sources with eC compiler.
About this and other new features I will blog separately some later.
LibC/LibeC:
We will include the several memory allocation models, we will provide the standart alloc() and free(), but with it we're can offer the wrappers and callbacks to make the internal memory allocator simply and faster. You will be able to create struct mem_alloc_t and use it with your own way, with the libc/libec standart functions/politics or assign your function that preffered for your application. Also the libs will have several new functions for allocating memory - rt_alloc(), local_alloc(), ext_alloc(), func_alloc(), there functions will provide the more flexible parallel programming issues.
In addition, libs will be extended for string operating functions, the libec will have the standart high customizable parser.
coreutils:
Core utils will includes tools for parallel task management, via this tools you can get ran processes via your cluster by the your custom way, run nonlocal architecture binaries on the needed node and other features.

See for next blog messages, I will give here more examples and will describe all things written above.

offtopic: m$ the big technologies brake

2007-01-24T12:39:00.000+02:00

Microsoft is a big shit deals company, but it's sounds like fanatic scream.
Okay, take a look for the arguments.
The first of all and generally people says - "hey, solution from microsoft are totally used and it tells that it's a better solutions" - it's a typical disunderstanding about technology and marketing targeted *only* for domination and "big money must came to us totally". We're can compare non-IT products, but products that really depends on the technologies, in example - every day used thing A. What are you doing before the "big bought" ? You compare the avialable things on the market, what are principles of your choose? Generally you looking for a better look&feel, after this you are look for the functionality, price, quality etc ... Are you sure on this? I don't sure, the first you spamed via the advertisment and marketing action like - "hey look at this, it's a used by everyone, it's better ..." and for more than 60 percent this thing will be preffered on your choose.
The same deal are applicable for the IT market, the end-user (non proffesional) will choose the product that has a more advertisment and if advertisment and marketing will be more aggressive the customer will be more targeted to the product. So, imaginate that a millions end-users attacked via the aggressive marketing, theirs don't thinking about functionality, there are thinking "if an advertisment is so sounds good, then product is good too". It's not a problem of the end-users it's a just big lack for bad business. Microsoft uses this way, it's mean that they don't improve the new features, stability, innovation and via intercepting the whole market they are kill all the ways to get the another concurrents, following this way the new thing in the business came - "Why we should to do something new and better if our products are buying ?". There are no concurrents and if there are coming to the market the microsoft killing its via the agressive marketing and brow up the set of the myth like "in the foo OS no stable, no features, it's a bad - choose our ones", or following with the another way - just buying another companies and closing the projects (remember Xenix for example).
But if you can see there are no fundamentally changes on the microsoft products long the many years. New GUI? New incompatible within formats? New animations? New installation programs? it's a things just for emulating that they doing something - for customers, otherwise customers didn't buy the new products and didn't pay for the microsoft.
But there are some limitations, the first leak of this business strategy that world contains not only end-users, the second leak is that selling products an ugly, the third leak that anyway microsoft must support their products, in short words I mean that there are many professional people that can analyze with system methods exiting solution and tell - "hey! folks! it's a big piece of shit!" and they are will be right. The other thing it's a critical industry that includes an industry automatic systems, science research that needs no for good GUI or pretty looks pooks, they are need for stable and featured product, that has a functionality, it's a 24x7 servers, embedded systems, industrial system, computer clusters, distributed networks, and other critical systems.
In this case microsoft will never shift out other, better solution from this segments. But thinks, that segments has a new innovations and for working with them end-user need have a features and new functionality, but microsoft doesn't do it or do it too slowly and ugly (remember when microsoft approved the tcp/ip stack and improve network support for their products and how it's ugly implemented for the current time). Microsoft sets a technology brake for others including developers, end-users, science, etc ...
But it cannot be eternal process, and now microsoft marketing department know about it, but there doesn't follow via the right way - the right way is a perform a better solutions in a products, compare win 200 with win xp - do you something new, excluding GUI look ? There are nothing, take a look for vista_ something new, excluding poor resource using and new GUI ? The answer is no, so why the microsoft will lay to the customers that linux is bad, that linux is old, that price of the linux solution is big - okay okay, but what the things are microsoft can offer for exchanging unixes ? There are nothing.
Think deeper, microsoft will die or redirected to the support and porting their products to the linux and get this segment.

Questions?

Microkernels: the bad or good ?

2007-01-23T12:23:00.000+02:00

Last days I've summarized my opinions about different architectures of the operating systems kernels such as classic monolitic kernel, as an exokernel and microkernel.
Summarizing, all concepts have a self weak points and we're need to select a better solution that will correlating a perfomance, stability and security.
In this case I take a look to the microkernel architecture.
The some myth with the microkernels:
- Low perfomance, do you ready to loss just 5-10% of perfomance switching to the stable OS? I think your reply is yes. Many systems use there resources for the overloaded interface or architecture addons that trying to do a best stability. In example a nooks layer in the linux kernel, or solution from the microsoft research - singularity, that not so good, because it's takes points from the perfomance and look like an ugly addon to the worth architecture.
- Non trivial implementation, it's sounds with smiley. On the same deal it's not a true. Just compare simply of the GNU mach microkernel with linux kernel, or minix3 with the *bsd kernel. Of couse you can said that - "the linux kernel contains a big set of the drivers for devices, filesystems etc ...", but just think how it's hard to maintain this. Microkernel haven't a large number of the source code lines, it's a simple to review it, fix and update with new things, that makes it more pretty for maintain and fixing.
Just imaginate that you have a stable and secure system with all included innovations if needed. It's like a software on your mp3/ogg player that can works stable for many times without updates and many more things that you need on the linux or m$ windows.
All kernel servers are separated from each other and device driver in example cannot down all the system, in example you are working with the big document and some device driver (for your scsi controller in example) falling off, in this case all the request in the microkernel are cached and system process reload this driver and reply the requests, and you are continuing the job thinking that nothing happened, on the classic kernels you will take a look to the kernel panic or BSoD and all your job will be lost.
In addition, did you see a good abstraction in the Operating System, where you don't think about file system location? This is a simple job for the microkernel architecture.
In example, you came to the office and turn on your notebook the system detects your placement and starting to work with the office shared filesystem, and you don't care where is it located? What I need to mount/ add the volume to do this job - I really don't care. On both systems *nix and m$ windows I need to mount a NFS volume or add the logical volume via windows network and take care with back up ones or other things.
By the way, with the microkernel you don't need to have hardcoded things that can affect your perfomance and other varios things, all is modular.
Like a postscript I want to tell you - user friendly system is a not a system with pretty look interface, it's a stable and simple system for both end-users and developers, where you don't think about system security and permanent drivers installations and rebooting.

Jari OS introduction

2006-10-16T12:01:00.000+03:00

How I wrote later, I don't like anything from the existing operating system.
There are no innovations on theirs.
So, you can tells to everybody that all is sucks, but do nothing - it's a not a good way to make a better something that sucks for your opinion.
Otherwise, I decide to realize a better look to the operating systems wide in the my research project called "Jari".
Jari is an OS that will contains a new way of kernels, servers set, system libs, core utils and C compiler, maybe if the project will have a success it will be contain a Lisp compiler and most of the libs and users program will be written on it, but it's a long future plans.
Now I'm working with my friend on the kernel and servers.
Kernel has a three rings - the kernel space, the servers space and the user space levels, it's a not a microkernel in the basic opinion, it's a advanced microkernel model.
In the Jari we're have a abstraction atom - is a file, server is a file, cpu is a file etc , but it's not looks like a standart UNIX paradigm, in the Jari we're have an advanced VFS, in example not on all unixes we're have a VFS.
So, just imaginate - on the Jari we're have an exec() on the VFS.
Another thing - we're don't have a big and hard API for intercommunicating with kernel or server, this is a protocol tasks (it's called JP(Jari Protocol)). In example - for running process A on the friend node B you just need to make a frame of the protocol and all. Yes, here we're have a not so light caching and buffering mechanism, but it's a not so hard - in the Jari we're use a development model that mean - "each of the system makes a small task, but many system does a big task" ie each of the kernel part makes a simple task, but all the kernel do any tasks by each of other system.
And in addition, I'm developing now a new filesystem for supporting all the Jari paradigms, but generally we're separate a task ie fs doesn't makes a things on the IPC object or anything other.

To be continued ...

Operating Systems: Crisis or reburning ?

2006-09-27T12:38:00.000+03:00

Many people told me about good sides of the different OSes, but I'm don't find something new and ready-for-use innovation features in the OSes.
All unixes family has an old philosophy that doesn't support and can't support a good cluster technology. There are a problem of the architecture limits. Developers said -"Hey, we must to do threads it's a good and fast", but nobody from theirs all don't think that threads cannot migrate between nodes in example. Theirs don't thinks about flexibility in generally.
Yes, now I talking about kernels, take a look for the linux kernel, it's a huge trash of the infinite features. They includes a buggy source code for features, *bsd otherwise, they don't include something new, but trying to make a sense of the security and stable features. But we don't have an innovations both in linux or *bsd.
The main reason is the really old architecture.
I'm don't speak about closed OSes such as Solaris or non-unix OS m$ windows - it's a deadly OSes without future.
So, I think that in the time we will see new concept of the kernels and OSes generally, but who ia ready fir this?

bash completion mini-HOW-TO

2006-09-22T15:07:00.000+03:00

So, while zsh doesn't supports an unicode, I've tried to add completion functionality to the bash.
I think that this maybe useful for somebody.
There are lines from my ~/.bashrc file:

shopt -s extglob #it's a needable for the bash completion support
set +o nounset

complete -A hostname rsh rcp telnet r ftp ping fail #the standart completion
complete -A export printenv
complete -A variable export local readonly unset
complete -A enabled builtin
complete -A alias alias unalias
complete -A function function
complete -A user su mail finger

complete -A helptopic help # currently same as builtins
complete -A shopt shopt
complete -A stopped -P '%' bg
complete -A job -P '%' fg jobs disown

complete -A directory mkdir rmdir
complete -A directory -o default cd

complete -f -o default -X '*.+(zip|ZIP)' zip #there is completion for the some tools that works with filenames
complete -f -o default -X '!*.+(zip|ZIP)' unzip
complete -f -o default -X '*.+(z|Z)' compress
complete -f -o default -X '!*.+(z|Z)' uncompress
complete -f -o default -X '*.+(gz|GZ)' gzip
complete -f -o default -X '!*.+(gz|GZ)' gunzip
complete -f -o default -X '*.+(bz2|BZ2)' bzip2
complete -f -o default -X '!*.+(bz2|BZ2)' bunzip2

complete -f -o default -X '!*.ps' gs ghostview ps2pdf ps2ascii
complete -f -o default -X '!*.dvi' dvips dvipdf xdvi dviselect dvitype
complete -f -o default -X '!*.pdf' acroread pdf2ps
complete -f -o default -X '!*.+(pdf|ps)' gv
complete -f -o default -X '!*.texi*' makeinfo texi2dvi texi2html texi2pdf
complete -f -o default -X '!*.tex' tex latex slitex
complete -f -o default -X '!*.lyx' lyx
complete -f -o default -X '!*.+(htm*|HTM*)' lynx html2ps

complete -f -o default -X '!*.+(jp*g|gif|xpm|png|bmp)' xv gimp
complete -f -o default -X '!*.+(mp3|MP3)' mpg123 mpg321
complete -f -o default -X '!*.+(ogg|OGG)' ogg123

complete -f -o default -X '!*.pl' perl perl5

_get_longopts ()
{
$1 --help | sed -e '/--/!d' -e 's/.*--$[^[:space:].,]*$.*/--\1/'| grep ^"$2" |sort -u ;
}

_longopts_func () #there are functions for the long function in example for configure
{
case "${2:-*}" in
-*) ;;
*) return ;;
esac

case "$1" in
\~*) eval cmd="$1" ;;
*) cmd="$1" ;;
esac
COMPREPLY=( $(_get_longopts ${1} ${2} ) )
}
complete -o default -F _longopts_func configure bash
complete -o default -F _longopts_func wget id info a2ps ls recode

_make_targets ()
{
local mdef makef gcmd cur prev i

COMPREPLY=()
cur=${COMP_WORDS[COMP_CWORD]}
prev=${COMP_WORDS[COMP_CWORD-1]}

# if prev argument is -f, return possible filename completions.
# we could be a little smarter here and return matches against
# `makefile Makefile *.mk', whatever exists
case "$prev" in
-*f) COMPREPLY=( $(compgen -f $cur ) ); return 0;;
esac

# if we want an option, return the possible posix options
case "$cur" in
-) COMPREPLY=(-e -f -i -k -n -p -q -r -S -s -t); return 0;;
esac

# make reads `makefile' before `Makefile'
if [ -f makefile ]; then
mdef=makefile
elif [ -f Makefile ]; then
mdef=Makefile
else
mdef=*.mk # local convention
fi

# before we scan for targets, see if a makefile name was specified
# with -f
for (( i=0; i < ${#COMP_WORDS[@]}; i++ )); do if [[ ${COMP_WORDS[i]} == -*f ]]; then eval makef=${COMP_WORDS[i+1]} # eval for tilde expansion break fi done [ -z "$makef" ] && makef=$mdef # if we have a partial word to complete, restrict completions to # matches of that word if [ -n "$2" ]; then gcmd='grep "^$2"' ; else gcmd=cat ; fi # if we don't want to use *.mk, we can take out the cat and use # test -f $makef and input redirection COMPREPLY=( $(cat $makef 2>/dev/null | awk 'BEGIN {FS=":"} /^[^.# ][^=]*:/ {print $1}' | tr -s ' ' '\012' | sort -u | eval $gcmd ) )
}

complete -F _make_targets -X '+($*|*.[cho])' make gmake pmake

# cvs(1) completion
_cvs ()
{
local cur prev
COMPREPLY=()
cur=${COMP_WORDS[COMP_CWORD]}
prev=${COMP_WORDS[COMP_CWORD-1]}

if [ $COMP_CWORD -eq 1 ] || [ "${prev:0:1}" = "-" ]; then
COMPREPLY=( $( compgen -W 'add admin checkout commit diff export history import log rdiff release remove rtag status tag update' $cur ))
else
COMPREPLY=( $( compgen -f $cur ))
fi
return 0
}
complete -F _cvs cvs

# svn completion , I'm not sure that this all is right
_svn ()
{
local cur prev
COMPREPLY=()
cur=${COMP_WORDS[COMP_CWORD]}
prev=${COMP_WORDS[COMP_CWORD-1]}

if [ $COMP_CWORD -eq 1 ] || [ "${prev:0:1}" = "-" ]; then
COMPREPLY=( $( compgen -W 'add admin checkout commit diff export history import log rdiff release remove rtag status tag update' $cur ))
else
COMPREPLY=( $( compgen -f $cur ))
fi
return 0
}
complete -F _svn svn

_killall ()
{
local cur prev
COMPREPLY=()
cur=${COMP_WORDS[COMP_CWORD]}

# get a list of processes (the first sed evaluation
# takes care of swapped out processes, the second
# takes care of getting the basename of the process)
COMPREPLY=( $( /usr/bin/ps -u $USER -o comm | sed -e '1,1d' -e 's#[]\[]##g' -e 's#^.*/##'| awk '{if ($0 ~ /^'$cur'/) print $0}' ))

return 0
}

complete -F _killall killall killps

# A meta-command completion function for commands like sudo(8), which need to
# first complete on a command, then complete according to that command's own
# completion definition - currently not quite foolproof (e.g. mount and umount
# don't work properly), but still quite useful - By Ian McDonald, modified by me.

_my_command()
{
local cur func cline cspec

COMPREPLY=()
cur=${COMP_WORDS[COMP_CWORD]}

if [ $COMP_CWORD = 1 ]; then
COMPREPLY=( $( compgen -c $cur ) )
elif complete -p ${COMP_WORDS[1]} &>/dev/null; then
cspec=$( complete -p ${COMP_WORDS[1]} )
if [ "${cspec%%-F *}" != "${cspec}" ]; then
# complete -F
# COMP_CWORD and COMP_WORDS() are not read-only,
# so we can set them before handing off to regular
# completion routine
# set current token number to 1 less than now
COMP_CWORD=$(( $COMP_CWORD - 1 ))
# get function name
func=${cspec#*-F }
func=${func%% *}
# get current command line minus initial command
cline="${COMP_LINE#$1 }"
# split current command line tokens into array
COMP_WORDS=( $cline )
$func $cline
elif [ "${cspec#*-[abcdefgjkvu]}" != "" ]; then
# complete -[abcdefgjkvu]
#func=$( echo $cspec | sed -e 's/^.*$-[abcdefgjkvu]$.*$/\1/' )
func=$( echo $cspec | sed -e 's/^complete//' -e 's/[^ ]*$//' )
COMPREPLY=( $( eval compgen $func $cur ) )
elif [ "${cspec#*-A}" != "$cspec" ]; then
# complete -A
func=${cspec#*-A }
func=${func%% *}
COMPREPLY=( $( compgen -A $func $cur ) )
fi
else
COMPREPLY=( $( compgen -f $cur ) )
fi
}

complete -o default -F _my_command nohup exec eval trace truss strace sotruss gdb
complete -o default -F _my_command command type which man nice