Obviously you have already done a lot of thinking and work on this, I love it so far. First let me say that this matches exactly how I would proceed, and you may already have some of it in mind, but let me try to cover some areas you may not have thought about.

Since the early releases need to be research vehicles, and I am thinking about your Designed for Rapid Development and Easy Maintainability point. Produce a reference version with these characteristics and allow others to translate it into many languages and implementations. If you do what your put in this section well, then it is the reference port from which systems engineers can deploy systems at many scales. By defining the protocols abstractly and in language independent ways, you will facilitate not only rapid deployment, but also adaptation into a variety of implementations and deployments.

Perhaps it is because I do as much systems engineering and architecture and operations as development, so I think about how systems will be deployed and how they need to evolve as operational entities. The most distributed approach to deployment would be full peer-to-peer with no central servers anywhere.

I know you have also looked at Identity Commons, and the XDI, XRI models dovetail easily. The two projects, identities and currencies (flows) seem inextricably complementary to me. ID Commons has flows all over the place, and identities are a key attribute of flows. The access control models based on link agreements seem a natural model for currencies, and the information flows in any identity system or application seem a lot like currency flows.

I see them as different aspects of the same problem.

Mentioning peer to peer and link agreements leads me to another set of issues that really aren't about software per se, but about the organizational and social agreements that will support eventual deployment. This is like moving from banking software to banking, so I thought I'd ask whether it should be here or a new thread.

Gerry, Let's take the social contracts to another thread so this one can focus on the software.

I am not sure I can be of any assistance - don't understand the first thing about programming - but I am very interested in tracking transactions from multiple points of view.

Arthur said:

The data and related triggers and scripts must easily accommodate the layers of relationships and cross-connections that we use to represent problems.

Let me know if I can help.

The scope here really is astounding. I love it!

David, I know you may not be a programmer, but apparently it's in your genes. It is a great pleasure to be able to work with your son and he's contributing quite a bit to this project. My challenge, as you know, is seeing that he gets paid for it.

Arthur,

Thank you for those kind words. It appears that he would rather work with you on this project than do something "boring" with a regular paycheck. In the end, I think that will be a genetic advantage (I have to think that - that's the part he got from me.)

I feel like there will be some exciting projects coming together soon - and I'm thinking that Congo will be an integral part.

Arthur Brock said:

Gerry, Let's take the social contracts to another thread so this one can focus on the software.

Good plan. Over here.

That's really great that your son is working on this, and that you are supporting him in doing this. Would that I had such support from my dad.

I'll be posting the underlying data structures with some explanation soon. But for the moment here is a link to the code for the flexible data engine and Auto-Generated documentation (which we have to clean up). We will continue to generate automatic updates to this documentation.

It is not a friendly nor accessible doc... but rather the code documentation output through Doxygen: http://CongoDocs.geekgene.com

So, this part implements a distributed database including transports and an application interface?

Are the C++ and python sides different implementations or parts of one?

Well... now that I look at it... It looks like the whole source tree didn't get published yet. This is the API. We have two Congo clients at the moment, one in C++ one in Python. I'll have to get the documents updated with the server code also.

We did a bunch of our prototyping in Python... and then rebuilt in C++ to have something compiled, optimized, and with low-level hooks. I'll be back with more goodies.

I would really love for someone to give me at least 2 or 3 REAL LIFE scenerios where this technology would have a practical use. Maybe this is sort of like giant PEER-TO-PEER "knowladge" network. Instead of sharing music, you simply share "information"? Am I correct?

Yes, that could be one application. You might want to check out the other thread I started and linked above for issues around how this software might be deployed.

Next installment of info... This is a high level diagram of our software architecture. You'll see that the currency engine is simply a part of a greater data engine framework.

Konrad,

I think you're probably on the right track in thinking of it as a P2P "knowledge" network instead of simple files or predictable columns and rows. It's great for tasks where you may encounter unpredictable data or relationships in the data.

You might think of folksonomies, knowledge-bases, qualitative feedback systems, or content management tools which can adapt to include new kinds of content. Of course it can certainly handle normalized data as well... but it's quite convenient when encountering the unpredictable. :)

Oh yeah... and let me clarify regarding the Doxygen source docs.

It is the server, client and shared code (more easiliy seen when you look at Directories). We're going to work on getting it more accessibly commented, but in any case, it's start.

Next I'll post the crazy data tables for the atomic element Db.

Arthur Brock said:

Konrad,

I think you're probably on the right track in thinking of it as a P2P "knowledge" network instead of simple files or predictable columns and rows. It's great for tasks where you may encounter unpredictable data or relationships in the data.

You might think of folksonomies, knowledge-bases, qualitative feedback systems, or content management tools which can adapt to include new kinds of content. Of course it can certainly handle normalized data as well... but it's quite convenient when encountering the unpredictable. :)

RE:

I am actually in the end of of development of my own system for content maangement, it is a commercial application of an idea that you should be able to create an "object template" for real world or abstract items... for instance all the shoes have certian list of their own attributes. So to me it is just logical that there should be database table that stores "shoes", with their own unique parameters. A separate table should be in place for all the other objects (cars, air planes, cam coders). Using internet you should be able to easily search for those objects. Also there is an idea that some bigger objects may consist of other smaller objects with their own properties. Car {Wheels, Engine, Mirror etc} I would suggest all of the objects should their own "templates" of parameters. They should also be searchable, with ability to interconect them together. Creating sort of a "real world" representation on the Internet. Making technology like that peer to peer would benefit all of the humanity. Ebay uses this type of idea to allow people to list certian items with specific parameters. Google base also does something similiar.

http://www.car-engines.com/

http://www.car-engines.net/

http://www.car-wheels.net/

Note that single words often go to sites with more "structure":

http://www.cars.com/

hoewever -- with very broad concepts (e.g. engines, motors), ambiguities may arise:

http://www.motors.org/

Konrad Szpirak said:

... So to me it is just logical that there should be database table that stores "shoes", with their own unique parameters. A separate table should be in place for all the other objects (cars, air planes, cam coders).

The ConGO data engine is made in that spirit, but not precisely that implementation approach. It might help to not picture it as a database but as a huge hash table... Data Elements get thrown into the hash and can be accessed again. Relationships or namespaces are defined distinct from the location of an element.

Think of the Linux file system for example... Every file is a numbered inode. The namespace (e.g. /home/artbrock/images/extreme_close_up.jpg) points to the ID of that inode. We can even create a symbolic link elsewhere (e.g. /home/konrad/friends/arthur/picture.jpg) that points to the precise same inode ID. One element addressable by two different paths in the namespace. We're doing that to databases... with multiple records (or instances in CongoSpeak) instead of a single file.

It sort of warps the brain a bit to get used to it. We throw around the term CongoTree (instead of table) to refer to the namespace/instance path by which you reach an element. For example, we took a few moments to design a folksonomy bolt-on tool for web sites. There are two namespaces the tag word tree which contains each object ID that has received a specific tag and the object tree which contains the tags that an object has received. The tags are symbolically linked so that if you edit one (to correct spelling or add a German translation), you only have to change it in one place. But the tree is easily queried from either side. The hash space is populated sparsely so that you can add a new property to one tag (shoes) but it isn't expected anywhere else.

So we're accomplishing what you're describing (shoes getting a different set of properties than camcorders) without creating more tables to maintain or altering the actual database table structures. We simply add new namespace entries.

Does that description communicate effectively?

It reminds me that I should get the our data table layout posted up here soon. :)

Hi, I thought that maybe I could come online and explain a little more about how the code is organized in the online docs, as well as the layout of congo as it stands. The site really does contain all of our code, but the docs are organized more for us than general consumption at the moment, something I'm working on now...

There are two main sets of core congo documentation for now, the c++, and the python. The third set, labeled libbu++ is a support library of c++ code that is non-application specific, but referenced heavilly from the c++ code.

Within the python code there are a number of utilities for interacting with the congo database system, but all of it is client side. There is also a very flexible and easy to use CGI system for doing web development and "bolt-ons" with python and congo, on any web site.

Within the c++ code there is the master congo server, named simply "congod" and the c++ client library and command line access and management tool called "congo". You will also notice a section of "shared" code, which is used by both the congo server and c++ client.

Finally, nestled within the c++ section is another support library for simple thread handling from c++ called libito (ito being a japanese word for silken thread, if I'm not mistaken).

At the moment the java client is lagging the furthest behind, so we havn't bothered to post the somewhat lacking codebase for that...

The docs are coming along, but all of the code is exposed. In a day or so the documentation should be organized a little better, plus it should auto-update every night, and we may even add other formats, like pdf.

***edit:

I already redid the layout. You can now view the server, c++ client, and python client. Each contains all of the extra libraries that you need to know about (that we wrote), as mentioned above. you can also view the docs online or download a pdf!

Arthur Brock said:

There are two namespaces:

• the tag word tree which contains each object ID that has received a specific tag

and

• the object tree which contains the tags that an object has received.

This is an old "problem" in semantics (see the seminal article by Gottlob Frege [late 19th Century], in which he analyzes refering to Venus as either the "Morning star" and/or the "Evening Star"). Note how this example very well illustrates how tags are / can be quite context sensitive, such that an individual tag e.g. "close up" can be interpreted quite differently e.g. WRT either photography vs. surgical operations. Note also that such ambiguities are very language dependant.

Mike did update the CongoDocs at http://congodocs.geekgene.com - If you're having trouble seeing a difference in the first couple screens, try refreshing your browser. Mine was caching the old version.

Arthur wrote:

Think of the Linux file system for example... Every file is a numbered inode. The namespace (e.g. /home/artbrock/images/extreme_close_up.jpg) points to the ID of that inode. We can even create a symbolic link elsewhere (e.g. /home/konrad/friends/arthur/picture.jpg) that points to the precise same inode ID. One element addressable by two different paths in the namespace. We're doing that to databases... with multiple records (or instances in CongoSpeak) instead of a single file.

Forgive me for picking nits here, but this is a software discussion where it is important. What you are describing here is more aligned with hard links as apposed to soft or symbolic links. If the link uses the number, it is a hard link and is completely independent of the namespace. A symbolic link uses the namespace for the linking. Either way you point to only one object, but with a symbolic link if you move the destination object (change its name), you break the link unless you run around updating all of them. With hard links, all the namespace objects point to the same ID, so you change the object and all the links see it. If you want to branch the object you have to create a duplicate with a newly created ID.

You can also run into the opposite problem where you have created two objects (one for Morning star, one for Evening star) and now you want to merge them together.

Gerry said:

What you are describing here is more aligned with hard links as apposed to soft or symbolic links.

Fair enough, Gerry.

You can also run into the opposite problem where you have created two objects (one for Morning star, one for Evening star) and now you want to merge them together.

True. But the cool thing about having sparse data sets with variable properties is that you could decide to merge them later when you discover they're the same thing. If all the properties were already identical, then just link. If there are different properties, replicate some onto the new object and Voila! The only problem is if some matching properties conflict... then you'll need to do some intelligent resolution to them.

Yes, the merging should be straightforward in most cases, and in the worst case you keep both values, flag the conflict, and a human has to decide how to resolve it.