Cassandra & Distributed File System

Unlike HBase, Cassandra has no dependencies on a distributed file system. However, in a sense it implements a partitioned strategy that effectively replaces any (otherwise) required hard dependence on distributed file system capabilities.

This statement should not be taken as an endorsement of one or the other. Instead, it is simply an important architectural observation that directly impacts the overall system architecture.

Hadoop Combiners and Map-Reduce

According to the Hadoop documentation, a Hadoop Combiner can be used to speedup Hadoop Map-Reduce if the Reduce function is both commutative and associative. It would be interesting to write a program that introspects a Hadoop Reduce function to draw a conclusion regarding the validity of this constraint for the use of Hadoop Combiner but it will not be trivial.

Cloud Computing--No Single Point of Failure

A UC Berkeley technical report on cloud computing observes:

Just as large Internet service providers use multiple network providers so that failure by a single company will not take them off the air, we believe the only plausible solution to very high availability is multiple Cloud Computing providers. The high-availability computing community has long followed the mantra “no single source of failure,” yet the management of a Cloud Computing service by a single company is in fact a single point of failure. Even if the company has multiple datacenters in different geographic regions using different network providers, it may have common software infrastructure and accounting systems, or the company may even go out of business. Large customers will be reluctant to migrate to Cloud Computing without a business-continuity strategy for such situations. We believe the best chance for independent software stacks is for them to be provided by different companies, as it has been difficult for one company to justify creating and maintain two stacks in the name of software dependability

Read more of the report for other astute observations.

Design Principles for Modern Distributed Systems

The design principles used for Amazon's S3 are generally applicable to modern distributed systems design. Quoting the S3 design principles:

Amazon S3 Design Principles

The following principles of distributed system design were used to meet Amazon S3 requirements:

• Decentralization: Use fully decentralized techniques to remove scaling bottlenecks and single points of failure.

• Asynchrony: The system makes progress under all circumstances.

• Autonomy: The system is designed such that individual components can make decisions based on local information.

• Local responsibility: Each individual component is responsible for achieving its consistency; this is never the burden of its peers.

• Controlled concurrency: Operations are designed such that no or limited concurrency control is required.

• Failure tolerant: The system considers the failure of components to be a normal mode of operation, and continues operation with no or minimal interruption.

• Controlled parallelism: Abstractions used in the system are of such granularity that parallelism can be used to improve performance and robustness of recovery or the introduction of new nodes.

• Decompose into small well-understood building blocks: Do not try to provide a single service that does everything for everyone, but instead build small components that can be used as building blocks for other services.

• Symmetry: Nodes in the system are identical in terms of functionality, and require no or minimal node-specific configuration to function.

• Simplicity: The system should be made as simple as possible (but no simpler).

Elasticity in Cloud Computing

As Berkeley computer scientists have noted in a recent technical report, cloud computing (as utility) providers bring an economics where "using 1000 servers for one hour costs no more than using one server for 1000 hours." Economically, this translates into an elasticity of resources" without paying a premium for large scale ... unprecedented in the history of IT."

Interactive GAE / AWS application

Out of pure curiosity, it would be interesting to build applications with applications parts interacting with both GAE and AWS at the same time.

Building PostgreSQL on Windows

If you want to get and build PostgreSQL on Windows, try using the "git" repositories here. The repository that works with the anonymous cvs is apparently the root repository and the "git" ones are "mirrors" but it turns out that the root cvs repository has some Windows end-of-line characters that cause the build to fail. The repository you can access through "git" seems to be missing these and the build goes through. You need to ensure your cygwin installation includes readline, bison and flex.

A Brief Comparision of Concurrency in Grand Central Dispatch (GCD) and in Scala

This is a first, brief note on the general similarities and differences in how concurrency constructs in Apple's GCD (Grand Central Dispatch) compare with similar constructs in Scala.

Both Scala and GCD require Closures / Code Blocks as well as Mutability / Immutability as the language-level foundation for their concurrency models.

• Scala has supported closures (or code blocks) and mutability declarations as basic primitives in its more general functional extension to Java.
  
• Apple has proposed the addition of elements to the C language that would bring support for code blocks and mutability to the standard C language. Apple has also modified the GCC 4.2 (C compiler) to accommodate the addition of these elements. Apple has made these modifications available under the MIT license. Apple has used these extensions in OS-X 10.6.
  

Both Scala and GCD support an event driven concurrency model.

Both Scala and GCD support "segmented programming". The segmented programming model is very popular and the accepted norm in GUI programming, where the developer implements, essentially, a series of call-backs that are called in response to various GUI events. Segmented programming model is an excellent model when we wish to support concurrent execution, as is often the case with GUI development. (For more on segmented programming model, see presentation Ulf Lange and I gave in Java One 2002.)

The major difference between GCD and Scala concurrency model is that GCD relies on dispatch/queue paradigm while Scala relies on message/actor paradigm.

I believe any complexity in using either GCD or Scala has to do primarily with learning the proper use of closures and immutability.

There are a few other things that separate GCD and Scala.

Scala is a natural evolution of Java as Java was a natural evolution of C or C++. For this reason, Scala provides familiar grounds to the Java programmer. It will be very familiar to Java programmers who are willing to learn about functional features as well as more complete static typing in Scala which helps with program conciseness. (As a Java programmer, I could not put down the first Scala book I picked up. All enhancements, modifications and changes that Scala has brought with respect to Java make sense, and the resulting conciseness in Scala proves to be a blessing when it comes to development work.)

GCD relies not only on the C language extension Apple has proposed (and the C compiler changes it has made available under a very permissive licensing model) but also on the specific APIs for GCD. These APIs are particular suited to the Objective-C and Mac OS-X environment. However, the major part of GCD API can be implemented in all environments and all of it will make sense in any Unix/Linux based system.

It would be good to see Apple's proposed changes to the C language adopted by the standards committee that looks over C.

Natural Evolution ...

C-->C++-->Java-->Scala

Any, Nothing, Scala and Java

The type verification engine in Scala outdoes the one in Java.

To enable this relative strength, Scala has defined, among other features, a closed type system, with the inclusion of "Any" and "Nothing".

"Any" is the super-type of all types. This is nothing new. Java has this, too, essentially speaking.

"Nothing" is the subtype of all types. When it comes to classic and modern type theory, there's nothing new about "Nothing" either. However, "Nothing"-ness is revolutionary new when thinking of how Java has treated types.

Java designers -- perhaps for the sake of the love of trees (I mean "trees" as a graph concept) and the love of "simplicity" of type system that characterizes OS and language designers wish to serve common programmers in general -- did not stipulate "Nothing". Had they done so, we would have had a completely different language in its characteristics from what became Java.

"Nothing" is a simple change but the consequences on the type system and language features are widespread.

Hadoop Tutorial in a virtual machine

Cloudera virtual machine can be used to play with Hive and HDFS environment. See here.

Configuring Java and JDK on Snow-Leopard

You can find the best explanations and backward configurations at http://wiki.oneswarm.org/index.php/OS_X_10.6_Snow_Leopard

Who says Java is slow?

"Slowness of Java" proved to be a myth even back in 1996-1997, when Java ORBs kept beating C++ orbs in various RPC benchmarks.

Consider also, that in July 2008, Hadoop won the Terabyte Sort Benchmark: One of Yahoo's Hadoop clusters sorted 1 terabyte of data in 209 seconds. This beat the previous record of 297 seconds in the annual general purpose (Daytona) terabyte sort benchmark.

It was the first time that either a Java or an open source program had ever won.

Using SVN to find diffs on repositories

Try something like the following :

svn diff -r779768:779000

which will produce file names and diffs between those two revision numbers of a code repository under svn control.

SOA . . . Is that All ?

I originally wrote this entry on October 5, 2004, and published it on blogs.sun.com.

So, do I really get to cook SOA with these 10 ingredients? On ingredient number 6 (Governance), at least, I'd recommend another look.

Production or Maintenance & Support

In The Cathedral & The Bazaar, Eric S. Raymond estimates that 75% to 90% of software business *effort* is not spent on the production of "products" but on support and maintenance of software products and systems. The sources of Raymond's claim are interesting in themselves ...

Bugs Lead to Existential Questions

I originally wrote this entry on September 30, 2004, and published it on blogs.sun.com.

There are some who believe that we only switch from our "being" mode to our "thinking" mode of existence when something breaks down.

Let me give an example to make things clear.

Say, you're walking in Central Park in New York City towards a bench to sit for a moment.

As you do this, you're not constantly thinking: "Oh, this is a bench; now I'm 10 yards from it; it has 4 legs; it is painted green ; now I'm 4 hards from it; it is made of wooden planks and the legs are steel, etc. . . ." (Are Central Park benches green and made of wooden planks and steel? Or are they coming in concrete these days?)

If we thought in this fashion and of all these details, we'll go mad very quickly. Instead, we simply go and sit on the bench and enjoy the fresh air. That's what I like to call the "being" mode of existence.

Now, if the bench breaks beneath you as you sit, it will probably lead you to fall.

At that point, you will start going over everything that is supposed to make it a bench and wonder what went wrong, what "broke," what was not as it was to be. That's what I like to call the "thinking" mode of existence. A mode that deals with bugs in the existential environment.

So, it should be no wonder that the best way to start learning and thinking about existing, complex code and the process around its production, is to start by debugging it. Once the code is known and navigable, we become comfortable and at home, returning to the "being" mode. We become the code and know it as us.

Which mode is prior to the other? For which mode are we best tuned? Which is a "better" mode? Is code a place to be or a place to think? Is code even a place?

I don't know the answers to these questions.

What I do know is that every software engineer (or urban professional for that matter) needs to get out some times to see the world, take a bicycle ride (1, 2) or go on a simple stroll in the park, to sit on a bench, preferably made of wooden planks that won't break.

Open Source Society

I originally wrote this entry on September 19, 2004 and published it on blogs.sun.com.

We (and this does not include just the U.S.) are already an open-source society to a very large extent. Information is widely available for those who care to find out and some have proven it possible to do so by their own example.

However, open dialogue matters well above and beyond open source.

Mixing, on the importance of which to innovation Lawerence Lessig has built a whole case, is simply an instance of open dialogue.

Open-source (widely available) information might be a pre-requisite for substantive dialogue but it neither replaces or guarantees it nor leads to it.

Finally, there're those who believe that what matters most is not cyber-dialogue but committed, emboddied dialogue and responsible action, as Hubert Dreyfus has noted in his analysis of the Internet.

Open Dialogue Code

I originally wrote this entry on September 19, 2004 and published it on blogs.sun.com.

It's funny how certain corporate strategists pay more detailed attention to the press than to their own corporate purpose. (Some may even believe that all their problems stem from bad press.)

So, now we read in the reports (Reuters and the WSJ) that Microsoft has decided to open source its Office software to certain governments and under certain conditions. The same reports say that Microsoft has apparently done this to combat the advances of Linux, its "open-source" desk-top OS rival.

Under the program, Microsoft doesn't completely lift the veil. Governments are able to see 90% of the source code. The bulk of the rest is code where a third party owns the copyright, according to Microsoft. The company also holds back from exposing code that relates to antipiracy technology. (The WSJ, Sept. 20, 2004)

Even if source code is to be shared with more governments and under much less strict conditions, the strategic threat that Microsoft faces will persist. The problem is not an inability to see the code but an inability to participate in the dialog that ends up in it.

As I've said earlier, what matters is not openness of the source (whatever that means) but the openness of the dialogue about the source code, its use and evolution.

And here is another question worth thinking about. Would government regulations around the world ask for "open-source" code or "open-dialog" code? Most probably, it would be the former because "open-source" is a property more measurable.

A Real Student Bargain for JavaOne 2009

This year, JavaOne let's students get in free!

Plus, they can sign up for CommunityOne at the same time.
Both passes are full conference passes with access to everything!
Educators can also get in free when they bring 10 students with them to the conference.
If they aren't able to do this, they still get a fantastic deal of $895 for a full conference pass.

What is a Full Pass?

All the benefits of a FULL conference attendee, which includes: JavaOne general and technical sessions, panel discussions, HOLs (Hands-on-labs), BOFs (Birds-of-a-feather), the Pavilion and the After Dark Bash / Networking Mixer! Not to mention a "hang space" (which includes video games, movies & more!), plus you will be able to check email any time at one of our many hacker stations.

PLUS

Admission to the CommunityOne Deep-Dives. What does this mean? FREE training by our very own SLS on some really great topics, plus hugely discounted certification (vouchers will be handed out at the event)- certification can even be done on-site! There will also be numerous parties and chances for you to network with the best and brightest in the industry, from community leaders to "Rockstars" and everyone in between.