Octopus Scanner: Java Build Tools and Malware

Alvaro Muñoz recently posted "The Octopus Scanner Malware: Attacking the open source supply chain" on the GitHub Security Lab site. I found this post to be interesting for a number of reasons, including its detailed coverage of how the Octopus Scanner malware works and how it was discovered, how the GitHub Security Incident Report Team (SIRT) went about addressing it, how it affected a popular Java IDE, and how GitHub works to detect and address risks to open source software deployed on its site.

Muñoz calls Octopus Scanner "an OSS supply chain malware" and writes that 26 GitHub-hosted open source projects "were backdoored by this malware." In the post, Muñoz describes in detail how Octopus Scanner works. The post in its entirety is worth reading, but here are some highlights describing Octopus Scanner:

• "Octopus Scanner malware is only interested in the pre-jar and post-jar tasks" of "a NetBeans project build."
• "The malware disguises itself as an ocs.txt file, but we can easily determine is is actually a Java Archive (JAR) file."
• "The malware also infected any JAR files that were available in the project, such as dependencies - not necessarily just build artifacts."
• "The octopus.dat payload is the binary that actually performs the NetBeans build infections."
• "cache.dat is responsible for backdooring the built classes so that when these classes get executed, they will infect the underlying system."
• Octopus Scanner is designed to target "UNIX-like systems", MacOS, and Windows.

I found the highly detailed approach to diagnosing Octopus Scanner's behavior in the GitHub post to be fascinating and insightful reading. It was especially insightful to see the tools and methods used to understand better how Octopus Scanner behaves. For example, they used ClassFileTransformer and "a Bytecode manipulation library such as Javassist or ByteBuddy to inject our analysis code" into the "class responsible for decrypting the blob ... right before it actually gets loaded into the JVM."

Besides the interesting details of how Octopus Scanner works and how it was discovered and researched, other interesting insights in this GitHub post are related to the risks open source builds face. Muñoz writes, "Infecting build artifacts is a means to infect more hosts since the infected project will most likely get built by other systems and the build artifacts will probably be loaded and executed on other systems as well." Muñoz adds, "In an OSS context, it gives the malware an effective means of transmission since the affected projects will presumably get cloned, forked, and used on potentially many different systems. The actual artifacts of these builds may spread even further in a way that is disconnected from the original build process and harder to track down after the fact."

Muñoz opens the post and concludes the post with some discussions about this and other attempts at sabotaging open source products and their builds. One chilling thought is included in the Conclusion: "Since the primary-infected users are developers, the access that is gained is of high interest to attackers since developers generally have access to additional projects, production environments, database passwords, and other critical assets. There is a huge potential for escalation of access, which is a core attacker objective in most cases."

Apache NetBeans?

It's fairly common to have significant announcements related to the world of Java released in the days and weeks leading up to JavaOne. With that in mind, it's not surprising that we're seeing some significant Java-related announcements just prior to JavaOne 2016 that begins next week. One announcement is Mark Reinhold's Proposed schedule change for JDK 9 in which Reinhold proposes "a four-month extension of the JDK 9 schedule, moving the General Availability (GA) milestone to July 2017." Another major proposal, the subject of this post, is the proposal by Oracle for Oracle to "contribut[e] the NetBeans IDE as a new open-source project within the Apache Incubator."

The Apache NetBeans proposal is summarized on NetBeans.org, but additional details are available on Apache Software Foundation's Incubator Wiki page called NetBeansProposal. The NetBeansProposal Wiki page provides several details related to the benefits, costs, and risks associated with moving NetBeans to the Apache Software Foundation. Additional views on this proposal that summarize or interpret the proposal can be found in online resources such as Proposal has NetBeans moving to Apache Incubator, Oracle's NetBeans Headed to The Apache Software Foundation, Oracle no more - NetBeans is moving to Apache, Java founder James Gosling endorses Apache takeover of NetBeans Java IDE, An unexpected proposal: Oracle bids farewell to NetBeans, and Oracle Proposes NetBeans IDE as Apache Incubator Project. There are also two Reddit threads on this subject on the subreddits programming and java.

I've felt for some time that the open source projects I'm most willing to "take a chance on" and recommend to management and customers are those that have either strong corporate sponsorship or are affiliated with an established and successful umbrella organization such as Apache Software Foundation. Therefore, although I don't like to see NetBeans lose the corporate backing and investment of Oracle, the Apache Software Foundation does provide a home for NetBeans to continue being a successful project.

Like many software developers who have been working in this area for years, I've been using Apache Software Foundation projects for most of those years. The liberal Apache 2 license is welcoming and uncomplicated. The projects tend to be well run and well used. On occasion when projects are no longer active, the ASF is fairly timely in moving such projects to the Apache Attic. Projects associated with ASF tend to enjoy benefits often associated with open source such as multiple contributors including multiple reviewers and real-life "testers." Many of the ASF projects enjoy a large community with the accompanying benefits of a large community such as improved main site documentation as well as third-party supplemental documentation with blogs, books, and articles. Of course, NetBeans already enjoys much of this, so moving to ASF might be more of an approach to retain some of the advantages it already enjoys while at the same time potentially encouraging greater community collaboration.

The Apache Software Foundation projects I've used over the years seem to come from two different types of origins. Some of them have been associated with ASF from their beginning or almost their beginning while others were popular projects already when they were moved to the ASF. NetBeans falls in the latter category with other projects that I used before they went to ASF such as Groovy (from SpringSource/Pivotal) and Flex (from Adobe). It seems likely that Oracle has proposed donating NetBeans to Apache Software Foundation for the same reasons that Pivotal and Adobe donated Groovy and Flex respectively to Apache Software Foundation.

The examples just mentioned (Adobe|Flex, Pivotal|Groovy, and Oracle|NetBeans) are just a subset of examples that could be cited in which corporations who are the sponsors and dominant contributors have given away the open source project, typically with the intent to spend fewer resources managing that project. If NetBeans is able to enjoy significant community contributions, the disadvantages of reduced corporate sponsorship might be at least partially offset. Some of this, of course, depends on what level of involvement Oracle supports its employees in contributing to NetBeans.

When Oracle acquired Sun, many of us wondered about the future of GlassFish (Oracle had already acquired WebLogic from BEA) and NetBeans (Oracle already had a free, but not open source, Java IDE in JDeveloper). Oracle announced in 2013 that GlassFish 4.x would not be available as a commercial offering and would only continue as an unsupported Java EE reference implementation (though third-party support can be found for the "drop-in replacement" Payara Server). Although there are some advantages to this "developer-friendly" reference implementation in terms of trying new Java EE features and learning Java EE concepts, most Java EE developers I'm aware of who use an open source Java EE application server for production have moved to WildFly. Given this, I've been happy to see NetBeans moving along and being supported as well as it has for as many years as it has.

One potentially new prospect for NetBeans is being the basis for more specialized IDEs. Eclipse has long been the basis of specialized IDEs and development tool suites such as Spring Tool Suite (Spring IDE), Oracle Enterprise Pack for Eclipse, Adobe Flash Builder, Red Hat JBoss Developer Studio, and Zend Studio. Similarly, Android Studio is built on IntelliJ IDEA. Although there are already tools based on NetBeans (such as VisualVM), NetBeans's independence from Oracle may seem more attractive to some for future tools' development.

At the time of this writing, the NetBeansProposal Wiki page already lists 63 people in "the initial list of individual contributors" (including 26 people contributors associated with Oracle). That, along with the extensive resources already available related to NetBeans, encourage me and make me think that NetBeans could be a successful and thriving Apache Software Foundation project. I certainly prefer NetBeans's chances as an Apache Software Foundation project over its chances if it existed in a state similar to that placed upon GlassFish.

We Java developers are fortunate to have multiple very strong IDEs available for our use. It's in our best interest if they can each remain strong and viable as all the IDEs (and the developers who use them) benefit from the competition and from the innovation that talented developers working on these IDEs bring to our development experience. Each of the IDEs offers different advantages and has different strengths and I'm hoping that we can benefit from NetBeans's current strengths and future strengths for years to come.

The End of Google Code

In the 21 January 2014 post Google Code is dead, Evert Pot referenced the post A Change to Google Code Download Service and wrote that "It's been sort of obvious for a while that [Google] stopped caring about their code hosting." The title of Pot's post was borne out with the announcement this past week that Google is Bidding farewell to Google Code.

According to the post Bidding farewell to Google Code on the Google Open Source Blog, "To meet developers where they are, we ourselves migrated nearly a thousand of our own open source projects from Google Code to GitHub." That post also outlines the final days of Google Code. No new projects can be created (as of 12 May 2015) and the site will become read-only on 24 August 2015 with closure of the project hosting on 25 January 2016 (though tarballs will be available throughout 2016).

I recently posted on the fall of Codehaus and mentioned several useful projects that were (or are) hosted there. Google Code also saw its share of important and useful projects in its heyday. These include Google's Guava (now on GitHub), Mockito (now on GitHub), charts4j (now on GitHub), easyb, Google's Go programming language (now at https://golang.org/), Google's Google Web Toolkit (now at http://www.gwtproject.org/), and Google's Chromium(now at http://www.chromium.org/).

Coman Hamilton concludes his article Google Code is dead – but today is a good day for open source with the statement, "Rather than lament the loss of one significant member of the open-source hosting community, we should rejoice in the fact that there are so many other great open-source hosters, that not even Google can compete."

Total Bummer: Pivotal Drops Groovy

Pivotal announced today that Groovy 2.4 And Grails 3.0 will be the last major releases under Pivotal sponsorship. This is big news that has not surprisingly created a lot of buzz in the blogosphere. In this post, I describe some of the questions that others and I are wondering about and speculate on Groovy's future.

After reading multiple Reddit references to this announcement, my initial thought was to see what Guillaume Laforge had to say about this. Apparently, a lot of people had the same idea because I encountered a 503 error when trying to access his page.

Fortunately, I did not have to wait for Laforge's blog to be available to get more insight from him on this announcement because there were a couple of interviews with him regarding the announcement already online: Voxxed.com's Pivotal’s "Sad and Odd" Decision to Set Groovy Adrift and InfoQ's Pivotal Pulls Groovy/Grails Funding. Since that time, Laforge's blog is available again and has a post on the subject called The Groovy project is looking for a new home. Anothrt person frequently and deservedly associated with Groovy, Graeme Rocher, has also posted on the subject: The Future of Groovy and Grails Sponsorship.

Laforge and Rocher were co-founders of G2One, which was acquired by SpringSource in late 2008. VMWare then acquired SpringSource about one year later (and VMWare had been owned by EMC since late 2003). EMC would later announce the spin-off of Pivotal in 2013 and Pivotal today announced the dropping of Groovy support as of 21 March 2015.

Questions, Answers, and Speculations

The posts referenced here in my post have collectively answered some of my questions about Groovy and at the same time presented additional questions.

Why is Pivotal dropping financial support of Groovy and Grails?

Answer: Pivotal's announcement: "The decision to conclude its sponsorship of Groovy and Grails is part of Pivotal’s larger strategy to concentrate resources on accelerating both commercial and open source projects that support its growing traction in Platform-as-a-Service, Data, and Agile development. Pivotal has determined that the time is right to let further development of Groovy and Grails be led by other interested parties in the open source community who can best serve the goals of those projects."

Who Might Sponsor Groovy and/or Grails Development?

Speculation: Many organizations benefit from Groovy and Gravy, but many probably aren't prepared to invest as fully in their development as G2One, SpringSource, VMWare, and even Pivotal have been. An example of an organization with an obvious vested interest in Groovy's future is GradleWare. Ken Kousen has tweeted and written a blog post on the opportunity of acquiring Groovy and Grails sponsorship.

What does this announcement mean for Groovy's future?

Answer Mixed with Speculation: Based on Laforge's and Rocher's posts, it seems clear that the core developers plan to continue working on Groovy. However, it is understandable that if this effort is not funded (sponsored), it will have to be at a slower pace than before (I have found through personal experience that home projects take a lot longer to complete than paid projects). I believe that Groovy has strong momentum already that will continue for some time. It is vital to Gradle, is used with other open source projects and tools such as SoapUI, and could have a promising future running on Android. I primarily use Groovy for scripting and simple "glue" tools in Java applications. The language is mature and serves these purposes well and I see no reason to stop using it at this time.

What does this mean for the future of the Spring Framework?

Speculation: There is some concern that perhaps Spring Framework could be jettisoned next from Pivotal. This seems unlikely to me, but I didn't expect Pivotal to drop Groovy either. As much as I love Groovy and as much effect on Java and JVM development as I acknowledge it has had, I think Spring Framework has been even more pervasive in Java EE development than Groovy and Grails have been in Java SE and Java EE development. That stated, Pivotal has shown that they are willing to, as most successful business are, drop a product offering that is perceived as not benefiting their strategy and bottom line. I can certainly understand if this development concerns users of Spring.

Is Standards-Based More Important than Being Open Source?

Answer: This is a difficult question to answer that often depends on numerous contextual factors including the tools being compared, the expected length of life of the products being built, etc. Fortunately, we often don't have to choose between these as many reference implementations in the Java world are also open source. However, a point can be made that any product that is not standard (including commercial or proprietary) is subject to losing support or not being available any longer. The theory is that if standards-based products are used, one can then shift to another implementation of that standard if necessary. However, a standard is only as good as its implementations and if there is only one realistic implementation of a standard, there's not much of an advantage of transferability there.

Conclusion

Although I understand Pivotal's motivation for dropping Groovy, I am still sorry to hear that news. I appreciate the effort that key Groovy contributes such as Laforge and Rocher have made and I appreciate the companies that have sponsored that work. Through this sponsorship and work, we have a really nice language to use for scripting and other purposes. I hope that a sponsor can be found for Groovy, but I plan to continue to use it either way.

Improving LibreOffice with Coverity Scan

Coverity, Inc. issued a press release this morning announcing that "the LibreOffice team" has "analyzed more than 9 million lines of code to find and fix more than 6,000 defects." In the press release, Zack Samocha, senior director of products for Coverity, states, "LibreOffice’s remarkable results after just two years [since October 2012] of using the Coverity Scan service reiterates the mission criticality of software testing for the open source community to find and fix software defects early."

This press release cites the Coverity Scan 2013 Open Source Report in explaining the degree of success the LibreOffice team has achieved. Specifically, according to the press release, the LibreOffice team has "reduced the project’s defect density from .8 to .08."

I was curious about some of the specific details associated with LibreOffice's use of Coverity Scan to reduce defects and improve quality and so took advantage of an offer to ask Zack Samocha some questions. The remainder of this post indicates my questions, Zack's answers, and some related references.

Q: What programming languages are used for LibreOffice (all/mostly C++ or some Java or other languages)?
A: The language used for LibreOffice is mostly C++.

Q: What is an example of one of the most common types of defects discovered and fixed in LibreOffice?
A: The top issues were:

• Error handling issues = 2271
• Null pointer dereferences = 1796
• Uninitialized members = 1145

Q: Is this typical of other open source projects analyzed with Coverity Scan?
A: This is comparable for OSS projects [with more than] 1 million lines of code (LOC)

Q: What is an example of one of the most serious types of defects (high-impact) discovered and fixed? Is this typical of other open source projects analyzed with Coverity Scan?
A: The most serious are memory related. For example, memory-illegal accesses (there were 23) and memory–corruptions (there were 17). This amount is common for such large code base.

Q: Are there any additional metrics regarding the fixes to LibreOffice using Coverity Scan such as number of developers or number of person hours spent on the effort? Is there any estimate of how much of this effort was identifying the issues (running the scan) versus fixing them and testing the fixes?
A: In the past year, LibreOffice fixed more than 8,500 defects, assuming at least one hour per defects, which is conservative. That's about 365 days of work for a single developer.

Q: How does Coverity Scan differ from FindBugs, PMD, other static analysis tools, and IDEs' built-in static analysis support? What advantages does Coverity Scan offer instead of or in addition to those tools?
A: At Coverity, we believe in open source collaboration. Coverity complements Findbugs, PMD and others. In fact, in Coverity Scan and our enterprise products, we provide FindBugs defects in the same workflow as defects found by Coverity development testing solutions. Coverity and FindBugs are looking for different things. Coverity is designed to find critical, crash-causing defects where FindBugs is best suited for find coding-style and best practice-type issues. To illustrate the point, we conducted an experiment with the Jenkins open source build server under a controlled environment. We found 296 critical defects while FingBugs found 1010 coding style issues. There were only 30 defects that were found by both solutions. Coverity analysis tends to be more inter procedural, in addition Coverity covers OWASP10 for Security issues.

Related References

• LibreOffice Coverity Defect Density
  • "We run LibreOffice through Coverity approximately once a week."
• LibreOffice Happy To Work With Coverity Scan Results
  • "The work on using the Coverity reports for LibreOffice is done by a variety of LibreOffice developers, some on the building and testing, others on the other work to fix the issues. In the first months many hundred improvements have been made, making LibreOffice more robust, better."
• LibreOffice 4.3, The Almost Perfect Office Suite
• What Does OWASP Top 10 Coverage Mean to You ... and Do You Have It?
• OWASP Top Ten Project

Coverity Scan 2013 Open Source Report

The Heartbleed Bug has received significant attention lately and has reignited discussions regarding open source security issues and open source quality issues. The article Heartbleed: Open source's worst hour goes so far as to open with the sentiment that Heartbleed is "open source software's biggest failure to date." In the midst of this discussion, the Coverity Scan^TM 2013 Open Source Report has been released and provides another interesting source of input for the discussion.

Coverity Scan's^TM main page states that it uses static analysis to "find and fix defects in your C/C++ or Java open source project for free." Coverity, which was recently acquired by Synopsys, originally teamed up with the Department of Homeland Security to develop the Coverity Scan^TM as part of the "Open Source Code Hardening Project." Last year's edition, the Coverity Scan: 2012 Open Source Report, found that "Code quality for open source software continues to mirror that of proprietary software–and both continue to surpass the accepted industry standard for good software quality." The just-released 2013 Coverity Scan^TM Open Source Report reports a change this year, "Open source code quality surpasses proprietary code quality in C/C++ projects."

Although the Coverity Scan^TM Open Source Report has mainly focused on the "state of open source software quality" in terms of C/C++ projects and Linux in the past, the 2013 report also adds Java-based open source projects Apache Cassandra, Apache CloudStack, Apache Hadoop, and Apache HBase. The report acknowledges that "we are still in the early days of working with Java projects" and looks at some possible explanations for the Java code that was analyzed having higher defect rates than the C/C++ code that was analyzed. These reasons include Java source code being new to the analysis (and thus not benefiting from being able to address previous results) and the use of FindBugs ("Many of the FindBugs checkers generate large quantities of results, in particular in the areas of dodgy code, performance and bad practices").

One of the other "key differences" analyzed in the 2013 Coverity Scan Report^TM is a lower percentage of "resource leaks" being fixed in analyzed Java code than in analyzed C/C++ code. The report's authors postulate that this might be explained by Java developers relying more on "some of the built-in protections in the language, such as the garbage collection." The authors point out potential fallacies of those types of reliance.

The 2013 Coverity Scan Report^TM includes an interesting assessment, "Quality concerns are no longer a barrier to open source adoption in the enterprise. In fact, the quality of the open source code for Coverity Scan participants can be higher than the proprietary code included in an enterprise product." Although not all open source is created equal and although product A is not necessarily superior to product B simply because the former is open source and the latter is proprietary, it is interesting to see more empirically driven studies demonstrating advantages of open source rather than relying on opinion, wishful thinking, and anecdotal evidence.

JSF Open Source Throwdown

Optimus Prime's post IceFaces Copies PrimeFaces Line by Line and Brian McKinney's response (New ACE Component Origins) have led to a controversial discussion on the blogosphere regarding legalities and ethics in an open source context. It is interesting to read the perspectives of those associated with PrimeFaces, the perspectives of those associated with ICEfaces, and the community reaction (and here).

Although I tend not to use JavaServer Faces, I have still found the general discussion interesting because the points and counterpoints being made could apply to any open source development projects. Perspectives on what is legal, what is ethical, and what is morale in open source appear to differ greatly among users of these products. Some people think that everything that has been done is legal and ethical. Others believe it is legal but unethical and still others believe there may be copyright infringement even if there are no license issues.

There are numerous good points made in these online discussions that made me think more about the nature of open source. There is no question that there are numerous different perspectives on what's right and wrong in open source, but I'd like to see opinions from well-known open source advocates such as Simon Phipps and Richard Stallman on the matter.

This recent open source controversy is also a reminder of the importance of choosing the appropriate license for one's open source projects. There is always a tenuous balance between wanting to offer a very liberal license to increase adoption and wanting to protect one's work and get appropriate credit/attribution for that work. The Apache Software Foundation license is very friendly to other users, but in this case it seems it may be a little more liberal than the folks at PrimeFaces would have liked.

The Problem of the Open Source Commons: Harsh Economic Realities of Open Source Software

The apparent ultimate fate of Sun Microsystems may be the most obvious example of the difficulty associated with earning significant revenues developing and sponsoring open source software. A particular open source product may be considered highly valuable by many users who, given the appropriate license agreement, never actually back up this perceived value with any type of monetary compensation. This touches on a central debate about open source licenses that provide software with very few restrictions on end use and do not require any type of payment versus open source licenses that restrict end-user freedom to a certain degree and require certain payments as a form of acknowledgement of the value received with use of that open source product.

As an end user of an open source product, I obviously prefer no strings attached and no license fee required. However, as a realist and as someone who is paid for at least some of his own software development, I do understand the desire and even the need for those who contribute significantly to open source to be compensated by those who use that open source.

I touched on this a little bit when I questioned which single entity (or even consortium) would be willing to pay Oracle for MySQL the same price that Sun paid for it (or even in the same ballpark). My speculation then is that MySQL is worth far more to a wide, diverse collective/community than it is to any one owner (except perhaps Oracle). The reason for this is that while MySQL is a heavily deployed database, it still does not garner the types of revenues most entities would need to justify its expensive price tag. This is a single example of the problem of the open source commons -- many of the products are highly valued by their users, but without any or with too little monetary compensation to reflect how valued these products truly are.

Rightly or wrongly, "value" is often measured in terms of dollars. The phenomenon of modern day open source software (not all, but a large and growing portion) is that this basic tenet has been largely ignored. We have enjoyed over the past several years access to many fine open source products in which we have to give absolutely nothing back. Some of us are old enough to remember that it wasn't always this way and is in many ways a recent phenomenon. Others in our profession are young enough to have never known differently. To them, it may seem strange to have to pay for open source.

Regardless of how old or young we are, most of us have gotten used to freely available software such as the JDK implementation, the Spring Framework, a host of useful Apache Software Foundation products, a host of Sun-sponsored products (NetBeans, GlassFish, MySQL, etc.), Eclipse, and on and on.

We are grateful for those who spend many hours contributing to these fine products, but often not grateful enough to actually financially support the same products. I try not to be too hard on myself for this; in many ways it is human nature to take the best deal you can get when it is offered. No laws are being broken (assuming the license allows for free use) and my little contribution wouldn't make a difference anyway. Besides, why would I pay for something voluntarily when I can have it for free?

The above paragraph summarizes the harsh economic realities of the open source software development world. With thinking like that contained in the paragraph immediately above, how can one expect to make enough off of open source development to live on other than via corporate sponsorship (like SpringSource, Sun, etc.)?

There have been several recent online events that have made this problem of the open source commons even more apparent. I believe the tough worldwide economy has accentuated the problem of the open source commons. One of the banner examples of what's good about the "open" world, Wikipedia, is currently prominently featuring a plea for financial contributions.

In a recent post called Funding Clojure 2010, Clojure creator and primary contributor Rich Hickey lays out a firsthand account of the effects of the current prevailing treatment of open source software by its users. I currently don't use Clojure, but it is still remarkable insight into the challenges of making money developing open source and well worth the read even by those not interested in Clojure.

A few months ago, the creator of and primary contributor to JFreeChart posted on his blog that he had accepted an employment contract because of difficulty making a sufficient living selling copies of the JFreeChart Developer Guide. Even with this setback, I was pretty impressed that he was able make a living for seven years. It seems to buck the trend of most open source development not sponsored by corporations who sell software.

One of the biggest problems is that many people think that "open source" and "free" are the same thing. It is true that the trend seems to be moving toward open source products being free of cost as well, but the distinction of "free beer" versus "free speech" is still important. It is important to distinguish between free of cost (gratis) and free of usage restrictions (libre) to understand that one might still pay for an open source product just as he or she would for a closed-source product.

I believe that open source development and use is at a crossroads. The open source development world is facing its own tragedy of the commons. We essentially need individuals and users to do something that does not seem immediately in their self interest (paying for a product that in no way requires them to pay for it) for the greater good and their own long-term good (because of the continuing viability of freely available open source). The problem is that many of us would rather that everyone else pay for it so that we don't need to.

These "tragedies of the commons" are difficult to resolve because they do pit individual (and often short-term) self interest against long-term community benefit. But, just as with other problems of the commons, if we don't find a way to do something about it, the true tragedy will be the loss of benefit to the entire community.

Benefits of Using Open Source: Potential Versus Realizable

There are many advertised benefits of using open source. These potential and advertised benefits include business flexibility, freedom/liberty/libre, (often) no/reduced license costs (gratis), improved security through disclosure, improved testing in real situations, improved code review by multiple reviewers (many eyeballs), high quality, the ability to learn from the code (both about how a particular product works and general coding principles), and even betterment of society. However, I have found that the realization of these potential benefits of open source depends largely on the open source product itself. Indeed, there are many open source projects that enjoy very few of these benefits.

In the remainder of this blog posting, I will look at how the nature of a particular open source product often determines the ability to realize the general advertised benefits of open source associated with that product.

Business Flexibility

It has been argued that an advantage of open source is greater flexibility in switching implementations (no vendor lock-in). However, the ability to switch implementations often depends more on the degree to which a product (open or closed) is based on standards than on whether the product itself in open source or closed source. It is certainly true that a closed source standard-compliant implementation will be easier to switch out from than a non-standard open source implementation. The standards reference implementations in the Java space are open source and therefore are good examples of open source products that do support great freedom to switch out implementation. For example, one can, in theory, easily switch from Tomcat or GlassFish to WebSphere or WebLogic or switch the other way. This does not necessarily have anything to do with Tomcat and GlassFish being open source, but instead has to do with all these products (including the closed-source WebSphere and WebLogic) implementing the same standard.


No Cost / Free Beer / Gratis

Some definitions of open source explicitly require that open source software be freely available without any limitation or cost (free beer). However, others aren't as concerned about the "free beer" aspect as they are about the "free speech" aspect of open source. Although most products that are advertised as open source do seem to be available for no cost, this should not be assumed without careful reading of the product's license. Moreover, there are many proprietary and closed source products that are also free of charge. These include products such as JDeveloper (a Java IDE that is not open source but is available without charge), Oracle Database Express Edition, Adobe Reader, Flash Player, Microsoft Internet Explorer, etc.

As another example, I was using Flex regularly back in its closed-source-but-freely-available days as Flex 2 and really liked it even then. Although I was excited to see Flex 3 open sourced, it really didn't change my use of Flex much in terms of its daily use. I was honestly more interested in Flex 3's new features than the fact that it was now open source.

As I have posted about earlier, I believe that open source contributors have a right to expect compensation for their work and creativity. This compensation for open source contributors often comes from support and service contracts, sales of documentation, sales of related books and articles, and consulting sales.

In the end, price may be a benefit of a particular open source product over a particular closed source product, but there are usually many factors to consider such as the total cost of using each product (training, service contracts, etc.). With freely available closed source products in the mix, it is certainly possible to find a situation in which use of the open source product is no cheaper than using its closed source alternative.


Freedom / Free Speech / Libre

The ability change the code is something that certainly differentiates open source from closed source. However, even this obvious differentiating benefit deserves closer examination when determining how much of this benefit is actually realizable. One consideration is that some open source licenses have restrictions on changes to open source products that are included in distributed products. In some cases, these restrictions may be deemed unacceptable, making this particular benefit unrealizable for that organization with that particular open source product.

A second consideration is to truly evaluate the likelihood of actually changing the open source product's code to suit one's own needs. It may be that an individual or organization has no qualms about this and, in such a case, they will be able to readily realize this benefit. However, if the individual or organization is reluctant to actually make changes to source code of an externally-provided open source product because of needs such as greater testing, greater maintenance costs, additional efforts to integrate with future versions of the product, and so forth, then this benefit may lose its degree of realization.


The Many Benefits of 'Many Eyes'

Several of the benefits commonly associated with use of open source products depend on the presumption of "many eyes" reviewing open source code. It is argued that open source products are inherently more secure than closed source products because there are many more reviewers of the source code. The argument of many more reviewers of code also leads to the advertised benefit of better code reviews and better quality in open source products. With many more users of the open source product, the argument continues that open source is better tested than proprietary software. Finally, another benefit often attributed to open source is a greater likelihood of having bugs fixed quicker. All of this, however, hinges on the presumption that there are indeed 'many eyes' reviewing and using the open source product. What happens when that is not the case? What happens if no one is actively supporting a particular open source product? Do these benefits still apply?

One definite advantage of open source is that one always can look at the source code directly. However, I think many end users of open source don't actually look at source code that often, but instead rely on the assumption that other users of the open source will have reviewed the code, looked for security holes in the code, and run the code in their applications. This assumption is more likely to be correct or appropriate for open source products with large communities than it is for open source products with very small communities.

Open source projects that enjoy large communities are more likely to allow potential open benefits to be realizable benefits. Large communities tend to mean "many eyes," which in turn can lead to less likely security holes, to more highly reviewed code, and to code used more frequently in a wider variety of situations and loads.

At the other extreme are the open source projects with a community of one, the open source project founder. It is unlikely that these open source products with one creator (who is also the product's only user) have anywhere near the amount of review and testing as most proprietary products.

Several people have pointed out that open source does not necessarily make something more secure. This is covered in Many Eyes, But How Many Brains?. A similar argument against stating that open source always means higher quality software is made in Open Source Does Not Make Better Code. Better Programmers Make Better Code.


Learning from the Code

This is probably the generally advertised open source benefit that most easily translates to a realized benefit. Having access to source code can be instrumental in learning how the code works. Viewing the source code can help one better understand how to use the library it underlies. For example, when Michael Martin and I were writing the article Visualize Your Oracle Database Data with JFreeChart, we figured out some minor details of the JFreeChart API by looking at JFreeChart's source code.

Having access to the source code can be useful when debugging or working on performance tuning. It can also be useful for learning how to accomplish certain things in your own code. I remember learning HTML (along with many bad habits) in the early nineties by viewing the source code of others' web pages.


All Open Source is NOT Equal


Even though I use many great open source software products and know of many more, there are many times that number of open source products out there without much of a community and without much to offer yet. It is not appropriate to lump the good and the bad together, but instead good open source should be differentiated from bad or immature open source. When considering how many immature or unused open source projects are out there, consider that SourceForge alone has more than 230,000 software projects (as of February 2009). There are several really good and useful projects on SourceForge, but the degree of value, maturity, or even level of support per project varies greatly. This is true across many open source repositories.


Conclusion

Nothing in this blog posting should be interpreted to mean that I'm against open source. On the contrary, I love using strong open source products with large, vibrant, and active communities. My only point of this posting is that it is too simplistic to argue that open source is always better than closed source. Instead, like most things in life, it is more complicated than that. It really depends on the individual products being compared. The state of being open or closed source can be considered as one of the criteria in selection, but should not normally be the deciding factor. All else being equal (or even close to equal), I typically favor open source products, but I don't blindly select an open source product over its closed source alternative simply because one is open source and one is closed source.

Most of the products I choose to use regularly are open source (Java, Flex, AIR, BlazeDS, OpenLaszlo, Ant, NetBeans, Eclipse, log4j, JUnit, Spring Framework, GlassFish, JEdit, JFreeChart, Ruby on Rails, Tomcat, Apache HTTP Server, various flavors of Linux, Firefox, and on and on). Nearly all of them have very large and active communities. Most of them also have strong sponsors (such as Apache Software Foundation, Sun, Adobe, SpringSource, Oracle, etc.) who provide staffing (in many cases) and common direction for them.

All open source is not created equal. Some open source products are really good; others aren't very good at all. In the end, my favorite open source products are the ones that can take on any closed source alternatives toe to toe and come out equal to or ahead even without considering that they happen to be open source. In those cases, I am typically indirectly and perhaps even unknowingly enjoying the realized benefits of the open source nature of the product without focusing on that open source nature. After all, it matters more to me what works best, works most efficiently, is easiest to use and maintain, is most tested, and is most easy to switch away from if needed. Open source is often a characteristic of products that meet these expectations, but not all open source products do allow these potential benefits to be actually or easily realized.

The one benefit you can count on in just about any major open source product is the ability to see and, in most cases, change the source code if necessary. This often may not be a very attractive scenario, but it is the one benefit that can be always be realized with open source and never realized with closed source.

Fourth Day at Colorado Software Summit 2008

Today (Thursday, 23 October 2008) was the fourth day of presentations at Colorado Software Summit 2008. As usual, I find myself feeling like there is very little room left to cram any more information into my brain and process it coherently. As with all my summaries of Colorado Software Summit presentations, it is important to note that any errors are probably a result of my misunderstanding or misinterpretation of what was stated.


Simon Phipps - The Adoption-Led Market: The Third Wave of Open Source

Thursday mornings at Colorado Software Summit traditionally begin with a keynote presentation from Simon Phipps and this year was no different. It is no surprise that, as Chief Open Source Officer at Sun Microsystems, Simon's annual keynote addresses emphasize open source. The title of this year's keynote address was The Adoption-Led Market: The Third Wave of Open Source.

In his address, Simon used Third Wave of Open Source to describe commercial and government embrace of open source. Simon referred to the first wave of open source as the wave when "fanatics" or "specialists" (depending on your perspective) were the only ones involved. The second wave of open source occurred when forward-thinking software developers started understanding the advantages of open source and now the third wave involves commercial and governmental organizations embracing open source.

Simon described the current procurement methods used by many organizations and stated that he believes that within ten years the majority of organizations will be using adoption rather than procurement to try out products and then adopt the product once it has been proven. Standards and open source together make this possible because an organization can try out the standard-based open source product with the freedom to switch to an alternate implementation of the same standard. Organizations also have the freedom to choose whether to purchase subscriptions for support and other benefits Simon outlined that often go with subscriptions. I have noticed that we already seeing this model with open source products from SpringSource (Spring Framework), Adobe (Flex), Laszlo Systems (OpenLaszlo), Sun (NetBeans and GlassFish), etc.

Simon made many more good points and outlined five lessons of 2008 related to open source freedom. He also referenced Alvin Toffler (The Third Wave) and pointed out the usefulness of ScribeFire 3.1.3. Among many other things Simon covered, he discussed freedom through substitutability rather than interoperability, the standardization processes, cloud computing, and much more. There were a lot of questions and comments, which usually reflects a lot of interest in the topic.


My Last Presentation of "Applying Flash to Java: Flex and OpenLaszlo"

In my third and final time presenting Applying Flash to Java: Flex and OpenLaszlo, I was happy to be able to make a segue from Simon Phipps' keynote about the evolution to an adoption-led market fueled by the compelling combination of open source and standards to Flex and Open Laszlo. These fine RIA frameworks are available for no charge and are open source, but support and other features can be purchased for organizations that desire that support. In other words, organizations have the freedom to use Flex or OpenLaszlo either on their own or by paying for levels of support.


Eric Schreiner - Real World Groovy: How to Add Scripting Functionality to Your (Existing) Application

After another excellent lunch and a brief walk around the Keystone Resort Lake, I attended Eric Schreiner's Real World Groovy - How to Add Scripting Functionality to Your (Existing) Application. While I have had a cursory awareness of Groovy and what it is, the presentation gave me a clearer perspective on Groovy. Eric pointed out that a differentiating aspect of Groovy is that Groovy compiles to Java .class files.

I also thought it was interesting that Groovy will autogenerate get/set methods for data members if they are not provided. Groovy's API is documented at http://groovy.codehaus.org/api/index.html, but what may be even more interesting is that Groovy extends the Java SDK and these extensions are documented at http://groovy.codehaus.org/groovy-jdk/. An example that Eric showed is the File.getText() method. Eric pointed out that Java SE's File class does not have a getText() method but Groovy adds one to it.


Matthew Wakeling - Performance Engineering from Scratch

Matthew Wakeling's excellent presentation Performance Engineering from Scratch was next on my schedule. I had changed my mind from going to another session to go to this one instead after a colleague told me how good it was. This is another advantage of the conference offering three presentations of each topic -- it allows word of mouth to help guide decisions about which presentations to attend.

Matthew's presentation had high ambitions of covering performance theory (including Big O notation), sorting algorithms, lookup algorithms, concurrency issues, database performance issues, and "funky stuff." It was useful to be reminded of some of the Computer Science 101 principles I have not thought about for a while and to learn some new things as well. I especially liked Matthew's coverage of the algorithm implementations used with various Java Collections and the strength of each.

Matthew's presentation included a reference to Tim Bray's On the Goodness of Binary Search and to the Java implementation java.util.Arrays.binarySearch(...). He also talked about the usefulness of TreeMap, TreeSet, HashMap, HashSet, LinkedHashMap, LinkedHashSet, java.util.BitSet, and several useful Java concurrent collections.

Matthew discussed behavior of String.equals and Object.hashCode versus String.compareTo and observed that String.compareTo terminates its checking of equality in characters as soon as it sees a mismatch. Using StringBuffer (or StringBuilder) instead of String for concatenation came up as an "obvious" performance issue. There was also some discussion of the potential performance advantages of moving significant functionality into the database versus the potential lower cost of scaling hardware running Java middleware and allowing that middleware to do more processing.

Matthew discussed the performance implications of reflection and emphasized the cost of catching exceptions related to reflection. Matthew showed several diagrams comparing performance of different collections and algorithms on those collections. He said he will be putting these tests on the conference CD and I am looking forward to trying them out when the CD arrives. The animations on these slides that showed the algorithm sorts and binary searches. These animations likely cost Matthew significant effort, but they were really good at helping me to see very clearly and easily how the various algorithms work. It was clearest presentation of search and sort algorithms that I have ever seen.


Matt Raible - What's Coming in Spring 3.0

As Matt Raible explained in his blog entry The Colorado Software Summit and Spring 3.0's SVN, Spring source code is contained in two repositories. There is also an explanation of what is delaying the release of Spring 3's early release versions. Matt covered the basics of the Spring Framework, the history of the Spring Framework, and outlined the anticipated improvements coming in Spring 3.0 in his What's Coming in Spring 3.0 presentation.

Matt mentioned several useful software development resources in his presentation including the Spring Kick-Start project he and Thomas Risberg host on Google Code for helping learn to use Spring quickly, JavaRebel for reloading class files dynamically, the DZone reference card on Spring Framework Annotations, and a reference explaining the Unified Expression Language (unifies JSP and JSF expression languages).

Matt has summarized this presentation on his blog and that summary includes a copy of the actual presentation.


Overall Comments for the Day

Thursday at the Colorado Software Summit is always the day I really start to feel myself being saturated with new knowledge. I have many things on my to-do list now to look into further or try out and am looking forward to it. I still have one presentation to give. I will be presenting JMX Circa 2008 for the third and final time at this conference first thing tomorrow morning. It will be interesting to see how many people attend, especially considering that it is being held in one of the very long conference rooms (Red Cloud Peak).

With only two sessions left in this year's conference, I have noticed that many of the conference attendees seem to be using Flex, Google Web Toolkit, or one of the Ajax/DHTML frameworks for their RIAs. I only heard one person state that she was using JavaFX. It has been very rewarding to talk to other people who are using Flex, OpenLaszlo, and JMX (my topics at the conference) and to hear some of the creative things they are doing with these technologies.

I did not win the iPod or any of the other items in the Thursday larger-than-other-days daily raffle, but it was still a great day to be at the Colorado Software Summit, to be in beautiful Keystone, Colorado, and to be learning new things.

Standardization: The Dangerous Relationship for Open Source

There are many examples in software development of open source products introducing useful concepts and approaches that are later standardized. It is often the case that the creative developers behind the most successful open source products can deliver innovative features much quicker than a standards committee could ever hope to do. Standards are, by their nature, more difficult to create because they require consensus of so many different organizations and people with different agendas and ideas of what is best. However, standards creators can adopt tried and proven features introduced by open source (and commercial) offerings once the value of the features is proven.

Ironically, once a standardized approach provides the desirable features originally offered by the open source project, the open source product's future is often eclipsed by that of the standardized approach. In this blog entry, I'll look at some of the open source products that have provided great benefit to only be overtaken in use by the standards they inspired. I will also look at some open source products that managed to survive and even thrive against the standardized competition offering the same features. From these products, I will observe what I believe makes these products successful.


XDoclet and Apache Commons Attributes

XDoclet is one of the most obvious examples of an open source product providing a useful feature, but being "overcome by events" when standardized approaches adopted that feature. XDoclet's popularity rose rapidly as J2EE (before it was Java EE) developers recognized many benefits of being able to annotate source code to have deployment descriptors and boilerplate interfaces generated from the source code. However, once annotations were formally added to the Java programming language with J2SE 5, use of XDoclet fell precipitously because its main benefits and features could be achieved using standards. XDoclet's main page currently shows its "Last Published" date as 5 May 2005 and its most recent news item is dated 23 October 2004 (XDoclet 1.2.2 released).

Like XDoclet, Apache Commons Attributes became less necessary with the adoption of annotations in Java. As of this writing, the last news update on the Commons Attributes page is dated 3 August 2006 (release of Attributes 2.2) and the "Last Published" date is 1 August 2007. According to Jürgen Höller's presentation Spring 2.5 on the Way to 3.0, it appears that Spring 3 will "prune" Commons Attributes support.

Of course, not everyone is using J2SE 5 or Java SE 6 and many legacy Java applications running in newer JREs may not have been ported specifically to use new features, so it is not surprising that XDoclet and Commons Attributes are still used, albeit at a much lower rate than in their peak years.


Log4j

I must admit my surprise at the resiliency of Log4j despite the introduction of many Log4j-like concepts with the JDK 1.4 introduction of java.util.logging as a standard, built-in approach to logging from Java applications. The "Last Published" date for the main Log4j web page is, as of this writing, 1 September 2007, and the latest news item is dated 29 August 2007 (release of log4j 1.2.15 and log4j extras 1.0).

Although I know many developers on applications using JDK 1.4, J2SE 5, and Java SE 6 that still prefer log4j (because of features in Log4j that are not in java.util.logging), the lack of recent changes to log4j and the changes of direction involved with Log4j 1.2, Log4j 1.3 (abandoned), and Log4j 2.0 (experimental) lead me to believe that some of the steam is running out for log4j. This may be partly due to the feeling that there is not much more that needs to be added to an already useful logging tool, but it may also be due at least partially to the existence of the standardized java.util.logging package. The standard Java logging package has started to add numerous useful features such as the ability, via JMX, to set the logging level dynamically and remotely.

Another open source project, Apache Commons Logging, allows developers to use both or move between Log4j and java.util.logging. The presence of this project illustrates that log4j has not gone away.


JDOM

JDOM was wildly popular for some time because this non-standard library was so much easier to use than other available approaches. In particular, JDOM's strategy of supporting Java-specific APIs rather than the more XML-general APIs supported by alternatives made it much more approachable for a large number of Java developers.

JDOM's latest news entry is the release of JDOM 1.1 on 18 November 2007. While JDOM still seems to be used on a large number of Java-based projects, most of my anecdotal experience has been with new applications using more standard approaches such as Java API for XML Processing (JAXP) and Java API for XML Binding (JAXB). JAXB, in particular, does a nice job of abstracting XML specifics so that Java developers can focus on using Java objects bound to XML rather than on the XML. The addition of XPath support to Java with the javax.xml.xpath package has provided Java developers with yet another standard approach for querying XML from Java.


Hibernate

Hibernate is one of the best examples of a non-standard open source project becoming a de facto standard. Hibernate might be most simply described as a Java object-relational mapping (ORM) framework. Hibernate heavily inspired the newer and actual standard Java Persistence API (JPA). Hibernate has remained a major force in the ORM world partly because its developers made the decision to tweak Hibernate to provide a JPA implementation. In other words, rather than Hibernate competing against JPA, Hibernate is an implementation of JPA.


Spring Framework

Spring Framework founder Rod Johnson's book J2EE Design and Development (and its sequel J2EE Without EJB) not only laid the groundwork for the Spring Framework, but also had enormous impact on the enterprise Java community. Not only did the Spring Framework help bring inversion of control (IoC) and dependency injection (DI) into the Java mainstream, but the framework also helped reinforce the benefits of using regular Java classes rather than numerous interfaces and class inheritance hierarchies.

JSR-220 adopted many of the features that were most popular in Spring and incorporated them into EJB 3 and Java EE 5. These improvements made Java EE much easier to use, but I believe that many enterprise Java developers are still jaded by their experience with EJB 2.x and have not adopted Java EE 5/EJB3/JPA as quickly as one might have expected.

Besides the fact that negative experiences with EJB 2.x may have jaded many of us and turned us cynical about EJBs, I think there are other explanations for Spring's success despite Java EE adopting many of Spring's concepts and features. These reasons for Spring's success include its great support for reducing boilerplate code related to Java APIs such as JDBC, JMX, JMS, RMI, and many others. The Spring Framework has continued to offer cutting edge features like OSGi support to continually differentiate it from more standard alternatives. Also, once many developers had switched from EJB 2.x to Spring, it would require enough pain or dissatisfaction with Spring to return to EJB.

I am curious to see if recent SpringSource announcements regarding their new Enterprise Maintenance Policy and their new licensing models for new products and the recent blogs questioning the openness of Spring impact the number of developers and applications using Spring. While I believe that these developments undoubtedly will turn a certain number of developers away from Spring, the real question is how many that will be. It is highly possible that Spring will lose a relatively large fraction of users/developers, but that SpringSource will still be monetarily better off with the smaller user base. I believe that these recent developments in the Spring community may be enough of a reason in some developers' minds to try out alternatives, including returning to EJB.


Other Examples

There are many more examples of where an open source product has brought forth great innovation, has inspired or directly fed into a standard approach, and then has fallen by the wayside as the standard alternative satisfies the same needs in a standard way. Examples include Doug Lea's util.concurrent library (fed into and replaced by java.util.concurrent package) and MC4J (replaced largely by JConsole and VisualVM, which are delivered with Sun's Java SE 6 JVM).


Observations

From this quick tour of open source products which have had more standardized approaches adopt some of their biggest benefits and most valuable features, a few observations can be made regarding what keeps these products alive.

1. Open source frameworks and toolkits that offer more than the standardized version offers may survive because they do offer these "extras" that make using the less standard product preferable even to using the more standardized product. Examples of this include log4j providing more than java.util.logging and Spring Framework offering more than just the dependency injection and POJO-orientation now available in EJB3. Other examples include the continued popularity of Apache XMLBeans and Castor despite the availability of standardized JAXB.

2. Strength and commitment of the open source product's community also have an effect. The commitment and interest of the primary developers of the open source product are particularly important in the success of the product when more standardized approaches offer many of the same benefits. The open source products must continue to innovate and offer features the standard counterparts don't.

3. Perhaps most importantly, an non-standard open source project is most likely to survive once a standard alternative adopts it best features and concepts if the open source product is willing to adapt to complement the standard approach and add value to it. This is the strategy that Hibernate seems to have taken and it is true (even if not all that common) that the Spring Framework can be used with a Java EE application server and with EJBs.


Conclusion

Because standards bodies are, by their nature, almost always going to generate innovation at a slower pace than a few creative and like-minded individuals can, it is not surprising that standards have and will continue to adopt some of their best features from proven open source frameworks. Also, because many of these open source projects are started simply because a (usually painful) gap is identified in the existing standards, it is also not surprising that many of these projects will be less necessary once the standard alternatives do adopt the features they provide. In fact, I think there are few things that can be more satisfying to the ego of an open source product founder than having his or her ideas accepted enthusiastically and adopted into standards, even if it means a lessening importance of his or her open source product. However, anyone starting an open source product to satisfy a glaring omission in the current standards implementations must realize that his or her success will often be the very thing that eventually leads to that very product losing its significance in the software development world.


Additional Resources

* Open Standards, Open Source

* Open Source and Standards

* Open Source and Open Standards

Thoughts on SpringSource's Enterprise Maintenance Policy

UPDATE (11 October 2008): SpringSource has tweaked the policy described below to a significant enough extent that I believe it the altered new policy will be palatable to the vast majority of developers using the Spring Framework. More information on this is available in SpringSource Regains Footing with Support Change and A Question of Balance: Tuning the Maintenance Policy.

---

The recent mini controversy surrounding SpringSource's Enterprise Maintenance Policy announcement provides interesting material for a look at how different people view open source differently. I have talked with professional software developers who still equate "open source" with "free software" rather than recognizing the classic differentiation between "free speech" and "free beer." In fact, I used to be one of these individuals until I attended multiple editions of the Colorado Software Summit in the early 2000s in which Simon Phipps explained each year different ways of looking at what it means to be open source and to be part of an open source community.

The Spring Framework has become wildly successful and widely adopted. There has been some discussion regarding how much "community" participation there is in Spring versus SpringSource direct involvement with Spring. In this blog entry, I intend to examine competing arguments that have been or might be made in this controversy. I can understand the motivations of the principals of both sides of the argument and I want to look at the issue from both points of view. I also intend to look at some possible results of this SpringSource announcement.

This recent announcement by SpringSource and the reactions to the announcement have caused me to reflect on my own participation in the Spring community. I think in may ways I may be representative of the "typical" Spring "community" member. I have never contributed a line of code to Spring and have not even written up a defect or enhancement request (JIRA). However, I feel like I have at least contributed a little to the Spring community by writing an article on Spring for Oracle Technology Network called Add Some Spring to Your Oracle JDBC Access (November 2005) and blogging on Spring occasionally. In addition, as some have stated in their feedback to the recent announcement, I have purchased Spring-related books and told colleagues about Spring. However, these are, in some respects, indirect support rather than direct contributions to the framework itself.

There is no question that we become accustomed to getting what we have regularly received in the past and do not like to hear that this freely given benefit will be removed or have some cost applied to it. For example, when an employer takes away a benefit we have received in the past, few employees will "welcome" that. From this very human perspective, it is natural to not "welcome" the news that SpringSource will not be publishing releases of the Spring Framework to non-Enterprise users after a major release has been out for three months. After all, if I have been enjoying the benefits of polished releases of software for years, why would I welcome extra work on my part or a subscription fee to continue to receive that same benefit?

My understanding is that bug fixes will continue to be available in the repository trunk, but that they will no longer be available in a binary form after the 3-month period following each major release. For many organizations that use Spring, this is a minor new hindrance to using the framework, but probably not enough of a hindrance to stop using the framework altogether unless something else is available for which the benefit-to-cost ratio seems better.

While the Spring Framework's popularity has grown tremendously, there are a growing number of popular alternatives. For example, Java EE 5 (including EJB3) has borrowed heavily from Spring and, in my opinion, is a now a reasonable alternative. Many of the issues that drove developers from EJB 2.x to Spring have now been resolved. There are also other non-standard but popular frameworks like Google's Guice that are available. What developers need to ask themselves is if the added cost of using Spring from here on out (either the cost of the enterprise support or the cost of building Spring releases) is still worth the benefits of using Spring and if the ratio of these benefits to the costs is still better than the ratio of using an alternative to the cost of using that alternative. For example, a developer may choose to use the simplified Java EE 5 on future projects because the extra effort associated with Java EE is so significantly reduced and the ability to write standard code regardless of providers is a desirable advantage. In fact, with a product like GlassFish, that developer can have an open source and standard Java EE 5 implementation.

Developers will likely have some options between becoming SpringSource Enterprise subscribers and ditching Spring altogether for an alternative. Many major vendors already use Spring in their products. As long as these vendors of IDEs (such as NetBeans 6.1 support and Spring Extension for JDeveloper), application servers (such as WebLogic and Oracle Containers for Java EE), other frameworks (such as AppFuse), and other products continue releasing their Spring-based and Spring-supporting products with new versions of Spring, some of the bug fixes and updates may become available to end-user developers via these third-party products. There is always the possibility that these third-party products could drop support for Spring due to this announcement, but the announcement does not seem onerous enough to me to cause that to happen. Finally, there has already been talk of starting a community effort to fill in the relatively small gap being left by this SpringSource announcement. This proposed project would, in theory, be able to take the source code on the publicly exposed repository and build it into a finished distributable like SpringSource does today. There has even been talk of a complete fork off of Spring.

Some developers are feeling trapped. They have tied their products heavily to Spring and so it is not as easy as simply switching to an alternative. Ironically, Spring creator Rod Johnson pointed out in the excellent book J2EE Design and Development that good design includes isolating non-standard and changeable code from standard and stable code. If developers followed this advice with Spring (and in many ways Spring does encourage this isolation), then the migration would be easier than it would have been from proprietary libraries or other third-party frameworks. Even so, the question to ask if if the new cost of using Spring (three year subscription or building Spring on one's own) is enough to justify the certain cost associated with switching to an alternative.

For some larger organizations, this actually might be welcome news because there are still large corporations that prefer professional support. Some clients and some organizations have been reluctant to use any open source products that did not have professional support. They are likely to already be using Spring's Enterprise support and so this won't impact them. Others may have not chosen this option, but do not mind paying the subscription fee and consider it a relatively small cost.

In his blog entry Open Source, Open Strategy: The SpringSource Manifesto, Rod Johnson covers the history of SpringSource (including its days as Interface21) and outlines reasoning behind using a different license for some of the newer products released by SpringSource. He makes valid points such as the fact that many companies sell products using the Spring Framework for significant portions of their functionality. An important and highlighted note in this blog entry is this statement: "We are not changing and will not change the license of any existing project." He goes on to add that this includes the Spring Framework and Spring Security. I think this is significant. The recent Enterprise Maintenance announcement is less of a blow if existing and future Spring Framework code remains open source under the Apache license.

Some have charged that SpringSource is only making this change in support because of their wide adoption and the dependencies people have on the framework. It seems that it is obvious that a move like this is much more likely to be successful after a framework has been evangelized and adopted rather than in its infancy, so I don't think any of us should be too surprised. SpringSource certainly does have some leverage right now and it is certainly greater than it would have been a number of years ago. In fact, in the blog entry Pumping It Dry: $200 a Barrel and $25,000 per CPU, Rod Johnson points out similar tactics in terms of revenue raising implemented by Oracle after their acquiring BEA.

My biggest concern about the recent SpringSource announcement is not the content of the announcement itself. With the assertion in the summer blog about the license for Spring Framework not changing and with Rod Johnson's assertion at TheServerSide's New Spring Maintenance Policy that "SpringSource continues to expose our open source code", I do not see any reason for immediate panic or knee-jerk reactions. Instead, my much bigger concern is worry that this might only be the first in a long series of steps that make Spring less attractive. It will definitely enter my mind when trying to decide how dependent on Spring I want my application to be. In other words, I may choose to use options where I don't need to implement a Spring interface.

It seems obvious that this decision was made to increase SpringSource revenues. I understand that and empathize with the talented individuals who have sacrificed to make Spring what it is today. They definitely deserve some payback for the sacrifice. Making money on open source development is a tricky business model and it is not surprising that it occasionally needs tweaking. That being said, what happens if this move is not enough to garner that payback? Will the license be changed in the future for new versions/releases or will more other much more drastic measures be taken to gain the desired revenue? I like to think that won't happen, but the recent announcement at least reminds one of that possibility.

The long-term implications of this announcement are interesting to ponder. The blog entry SpringSource Will Charge for Updates to Spring, What Comes Next? provides some possible next steps that SpringSource might take in this same direction. While that blog entry focuses on what SpringSource may do, I also wonder how the community will evolve as a result of this announcement.

The following are some questions whose answers we will likely not know for certain until some time has played out.

* Will there be a dramatic decrease in the number of developers and projects using SpringSource products (especially the Spring Framework)?

* Will there be fewer blog entries, articles, and other coverage of Spring Framework due to this policy change?

* Will the developers that stop using Spring be missed by SpringSource if they weren't providing any type of direct monetary benefit to SpringSource?

* Will the developers that stop using Spring be missed by the community if they weren't contributing directly or indirectly to the community?

* What percentage of the developers that stop using Spring or reduce their usage of Spring will be contributors who have performed useful testing and bug and enhancement reporting?

* Will the new policy allow SpringSource to retain and acquire the necessary talent and resources to make the products better for users?

* Will this announcement be the beginning of a bright new era or the beginning of the end of Spring's rapidly growing dominance?

I won't be surprised if this new policy is the subject of some hallway discussion and even some Q&A discussion at the Colorado Software Summit 2008 that occurs four weeks from now. With Matt Raible speaking on What's Coming in Spring 3.0, with Chris Richardson (author of POJOs in Action) speaking, and with a SpringSource employee (Filip Hanik) presenting, it seems likely this will come up.

No product is perfect and the Spring Framework is no different. However, many enterprise Java developers have found it to be extremely useful and helpful and a welcome release from the complications of EJB 2.x. When evaluating a product, I like to look at the entire package including all advantages and disadvantages. The Spring Enterprise Management policy should be considered along with Spring's advantages and disadvantages when deciding whether or not to use Spring. For some, it may be an advantage, for some it may be considered a significant disadvantage, and for many of us it will probably only be a minor disadvantage. The more interesting questions involve the effect it will have on Spring, on the Spring community, and if it is just the first of additional and even more dramatic steps in this direction.